๐ฎ๐ฉ Baca dalam Bahasa Indonesia
The Question
Someone asked me: "If you had to stress test a search endpoint, what would you use? I want to judge its semantic search and its typo tolerance."
That sentence holds two different tests:
- Quality: does the right article come back when the query is misspelled, paraphrased or in another language? That is an evaluation with labelled data, not a load test.
- Load: how many searches per second does it handle before latency breaks, and do the results get worse when it is busy?
Rather than answer in the abstract, I ran both against something real: the search over this blog. 55 articles, 49 of them in English and Indonesian, 6 in only one language. Everything below can be reproduced from the repository (tools/search-eval), and every number comes from those runs.
What came out of it, in short:
- Embedding the full text of each article (the default way to use
semantic_text) scored no better than plain BM25: nDCG@10 0.739 against 0.756. - Embedding only the title and description of each article, with the same model, raised it to 0.888. Cross-language queries went from 0.034 to 0.643.
- Typo tolerance mostly came for free. Plain BM25 kept its quality on misspelled multi-word queries;
fuzziness: AUTOmainly made the top result more stable, and it turned a correct "no results" into wrong results. - No configuration could tell a nonsense query apart by its score. "Resep rendang padang" (a beef curry recipe) scored 0.918 in semantic search, the same as real answers.
- A load test that repeats the same queries measures Elasticsearch's inference cache, not the model. With the cache off, one model allocation topped out at about 13 inferences per second, and past that point requests waited 10 seconds and failed with HTTP 408.
1. The Setup
| Part | Choice |
|---|---|
| Engine | Elasticsearch 9.1.0, single node in Docker (1 GB heap), on an Apple Silicon laptop |
| Model | .multilingual-e5-small, Elastic's packaged multilingual E5, one allocation with two threads |
| Semantic field | semantic_text, which chunks text (sentence-based, 250 words, 1 sentence overlap) and embeds every chunk |
| Lexical fields | titles, descriptions and bodies, with the english and indonesian analyzers |
| Corpus | one document per article; both language versions go into the same document |
I indexed the articles as plain text: code blocks, links and the "Series Navigation" blocks removed. Those blocks list the titles of every other part of a series, so leaving them in makes every part of a series match every other part's keywords.
Indexing cost
Indexing 55 articles took 875.6 seconds. Almost all of that is the model embedding about 94,000 words on CPU. BM25 indexing of the same text takes a second. Semantic search moves cost to ingest time: re-indexing after a model or chunking change is not free, so plan for it.
First pitfall: the model download
Elasticsearch downloads the 470 MB model itself. On my connection (about 350 KB/s) the download broke after five minutes with Model importing failed due to an IOException [java.io.IOException: Premature EOF]. The fix that works on any connection: download the three model files with a client that can resume (curl -C -) and point Elasticsearch at the folder with xpack.ml.model_repository=file://.... This is the documented approach for air-gapped clusters, and it is the one I would use in production anyway, so deployments do not depend on a download at startup.
2. The Golden Set
A golden set is a list of queries with the documents that should come back, graded:
- 3: the article that answers the query
- 2: clearly relevant, a good second result
- 1: partly relevant, fine anywhere in the top 10
I wrote 47 queries in six kinds, because "is search good?" is really six different questions:
| Kind | n | Example | What it tests |
|---|---|---|---|
| exact | 13 | optimistic locking spring boot exam | words from the article itself |
| paraphrase | 12 | stop two requests from overwriting the same exam answer at the same time | same intent, different words |
| crosslang | 6 | writing someone else's final thesis for money โ an Indonesian-only article about joki skripsi | query language differs from the article's only language |
| slang | 5 | gmn cara deploy fastapi ke vps pake docker | informal Indonesian with abbreviations |
| spacing | 6 | nest js api, rubyonrails clean architecture | product names split or joined |
| negative | 5 | resep rendang padang, flutter bloc state management | nothing in the blog answers it |
Three of the negatives are near misses on purpose. The blog has React Native, Python and Android-adjacent articles, so flutter bloc state management, golang goroutine worker pool and kotlin jetpack compose android app sound like they belong. A search engine that confidently returns something for them is misleading its users.
Typos are generated, not hand-written
Hand-written typos tend to be the ones the author thinks of. I generated them instead (lib/typos.js): for each exact, paraphrase and slang query, one variant with one edit and one with two edits. An edit is a deletion, a transposition, or an insertion or substitution with a neighbouring key on a QWERTY keyboard. The first letter is never touched, and words shorter than four letters are left alone. The generator is seeded, so the same seed always gives the same 60 variants:
optimistic locking spring boolt exam (1 edit)
optimistic lockin sprint boot exam (2 edits)
watching studengs live while they take an online tyest
gmn cara deploy fastapi ke vps pake docklee
Together that is 107 queries.
3. What Gets Measured
| Metric | Question it answers |
|---|---|
| nDCG@10 | Are the best documents near the top? Graded, so a 3 at rank 1 counts more than a 1. |
| Recall@10 | Did every relevant document make the first page? |
| MRR | How far down is the first good (rated 2+) result? |
| Zero-result rate | How often does a real query return nothing? |
| Typo robustness | For each typo variant compared with its clean query: nDCG ratio, overlap of the two top-10 lists, and whether the top result stayed the same. |
| Negative leakage | If you cut results below the score of the weakest 10% of real answers, how many nonsense queries still show results? |
With a few dozen queries per kind the numbers are noisy, so the evaluator also reports a 95% bootstrap confidence interval for the overall score. A difference smaller than that interval is not a finding.
4. Four Configurations, and a Surprise
I started with four configurations:
bm25: plain BM25 across titles (boost 3), descriptions (boost 2) and bodiesbm25_fuzzy: the same withfuzziness: AUTOsemantic: asemanticquery on the full-textsemantic_textfieldhybrid: Reciprocal Rank Fusion (RRF) ofbm25_fuzzyandsemantic
The surprise was semantic search on cross-language queries, the one thing a multilingual model should be good at. It was the worst of the four. For four of the six cross-language queries, the right article ranked between 31st and 46th out of 55. The query about writing a thesis for money came back with an article about IDOR vulnerabilities.
Hypothesis 1: missing E5 prefixes (ruled out)
E5 models are trained with query: and passage: prefixes, and work worse without them. So I checked whether Elasticsearch adds them. With the inference API, an embedding requested with input_type: search is identical to one of the text with query: typed in front (cosine 1.0000), and differs from an ingest embedding of the same text. Elasticsearch handles the prefixes. Not the cause.
Hypothesis 2: chunk dilution (confirmed)
With semantic_text, an article becomes many 250-word chunks, and a query is scored against those chunks: a document is as good as its best-matching chunks. A long article has many generic passages (setup steps, conclusions, "in this series") and one of them is always somewhat close to any query. Short, single-language articles have fewer chances. All six cross-language targets are single-language articles.
To test it, I embedded only each article's title and description (both languages) and ranked the articles by cosine similarity to each paraphrase and cross-language query. Same model, same queries:
| Query | Best answer's rank, title + description | Best answer's rank, full-text chunks |
|---|---|---|
c04 when should an early stage company hire a contract frontend engineer | 2 | 46 |
c06 writing someone else's final thesis for money | 3 | 38 |
c01 alasan tidak memakai kontainer saat hosting aplikasi next | 13 | 31 |
p07 hosting a react framework site on a cheap server without containers | 1 | 9 |
| MRR of the best answer, 18 queries | 0.693 | 0.486 |
The model understood the queries. The way the scores were aggregated over chunks buried the answers.
So I added a second field, summary_semantic (titles and descriptions in both languages), and two configurations that use it:
semantic_summary: semantic query on the summary field onlyhybrid_summary: RRF ofbm25_fuzzy, full-textsemanticandsemantic_summary
Adding the field did not need a full re-index. A mapping update plus a bulk partial update of that one field took 39.9 seconds, against 875.6 seconds for the full text, because only the new field was embedded.
5. Fixing the Golden Set Too
The evaluator lists every unlabelled document that showed up in a top 5 (out/unrated.json). After the first run there were unlabelled documents for 42 queries. Most were noise. Some were not: for watching students live while they take an online test, semantic search returned Laravel E-Learning Part 6, which has a whole section called "Live Exam Monitor Widget". I had not labelled it.
This is how evaluation sets are normally built (pooling): take the top results of every system under test as candidates, then judge them by hand. My rule was to add a label only if the article has a section that answers the query, checked in the article text. Mentions in passing did not count. That added 17 labels, for 84 in total. Without this step a configuration that finds relevant documents nobody labelled gets penalised for it, and semantic search does that more often than BM25.
6. Results
All numbers below are from the final run: six configurations, final labels, 107 queries.
nDCG@10 per query kind
| Kind | n | bm25 | bm25_fuzzy | semantic | hybrid | semantic_summary | hybrid_summary |
|---|---|---|---|---|---|---|---|
| exact | 13 | 0.980 | 0.980 | 0.970 | 0.986 | 0.987 | 0.990 |
| typo, 1 edit | 30 | 0.816 | 0.796 | 0.851 | 0.882 | 0.888 | 0.924 |
| typo, 2 edits | 30 | 0.779 | 0.773 | 0.784 | 0.821 | 0.812 | 0.877 |
| paraphrase | 12 | 0.599 | 0.527 | 0.749 | 0.712 | 0.815 | 0.845 |
| crosslang | 6 | 0.288 | 0.398 | 0.034 | 0.271 | 0.643 | 0.330 |
| slang | 5 | 0.910 | 0.910 | 0.729 | 0.989 | 0.973 | 0.988 |
| spacing | 6 | 0.927 | 0.868 | 0.936 | 0.997 | 0.992 | 0.997 |
| all clean queries | 42 | 0.756 | 0.743 | 0.739 | 0.807 | 0.888 | 0.855 |
| 95% CI | 0.65โ0.85 | 0.64โ0.84 | 0.64โ0.84 | 0.71โ0.90 | 0.82โ0.95 | 0.76โ0.93 |
Typo robustness
Each typo variant compared with its clean query:
| Config | 1 edit: nDCG ratio | 1 edit: same top result | 2 edits: nDCG ratio | 2 edits: same top result |
|---|---|---|---|---|
| bm25 | 1.00 | 90% | 0.95 | 87% |
| bm25_fuzzy | 1.01 | 100% | 0.98 | 87% |
| semantic | 1.01 | 80% | 0.93 | 67% |
| hybrid | 1.01 | 83% | 0.94 | 70% |
| semantic_summary | 0.97 | 90% | 0.89 | 77% |
| hybrid_summary | 0.99 | 100% | 0.94 | 87% |
(A ratio slightly above 1 is noise: a typo variant happened to rank a relevant document higher than its clean query did.)
Negative queries
| Config | Score cut-off (weakest 10% of real answers) | Negatives above the cut-off |
|---|---|---|
| bm25 | 12.652 | 1 of 5 (flutter bloc state management) |
| bm25_fuzzy | 12.876 | 1 of 5 (same) |
| semantic | 0.921 | 1 of 5; the other four between 0.910 and 0.918 |
| semantic_summary | 0.905 | 2 of 5 (resep rendang padang 0.914, kotlin jetpack compose android app 0.909) |
| hybrid, hybrid_summary | n/a | RRF scores come from ranks, not from similarity |
7. What the Numbers Say
Typo tolerance: mostly free, and fuzziness has a price
On multi-word queries, plain BM25 already tolerates typos: one misspelled word out of five still leaves four that match. Its nDCG ratio for one-edit typos is 1.00. What fuzziness: AUTO adds is stability. The top result stayed the same for 100% of one-edit typos instead of 90%.
The price:
- Paraphrases got worse, from 0.599 to 0.527. Fuzzy matching lets unrelated words that are one edit apart match too.
- "No results" disappeared.
resep rendang padangreturns nothing with plain BM25, which is correct. With fuzziness it returns 38 articles: highlighting shows resep matching "reset", rendang matching "rancang" (Indonesian for "design") and padang matching "pandang" and "papan". Every one of those is a legitimate one- or two-edit match, and every one is wrong. - It costs latency: the median
tookwent from about 2 ms to 16 ms.
Typo tolerance matters most for one- or two-word queries, where a single typo is the whole query. If your users mostly type short queries (product names, people), fuzziness earns its cost. For sentence-like queries the semantic side already covers it.
Semantic over full text was not better than BM25
Overall 0.739 against 0.756, with overlapping intervals. Semantic search did help paraphrases (0.749 against 0.599), and hurt cross-language and slang queries. Turning semantic_text on is not an upgrade by itself.
What you embed matters more than which model
The same small model, given one short passage per article instead of dozens of chunks, went from 0.739 to 0.888. Its interval (0.82โ0.95) barely touches the full-text one (0.64โ0.84), so this is the one overall difference in this study I would call real. For a blog, the title and description say what the article is about; the body says many other things too.
The trade-off shows in Recall@10: semantic_summary finds the best answer first (MRR 0.89 on paraphrases) but misses more of the secondary ones (Recall@10 0.65 against 0.85), because a summary does not mention everything an article covers.
RRF can also make things worse
hybrid_summary is the best or tied for best on five of the seven kinds, and the most typo-robust. But on cross-language queries it drops to 0.330, against 0.643 for the summary alone. RRF only sees ranks: when BM25 and full-text semantic both rank the wrong articles first, they outvote the one retriever that got it right. Fusion averages opinions; it does not pick the expert.
No score threshold separates nonsense from answers
E5 similarities sit in a narrow band: real answers and a beef curry recipe both land between roughly 0.90 and 0.94. A min_score would cut real answers before it cut the nonsense. If "no results" matters to your product, it needs a different mechanism: requiring a lexical match (for example minimum_should_match on the BM25 side), a reranker trained for relevance, or a classifier for "is this query in scope".
What I would ship for this blog
hybrid_summary, with two follow-ups: a better answer for cross-language queries (weighting the summary retriever higher, or translating titles at index time), and a separate "no results" rule. I would not tune weights on these 47 queries and call the result proof: with this few queries, tuning on the test set is how you overfit.
8. The Load Test
The k6 script (k6/search-load.js) does three things a plain "hammer one URL" test does not:
- It sends the query mix, not one query. A single repeated query only tests a cache.
- It scores every response with nDCG@10 against the golden labels, so "quality under load" is a metric next to latency, not a guess.
- It tags every request with its load step (6 rps, 12 rps, ...), so the summary shows where latency breaks instead of one average over the whole run.
The stress profile ramps through four steps (30 seconds up, one minute held), with a p95 budget of one second and a 1% error budget.
The inference cache will lie to you
After a few runs, semantic queries that had taken 60โ80 ms took 2 ms. The deployment stats explained it: a trained model deployment keeps an inference cache, by default as large as the model (448 MB here), and 1,286 of the first 1,920 inferences were answered from it. A load test that replays the same 107 queries measures that cache.
My first fix was to give every request a fresh random typo. It was not enough: a short query only has a few dozen one-edit variants, and after a few thousand requests they repeat. In that run 53% of inferences still hit the cache. For the cold numbers below, I restarted the deployment with cache_size=0 and verified that the cache hit count stayed at zero.
Results
All runs on the same laptop, one model allocation with two threads.
| Run | Step | p95 latency | Failed | nDCG@10 (successful responses) |
|---|---|---|---|---|
bm25_fuzzy, no model | 10 / 20 / 30 / 40 rps | 40 / 32 / 31 / 30 ms | 0% | 0.78 |
hybrid_summary, golden queries, cache on (100% hits) | 10 / 20 / 30 / 40 rps | 90 / 89 / 88 / 89 ms | 0% | 0.88 |
semantic_summary, fresh typos, cache off | 6 / 12 rps | 113 / 115 ms | 0% | 0.87 |
| 18 / 24 rps | 10 s (timeouts) | 53% of the whole run | ||
hybrid_summary, fresh typos, cache off | 3 / 6 rps | 299 / 368 ms | 0% | 0.90โ0.91 |
| 9 / 12 rps | 10 s (timeouts) | 57% of the whole run |
What the load test says
The model is the bottleneck, and it has a hard ceiling. Without the cache, one inference took 75 ms on average. One allocation runs one inference at a time, so it tops out at about 1000 / 75 โ 13 inferences per second. That matches the data: semantic_summary (one inference per query) breaks between 12 and 18 rps, and hybrid_summary breaks between 6 and 9 rps. BM25 never came close to a limit at 40 rps.
hybrid_summary runs the model twice per query. The inference count rose by two per request: each semantic clause embeds the query text on its own, even when the text is the same. Two semantic retrievers halve the capacity. A query embedding computed once and reused would avoid it.
Past the ceiling, requests do not fail fast. They queue behind the model and fail after exactly 10 seconds with HTTP 408: timeout [10s] waiting for inference result. I reproduced it with 200 concurrent unique queries: 44 succeeded, 156 got the 408. For a user that is a ten-second spinner followed by an error. The application in front of Elasticsearch should use a shorter timeout and fall back to the BM25 results, which cost 30 ms and never hit the model.
Quality did not degrade before the failures started. Per kind, the golden-query run under load matched the offline evaluation almost exactly (typo1 0.927 against 0.924, paraphrase 0.846 against 0.845, crosslang 0.321 against 0.330). There was no silent degradation; the failure mode is errors, not worse results. That is also why nDCG of successful responses has to be read together with the error rate: a request that times out is a zero for the user, and it does not appear in that average.
Capacity planning uses the cold numbers. The cache makes popular queries cheap (90 ms flat at 40 rps), but long-tail queries are mostly unique, and the cold ceiling is what a burst of them hits. Throughput scales with number_of_allocations (each needs its own CPU threads) and per-query latency with threads_per_allocation; on a single small node, the cheapest win is not running the model twice per query.
9. Checklist
- Split quality from load. A load test answers "how fast", a golden set answers "how good". You need both.
- Label by query kind, including negatives and near misses, and report every kind separately. One overall number hides a 0.034.
- Generate typos, seeded, with a known number of edits. Compare each variant with its clean query.
- Pool unlabelled results from every configuration and judge them, or you will penalise the system that finds what you forgot.
- Report confidence intervals. With 5 or 6 queries per kind, most per-kind differences are noise.
- Check what is embedded, not only which model. Chunk dilution is invisible until you compare against a short summary.
- Do not trust
min_scorewith embeddings without measuring negatives. - Break the inference cache in load tests, or you are benchmarking a hash map.
- Measure relevance during the load test, not only latency.
10. Reproducing It
# Elasticsearch 9.1 with the model loaded from a local folder (see tools/search-eval/README.md)
cd tools/search-eval
node make-queries.mjs # golden set + 60 seeded typo variants -> out/queries.json
node setup-index.mjs # inference endpoint, index, 55 articles (~15 min on CPU)
node evaluate.mjs # six configurations -> out/eval-report.md
# cold run: restart the model deployment with cache_size=0 first (README, "Measuring the cold cost")
docker run --rm -i --network container:es-search-eval -v "$PWD":/work -w /work \
grafana/k6:1.0.0 run -e CONFIG=hybrid_summary -e QUERY_MODE=fresh \
-e PROFILE=stress -e PEAK=12 -e P95=1000 k6/search-load.js
To test your own application rather than Elasticsearch directly, both tools take your endpoint: node evaluate.mjs --api "https://your-app/api/search?q={q}" and k6 run -e API_URL=.... Replace golden-set.json with queries and labels from your own domain; the rest stays the same.
Limitations
- 55 documents, one labeller (me), 47 queries. The method transfers; the exact numbers do not.
- One laptop running Elasticsearch, the load generator and other containers at once. Latency numbers are relative between configurations, not a capacity plan for a server.
- The trial license was enabled so that every machine learning feature was available; check what your subscription includes before relying on these features in production.

Written by Muhammad Fari Madyan
Senior Full Stack Engineer and co-founder of LampungDev. I write up what I build so others can learn from it.
