Elasticsearch – The Search API and Query DSL

Everything so far has been about getting data in. This is where it pays off. The search API is one endpoint and a JSON body, and the body is the Query DSL — a small language that composes well once you know its four or five shapes.

The request

GET /stayhub-properties/_search
{
  "query":  { "match": { "title": "cabin" } },
  "from":   0,
  "size":   20,
  "sort":   [ { "price_per_night": "asc" } ],
  "_source": ["title", "city", "price_per_night"]
}

Five top-level keys, and they are separate concerns on purpose. query decides which documents match and how well. from/size decide which slice comes back. sort decides the order. _source decides how much of each document to send. You can change any one without touching the others.

There is also a URI form, which is handy at a terminal and nowhere else:

curl -s "localhost:9200/stayhub-properties/_search?q=cabin&size=5&pretty"

Fine for a quick check. Do not build an application on it: q is parsed with the query-string syntax, which has its own operators, and a user typing a stray : or [ gets a parse error rather than a search.

The response

{
  "took": 2,
  "timed_out": false,
  "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 },
  "hits": {
    "total": { "value": 12, "relation": "eq" },
    "max_score": 8.64,
    "hits": [
      { "_index": "stayhub-properties-000002",
        "_id": "8f1e-...",
        "_score": 8.64,
        "_source": { "title": "Sunlit Loft in the Mission" } }
    ]
  }
}

took is the cluster's own query time in milliseconds. It excludes the network and your client's deserialisation. On the index used throughout this track a search reports took: 2 while the round trip measured from Python is 4.5 ms. Report the second number to anyone asking about latency.

_shards.failed is the field nobody reads. A search can partially fail — one shard errors, the rest answer — and you get HTTP 200 with fewer results than exist. Same shape as the bulk trap in lesson 8. If results matter, check it.

timed_out is the same idea for a different cause. There is a timeout parameter, and it does not cancel the search — it returns whatever was collected so far, with this flag set. Partial results that look complete unless you look.

match and term, one more time

Lesson 4 covered this from the mapping side. From the query side it is the single most common bug in Elasticsearch, so it is worth stating as a decision rule.

match analyses your query string with the field's analyzer, then looks for the resulting terms. Use it on text fields — anything a human typed.

term does not analyse anything. It looks for the exact term you gave. Use it on keyword fields, numbers, booleans and dates.

Using term on a text field returns nothing, silently, because the field contains lowercased tokens and you asked for the original string.

The match family

{ "match": { "title": "mountain cedar" } }

By default the terms are OR-ed: a document with either word matches, and one with both scores higher. Change that with operator:

{ "match": { "title": { "query": "mountain cedar", "operator": "and" } } }

Or ask for some of them, which is often the better answer than either extreme:

{ "match": { "title": { "query": "cedar mountain lakefront", "minimum_should_match": "2" } } }
// matches "Cedar Cabin with Mountain Views" — two of the three terms

minimum_should_match takes a number, a percentage ("75%"), or a combination expression. On a real search box, "at least 75% of the terms" behaves far better than and, which returns nothing the moment a user adds a word, or or, which returns everything.

match_phrase requires the terms adjacent and in order:

match         "mountain cedar"   ->  1 hit    (both words present, any order)
match_phrase  "mountain cedar"   ->  0 hits   (not adjacent, wrong order)
match_phrase  "mountain views"   ->  1 hit

This is what token positions from lesson 5 are for. slop loosens it — "slop": 2 allows terms to be up to two positions apart, so "mountain with views" would still match "mountain views".

match_phrase_prefix treats the last term as a prefix, which makes it the cheapest search-as-you-type there is. It is genuinely useful and it has a limit worth knowing: it expands the prefix against the index, capped by max_expansions (50 by default), so on a large index it silently considers only some of the matching terms.

multi_match

One query, several fields:

TEXT_FIELDS = ["city^3", "title^2", "description", "state", "country"]

The ^ is a boost. city^3 says a match on the city counts for three times as much as one in the description — a guest typing "Austin" means the place, not a description mentioning it in passing.

The parameter that decides its behaviour is type, and the default is not always what you want. Lesson 11 takes that apart properly, including a real bug it caused; for now it is enough to know that best_fields (the default) scores each field separately and keeps the best, which has consequences when you combine it with operator: and.

The other query types

The match family covers text. These cover everything else, and between them and bool you can express nearly any real search.

term and terms. One exact value, or a list of them:

{ "term":  { "property_type": "CABIN" } }
{ "terms": { "property_type": ["CABIN", "LOFT"] } }

terms is an OR. That matters when a filter panel has two boxes ticked, and lesson 10 covers why StayHub deliberately does not use it for amenities.

range, for numbers and dates:

{ "range": { "price_per_night": { "gte": 100, "lte": 250 } } }
{ "range": { "created_at": { "gte": "now-7d/d", "lt": "now/d" } } }

Date maths is evaluated by the cluster. now-7d/d is "seven days ago, rounded down to midnight", and the rounding is what makes the query cacheable — an unrounded now is a different value every millisecond, so nothing can be reused.

exists asks whether a field has any indexed value:

{ "exists": { "field": "rating_average" } }

Remember lesson 4: a null, an empty array and a value skipped by ignore_malformed all count as absent. There is no "is null" query, because there is no null in the index — only present and absent.

prefix, wildcard, regexp match against the term dictionary rather than a term:

{ "prefix":   { "city.raw": "San" } }
{ "wildcard": { "city.raw": "San*" } }

Use sparingly. A leading wildcard ("*Francisco") forces a scan of every term in the field, which is the same problem as LIKE '%...' in SQL and for the same reason. If you need prefix matching at speed, index prefixes with edge_ngram as lesson 5 showed.

ids fetches by document id inside a query, which is how you combine "these specific documents" with other clauses.

match_all and match_none do what they say. match_all is not a placeholder to be embarrassed about — a browse page with no search term is genuinely a match_all with filters, and it is what StayHub sends when the search box is empty:

    if not req.q:
        # No text query still needs a scoreable clause, or ES has nothing to rank by.
        return {"match_all": {}}

Letting users type operators

Two queries accept a syntax rather than a plain string. query_string is the full one: AND, OR, NOT, field prefixes, wildcards, ranges, fuzzy operators. It is powerful and it throws a parse error on malformed input — which means an unbalanced quote from a user is a 400, not a search.

simple_query_string is the one to expose. Same idea, a smaller syntax (+, |, -, ", *), and it never throws: bad syntax is treated as literal text.

{ "simple_query_string": {
    "query": "cabin -shared \"mountain views\"",
    "fields": ["title^2", "description"],
    "default_operator": "and"
} }

Whether to offer this at all is a product decision. Most users never type an operator, and the ones who do expect them to work exactly as Google's do, which they do not. For most search boxes a well-tuned multi_match beats teaching a syntax.

Choosing what comes back

_source is the whole stored document, and returning all of it for twenty hits is often the largest part of a response:

"_source": false
"_source": ["title", "city", "price_per_night"]
"_source": { "excludes": ["description"] }

The exclude form is usually the practical one: keep everything except the two fields that are long and are not shown in a result card. It costs nothing and can halve a response.

Note this is purely about transfer. The document is read from disk either way, so this saves bandwidth and serialisation, not I/O.

Pagination, and the wall

{ "from": 40, "size": 20 }   // page 3

Simple, and it stops working at a specific point:

{ "from": 10000, "size": 10 }

illegal_argument_exception
Result window is too large, from + size must be less than or equal to: [10000]
but was [10010]. ... This limit can be set by changing the [index.max_result_window]

The limit is not arbitrary. To return results 10,000 to 10,010, every shard must find and rank its top 10,010 matches and send them to the coordinating node, which merges and throws away 10,000 of them. The cost is borne per shard, so on a ten-shard index that is 100,100 documents ranked to return ten.

The error suggests raising index.max_result_window. Resist it. The setting exists to stop you doing this, and raising it converts a clear error into an out-of-memory failure on a bigger page.

The real answers are: do not let users page that deep — nobody reaches page 500 of a search, and a crawler doing so is a load problem to fix rather than to serve — or use search_after, which pages by the sort value of the last hit instead of an offset. Lesson 12 covers it, because it depends on understanding sort values.

You may still see scroll recommended. It is for exporting an entire index in a batch job, not for user-facing pagination: it holds a consistent snapshot open, which costs resources on every shard for as long as the scroll lives. For pagination use search_after; for export use a point-in-time plus search_after, which is what replaced scroll.

track_total_hits

By default, Elasticsearch stops counting matches at 10,000:

"total": { "value": 10000, "relation": "gte" }

relation is the field to read. "eq" means exact, "gte" means "at least this many, we stopped counting". Rendering "10,000 results" from that is wrong, and it is wrong in a way that looks suspiciously round.

Counting every match is real work, which is why it is off by default. Ask for it when you need it:

        "track_total_hits": True,

StayHub does, because the UI shows a page count, and honest pagination counts matter more than the microseconds it costs at this scale. On a large index, consider "track_total_hits": 1000 — count exactly up to a thousand, then stop — which gives you "About 1,000+ results" cheaply.

Highlighting

Marking the matched terms in the result is one block:

    return {
        "pre_tags": ["<mark>"],
        "post_tags": ["</mark>"],
        "encoder": "html",
        "fields": {
            "title": {"number_of_fragments": 0},  # 0 = return the whole field, marked up
            "city": {"number_of_fragments": 0},
            "description": {"fragment_size": 140, "number_of_fragments": 1},
        },
    }

The result arrives per hit, alongside _source:

"highlight": {
  "city":  ["<mark>San</mark> <mark>Francisco</mark>"],
  "title": ["Sunlit <mark>Loft</mark> in the Mission"],
  "description": ["A bright corner <mark>loft</mark> two blocks from Dolores Park."]
}

number_of_fragments: 0 returns the whole field marked up, which is right for a title. For a long description you want a fragment around the match instead, which is what fragment_size and a fragment count give you.

The setting that is not optional

encoder: "html". Here is why, with a description containing a script tag:

without an encoder:
  "A lovely <script>alert(1)</script> <mark>cabin</mark> retreat"

with encoder: "html":
  "A lovely &lt;script&gt;alert(1)&lt;&#x2F;script&gt; <mark>cabin</mark> retreat"

Highlighting inserts markup into the original text and hands you back a string your UI is expected to render as HTML. Without the encoder, any HTML that was in the source text comes through live. The encoder escapes the source first, then adds the marks.

This is a genuine cross-site scripting hole in a feature nobody thinks of as a security surface, and it is one setting.

One performance note: highlighting re-analyses the field per hit, so it is not free. StayHub only asks for it when there is a query term that could be marked:

    if req.q:
        body["highlight"] = _highlight_spec()

Collapsing: one result per group

A problem that turns up on every search page eventually. A host has eight near-identical listings in the same building, and a search for that neighbourhood returns all eight, pushing everything else off the page.

collapse solves it in one clause:

{
  "query": { "match": { "description": "loft" } },
  "collapse": {
    "field": "host_id",
    "inner_hits": { "name": "more", "size": 2 }
  }
}

One hit per host_id — the highest scoring one — with two more available under inner_hits for a "3 more from this host" link. The collapse field must be a keyword or a number, because it groups on terms.

One caveat that matters for pagination: hits.total still counts the uncollapsed matches. So a page showing 10 collapsed results out of a reported 80 is not lying, exactly, but it is not what a user will assume. If you need the group count, that is a cardinality aggregation alongside — lesson 13.

Did you mean?

Suggesters are a separate top-level key, and they can ride along with a normal search:

{
  "query": { "match": { "title": "cabbin" } },
  "suggest": {
    "did_you_mean": {
      "text": "cabbin",
      "term": { "field": "title" }
    }
  }
}

The term suggester works word by word against the terms actually in the index, so its suggestions are always things that would return results. phrase is the better one for a multi-word query — it scores whole candidate phrases rather than correcting each word in isolation, which stops "cabin in tahoe" being corrected into a phrase nobody wrote.

The practical pattern: run the search, and if it returns few or no hits, show the suggestion. The mistake is showing "did you mean" above a page of perfectly good results.

Note that fuzziness on the query and a suggester solve overlapping problems differently. Fuzziness silently finds the results; a suggester tells the user what you think they meant. A search box usually wants both, and it wants the fuzzy match not to be so broad that the suggester never fires.

Searching more than one index

GET /stayhub-properties,stayhub-experiences/_search
GET /logs-2026.08.*/_search
GET /_all/_search

Comma-separated names, wildcards, or an alias standing for several indexes. The results interleave and are ranked together, and _index on each hit tells you where it came from.

Two things to know. Fields with the same name must have compatible types across the indexes, or the search fails — which is the failure _field_caps from lesson 4 exists to predict. And relevance across indexes with very different sizes is unreliable, because term frequencies are per shard, so a term that is rare in one index and common in another scores inconsistently.

When it matters, "search_type": "dfs_query_then_fetch" gathers global term statistics first and ranks with those. It costs an extra round trip across the shards and it is the correct fix for "the same query ranks differently depending on which shard a document landed on".

Long-running and consistent searches

Two features that come up once a system gets large enough.

Point in time. A normal search sees whatever the index looks like at that instant, so paging through results while documents are being written can show a document twice or skip one. A PIT freezes a view of the index:

POST /stayhub-properties/_pit?keep_alive=1m
# { "id": "46ToAwMD..." }

Pass that id in subsequent searches (instead of an index name) and every page sees the same snapshot. Combined with search_after this is the modern replacement for scroll, and it is what you want for any export.

Async search. For a query over months of data that will take a minute, _async_search returns immediately with an id and lets you poll for partial results as shards report in. Right for analytics, wrong for a search box — if a user-facing search needs this, the fix is the query or the model, not asynchrony.

Tools for when a query misbehaves

_validate/query?explain tells you whether a query is even well-formed, and how it was parsed, without running it. Useful for a query built by string concatenation.

_explain takes one document id and one query and returns the scoring arithmetic. This is lesson 11's main tool.

"profile": true added to a search returns a per-shard breakdown of where the time went — which Lucene query ran, how long it built, how long it collected. The output is large and it is the only way to answer "which clause is slow" rather than guessing.

And _count when you want a number and no documents:

curl -s localhost:9200/stayhub-properties/_count -H 'Content-Type: application/json' \
  -d '{"query": {"term": {"property_type": "CABIN"}}}'

Several searches in one request

_msearch takes the same newline-delimited format as bulk and runs several searches in one round trip:

POST /_msearch
{"index": "stayhub-properties"}
{"query": {"term": {"property_type": "CABIN"}}, "size": 3}
{"index": "stayhub-properties"}
{"query": {"term": {"property_type": "LOFT"}}, "size": 3}

Right for a homepage that renders four independent carousels. Not a substitute for aggregations — if the searches differ only by a filter and you want counts, one aggregation does it in one pass. Lesson 13.

Guarding a search endpoint

Two parameters worth knowing, and one of them behaves differently from how it reads.

"timeout": "2s" does not abort the search. It returns whatever has been collected so far with timed_out: true. Useful as a backstop, useless as a guarantee.

"terminate_after": 100 stops each shard after it has collected that many matches. On a small index it does nothing at all:

{"terminate_after": 3, "size": 0}
-> total: {"value": 12, "relation": "eq"}, terminated_early: false

Twelve, not three, and terminated_early: false — the shard finished before it had a reason to stop. It only bites when there is genuinely more work than the limit, which is exactly when you want it and exactly why it is hard to test.

Neither replaces a client-side timeout. StayHub sets one on the client and turns a failure into an honest error rather than an empty result:

    if not es_available():
        raise ApiException(
            "Search is temporarily unavailable. Please try again in a moment.",
            status_code=503,
        )

That distinction matters more than it looks. "No listings match your search" and "search is broken" look identical to a user, and only one of them is their fault.

What actually makes a search slow

Worth knowing before you start optimising, because the usual first guess is wrong.

Deep pagination is the biggest one, and it is entirely self-inflicted. Every shard ranks from + size documents regardless of how many you keep.

Returning too much. Twenty full _source documents with long descriptions is often more time in serialisation and transfer than in the search itself. This is the cheapest thing to fix and the least often looked at.

Scoring things that do not need scoring. A clause in must computes a relevance contribution for every matching document and can never be cached. The same clause in filter does neither. Lesson 10.

Wildcards, regexes and scripts, which run against terms or documents rather than using the index.

Too many shards, so every search fans out further than it needs to and pays a merge across all of them. Lesson 18.

Note what is not on the list: the number of documents. An inverted index lookup barely cares whether there are ten thousand documents or ten million — that is the entire point of it. A search that got slower as the index grew is usually one of the five above becoming more expensive, not the lookup itself.

Measure before changing anything. "profile": true attributes time to individual query components per shard, and took versus your client's own timing tells you whether the problem is even inside Elasticsearch. On the index in this track, a search without facets is 4.5 ms from Python and reports took: 2 — more than half the wall clock is not the search.

Putting it together

StayHub's search body, assembled from everything above:

    body: dict[str, Any] = {
        "query": {"bool": {"must": [_text_clause(req)], "filter": _flatten(groups)}},
        "from": (req.page - 1) * req.page_size,
        "size": req.page_size,
        "track_total_hits": True,
    }
    if sort:
        body["sort"] = sort
    if req.q:
        body["highlight"] = _highlight_spec()
    if with_facets:
        body["aggs"] = build_aggs(req)
    return body

Note what is conditional. Highlighting only when there is a term to mark. Sort only when it is not the default. Aggregations only when the caller renders them — the result page wants facets, a map's pan handler does not, and asking for them anyway is the easiest way to double the cost of a search without noticing.

Two habits are worth adopting from that shape rather than from the details. Build the body as a data structure and assemble it from small named functions, not as a template string — a query built by string concatenation cannot be unit tested, and it is where injection bugs live. And keep each optional part behind the condition that justifies it, so the cost of a feature is paid only by the callers who use it.

Both matter more than they sound. A search body grows: filters, facets, highlighting, sorting, suggestions, collapse. The difference between a query builder you can still reason about after a year and one nobody wants to touch is almost entirely whether those pieces stayed separable.

The one piece not yet explained is bool, and specifically why the text query goes in must while everything else goes in filter. That is the next lesson, and it is the single decision with the largest effect on both correctness and speed.