Sorting looks like the simplest thing in the search API, and then the first field you try refuses outright. This lesson covers why, what to do about it, and the thing sort values are secretly for — paginating past the 10,000-result wall.
The basics
{
"query": { "match_all": {} },
"sort": [
{ "price_per_night": "asc" },
{ "rating_count": "desc" }
]
}A list, applied in order. Ties on the first key are broken by the second. Without any
sort, results come back by _score descending, which is why a relevance
search needs no sort at all.
The shorthand forms are worth recognising too, since most examples online use them.
"sort": "price_per_night" is a plain ascending sort on one field, and
"sort": ["price_per_night", "rating_count"] sorts ascending on both. The expanded
object form is the one to write in application code, because every option below — order,
missing, mode, unmapped type — only exists there.
Why sorting on your title field fails
{ "sort": [ { "title": "asc" } ] }illegal_argument_exception
Fielddata is disabled on [title] in [stayhub-properties-000001]. Text fields are not
optimised for operations that require per-document field data like aggregations and
sorting, so these operations are disabled by default.The error is unusually good, and the reason behind it is worth understanding because the same constraint governs aggregations in the next lesson.
The inverted index answers "which documents contain this term". Sorting needs the opposite: "what is this document's value". That is a different data structure, called doc values, built at index time for most field types.
text fields do not get doc values, because a text field does not have a
value — it has a list of tokens. "Cedar Cabin with Mountain Views" is five terms, and sorting
by them means nothing.
Elasticsearch can build the equivalent structure on the fly, in memory, which is what
fielddata means. It is disabled by default because on a large text field it consumes
enormous heap and is the classic way to make a cluster fall over. Enabling it is almost always the
wrong answer.
The right answer is the multi-field from lesson 3:
{ "sort": [ { "city.raw": "asc" } ] }
# ['Austin', 'Big Bear Lake', 'Brooklyn']The keyword sibling has one value, has doc values, and sorts. If you plan to sort or
group by a string, that is the mapping decision that makes it possible, and it has to be made before
the documents are indexed.
And it sorts by bytes, not by language
A keyword sort is a byte-order sort of the raw value. So uppercase sorts before
lowercase, and accented characters sort after every unaccented one — "Zurich" before
"Évian". For an internal tool nobody cares. For a user-facing A–Z list it looks broken.
The fix is an icu_collation_keyword field from the ICU analysis plugin, which stores
a locale-aware sort key. That is a plugin install and a mapping change, so it is worth deciding
early whether alphabetical order will ever be shown to a user.
What happens to _score
Add a sort to a scored query and look at the response:
{"query": {"match": {"title": "cabin"}}, "sort": [{"price_per_night": "asc"}]}
max_score: null
hit _score: [null]
hit sort: [[189.0]]_score is null. Not zero, not preserved — not computed at
all. Elasticsearch skips scoring entirely when it does not need it, which is a real optimisation and
a real trap.
The trap: every relevance signal you tuned in lesson 11 is gone the moment a user clicks "sort by
price". That is fine and expected. What is not fine is a "sort by relevance" option implemented as a
sort on some field, or a secondary sort added to a relevance search "just to make it stable" —
which silently demotes _score to a tie-breaker.
To keep scores while sorting, ask for them explicitly, or sort on _score first:
{ "track_scores": true, "sort": [ { "price_per_night": "asc" } ] }
{ "sort": [ "_score", { "rating_average": "desc" } ] }The second is a genuinely useful pattern: rank by relevance, break ties by rating.
Tie-breakers, and why you need one
elif req.sort == "rating":
# Tie-break on review count: a lone 5-star review should not outrank forty 4.9s.
sort = [{"rating_average": "desc"}, {"rating_count": "desc"}]Two reasons to add a second key, and the second one is not obvious.
The first is meaning. Sorting purely by average rating puts a listing with one five-star review above one with forty at 4.9. Review count as a tie-break fixes the ordering people actually expect.
The second is stability. When two documents have identical sort values, their relative order is undefined — and it can differ between requests, because it depends on segment layout and which shard replied first. With paging, that means a document can appear on both page 1 and page 2, or on neither.
The cure is a final tie-break on something unique:
{ "sort": [ { "price_per_night": "asc" }, { "public_id": "asc" } ] }This bug is invisible in testing, because it needs enough documents with equal values to matter, and it produces a support ticket that reads "sometimes a listing is missing".
Sorting is a mapping decision made early
Nearly everything above traces back to a choice made before any document was indexed, which is worth stating plainly because it is the practical lesson.
Sorting alphabetically needs a keyword field, so the multi-field has to be in the
mapping. Sorting correctly by locale needs an icu_collation_keyword, which needs a
plugin. Stable pagination needs a unique field to tie-break on, which means an id in the document
rather than relying on _id — sorting on _id works but is expensive
and discouraged, because it has no doc values. Sorting on a computed total needs that total stored.
And sorting by distance needs a geo_point, not two doubles.
None of those can be added to existing documents without a reindex, as lesson 3 established. So the useful exercise, before an index goes live, is to write down every ordering the UI might ever offer — including "A to Z" and "nearest first", which are the two people forget — and check the mapping supports each of them.
Missing values
A document with no value for the sort field has to go somewhere:
{ "sort": [ { "rating_average": { "order": "asc", "missing": "_last" } } ] }_last is the default for ascending, _first for descending — so
"missing" always sorts as though it were the worst value, which is usually right. You can substitute
a value instead: "missing": 0 treats unrated listings as zero-rated.
Note the difference from null_value in lesson 4. That one changes what is
indexed and affects every query. missing affects only this sort. Prefer
missing when the choice is presentational, which it usually is.
Fields that are not in the mapping
Sorting on a field that does not exist is an error, not an empty result:
query_shard_exception
No mapping found for [nope] in order to sort onThat is right for a typo and wrong when searching several indexes where only some have the field.
unmapped_type tells the shards without it to treat it as missing:
{ "sort": [ { "nope": { "order": "asc", "unmapped_type": "long" } } ] }
# ok — sort values: [9223372036854775807]That value is Long.MAX_VALUE, which is how "missing, sorted last ascending" is
represented. Seeing it in a sort array is a reliable sign that a field you expected is absent.
Sorting on a field with many values
Lesson 4 noted that any field can hold an array. So what does sorting on an array do?
{ "sort": [ { "prices": { "order": "asc", "mode": "min" } } ] }mode picks which of the values represents the document: min,
max, sum, avg, or median. The default is
min for ascending and max for descending — which is to say, the
value that puts the document earliest.
This is quietly the right behaviour for the common case: sorting listings by price when each has several seasonal rates should use the cheapest one, because that is the number a user is comparing. But it is a default worth stating explicitly in the query, because "why is this listing ranked at $78 when its page says $210" is a confusing bug to receive.
Sorting on a nested field needs a nested clause in the sort itself,
naming the path and optionally a filter — "sort by the rating of reviews written this year"
is expressible, and it is one of the few places where nested's cost buys something a flat field
cannot.
What sorting costs
Less than people fear, and in one specific case much more.
A sort on a numeric or keyword field with doc values is cheap. The values are stored in a
column-oriented structure built for exactly this, read sequentially, and the shard keeps only the
top from + size in a small heap. It is often faster than sorting by
_score, because no scoring happens at all.
What is expensive: script sorts, which run per document; sorts on fielddata-enabled
text fields, which build an in-memory structure over every term; and any sort combined with deep
pagination, where the heap the shard maintains has to hold from + size entries.
There is also an optimisation worth knowing about because it explains a puzzling result. When you
sort by a field, do not need a total count, and the index is sorted the same way, Elasticsearch can
stop early — it knows nothing later can rank higher. The tell is
"total": {"relation": "gte"} on a query you expected an exact count for. Asking for
track_total_hits: true disables the shortcut, which is the trade: an honest count, or
the speed.
Sorting by distance
sort.append(
{
"_geo_distance": {
"location": origin,
"order": "asc",
"unit": "km",
# A listing with no coordinates sorts last instead of failing the query.
"ignore_unmapped": True,
}
}
)A computed sort rather than a stored field, and lesson 14 covers the geo side. What belongs here is the trick it enables, because it applies to every sort key.
Sort values come back on every hit
Each hit carries the values it was sorted by, in the same order as the sort array:
"sort": [128.0, "6e6e5150-78f0-4a65-bc9d-eaae63aac4f6"]That is free information — Elasticsearch computed it in order to sort — and it is worth using rather than recomputing.
StayHub reads distance out of it instead of running a script field:
"""The `sort` array, and the index within it that carries the distance (or None).
Elasticsearch returns each hit's sort values in the same order as the sort array, so once a
`_geo_distance` entry is in there its position IS how you read the distance back out. That is
cheaper than a `script_field` and exact, because ES already computed it to sort by it.
"""Which is why a coordinate supplied without sort=distance still appends a geo entry
— and why an explicit "_score" has to come first, or the appended entry would
silently reorder a relevance search by proximity:
if req.sort != "distance":
# Distance is wanted as a NUMBER on every hit, not as the ordering. An explicit
# `_score` first is required once anything else is in the array, because a bare
# `[_geo_distance]` would silently reorder a relevance search by proximity.
if not sort:
sort = ["_score"]
geo_index = len(sort)search_after: the wall, solved
Lesson 9 left from + size failing at 10,000. Sort values are the way past it.
Instead of "skip 10,000 documents", say "start after this one". Page one is ordinary:
{
"size": 2,
"sort": [ { "price_per_night": "asc" }, { "public_id": "asc" } ]
}page1: [ (78.0, [78.0, "a05193d5-eb94-49bf-8d16-137682aaa33e"]),
(128.0, [128.0, "6e6e5150-78f0-4a65-bc9d-eaae63aac4f6"]) ]Take the last hit's sort array verbatim and hand it back:
{
"size": 2,
"sort": [ { "price_per_night": "asc" }, { "public_id": "asc" } ],
"search_after": [128.0, "6e6e5150-78f0-4a65-bc9d-eaae63aac4f6"]
}Now every shard seeks directly to that position instead of ranking everything before it. The cost is the same for page 2 and page 2,000 — which is the whole point.
Three requirements come with it. The sort must end in a unique tie-break, or the seek point is ambiguous and you will skip or repeat documents. You cannot jump to an arbitrary page, only forward one at a time — so this gives you infinite scroll and "next", not "go to page 57". And by default each page sees the index as it is now, so concurrent writes can shift things; pair it with a point-in-time from lesson 9 when consistency matters.
Implementing it without leaking the cursor
The sort array becomes the pagination cursor, and it is worth thinking for a moment about how it reaches the client. Handing back a raw JSON array of internal sort values works and exposes exactly what your sort keys are, including any internal identifier used as the tie-break.
The usual treatment is to encode it — base64 the array, or better, sign it — and
return it as an opaque nextCursor string. That keeps the API honest about what it is
(a resume token, not a page number) and lets you change the sort keys later without breaking
clients holding old cursors.
It also solves a subtler problem. A client that can construct arbitrary
search_after values can ask a shard to seek anywhere, which is harmless with your own
sort keys and less harmless once one of them is a user id. Opaque and validated beats transparent
and trusted.
One design consequence worth accepting up front: an API paginated this way cannot offer "page 57" and should not pretend to. Infinite scroll and a "load more" button are the honest UI, and they are also what users of a search result set actually do.
Sorting by something you did not store
Occasionally the order depends on a computation:
{ "sort": [ { "_script": {
"type": "number",
"script": { "source": "doc['price_per_night'].value + doc['cleaning_fee'].value" },
"order": "asc"
} } ] }It works, and it runs for every matching document, which makes it the slowest sort available. On a large result set it is a bad idea.
The alternative is nearly always better: compute the value at index time and store it. A
total_price field costs a few bytes per document and turns a script sort into a doc
values lookup. That is a general principle in Elasticsearch — work moved from query time to
index time is paid once per document instead of once per document per query.
The sort options a search page needs
Putting it together, StayHub's four:
if req.sort == "price_asc":
sort = [{"price_per_night": "asc"}]
elif req.sort == "price_desc":
sort = [{"price_per_night": "desc"}]
elif req.sort == "rating":
sort = [{"rating_average": "desc"}, {"rating_count": "desc"}]
elif req.sort == "distance" and origin:
sort = [] # the geo entry appended below becomes the primary sortAnd "relevance" leaves the list empty, which means _score — the default, and
the whole point of having a text query at all.
Two things worth copying from that. The sort choice is a small closed set validated at the API boundary, not a field name from the query string — letting a user name the sort field means letting them sort by a text field and get a 400, or by a field you never intended to expose. And "relevance" is the default, because a search page that defaults to price ascending is a search page where the cheapest listing always wins regardless of what was typed.
The last of those is worth one more sentence, because it is a decision with a name. A search page that defaults to anything but relevance has effectively turned off the search engine — the ranking work from lesson 11 does nothing, and the query becomes a filter. That is a legitimate choice for a browse page. It is rarely the right one for a page with a search box on it.
Sorting answers "in what order". The next lesson answers the question a filter panel asks instead: how many of each.