Aggregations are the other half of Elasticsearch. Search answers "which documents"; aggregations answer "how many, grouped how" — and they do it in the same request, over the same result set, which is the thing that makes a filter panel possible at all.
Most of this lesson is ordinary. The last third is the part almost every tutorial leaves out, and it is the part that decides whether your filter panel works.
Three kinds
Metric aggregations compute a number over a set of documents: avg,
sum, min, max, stats,
cardinality, percentiles.
Bucket aggregations group documents: terms, range,
date_histogram, histogram, filters, nested.
Pipeline aggregations operate on the output of other aggregations rather than on documents — a moving average over a date histogram, the bucket with the highest value.
Bucket aggregations can contain other aggregations, which is where the power is: group, then compute, then group again.
A first aggregation
{
"size": 0,
"aggs": {
"cities": { "terms": { "field": "city.raw", "size": 20 } }
}
}"size": 0 is doing real work there. Aggregations are computed regardless, so this
says "I want the counts, not the documents" — and skipping the fetch phase for twenty full
_source documents is a meaningful saving on a request that does not use them.
"aggregations": {
"cities": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 9,
"buckets": [
{ "key": "Austin", "doc_count": 1 },
{ "key": "Big Bear Lake", "doc_count": 1 },
{ "key": "Brooklyn", "doc_count": 1 }
]
}
}Two fields above the buckets that matter and are routinely ignored. Both are covered below.
Aggregate on keyword, never on text
{ "aggs": { "c": { "terms": { "field": "city" } } } }Fielddata is disabled on [city]. Text fields are not optimised for operations that
require per-document field data like aggregations and sorting...The same error as sorting in lesson 12, for the same reason: aggregations need doc values, and a
text field has tokens rather than a value.
Even if you enable fielddata, the result is wrong rather than slow. Aggregating a
text field buckets its terms, so "San Francisco" produces buckets for san and
francisco, and a city with a two-word name is counted twice under names nobody
recognises.
This is what the multi-field is for, and StayHub says so at the point of use:
# `city.raw`, not `city`. The analyzed `text` field would count "san", "francisco"
# and "san francisco" as three different cities — aggregations run on TERMS, and
# the terms of a text field are its tokens. That is what the `.raw` multi-field in
# the mapping is for.
"cities": scoped("city", {"terms": {"field": "city.raw", "size": 20}}),terms is approximate, and it tells you so
This surprises people, and it is why those two extra fields exist.
A terms aggregation runs on every shard independently. Each shard returns its own top
N, and the coordinating node adds them up. A term that is 11th on every shard but would be 3rd
overall can be missing entirely, and a term that is returned may have an undercounted total, because
some shard did not include it in its top N.
doc_count_error_upper_bound is the worst-case error on the counts — 0 means
exact. sum_other_doc_count is how many documents fell outside the returned buckets:
doc_count_error_upper_bound: 0
sum_other_doc_count: 9
buckets: 3Three buckets asked for, nine documents in cities not shown. If your UI says "showing all cities", that number is how you know whether it is lying.
The lever is shard_size — how many buckets each shard returns before the merge,
defaulting to a multiple of size. Raising it improves accuracy and costs memory. On a
single-shard index none of this applies and the counts are exact, which is one more reason a small
index with one shard is a pleasant place to learn.
cardinality has the same caveat in a stronger form: it is a
HyperLogLog++ approximation, accurate to within a percent or so, and it is designed that way because
counting exact distinct values across shards would mean shipping every value to one node.
{ "aggs": { "cities": { "cardinality": { "field": "city.raw" } } } }
# { "value": 12 }Do not use it for anything that must reconcile — billing, an invoice line, a compliance count. Do use it for "roughly how many distinct users", which is what it is for.
Nesting
Bucket first, then compute inside each bucket:
{
"size": 0,
"aggs": {
"by_type": {
"terms": { "field": "property_type", "size": 3 },
"aggs": {
"avg_price": { "avg": { "field": "price_per_night" } }
}
}
}
}HOUSE 5 docs avg $235.60
APARTMENT 2 docs avg $186.50
CABIN 2 docs avg $265.50Nest as deeply as you like — city, then property type, then price statistics. The warning is
that bucket counts multiply. Twenty cities by ten types by twelve months is 2,400 buckets, each
carrying its sub-aggregations, all held in memory while the request runs. There is a
search.max_buckets limit (65,536) that exists to stop a single query taking a node
down, and hitting it means the query needs rethinking rather than the limit raising.
The bucket types worth knowing
range puts documents into bounds you define. The bounds are half-open —
from inclusive, to exclusive — which is exactly right for money and
catches people who expect both ends inclusive:
PRICE_RANGES: list[dict[str, Any]] = [
{"key": "0-100", "to": 100.0},
{"key": "100-200", "from": 100.0, "to": 200.0},
{"key": "200-350", "from": 200.0, "to": 350.0},
{"key": "350-500", "from": 350.0, "to": 500.0},
{"key": "500+", "from": 500.0},
]Note these are hard-coded, and the comment above them explains why: a filter panel whose buckets move as you filter is unusable, because the row you were aiming at slides away as you click.
date_histogram buckets by time, and calendar_interval is the one to
use over fixed_interval for anything a human reads — a calendar month is a real
month, not 30 days:
{ "date_histogram": {
"field": "created_at",
"calendar_interval": "month",
"min_doc_count": 0,
"time_zone": "America/Los_Angeles"
} }min_doc_count: 0 keeps empty buckets, which a chart needs and a list does not. And
time_zone matters more than it looks: without it, "daily signups" are bucketed by UTC
midnight, so every chart is shifted by your users' offset and nobody can say why the numbers
disagree with the dashboard.
filters is the escape hatch — arbitrary named buckets from arbitrary queries,
for groupings that are not a field:
{ "filters": { "filters": {
"budget": { "range": { "price_per_night": { "lt": 120 } } },
"family": { "range": { "max_guests": { "gte": 5 } } }
} } }Pipeline aggregations
The third family, and the one most people never reach for. A pipeline aggregation takes the output of a sibling or parent aggregation rather than documents:
{
"size": 0,
"aggs": {
"monthly": {
"date_histogram": { "field": "created_at", "calendar_interval": "month" },
"aggs": { "revenue": { "sum": { "field": "price_per_night" } } }
},
"trend": { "moving_fn": { "buckets_path": "monthly>revenue",
"window": 3, "script": "MovingFunctions.unweightedAvg(values)" } },
"best": { "max_bucket": { "buckets_path": "monthly>revenue" } },
"running": { "cumulative_sum": { "buckets_path": "monthly>revenue" } }
}
}A three-month moving average, the best month, and a running total — computed in the cluster
rather than in your application. buckets_path is the syntax to learn: >
descends into a sub-aggregation, and the last segment names the metric.
Whether to use them is a genuine judgement call. Doing the arithmetic in your application is easier to read, easier to test, and moves a small amount of data. Doing it in the cluster is right when the intermediate buckets are large and the result is small — a moving average over ten thousand daily buckets is worth computing where the data already is.
Getting documents out of a bucket
Two aggregations return documents rather than numbers, and they answer questions that otherwise need a second request.
top_hits nested inside a bucket returns the best few documents per group
— "the three highest-rated listings in each city" in one query:
{ "terms": { "field": "city.raw", "size": 5 },
"aggs": { "best": { "top_hits": {
"size": 3,
"sort": [ { "rating_average": "desc" } ],
"_source": ["title", "price_per_night"]
} } } }This is the aggregation-side answer to the same problem collapse solved in lesson 9.
Use collapse when you want one flat, paginated result list; use
top_hits when you want genuine groups with their counts.
significant_terms is the more interesting one. Rather than returning the most common
terms, it returns the ones that are unusually common in this result set compared to the
index as a whole. Applied to search results it produces "people who searched for this also cared
about..." with no model and no training data.
Faceted search, and the part nobody explains
Here is the whole reason aggregations exist on a search page. A filter panel shows "Austin (48) · Denver (31)", and the counts have to answer the question the user is actually asking, which is "how many results would I get if I picked this instead".
The obvious implementation does not do that.
Put the aggregation in the same request as the query and it is scoped to the query — including the filters. So the moment a guest ticks "Austin", the city aggregation only sees Austin listings, the city list collapses to a single row, and there is now no way to switch cities without clearing the filter first.
Same for every multi-select. Tick "wifi" and every other amenity count becomes "wifi AND that", which reads as though the catalogue shrank.
The fix: each facet blind to its own filter
def scoped(exclude: str, inner: dict[str, Any]) -> dict[str, Any]:
return {
"filter": {"bool": {"must": [text], "filter": _flatten(groups, exclude=exclude)}},
"aggs": {"buckets": inner},
}
return {
"facets": {
"global": {},
"aggs": {
"cities": scoped("city", {"terms": {"field": "city.raw", "size": 20}}),
"property_types": scoped("property_type", {"terms": {"field": "property_type", "size": 20}}),
...
"amenities": scoped("amenities", {"terms": {"field": "amenities", "size": 30}}),
"price_ranges": scoped(
"price", {"range": {"field": "price_per_night", "keyed": False, "ranges": PRICE_RANGES}}
),
"price_stats": scoped("price", {"stats": {"field": "price_per_night"}}),
},
}
}Three layers, and each one is load-bearing:
global -> escape the main query entirely
filter -> re-apply text + all filters EXCEPT this facet's
terms/range -> countglobal is what makes the re-application possible. Without it the aggregation is
still scoped to the main query, and dropping a clause changes nothing at all.
This is also why lesson 10 built the filters grouped by control rather than as a flat list: you cannot un-apply a clause that has forgotten which control produced it.
What it looks like when it works
Filtering by one amenity on the twelve-listing index:
?amenities=hot-tub -> 3 results
amenities facet (own filter dropped):
kitchen 12 · wifi 12 · free-parking 8 · heating 7 · washer 7 ...
cities facet (narrowed, as it should be):
Big Bear Lake 1 · Joshua Tree 1 · Lake Tahoe 1The amenities list stays complete, so the guest can switch. The cities list narrows to the three cities that have hot tubs, which is the useful answer to "where could I go next".
post_filter, the simpler version
There is a lighter-weight approach for the single-facet case:
{
"query": { "match_all": {} },
"post_filter": { "term": { "property_type": "CABIN" } },
"aggs": { "t": { "terms": { "field": "property_type" } } }
}
# hits: 2
# buckets: HOUSE 5, APARTMENT 2, CABIN 2, CONDO 1, LOFT 1, VILLA 1post_filter runs after the aggregations, so the hits are filtered and the
counts are not. Two hits, six full buckets.
That is exactly right when there is one facet. It stops working with several, because
post_filter is all-or-nothing — every facet is computed without every filter, so
the counts stop reflecting the other selections. The global-plus-filter
construction is the general answer, and post_filter is the one to reach for when there
genuinely is only one multi-select.
What it costs
Facets are not free, and the arithmetic is easy: one filter aggregation per facet,
all inside one request. Measured on the twelve-listing index, median of 30 round trips:
search without facets 4.5 ms
search with facets 11.0 msRoughly two and a half times, for six facets. Fine at this scale, and a real cost at a larger one — which is why it is a parameter rather than a default behaviour:
# Facets cost one `filter` aggregation per control (see queries.build_aggs). The result page
# wants them; "load more" and the map's pan handler do not, and asking for them anyway is the
# easiest accidental way to double a search's cost.
facets: bool = Query(default=True),On a large index the options are to cut it down to the facets the panel actually shows, or to accept self-confirming counts on the single-select controls where nobody notices the difference.
Making them cheaper
Four things help, roughly in order of how much.
Ask for fewer facets. The cost is per aggregation, so a panel that computes eight facets and displays four is paying double. This sounds too obvious to mention and it is the most common cause.
Lower size on the terms aggregations. A facet showing
the top ten values does not need "size": 100, and the shard-level work scales with it.
Use execution_hint deliberately on high-cardinality fields. The
default builds a map of global ordinals, which is fast and memory-hungry;
"execution_hint": "map" is the opposite trade and is better when the field has many
values but the query matches few documents.
Cache what you can. A request with "size": 0 is eligible for the
shard request cache, which caches the whole aggregation result per shard and is invalidated on
refresh. That makes an unfiltered facet computation — the one every user sees on the landing
page — essentially free after the first request. Note the condition: it only applies when no
documents are returned, which is one more reason to set "size": 0 when you mean it.
Reading the response
The nesting in the request is mirrored in the response, so parsing it means walking the same names back down:
def _facets(aggs: dict[str, Any] | None) -> Facets | None:
if not aggs or "facets" not in aggs:
return None
f = aggs["facets"]
def inner(name: str) -> dict[str, Any] | None:
return f.get(name, {}).get("buckets")Two details worth stealing. Everything uses .get rather than indexing, because an
aggregation over an empty result set returns nulls for min, max and
avg rather than omitting them — an empty result is not an error. And the
facets are turned into a typed object at the boundary rather than passed through as raw
Elasticsearch JSON, so the shape of a response does not become an accidental part of your public
API.
The typed boundary earns its place a second time here. A facet response is the part of a search
API most likely to be consumed by a frontend written by someone else, and Elasticsearch's shape
— buckets nested under an aggregation name, keys that are sometimes labels and sometimes
values, from and to present only on range buckets — is an
implementation detail you do not want to promise to keep:
class FacetBucket(ApiModel):
"""One row of a filter panel: a value and how many listings would remain if you ticked it."""
key: str
count: int
# Present on range facets only ("under $100"), so the UI can render the bounds it filters on.
from_: Decimal | None = Field(default=None, alias="from")
to: Decimal | None = NoneOne reader handles both bucket shapes because both arrive as key — a range
bucket's key is the label you defined, a terms bucket's key is the value itself.
The mistakes
Aggregating on a text field. An error if you are lucky, wrong buckets if
somebody enabled fielddata.
Trusting terms counts on a multi-shard index without reading
doc_count_error_upper_bound.
Forgetting "size": 0 when you only want counts, and paying for
twenty documents on every request.
Facets that include their own filter, which is the one that makes a filter panel a dead end.
Aggregating over a nested field without a nested
aggregation, which silently aggregates the flattened values — the same wrong-pairing
problem lesson 4 showed, arriving in the counts instead of the hits.
Using cardinality where an exact count is required. It is an
estimate, and it says so in the documentation and nowhere in the response.