Everything up to here was about making search work. This is about what to settle before real users depend on it — roughly in the order the decisions bite, and with the ones that cannot be changed later marked as such.
Shard count, which you cannot change
Start here because number_of_shards is a static setting: it is fixed at index
creation, and changing it means a reindex.
The instinct is that more shards means more parallelism means more speed. It usually does not. Every shard is a complete Lucene index with its own memory, file handles and merge activity; every search fans out to all of them and pays a merge across the results. Over-sharding a small index makes it slower and uses more memory for nothing.
The working guidance is 10–50GB per shard, and no more than about 20 shards per GB of heap on a node. So estimate your index size in a year, divide by 30GB, and round up. If that gives you one, use one.
"settings": {
# One shard: this is a demo, and a single shard also makes relevance scores stable.
# Scores are computed per shard, so small multi-shard indexes give oddly inconsistent
# ordering — the classic "why did the same query rank differently?" surprise.
"number_of_shards": 1,
"number_of_replicas": 0,For data that grows forever — logs, events — do not size one enormous index. Use rollover from lesson 15, so each generation stays a sensible size and old ones can be deleted by dropping an index rather than by a delete-by-query.
Replicas, which you can
number_of_replicas is dynamic, and it is what green actually
promises.
Zero replicas means one node failure loses data. That is correct for a derived index that can be rebuilt and for a development cluster, and wrong for anything else. One replica — the default — survives one node loss and roughly doubles read capacity. Two is for clusters where a second failure during recovery is a real concern.
Two practical notes. Replicas cost write throughput and disk proportionally, so setting them to zero during a bulk load and back afterwards is a genuine speedup. And a replica needs somewhere to live: on a single-node cluster it can never be assigned, and asking for one is how you end up permanently yellow and stop noticing yellow.
Heap, and the two rules
Give the JVM half the machine's RAM, and no more than about 30GB. Both halves matter.
The half is because Lucene reads its index files through the operating system's page cache, so the memory you do not give the JVM is doing real work. A node with all its RAM in the heap reads from disk constantly.
The 30GB ceiling is because the JVM uses compressed object pointers below roughly 32GB. Cross that line and pointers get larger, so a 33GB heap holds fewer objects than a 30GB one. If a machine has 128GB, run several nodes rather than one enormous heap.
ES_JAVA_OPTS: "-Xms512m -Xmx512m"Min and max the same, always. A growing heap means the JVM spends time resizing and the operating system may hand out memory it later wants back.
Also: disable swap. A swapped-out JVM heap causes garbage collection pauses measured in seconds,
and a node that pauses for seconds is treated as dead by the rest of the cluster, which then starts
reallocating its shards. bootstrap.memory_lock: true, or no swap at all.
Security, past the point most guides stop
Turning xpack.security on is the first step and it is not the interesting one. Most
guides then hand every client the elastic superuser, which is roughly the same as
giving every service your database's root password.
StayHub's search code needs to read and write one alias. So that is the role:
ROLE_BODY = {
"cluster": ["monitor"], # _cluster/health for the readiness probe. NOT `manage`.
"indices": [
{
# A wildcard over the generations, so a reindex into `-000003` needs no new grant.
# It is still scoped: this key cannot touch an index called anything else.
"names": [f"{ALIAS}*"],
"privileges": ["read", "view_index_metadata", "index", "delete", "manage"],
}
],
}Then an API key derived from it, rather than a password:
result = es.security.create_api_key(
body={
"name": KEY_NAME,
"role_descriptors": {ROLE: ROLE_BODY},
# A key with no expiry is a credential you will still be running in three years.
"expiration": "90d",
}
)API keys beat passwords for a service because they are individually revocable, they carry their own scope, and they expire. A password shared between three services cannot be rotated without coordinating three deploys.
One subtlety worth knowing before it confuses you:
# ⚠️ `role_descriptors` here is an INTERSECTION with the creating user's privileges,
# not a grant on top of them. A key can never be more powerful than whoever made it —
# which is why minting keys from `elastic` works and minting them from a limited
# service account silently produces a key that can do less than you wrote.Check that the scope actually bites
A role you have not tested is a role you hope is right. Run the negative cases:
what the scoped key CAN do:
index a document: ok
search: ok
cluster health: yellow
what it CANNOT do — this is the whole point:
read someone else's index refused (403 unauthorized)
delete an unrelated index refused (403 unauthorized)
list users refused (403 unauthorized)
mint another API key refused (403 unauthorized)That last one matters most. A credential that can create credentials is not scoped, whatever its index privileges say.
And TLS
The demo's secure profile runs authentication over plain HTTP, and says so:
xpack.security.http.ssl.enabled: "false"That is a teaching shortcut. Basic auth over HTTP puts the password in every request in the clear, and an API key is a bearer token, so it is no better. Production needs HTTP TLS with the client trusting the CA — and transport TLS between nodes, which is separate and is what stops an attacker who can reach the transport port from simply joining the cluster as a node.
Then: bind to a private network, never expose 9200 publicly, and do not put credentials in a frontend bundle. Elasticsearch has no rate limiting and no query-cost limits of its own, so a publicly reachable cluster is a resource-exhaustion target even when authenticated.
Client configuration is part of production too
Half of what makes a search feature survive an incident is on your side of the connection.
return Elasticsearch(
settings.elasticsearch_url,
# A search that hangs must not hang the API. Better a fast failure the route can turn
# into "search is unavailable" than a request that ties up a worker for a minute.
request_timeout=5,
retry_on_timeout=True,
max_retries=2,
**_auth(),
)A timeout your users would accept waiting, not one Elasticsearch would like. Retries bounded. And a failure turned into an honest error rather than an empty result set:
if not es_available():
# An honest 503 beats an empty result set. "No listings match" and "search is down" look
# identical to a user otherwise, and the second one is not their fault.
raise ApiException(
"Search is temporarily unavailable. Please try again in a moment.",
status_code=503,
)And on the write side, the opposite decision, for the reason lesson 8 gave: a failed index must not fail the user's write, but it must be retried rather than merely logged.
Node roles, once there is more than one node
A single node does everything. As soon as there are several, splitting the roles matters more than adding capacity.
Dedicated master-eligible nodes. The master maintains cluster state — mappings, aliases, shard allocation. A master node that is also serving heavy searches can be starved of CPU or paused by garbage collection, and a master that stops responding triggers an election and a cascade of shard reallocation. Three small dedicated masters is the standard answer, and the number three is not arbitrary.
Always an odd number of master-eligible nodes. Cluster state changes need a majority. Two master-eligible nodes have no majority when they cannot see each other, so each half refuses to elect, and the cluster stops accepting changes. Three tolerates one loss; five tolerates two.
Coordinating-only nodes — no data, no master role — handle the fan-out and the merging of results. Worth adding when heavy aggregations are causing memory pressure on the data nodes, and not before.
Two more settings worth naming. discovery.seed_hosts tells a node where to look for
peers; cluster.initial_master_nodes is used only on the very first bootstrap
and must be removed afterwards — leaving it in place is a documented way to end up with two
clusters that both think they are the real one.
Circuit breakers, and what they are protecting you from
Elasticsearch has a set of memory circuit breakers that refuse a request rather than let it take the node down. When one trips you get a distinctive error:
429 circuit_breaking_exception
[parent] Data too large, data for [<http_request>] would be [1.9gb]
which is larger than the limit of [1.8gb]This is a good error. It means the cluster protected itself from an out-of-memory kill, which is what would otherwise have happened.
The causes are nearly always one of three: an aggregation with too many buckets, a
from + size that is too deep, or a bulk request that is too large. Raising the breaker
limit converts a refused request into a dead node, so the fix is the query.
Worth setting indices.query.bool.max_clause_count and
search.max_buckets deliberately rather than discovering the defaults during an
incident, and worth knowing that a generated query — a filter built from a user-supplied list
of ids, say — is the usual way to blow past a clause limit.
Index lifecycle management
For any index that grows forever, ILM automates what lesson 15's rollover does by hand: create a new generation when the current one reaches a size or age, move older ones to cheaper nodes, and eventually delete them.
PUT _ilm/policy/logs-policy
{ "policy": { "phases": {
"hot": { "actions": { "rollover": { "max_primary_shard_size": "30gb", "max_age": "7d" } } },
"warm": { "min_age": "7d", "actions": { "forcemerge": { "max_num_segments": 1 } } },
"delete": { "min_age": "90d", "actions": { "delete": {} } }
} } }The point is that max_primary_shard_size makes the shard-sizing decision automatic
— you state the target rather than guessing an index size a year out.
A derived search index with a bounded document count needs none of this. Logs, events and metrics need it from day one, because the alternative is discovering the disk is full on a Sunday.
What to monitor
Five cluster numbers, from lesson 16: status not green; unassigned shards persisting; heap above 85% on any node; disk above 80%; thread pool rejections above zero.
Two application numbers that no Elasticsearch metric will ever give you. Search latency
measured at your API, not took — on this track's index a search reports
took: 2 while the round trip from Python is 4.5 ms, and more than half
the wall clock is outside the cluster. And document count against the source of
truth, which is the only thing that catches a sync that silently stopped. An index missing
half its documents looks perfectly healthy to every cluster metric there is.
One number worth adding early: the rate of empty result sets. A jump in searches returning nothing is the signature of an analysis change, a mapping problem or a broken sync, and it usually shows up days before anyone files a ticket.
Upgrades
Worth planning once rather than improvising annually, because two of the constraints are awkward.
Elasticsearch supports a rolling upgrade within a major version and from the last minor of one major to the next major. Nodes are upgraded one at a time, and the standard practice is to disable shard allocation first so the cluster does not start relocating everything each time a node goes down — then re-enable it and wait for green before taking the next node.
The two constraints. An index created two majors ago cannot be read: 8.x reads indexes created by 7.x, not by 6.x. And a snapshot restores into its own major or the next, not further. Together they mean an index or an archive can quietly become unreadable by simply skipping a version, and the fix — reindex it forward while you still can — has to happen before the upgrade, not after.
For a derived index none of this is dangerous, because a rebuild produces a fresh index in the current format. It is one more thing that gets easier when Elasticsearch is not the source of truth.
Testing search like the rest of your code
Search tends to escape testing because it needs a cluster. Splitting the tests in two solves most of that.
Query construction is a pure function and needs no cluster at all — assert
on the body your code builds. That covers the decisions that are silently wrong: a
terms where a term belonged, a filter in must, a facet that
includes its own filter.
Behaviour needs a real cluster, and should skip cleanly without one:
es_required = pytest.mark.skipif(
not es_available(), reason="Elasticsearch is not running — `docker compose up -d elasticsearch`"
)Those tests index a handful of known documents into a throwaway index built from the real mapping, so analysis and multi-fields behave as they do in production. They are the only way to verify the claims that matter — that accent folding works, that a highlight is escaped, that two ticked amenities narrow rather than widen.
Both halves together are what makes it safe to change a query later, which is the whole point: search code that nobody dares modify stops improving.
Failure modes to rehearse
All of these are cheap to practise and expensive to meet for the first time in production.
Elasticsearch is down. Stop the container. Does the site still work? Does search return an honest error? Do writes still succeed? Are the missed index updates queued, or lost?
The disk fills. Fill it on a test cluster and watch the read-only block arrive.
The error is cluster_block_exception and it mentions nothing about disk.
A node dies. On a multi-node test cluster, stop one. Watch health go yellow,
watch _cat/recovery, and find out how long recovery actually takes.
The index is wrong. Time a full rebuild from your source of truth. If it is hours, that is your recovery time objective, and it is worth knowing before you quote a different number to someone.
A mapping change. Run the alias flip on a copy of production data before you need it. It is the operation most likely to be attempted for the first time under pressure.
The checklist
Cannot be changed later — decide before creating the index. Shard count,
sized from a year's growth. An explicit mapping with dynamic: strict. Analyzers.
Multi-fields on anything to be sorted, filtered exactly, or aggregated. A geo_point for
anything that might ever be searched by location. And an alias, always, even with
one index.
Cluster. Heap at half of RAM, under 30GB, min equals max. Swap off. Replicas at least 1. Dedicated master nodes once there are more than a handful of nodes.
Security. Security enabled. TLS on HTTP and transport. An API key per service,
scoped to its own indexes, with an expiry. The elastic user used for bootstrap and
nothing else. Nothing exposed publicly.
Data. A rebuild path that is tested and timed. A drift check against the source of truth. Snapshots if — and only if — the index is the only copy, with SLM, a retention floor, and a restore you have actually run. ILM for anything that grows without bound.
Application. A short timeout and bounded retries. An honest error when search is unavailable. Indexing that never fails a user's write, and never merely logs the failure either. Facets and highlighting requested only where they are rendered.
Monitoring. The five cluster numbers, the two application numbers, and the empty result rate.
The one that matters most
If only one thing from these eighteen lessons survives, make it the rule from lesson 1: the database is the source of truth and the index is a derived, disposable copy.
Everything easy about operating Elasticsearch comes from that. You can delete the index and rebuild it. A mapping change is a reindex rather than a migration with data at risk. An indexing failure can be logged and retried instead of failing a user's request. Zero replicas is a defensible choice. Snapshots are optional. A bug in the sync layer is repairable rather than permanent.
Every one of those stops being true the first time a field exists only in Elasticsearch. It happens gradually and it is very hard to walk back — so the useful discipline is to ask, of every field you add, whether you could rebuild it from somewhere else tomorrow morning.