Elasticsearch – The _cat APIs and Reading Cluster Health

The _cat APIs return aligned columns instead of JSON, which is why they are what you actually type when something is wrong. This lesson is the handful worth memorising, and more usefully the questions they answer — what yellow means, which shard is unassigned and why, and where the disk went.

The three parameters that make them usable

curl -s "localhost:9200/_cat/indices"                       # no header, all columns
curl -s "localhost:9200/_cat/indices?v"                     # with the header row
curl -s "localhost:9200/_cat/indices?v&h=index,docs.count,store.size"   # chosen columns
curl -s "localhost:9200/_cat/indices?v&h=index,store.size&s=store.size:desc"  # sorted

v adds the header. h selects columns. s sorts, with :desc for descending.

Two more worth knowing. help lists every available column for that endpoint, which is how you discover the two dozen fields _cat/nodes can show. And format=json turns the output back into JSON, which matters because the column layout is not a stable API — it changes between versions. Never parse the aligned output in a script.

Health first

curl -s "localhost:9200/_cat/health?v"

epoch      timestamp cluster        status node.total node.data shards pri relo init unassign
1787466514 06:28:34  docker-cluster green           1         1      1   1    0    0        0

The columns that matter are status, unassign, relo and init.

Green: every primary and every replica is assigned. Yellow: every primary is assigned, at least one replica is not — all your data is present and searchable, you have lost redundancy. Red: at least one primary is missing, so some of your data is not available and searches over it return partial results.

The distinction people get wrong is that red does not mean the cluster is down. It means part of it is. Searches keep working and quietly return fewer results — which brings back _shards.failed from lesson 9, the field nobody reads.

relo and init being non-zero is normal and temporary: shards are moving or being built. A cluster that sits with init above zero for a long time is one worth looking at.

Why a single-node cluster is yellow

The most common "problem" that is not one. Create an index that asks for a replica:

PUT /yellow-demo
{ "settings": { "number_of_shards": 1, "number_of_replicas": 1 } }

status yellow  ...  shards 2  pri 2  unassign 1  active_shards_percent 66.7%

_cat/shards says exactly which one:

index       shard prirep state      docs store  node
yellow-demo 0     p      STARTED       0  227b  1ca063334783
yellow-demo 0     r      UNASSIGNED

The primary (p) is fine. The replica (r) is unassigned, because a replica may never live on the same node as its primary — a copy on the same machine protects against nothing.

On one node this is arithmetic, not a fault. It is why the demo index in this track sets "number_of_replicas": 0: a single-node development cluster that asks for a replica is permanently yellow, and a permanently yellow cluster trains you to ignore the one signal you most need to notice.

allocation/explain: the answer, not a guess

When a shard is unassigned and the reason is not obvious, there is an API that simply tells you:

curl -s "localhost:9200/_cluster/allocation/explain?pretty" \
  -H 'Content-Type: application/json' \
  -d '{"index": "yellow-demo", "shard": 0, "primary": false}'

Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster.

node 1ca063334783: no
  decider: same_shard   NO
  a copy of this shard is already allocated to this node

Every node, every decider, and the reason each one said no. This is the single most useful diagnostic API in Elasticsearch and almost nobody reaches for it, because the instinct is to search the internet for a symptom instead.

Call it with no body at all and it explains the first unassigned shard it finds, which is usually the one you meant.

The deciders you will actually meet: same_shard (the replica-on-one-node case above), disk_threshold (the node is above the watermark), awareness (zone rules), filter (an index or cluster allocation filter excludes the node), and max_retry — which means allocation was attempted five times and failed, and the shard will not be retried until you say POST /_cluster/reroute?retry_failed=true.

The rest of the family

_cat/indices

health status index                     pri rep docs.count docs.deleted store.size
green  open   stayhub-properties-000001   1   0         12            1     43.4kb

docs.deleted is the column worth reading and the one nobody does. Deletes and updates leave tombstones until a merge rewrites the segment, so a large docs.deleted relative to docs.count means the index is carrying dead weight. A ratio near half on a heavily updated index is normal; a ratio that keeps climbing means merging is not keeping up.

status is open or close. A closed index does not serve searches and does not consume much memory — and a closed index that somebody closed during an incident and forgot is a genuinely mystifying "the data is gone" report.

_cat/nodes

curl -s "localhost:9200/_cat/nodes?v&h=ip,name,heap.percent,ram.percent,cpu,load_1m,node.role,master"

ip         name         heap.percent ram.percent cpu load_1m node.role   master
172.19.0.2 1ca063334783           57          99   3    6.43 cdfhilmrstw *

heap.percent is the number to watch. Sustained above 85% means garbage collection is running constantly and the node is about to become slow and then unresponsive. It is the single best early warning Elasticsearch offers.

ram.percent at 99 is not a problem — that is the operating system's file cache using memory that would otherwise be idle, and Lucene depends on it.

node.role is a letter per role: m master-eligible, d data, i ingest, c cold, and so on. The * in master marks the elected master. On a small cluster every node has every role, which is what the string of letters above means.

_cat/shards

One line per shard, and the endpoint to reach for when health is not green. The useful trick is filtering and sorting:

curl -s "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason&s=state"
curl -s "localhost:9200/_cat/shards?v&h=index,shard,store&s=store:desc"   # the biggest shards

unassigned.reason gives a one-word cause — NODE_LEFT, ALLOCATION_FAILED, INDEX_CREATED. It is the fast version of allocation/explain, and it is usually enough.

_cat/allocation

shards disk.indices disk.used disk.avail disk.total disk.percent node
     1       43.4kb   111.5gb    895.2gb   1006.8gb           11 1ca063334783

Shards and disk per node. This is where you find out that one node is holding most of the data and most of the shards, which is the usual explanation for one node being slow.

The disk.percent thresholds are worth memorising, because they are the incident nobody sees coming: at 85% Elasticsearch stops allocating new shards to the node, at 90% it starts moving shards away, and at 95% it sets every index in the cluster to read-only. That last one turns a slow disk fill into a total write outage, and the error — cluster_block_exception — does not mention disk at all.

_cat/thread_pool

curl -s "localhost:9200/_cat/thread_pool/write?v"

node_name    name  active queue rejected
1ca063334783 write      0     0        0

rejected is a cumulative counter since the node started, and any non-zero value means requests were refused — the 429 from lesson 8. Watch write during an indexing load and search during a traffic spike. A climbing queue with rising rejected means the answer is smaller batches or fewer of them, never more threads.

_cat/aliases and _cat/count

Two small ones that come up constantly once lesson 15's alias pattern is in place.

curl -s "localhost:9200/_cat/aliases?v"
curl -s "localhost:9200/_cat/count/stayhub-properties?v"

_cat/aliases shows which alias points at which index, plus any filter and routing attached — the fastest way to confirm a flip landed. _cat/count takes an index or an alias and returns the document count, which is the one-line version of the drift check.

There is also _cat/pending_tasks, which is nearly always empty and is very informative when it is not. Pending tasks are cluster-state changes queued behind the master — index creations, mapping updates, alias changes. A backlog there means the master is overloaded, which is a different problem from a slow data node and is usually caused by too many indexes, too many shards, or something creating indexes in a loop.

Beyond _cat

_cat is for reading at a terminal. Three JSON APIs are what you actually alert on.

GET /_cluster/health is the structured version of the first command in this lesson, and it takes parameters worth knowing: ?wait_for_status=green&timeout=30s blocks until the cluster is green or the timeout expires, which makes it a real readiness gate in a deploy script rather than a poll loop. ?level=indices breaks it down per index, so you can find the red one immediately.

GET /_nodes/stats is the firehose — JVM, garbage collection, thread pools, per-index operation counts, filesystem. It is what a monitoring agent scrapes, and it is far too much to read by hand.

GET /_cluster/stats is the cluster-wide summary: total documents, total store size, field count, node roles. Useful for capacity conversations and for noticing that the field count is climbing towards the limit from lesson 4.

And GET /_tasks?detailed&actions=*search lists what is running right now, which is how you find the one query that is eating the cluster. Tasks can be cancelled.

What a shard actually is, briefly

Half the output above is about shards, so it is worth pinning down what one is before reading more of it.

A shard is a complete, self-contained Lucene index. It has its own inverted index, its own doc values, its own segments on disk. An Elasticsearch index is a set of shards, and a search runs on every one of them in parallel and merges the results.

That explains several things that otherwise look arbitrary. Relevance scores are per shard, because term frequencies are per shard. terms aggregations are approximate, because each shard returns its own top N. A shard is the unit of allocation, so it is the thing that moves between nodes, and the reason a node "holding more shards" is a meaningful complaint. And a shard has a fixed cost in memory and file handles regardless of how much data is in it, which is why lesson 1 warned against over-sharding and lesson 18 comes back to it.

A primary shard takes writes first and replicates them. A replica is a copy that serves reads and can be promoted if the primary is lost. Replica count is changeable on a live index; shard count is not.

Recovering from red

Red means at least one primary shard is unavailable, and the correct first action is almost never the one people take.

Find out which.

curl -s "localhost:9200/_cluster/health?level=indices&pretty" | grep -B2 '"status" : "red"'
curl -s "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason&s=state"

Find out why, with allocation/explain. The answer usually falls into one of three categories, and each has a different response.

A node is gone and is coming back. Do nothing. Elasticsearch delays reallocating shards from a departed node for a minute by default, precisely so a restart does not trigger a cluster-wide rebalance. Waiting is the correct action, and impatience here causes far more damage than the outage would have.

A node is gone and is not coming back. If the shard had replicas, the cluster promotes one and recovers on its own. If it did not, the data is gone and the options are a restore from a snapshot or a rebuild from the source of truth — which is the whole argument of lessons 1 and 17.

Allocation is being refused. Disk watermarks, allocation filters, or max_retry. All three are fixable and allocation/explain names which.

The thing to resist is _cluster/reroute with allocate_empty_primary. It makes the cluster green by declaring an empty shard to be the primary, which is to say it makes the symptom go away by discarding the data. It is a genuine last resort for a case where the data is already unrecoverable and you need the index writable, and it is not a fix.

Reading a shard's story

Two more endpoints answer "what is this shard doing right now", which is the question during a recovery.

curl -s "localhost:9200/_cat/recovery?v&active_only=true"
curl -s "localhost:9200/_cat/segments?v&h=index,shard,segment,docs.count,size"

_cat/recovery shows shards currently being copied or replayed, with a percentage complete and bytes transferred. During a node restart this is the difference between "it is working, give it ten minutes" and "it is stuck".

_cat/segments lists the Lucene segments inside each shard. Mostly you do not need it, and it is the right place to look when an index is much larger than its documents or when a forcemerge is being considered — many small segments is what merging exists to fix.

Working through a real symptom

Someone reports that search is slow. The sequence that finds the cause fastest:

Is the cluster healthy? _cat/health?v. Yellow or red, or a non-zero unassign, and you are looking at a cluster problem rather than a query problem — go to _cat/shards and then allocation/explain.

Is a node struggling? _cat/nodes?v. Heap above 85%, or one node's CPU far above the others. If so, _cat/allocation?v to see whether that node is holding more than its share.

Is anything being rejected? _cat/thread_pool?v. Rejections mean the cluster is saturated and no query change will help.

Is it actually the query? Compare took with the time your client measured. If took is small and the wall clock is not, the cluster is not the problem — it is the network, the response size, or your own serialisation. If took is large, use "profile": true from lesson 9 to find out which clause.

Is it one query or all of them? _tasks?detailed&actions=*search while it is happening. One enormous aggregation from a dashboard is a much more common cause than general slowness.

That sequence goes from cheapest to most specific, and it distinguishes the three cases that matter — a broken cluster, a saturated cluster, and a bad query — before you have changed anything.

Making the cluster's state visible in your own tools

A search feature has one dependency that can be healthy, degraded or gone, and the application should be able to say which. The cheapest version is a ping behind a readiness check:

def es_available() -> bool:
    """Used by the health check and by the indexer, which must not crash writes when ES is down."""
    try:
        return bool(get_es().ping())
    except Exception:  # noqa: BLE001 — any transport failure means "not available"
        return False

That distinguishes "gone" from "fine", which is what lets the search endpoint return an honest 503 rather than an empty result set. It does not distinguish "degraded", and for that the useful addition is the cluster status, which is one call and worth surfacing in whatever dashboard your team already looks at:

curl -s "localhost:9200/_cluster/health?format=json" | jq '{status, unassigned_shards, active_shards_percent_as_number}'

The reason to put it beside your application's own metrics rather than in a separate Elasticsearch dashboard is mundane and important: during an incident, nobody opens the second dashboard. A yellow cluster next to a latency graph gets noticed; a yellow cluster in a tool you check weekly does not.

The same argument applies to the index-level facts worth alerting on. Document count against the source of truth, from lesson 8's drift check, belongs next to the same graphs — it is the one number that catches a sync that stopped, and no Elasticsearch health metric will ever show it, because from the cluster's point of view an index missing half its documents is perfectly healthy.

What to keep an eye on

Five numbers cover most of it. Cluster status not green. Unassigned shards above zero for more than a few minutes. Heap above 85% on any node. Disk above 80% on any node, which gives you room before the 85% watermark bites. And thread pool rejections greater than zero.

Two of the five deserve a rehearsal rather than just an alert. Fill a disk on a test cluster and watch the read-only block arrive, so the error message is familiar when it matters. And stop a node in a multi-node test cluster and watch the shards reallocate, so you know how long it takes and what "normal recovery" looks like in _cat/recovery. Both take twenty minutes and both turn an unfamiliar incident into a recognised one.

None of those needs a monitoring stack to start with. A script that hits _cluster/health and _cat/nodes?format=json every minute and shouts when one of the five is out of range is an afternoon's work and catches the failures that actually happen.