Elasticsearch – Snapshots, Restores and SLM

Before the mechanics, the question underneath: does your index need a backup at all?

If it is derived from a database — which lesson 1 argued it should be — then a snapshot restores you to a stale copy of something you can regenerate exactly. A rebuild is both more current and, at moderate size, faster.

Snapshots earn their place in two cases. When the index is the source of truth, which is normal for logs and metrics where nothing else holds the data. And when a rebuild is too slow to be an outage plan — a hundred million documents reindexed from Postgres is hours, restored from a snapshot it is minutes, because a restore copies segment files rather than re-analysing text.

The mechanics are the same either way, which is why they are worth knowing before you need them.

What a snapshot is

Not a dump. Elasticsearch copies the Lucene segment files into a repository, and because segments are immutable, a second snapshot only stores segments the first one did not already have.

Three consequences follow, and all three surprise people.

Snapshots are incremental but each one is complete. There is no "full versus incremental" distinction to manage. Snapshot 7 shares most of its files with snapshot 6 and can be restored on its own; deleting snapshot 6 keeps any files 7 still needs.

They are cheap after the first. A daily snapshot of a mostly-static index costs almost nothing, which makes daily the sensible default rather than a luxury.

A heavily updated index snapshots more than you expect, because merging rewrites segments and a rewritten segment is a new file even when its contents barely changed.

There is a fourth consequence worth stating because it affects planning: deleting a snapshot does not reliably free the space you expect. Files shared with a snapshot you are keeping stay where they are. Repository size therefore tracks the union of everything retained, not the sum of individual snapshot sizes, and estimating it by multiplying index size by snapshot count badly overshoots.

What a snapshot is not is a copy you can read. The files in the repository are internal Lucene structures with Elasticsearch metadata around them. There is no way to extract a document without restoring.

Registering a repository

A repository has to be declared before anything can be written to it, and for a filesystem repository the location has to be on an allow-list the cluster reads at startup:

      path.repo: "/usr/share/elasticsearch/snapshots"

That setting is why adding a backup location needs a restart. It is a guard against a compromised cluster being told to write files anywhere on the host, and it is worth knowing about before you are trying to add a repository during an incident.

    es.snapshot.create_repository(
        name=REPO,
        body={
            "type": "fs",
            "settings": {
                "location": REPO_LOCATION,
                "compress": True,
            },
        },
    )

compress applies to the metadata, not to the already-compressed segment files, so the win is small and free.

Then verify it, which does more than it looks:

    # `verify` writes and reads back a marker file from every node. On a single node this is
    # ceremony; on a cluster it is the check that catches the classic misconfiguration — a repo on
    # a local disk rather than shared storage, where every node writes its own private half of a
    # snapshot that can never be restored.
    result = es.snapshot.verify_repository(name=REPO)

That misconfiguration is the single most common way a backup strategy turns out not to be one. A filesystem repository must be shared storage mounted identically on every node — NFS, or the equivalent. Point it at a local path on each node and every snapshot succeeds, every listing looks right, and no restore will ever work.

$ python -m scripts.snapshot --register
registered stayhub-backups at /usr/share/elasticsearch/snapshots
verified on nodes: ['ZqoP1OK1T-2rZP1xJgNDyw']

In production, use object storage

The fs type is right for a single-node demo. Real clusters use repository-s3, repository-gcs or repository-azure:

PUT /_snapshot/prod-backups
{ "type": "s3", "settings": {
    "bucket": "example-es-snapshots",
    "base_path": "prod/",
    "server_side_encryption": true
} }

Everything below is identical. Only the registration body changes, which is the reason it is worth practising on the filesystem version first.

Taking one

    result = es.snapshot.create(
        repository=REPO,
        snapshot=name,
        body={
            # Snapshot the CONCRETE index. An alias is not a thing that can be snapshotted, but
            # `include_global_state` carries the alias definition, so a restore brings it back
            # pointing at the right index.
            "indices": concrete,
            "include_global_state": True,
        },
        wait_for_completion=True,
    )

The alias detail matters if you followed lesson 15. Aliases live in the cluster state, not in the index, so a snapshot of the index alone restores the data with no alias pointing at it — and an application configured to use the alias sees an empty cluster.

$ python -m scripts.snapshot --create
snap-20260823-053637: SUCCESS  1/1 shards
  indices: ['stayhub-properties-000002']
  took: 0ms

wait_for_completion=True is fine at this size and wrong at any real one. A large snapshot returns immediately without it and is monitored through GET /_snapshot/<repo>/<name>/_status.

Snapshotting does not block writes. The snapshot captures the state at the moment it starts, and indexing continues throughout — which is the same "there is a gap" property as the reindex in lesson 15, and it matters for the same reason.

Restoring

The step that catches people first: an open index cannot be restored over.

    # ⚠️ An OPEN index cannot be restored over. Elasticsearch refuses rather than serving half of
    # each. Closing it is the standard move; `rename_pattern` into a fresh name and flipping the
    # alias is the zero-downtime alternative, and is the same alias trick as scripts/reindex.py.
    es.options(ignore_status=404).indices.close(index=concrete)

Closing it means search stops for the duration. The alias alternative avoids that entirely: restore under a different name, verify it, then flip the alias — which is lesson 15's procedure with a snapshot as the source instead of a reindex:

POST /_snapshot/stayhub-backups/snap-20260823-053637/_restore
{
  "indices": "stayhub-properties-000002",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1"
}

And whichever route you take, reopen in a finally:

    finally:
        # Reopen even on failure — leaving the index closed is a silent outage that looks like an
        # empty result set.
        es.options(ignore_status=404).indices.open(index=concrete)

A closed index is not an error state anyone will recognise. Searches against it fail in a way that reads like a configuration problem, and a script that died halfway through a restore is a plausible cause nobody guesses.

Proving it works

A backup nobody has restored is a hypothesis. The full round trip on the demo index — destroy the data, restore, count:

$ curl -XPOST "localhost:9200/stayhub-properties/_delete_by_query?refresh=true" \
    -d '{"query": {"match_all": {}}}'
{"deleted": 12}

$ curl -s "localhost:9200/stayhub-properties/_count"
{"count": 0}

$ python -m scripts.snapshot --restore snap-20260823-053637
restoring snap-20260823-053637
restored 1/1 shards
documents now: 12

Twelve, to zero, to twelve. That takes two minutes to run and it is the only evidence that any of the configuration above is correct.

Do it on a schedule, not once. Restore into a scratch cluster monthly and compare document counts against production. Backups fail silently — a repository that filled up, a permission that changed, a node added without the shared mount — and every one of those looks perfectly healthy until the restore.

Restore options worth knowing

The restore body takes more than an index name, and three of the options come up regularly.

POST /_snapshot/stayhub-backups/snap-20260823-053637/_restore
{
  "indices": "stayhub-properties-*",
  "include_aliases": false,
  "index_settings": { "index.number_of_replicas": 0 },
  "ignore_index_settings": [ "index.refresh_interval" ]
}

include_aliases: false is the one to reach for when restoring production data into staging. Without it the restored index arrives carrying production's aliases, which in a cluster that already has an index behind that name is an immediate conflict — and in one that does not, is a staging cluster silently serving under production names.

index_settings overrides settings on the way in. Restoring with zero replicas and adding them afterwards is much faster, for the same reason it is faster during a bulk load.

ignore_index_settings drops a setting entirely so the restored index takes the default. Useful when the snapshot was taken from a cluster whose allocation filters or node attributes do not exist on the target — a restored index that references a node attribute nobody has will sit unassigned forever, and allocation/explain from lesson 16 is how you find that out.

What is in a snapshot besides indexes

include_global_state has appeared twice above without a full explanation, and it is worth one because it is the difference between a restore that works and one that restores data nothing can use.

The global state holds the cluster-level things: index templates, ingest pipelines, aliases, legacy settings, and stored scripts. Restore an index without it into a fresh cluster and you get the documents, with no template governing future indexes, no pipeline for enriching them, and no alias pointing at what you just restored.

Restoring with it into an existing cluster is the opposite hazard: it overwrites that cluster's templates and settings with the snapshot's. So the rule is roughly — include it when taking the snapshot, always; include it when restoring into an empty or dedicated cluster; leave it out when restoring one index into a cluster that is already running.

Automating it: SLM

A snapshot script on a cron on one machine is a backup plan that fails the day that machine is rebuilt. Snapshot Lifecycle Management runs inside the cluster and reports its own failures:

    es.slm.put_lifecycle(
        policy_id=POLICY,
        body={
            "schedule": "0 30 2 * * ?",  # 02:30 daily — cron with a seconds field, Quartz style
            "name": "<stayhub-{now/d}>",  # date maths in the name: stayhub-2026.08.22
            "repository": REPO,
            "config": {"indices": [f"{ALIAS}*"], "include_global_state": True},
            "retention": {
                "expire_after": "30d",
                "min_count": 7,
                "max_count": 30,
            },
        },
    )

Two details in there are easy to get wrong.

The schedule is a Quartz cron expression with a seconds field first, so it has six fields rather than five. A standard five-field crontab line pasted in either fails to parse or means something entirely different.

The name uses date maths in angle brackets, so each run produces stayhub-2026.08.22 rather than overwriting one name.

The retention setting that matters

                # ⚠️ `min_count` is the part that makes retention safe. Without it, a cluster that
                # was off for a month deletes every snapshot it has the moment it comes back,
                # because they are all older than `expire_after`.

Retention is evaluated as "delete anything older than expire_after", and on its own that is a rule with a catastrophic edge case. A cluster that was down, or a policy that stopped running, comes back to find every snapshot expired — and dutifully deletes all of them.

min_count is the floor: never go below this many, whatever their age. max_count is the ceiling. Always set the floor.

SLM reports its own health, which is the point of it:

curl -s "localhost:9200/_slm/policy/stayhub-nightly?pretty"
# "stats": { "snapshots_taken": 0, "snapshots_failed": 0 }, "next_execution_millis": ...

snapshots_failed is what to alert on. A backup that has been failing for three weeks is worse than no backup, because you believed you had one.

The volume permission that will stop you first

Worth including because it costs an hour and the error names neither permissions nor ownership.

Elasticsearch runs as uid 1000 inside its container. Docker creates a named volume owned by root unless the mount point already exists in the image, in which case it copies that directory's ownership. /usr/share/elasticsearch/data exists in the image, so the data volume is fine. A snapshot directory does not, so it arrives owned by root and the process cannot write to it.

The failure is not a permission error you can read:

500 repository_verification_exception
failed to create blob container

The fix is to set the ownership before Elasticsearch starts, which a one-shot container does cleanly:

  es-snapshot-init:
    image: busybox:1.36
...
    command: sh -c "mkdir -p /snapshots && chown -R 1000:0 /snapshots && chmod 775 /snapshots"
    volumes:
      - stayhub-essnapshots:/snapshots

  elasticsearch:
    depends_on:
      es-snapshot-init:
        condition: service_completed_successfully

The same class of problem exists in production with different clothes: an S3 bucket policy that allows PutObject but not DeleteObject produces snapshots that work and retention that silently never removes anything, until the bucket bill becomes the alert.

Whatever the storage, verify_repository is what tells you before the first snapshot rather than during the first restore.

Restoring somewhere else

The interesting uses of snapshots are not disaster recovery.

Populating a staging environment with real data: snapshot production, restore into staging, done. Far simpler than a data pipeline, and the reason to keep the repository credentials separate is that this means staging can read production's backups.

Migrating between clusters, including across cloud providers. Register the same repository on both, snapshot from one, restore into the other.

Version upgrades. Here the compatibility rule bites: a snapshot can be restored into the same major version or the next one, and no further. A 7.x snapshot restores into 8.x; it does not restore into 9.x. So a cluster upgraded across two majors cannot recover its old snapshots, and the archive you kept for compliance may be unreadable by the cluster you are running.

If snapshots are your only copy of anything, that constraint belongs in your upgrade plan explicitly — restore-and-re-snapshot at each major, or keep a cluster of the old version alive.

Searchable snapshots and frozen data

One more capability worth knowing exists, because it changes the calculation for time-series data.

A searchable snapshot mounts an index directly from the repository and queries it there, fetching and caching blocks on demand rather than holding a full copy on local disk. The index behaves like any other for reads — slower on a cold cache, and considerably cheaper, because object storage costs a fraction of what fast local disk does.

That is what the "frozen tier" means in practice: data old enough that nobody searches it often, kept queryable at storage prices instead of being deleted or archived somewhere unreadable. It is a commercial-licence feature, and the equivalent decision on any tier is the same one — how old does data have to be before it is worth trading query speed for cost.

For a derived search index none of this applies, because there is no old data: the index is whatever the database currently says. It matters exactly where snapshots mattered in the first place, which is when the index is the only copy.

A minimum viable backup setup

If you decide you need one, this is the shortest complete version.

Register a repository on shared or object storage and run verify_repository — not once, but as part of whatever provisions the cluster, so a node added later without the mount fails loudly.

Create an SLM policy with a daily schedule, include_global_state: true, and a retention block with min_count set. Alert on snapshots_failed from the policy stats.

Write the restore procedure down as a script, not a wiki page, and run it monthly against a scratch cluster. Compare document counts. That script is the backup; everything before it is a hypothesis.

And record which major version the snapshots were taken with, next to the snapshots. It is the one piece of information that becomes unrecoverable and matters most.

Deciding, honestly

Ask what a restore would actually give you.

If the answer is "a copy of the index as it was at 02:30, which I would then have to re-synchronise with the database anyway", you have a rebuild, not a backup, and the rebuild path is the one worth investing in — testing it, timing it, making sure it can run without a full outage.

If the answer is "the only copy of that data", then snapshots are not optional, and everything above applies: shared storage, verified, automated with SLM, with a floor on retention, and restored on a schedule to prove it.

Most search indexes are the first kind. The mistake is not knowing which kind yours is, and finding out during the incident.

The last lesson pulls this together with everything else worth settling before real users arrive — shard sizing, replicas, heap, credentials and what to monitor.