Elasticsearch – Installation and First Connection

Installing Elasticsearch is five minutes of work and one surprise. The surprise is that security has been on by default since 8.0, so every tutorial written before 2022 — including the one this post replaces — hands you a docker run that now answers 401 missing authentication credentials, or refuses to talk plain HTTP at all.

This lesson gets you a cluster you can develop against, then shows the secured version beside it so the difference is obvious rather than mysterious.

What changed in 8.0

Start a stock 8.x container with no configuration and it will, on first boot, generate a password for the built-in elastic user, generate a CA and a certificate, enable TLS on the HTTP layer, and print an enrollment token for Kibana. The endpoint is https://localhost:9200, not http, and an unauthenticated request is refused.

That is the correct default and it is a bad first hour. You end up debugging certificate trust before you have indexed a document. So the usual approach — and the one this track takes — is to turn security off for local development, deliberately and visibly, and turn it back on as a separate exercise once search itself makes sense. Lesson 18 does that properly.

Docker Compose, with the settings that matter

Use Compose rather than docker run, because you will want the flags again tomorrow and a shell history is not a configuration. This is the service the whole track runs against:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.3
    container_name: stayhub-elasticsearch
    restart: unless-stopped
...
    environment:
      discovery.type: single-node
      xpack.security.enabled: "false"
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
      path.repo: "/usr/share/elasticsearch/snapshots"
    ports:
      - "9200:9200"
    volumes:
      - stayhub-esdata:/usr/share/elasticsearch/data
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://localhost:9200/_cluster/health || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 20

Every line there is doing something.

discovery.type: single-node tells Elasticsearch not to look for peers and not to hold an election. Without it a single container waits for a quorum that will never arrive and logs "master not discovered yet" every ten seconds while refusing every request.

ES_JAVA_OPTS: "-Xms512m -Xmx512m" pins the JVM heap. Min and max are set to the same value on purpose — a growing heap means the JVM spends time resizing and the operating system may hand out memory it later wants back. The default is a quarter of the machine's RAM, which on a laptop running four other containers is more than you want to give it.

path.repo is the allow-list of directories snapshots may be written to. It is read at startup only, so adding a backup location later means a restart. Lesson 17.

The named volume is what makes the data survive docker compose down. Without it every restart is an empty cluster, which is briefly convenient and then very annoying.

The healthcheck matters more than it looks. Elasticsearch takes ten to thirty seconds to become ready, and anything that depends on it — a seed script, your API, a CI job — will otherwise fail on a connection refused that has nothing wrong with it.

Checking it is alive

docker compose up -d elasticsearch
curl -s localhost:9200 | head -12

A working cluster answers with its own description, and the two version numbers are worth reading rather than skipping:

{
  "name" : "afb55bfad1a7",
  "cluster_name" : "docker-cluster",
  "version" : {
    "number" : "8.15.3",
    "lucene_version" : "9.11.1"
  },
  "tagline" : "You Know, for Search"
}

Then health, which is the request you will type most often:

curl -s "localhost:9200/_cluster/health?pretty"
curl -s "localhost:9200/_cat/health?v"
curl -s "localhost:9200/_cat/indices?v"

The _cat family returns aligned columns instead of JSON, which is why it is what people actually type at a terminal. ?v adds the header row. Lesson 16 covers the rest of the family.

On a fresh single-node cluster with no indexes, health is green. Add an index that asks for a replica and it drops to yellow and stays there — a replica may never share a node with its primary, so on one node it can never be assigned. That is arithmetic, not a fault.

Running it with security on

Rather than switching the development cluster back and forth, run a second one beside it. The same image, a different port, security enabled:

  elasticsearch-secure:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.3
    container_name: stayhub-elasticsearch-secure
    profiles: ["secure"]
    environment:
      discovery.type: single-node
      xpack.security.enabled: "true"
      ELASTIC_PASSWORD: stayhub-elastic
      xpack.security.http.ssl.enabled: "false"
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
    ports:
      - "9201:9200"

ELASTIC_PASSWORD sets the built-in superuser's password at first start, which saves you fishing it out of the startup log or running bin/elasticsearch-reset-password. profiles: ["secure"] means the service only starts when you ask for it:

docker compose --profile secure up -d elasticsearch-secure

curl -s localhost:9201/_cluster/health
# {"error":{"type":"security_exception","reason":"missing authentication credentials..."}}

curl -s -u elastic:stayhub-elastic localhost:9201/_cluster/health
# {"cluster_name":"docker-cluster","status":"green", ...}

Note what is not here: xpack.security.http.ssl.enabled is false, so this is authentication over plain HTTP. That is a teaching shortcut and not a production configuration — basic auth over HTTP puts the password in every request in the clear, and an API key is a bearer token, so it is no better. Real deployments enable HTTP TLS and give the client the CA certificate. Roles and API keys work identically either way, which is why they are worth learning apart from the certificate work. Lesson 18.

One thing to know now: the elastic user is a bootstrap credential, not a runtime one. Anything holding it can delete every index in the cluster. Your application should get an API key scoped to the one index it uses.

What the untouched 8.x default actually does

The configuration above skips a flow you will meet eventually, so it is worth knowing what it skipped. Start the image with no security settings at all and the first boot prints a block like this, once, into the container log:

-> Password for the elastic user: kJ2*fQ7wRt+xZm0pLn9v
-> HTTP CA certificate SHA-256 fingerprint:
   a1b2c3...9f
-> Configure Kibana to use this cluster:
   eyJ2ZXIiOiI4LjE1LjMiLCJhZHIiOlsiMTcyLjE4...

Three separate things, and they are easy to confuse. The password is for the elastic superuser. The fingerprint is how a client verifies the self-signed CA the cluster just generated for itself. The long base64 blob is an enrollment token, which is a short-lived credential that lets Kibana or a second node join without you copying certificates by hand.

If you miss that block — and you will, because it scrolls past — both are recoverable from inside the container:

docker exec -it es bin/elasticsearch-reset-password -u elastic
docker exec -it es bin/elasticsearch-create-enrollment-token -s kibana

Connecting to that cluster from Python then needs the certificate as well as the password, and this is the shape to copy:

Elasticsearch(
    "https://localhost:9200",
    basic_auth=("elastic", password),
    # Either the CA file copied out of the container...
    ca_certs="./http_ca.crt",
    # ...or the fingerprint printed at first boot, which needs no file at all.
    # ssl_assert_fingerprint="a1b2c3...9f",
)

What you must not do is reach for verify_certs=False. It makes the error go away and it turns TLS into decoration — an encrypted channel to whoever answered, which is exactly what a certificate exists to prevent. It appears in a great many blog posts. Copy http_ca.crt out of the container instead: docker cp es:/usr/share/elasticsearch/config/certs/http_ca.crt .

Connecting from Python

pip install "elasticsearch==8.15.1"

Pin the client to your server's major version. The 8.x client refuses to talk to a 7.x cluster by design, and the failure is a clear error rather than a subtle one — but a 9.x client against 8.15 will surprise you eventually.

Build the client once per process. It owns a connection pool, so constructing one per request means a new pool per request:

@lru_cache
def get_es() -> Elasticsearch:
    """One client, reused. It holds a connection pool, so building a new one per request would
    open a new pool per request."""
    return Elasticsearch(
        settings.elasticsearch_url,
        request_timeout=5,
        retry_on_timeout=True,
        max_retries=2,
        **_auth(),
    )

request_timeout=5 is the line to argue about, and the argument is worth having early. The client's default is ten seconds and no application should use it. A search that hangs must not hang your API — better a fast failure the route can turn into "search is temporarily unavailable" than a request that occupies a worker for ten seconds while the queue behind it grows. Pick a number your users would accept waiting, not a number Elasticsearch would like.

Credentials, or the absence of them

The credential handling has one trap in it worth stating plainly:

def _auth() -> dict:
    if settings.elasticsearch_api_key:
        return {"api_key": settings.elasticsearch_api_key}
    if settings.elasticsearch_username and settings.elasticsearch_password:
        return {"basic_auth": (settings.elasticsearch_username, settings.elasticsearch_password)}
    return {}

Sending credentials to a cluster running with xpack.security.enabled: false is not a harmless no-op. It returns 401 missing authentication credentials — on a cluster that has no authentication at all. That error message sends people looking for a wrong password that does not exist. So: send nothing unless something is configured.

Also note that api_key and basic_auth are mutually exclusive. Passing both raises ValueError at construction time, which is a crash on the first search rather than a warning at startup.

A first document

The smoke test, worth running before you build anything on top:

curl -XPOST "localhost:9200/hello/_doc/1?refresh=true" \
  -H 'Content-Type: application/json' \
  -d '{"title": "Cedar Cabin with Mountain Views", "city": "Big Bear Lake"}'

curl -s "localhost:9200/hello/_search?q=cabin&pretty"

curl -XDELETE localhost:9200/hello

Two things happened without you asking. The index hello was created on first write, and its mapping was inferred from that one document. Both are conveniences that become problems at exactly the wrong moment, which is what lesson 3 is about.

?refresh=true is there because Elasticsearch is near-real-time: without it, the search a moment later can legitimately return nothing. Never use it in application code — lesson 7 explains what it costs.

Kibana, and whether you need it

Kibana is the web UI: dashboards, index management, and — the part that is actually useful while learning — the Dev Tools console, which is a query editor with autocomplete against your own mappings.

  kibana:
    image: docker.elastic.co/kibana/kibana:8.15.3
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      elasticsearch:
        condition: service_healthy

Two details there are the ones people get wrong. The host is http://elasticsearch:9200 — the service name and the container's own port, because Kibana is talking over the Compose network, not through your machine. Writing localhost:9200 there points Kibana at itself. And condition: service_healthy is why the healthcheck earlier was worth writing: Kibana started before Elasticsearch is ready will retry for a while and then give up.

With security on, Kibana also needs credentials, and specifically not the elastic user — there is a dedicated kibana_system account for it, whose password you set the same way you reset any other.

Whether you want it is a real question. Dev Tools is genuinely pleasant, and it is roughly half a gigabyte of RAM to have an editor. This track does not use it: every example here is a curl or a Python call, because that is what your application will be doing, and because a query you can only build by clicking is a query you cannot put in a code review.

What runs in production is not this

Compose is right for development and wrong for production, and it is worth knowing the three real options before you build habits around the wrong one.

Elastic Cloud is the managed service from the people who write it. You get the newest versions the day they ship, the commercial features, and someone else's pager. It is the default answer unless you have a reason.

Amazon OpenSearch Service is AWS's managed fork. If the rest of your infrastructure is in AWS, the IAM integration and the VPC story are worth a lot. The cost is that OpenSearch forked from Elasticsearch 7.10, so newer Elasticsearch features are not there and the clients are different packages. Everything up to lesson 14 in this track works unchanged; the operational lessons mostly do too, with different names.

Self-managed means you own the JVM tuning, the rolling upgrades, the snapshot repository, the certificate rotation and the capacity planning. It is entirely doable and it is a part-time job. Choose it when you have a compliance reason or a scale where the managed bill stops being reasonable, not to save money on a small cluster.

In all three cases the thing that actually changes is the URL and the credentials. Which is why both belong in configuration rather than in code:

    elasticsearch_url: str = "http://localhost:9200"
    elasticsearch_index: str = "stayhub-properties"
    elasticsearch_api_key: str = ""
    elasticsearch_username: str = ""
    elasticsearch_password: str = ""

One deliberate detail in there for later: elasticsearch_index is not an index. It is an alias, and lesson 15 is about why that distinction is the difference between a routine mapping change and an outage.

Where the data actually lives

Inside the container, everything is under /usr/share/elasticsearch/data — the Lucene segment files, the translog, and the cluster state. That is the directory the named volume is protecting, and it is the only thing worth backing up at the filesystem level. Even then, do not: taking a filesystem copy of a running cluster gives you a torn snapshot of files being written to. Use the snapshot API, which is lesson 17.

Disk sizing has one rule of thumb worth carrying: leave headroom. Elasticsearch watermarks disk usage and starts refusing to allocate shards at 85%, moves shards away at 90%, and at 95% it sets every index in the cluster to read-only. That last one is the incident nobody sees coming, because the symptom is writes failing with cluster_block_exception long after somebody noticed the disk was filling.

When it will not start

Three failures account for nearly all of them, and none announces itself clearly.

The container exits with code 137. That is the kernel killing it for using too much memory, not an Elasticsearch error. Give Docker Desktop more RAM or lower ES_JAVA_OPTS. There is often nothing useful in the logs at all, which is what makes it confusing.

max virtual memory areas vm.max_map_count [65530] is too low. Elasticsearch memory-maps its index files and needs a higher limit than the Linux default. On Docker Desktop this is usually already fine; on a Linux host it is sysctl -w vm.max_map_count=262144, and it needs to go in /etc/sysctl.conf to survive a reboot.

The port is already taken. 9200 is a popular port and the failure mode is not always a clean bind error — sometimes you connect to somebody else's cluster and spend twenty minutes wondering why your index is not there. curl -s localhost:9200 and read the cluster_name before assuming anything.

It starts, then the client cannot reach it. Almost always a network.host problem or a port mapping one. Inside Docker, check that the container is listening on all interfaces rather than its own loopback — docker logs will show publish_address {172.18.0.4:9200} if it is right and {127.0.0.1:9200} if it is not. From another container, the address is the service name and the container's own port, never localhost.

What you should have now

A cluster on 9200 you can develop against, a second one on 9201 that demands credentials so the difference is concrete, and a Python client with a timeout short enough to protect your API. That is the whole of the setup, and it is worth checking all three work before moving on — nearly every confusing error later in a search project turns out to be one of these three things half configured.

One habit to start now: keep the version you are running written down somewhere your code can see it. Elasticsearch changes real behaviour between minor versions, deprecates parameters with a warning header rather than an error, and removes them a version or two later. A track, a runbook or a bug report that does not say which version it was written against ages badly and silently.

With a cluster running and a client that can reach it, the next question is what shape the data should be — and the answer starts with not letting Elasticsearch guess.