What is Elasticsearch, and When Should You Use One?

Elasticsearch is a search engine you run alongside your database, not instead of it. That sentence is the whole of this lesson, and almost everything else in this track is a consequence of it.

It is worth being precise about what it gives you, because "fast search" is not an answer — your database is fast too. What Elasticsearch does that a relational database will not is answer questions about text, and rank the answers.

The query your database cannot answer

Suppose you have a table of short-let listings and a guest types cabin into a search box. In SQL you reach for:

SELECT * FROM properties
WHERE title ILIKE '%cabin%' OR description ILIKE '%cabin%';

This works, and it is wrong in four separate ways.

It cannot use an index. A B-tree index is sorted by the whole value, so it can find everything starting with "cabin" and nothing containing it. A leading wildcard means a full scan of every row, every time.

It has no idea which result is better. A listing titled "Cedar Cabin" and one whose description mentions "a short drive to the cabin rentals" come back in whatever order the scan produced.

It is exact. A guest who types cabbin gets nothing. So does one who types cabins, unless you thought to write ILIKE '%cabin%' rather than = 'cabin' — and a guest searching for Málaga without the accent gets nothing at all.

And it cannot tell you anything about the results as a set. A real search page wants "Austin (48) · Denver (31)" next to the results, which in SQL is a second query, and a third for the price ranges, and a fourth for the amenity counts.

What an inverted index actually is

A database index maps a row to its values. An inverted index maps a value to its rows — and it does it after chopping text into terms.

Index three listings titled "Cedar Cabin with Mountain Views", "Sunlit Loft in the Mission" and "Lakefront A-Frame", and what is stored is roughly:

term        documents
--------    ---------
a-frame     [3]
cabin       [1]
cedar       [1]
lakefront   [3]
loft        [2]
mission     [2]
mountain    [1]
sunlit      [2]
views       [1]

Now "which documents contain cabin?" is a lookup, not a scan, and it stays a lookup whether there are twelve documents or twelve million. That is the trick, and it is the only trick. Everything else Elasticsearch does is built on top of it.

Two things follow immediately, and both catch people out later.

The text you sent is not what is searched. "Cedar Cabin" became cedar and cabin — lowercased, split on the space. That transformation is called analysis, it is configurable, and when a search mysteriously returns nothing it is almost always because the analysis at index time and the analysis at query time disagreed. Lesson 5 is entirely about this.

The index is built when the document is written, not when it is queried. So indexing costs something, changing how a field is analysed does not retroactively change documents already stored, and a document is not searchable the instant you write it. Lessons 3, 7 and 15 all turn on some version of that.

Relevance, for free

Because the index already knows how many documents contain each term, it can score them. A term in 1 document out of 12 is a much stronger signal than one in 11 of 12 — that is inverse document frequency, and it is why searching for "cabin in the mountains" ranks on "cabin" and largely ignores "in" and "the" without you configuring anything.

Ask Elasticsearch to explain a score and it shows you the arithmetic. On the twelve-listing index this track uses, the top hit for cabin scores 4.3190, and it will tell you exactly why:

4.3190  score(freq=1.0), computed as boost * idf * tf from:
  4.4000    boost
  2.1595    idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:
    1.0000      n, number of documents containing term
   12.0000      N, total number of documents with field
  0.4545    tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl))

You do not need to understand that yet — lesson 11 takes it apart. The point is that relevance here is a computation you can inspect and adjust, not a black box. Very few arguments about search results survive contact with _explain.

What a document and a search actually look like

Before the vocabulary, it is worth seeing one round trip end to end, because the shape of the response explains several later lessons at a glance.

A document is just JSON. Here is a StayHub listing as it is stored — already flattened, so the amenities that live in a join table in Postgres are an array here:

{
  "public_id": "8f1e...",
  "title": "Sunlit Loft in the Mission",
  "description": "A bright corner loft two blocks from Dolores Park.",
  "city": "San Francisco",
  "country": "US",
  "property_type": "LOFT",
  "amenities": ["wifi", "kitchen"],
  "price_per_night": 250.0,
  "max_guests": 4,
  "rating_average": 4.8,
  "location": { "lat": 37.7599, "lon": -122.4148 }
}

A search is also JSON, sent to GET /<index>/_search:

{
  "query": {
    "bool": {
      "must":   [ { "match": { "title": "loft" } } ],
      "filter": [ { "range": { "price_per_night": { "lte": 300 } } } ]
    }
  },
  "size": 20
}

Two clauses, and the difference between them is the single most useful thing to know early. must scores — how well a listing matches "loft" affects its rank. filter only includes or excludes, contributes nothing to the score, and is cacheable as a result. "Under $300" is not a matter of degree, so it goes in filter. Lesson 10 is this idea in full.

And the response:

{
  "took": 2,
  "hits": {
    "total": { "value": 1, "relation": "eq" },
    "max_score": 8.64,
    "hits": [
      {
        "_index": "stayhub-properties-000002",
        "_id": "8f1e...",
        "_score": 8.64,
        "_source": { "title": "Sunlit Loft in the Mission", "city": "San Francisco" }
      }
    ]
  }
}

Four details in there are worth naming now.

took is the cluster's own timing in milliseconds — query execution only. It does not include the network, JSON serialisation, or the time your client spent waiting. On the index used throughout this track, a typical search reports took: 2 while the round trip measured from Python is 4.5 ms. Neither number is wrong; they measure different things, and quoting took as "our search takes 2ms" is how a latency budget goes missing.

total.relation is "eq" here, meaning the count is exact. By default Elasticsearch stops counting at 10,000 and returns "gte", because counting every match is work nobody usually needs. If your UI says "page 3 of 512", you need to ask for an exact count explicitly. Lesson 9.

_source is the original JSON, stored verbatim alongside the index. That is why a result can carry the cover image and the amenity list without a second lookup, and it is why denormalising is the right instinct here rather than a compromise.

And _index says stayhub-properties-000002, not stayhub-properties, even though that is the name the application searched. The application talks to an alias. That one decision is what makes a mapping change something you do on a Tuesday afternoon rather than an outage, and it is lesson 15.

What it costs

Running Elasticsearch means running a second datastore, and it brings three real problems.

The data is a copy, so it can be wrong. Something has to write every change into the index, and that something can fail, lag, or be skipped by a code path nobody remembered. Lesson 8 is about nothing else.

There are no transactions and no joins. You cannot index a document and update a row atomically. You cannot join two indexes at query time. Both constraints push you towards denormalised documents — one document per thing a user searches for, carrying everything the result card needs — which is a genuinely different way to model data. That is lesson 6.

It is near-real-time, not real-time. A document you just wrote is typically searchable about a second later. This is a deliberate trade for indexing throughput, and it is the reason a test that indexes a document and immediately searches for it finds nothing and looks like a broken query. Lesson 7.

So when should you use one?

Reach for it when the query is about text and the answer has an order: a search box over descriptions, autocomplete, "more like this", log search. Reach for it when one request has to return results and the counts that go beside them — faceted search is the case where Elasticsearch is not just faster but structurally simpler than the SQL equivalent. And reach for it for time-series and log data at volumes where a relational database stops being cheap.

Do not reach for it as your primary store. It has no foreign keys, no constraints, no transactions, and losing it should be an inconvenience rather than an incident.

Do not reach for it for exact lookups by primary key — your database already does that in under a millisecond and does not need a copy of the row to do it.

And do not reach for it because reporting queries are slow. That is usually an index, a materialised view, or a read replica, and all three are less work than a second datastore with a sync problem.

What about Postgres full-text search?

This is the honest comparison, because it is the alternative most teams actually have. Postgres has tsvector, tsquery, GIN indexes, pg_trgm for fuzzy matching and ts_rank for scoring. For a search box over a few hundred thousand rows it is genuinely good, and it has one enormous advantage: the data cannot drift, because there is only one copy.

Start there. Move when you hit something it does not do well — per-field boosting, decent relevance tuning you can inspect, faceted counts in the same round trip as the results, or analysis per language. Those are the things Elasticsearch is actually for, and "our search feels slow" is usually not one of them.

The same reasoning applies to OpenSearch, the fork of Elasticsearch 7.10 that AWS maintains. The query DSL in this track works on both; the divergence is mostly in licensing, newer features and tooling. Everything up to lesson 14 transfers unchanged.

The words you need

The vocabulary is small and mostly means what you would guess. It is worth pinning down now because every error message uses it.

A document is a JSON object. It is the unit of indexing and the unit of a result. In this track one document is one listing.

An index is a collection of documents that share a mapping. It is roughly a table, and the analogy is safe until you try to join two of them.

A mapping is the schema: which fields exist and what type each one is. You can let Elasticsearch infer it. Lesson 3 explains at length why you should not.

A shard is a piece of an index — a self-contained Lucene index holding some of the documents. An index is split across shards so it can outgrow one machine, and searches run on every shard in parallel and merge.

A replica is a copy of a shard on a different node. Replicas serve reads and take over when a node dies.

A node is one Elasticsearch process; a cluster is the nodes that have agreed to work together.

Shards: fewer than you think

The instinct is that more shards means more speed. It does not. Each shard is a separate Lucene index with its own overhead, every search fans out to all of them, and the results have to be merged. Over-sharding a small index makes it slower and uses more memory.

There is a second, subtler reason to keep the count low while you are learning: relevance scores are computed per shard. Term frequencies are local to a shard, so on a small multi-shard index the same query can rank documents differently depending on where they landed — the classic "why did this move?" surprise. The index in this track uses one shard partly for that reason:

"settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,

Green, yellow, red

Cluster health has three states and they mean exactly one thing each. Green: every shard and every replica is assigned. Yellow: every primary shard is assigned but at least one replica is not — all your data is there and searchable, you have lost redundancy. Red: at least one primary shard is missing, so some of your data is not available.

A single-node cluster that asks for one replica sits at yellow forever, because a replica may never live on the same node as its primary. That is not a fault to chase; it is arithmetic. Lesson 16 covers reading health properly.

The one rule to carry through the rest of the track

Your database is the source of truth. The index is a derived copy, and it should be disposable.

Everything gets easier once that is true. You can delete the index and rebuild it. You can change a mapping by building a new index and copying into it. A failed write to Elasticsearch can be logged and retried rather than failing the user's request, because the real data is already safe. In this track's demo app, the repair path is one function:

def rebuild_index(properties: list[Property], *, es: Elasticsearch | None = None) -> int:
    """Drop the index and refill it from Postgres. The repair path, and the seed path.

    Because the index is derived data, a full rebuild is always safe — this is the payoff for
    treating Postgres as the single source of truth.
    """

The moment something lives only in Elasticsearch, all of that stops being true. It happens gradually — a field the API writes straight to the index, a counter incremented in place — and it is very hard to walk back.

The app behind every example

Every query, mapping and measurement in this track comes from StayHub, a working short-let booking app. It is worth a sentence because it shapes what the examples look like: listings have a title and description to match on, a city that needs both loose matching and exact grouping, a price that has to sort correctly, amenities that behave like tick-boxes, and coordinates.

                 ┌── writes ──> FastAPI ──> Postgres
   React apps ───┤                              │
                 ├── reads  ──> Hasura ─────────┘
                 └── search ──> FastAPI ──> Elasticsearch
                                              ▲
                        sunk in application code from every write path

Postgres holds the listings. Elasticsearch holds a flattened copy of the published ones, written from the application's own code on every change. Search is the one read that does not go to the database — which is the entire point of maintaining the index.

What is in this track

Eighteen lessons, in order. Lessons 2 to 6 get you a cluster and a well-designed index. Lessons 7 and 8 cover writing to it and keeping it in step with a database. Lessons 9 to 14 are querying: the DSL, filters, relevance, sorting, aggregations and geo. Lessons 15 to 18 are running it — aliases and reindexing, reading a cluster's health, backups, and what to settle before you put it in front of users.

Each lesson assumes the ones before it and nothing else — no prior Lucene knowledge, and no Kibana. Everything here is done with HTTP requests and the Python client, because that is what your application will be doing, and because a query you can only build by clicking is a query you cannot put in a code review.

Everything is written against Elasticsearch 8.15.3 with the elasticsearch Python client 8.15.1. The version matters more than usual here: security is on by default from 8.0, which means most tutorials written before 2022 give you a docker run that now answers 401. That is where lesson 2 starts.