Concurrency – Double Booking and Distributed Locks

Two guests, one room, the same millisecond. It is the question every booking, ticketing and inventory system gets asked, and the answer everyone gives first is wrong — not slightly wrong, but wrong in a way that works perfectly in testing and fails under exactly the load it was built for.

This post is about that class of bug: where it comes from, the four mechanisms that fix it, and why the one people reach for first should be the last one you consider.

The bug

Here is the natural way to write it, and it is what almost every codebase contains before somebody notices:

if bookings.overlapping(property_id, check_in, check_out):
    raise ConflictException("Those dates are no longer available.")

booking = Booking(property_id=property_id, check_in=check_in, check_out=check_out)
db.add(booking)
db.commit()

Check, then write. Read it again and it still looks correct, because sequentially it is correct. Draw the timeline for two requests and the bug is obvious:

   time ──>

   guest A:  overlapping()? ─> NO ──────────> INSERT ✓
   guest B:      overlapping()? ─> NO ──────────> INSERT ✓
                              ▲
             both checks ran before either insert landed,
             so both saw an empty calendar

Nothing is wrong with either request. The gap between the check and the write is the bug, and it is measured in microseconds — which is why it never happens in development and happens constantly on a launch day.

This shape has a name: check-then-act, or a time-of-check-to-time-of-use race. Once you can see it, you find it everywhere — “is this username taken?”, “is there stock left?”, “has this coupon been used?”.

Three things that do not fix it, and all three get proposed:

• A lock in application code. A mutex protects threads in one process. The second request is on another machine.
• Wrapping it in a transaction. At Postgres’s default isolation level (read committed), the two transactions genuinely do not see each other’s uncommitted inserts. The consistency post covers why — this is a phantom read, and read committed does not promise to prevent it.
• Checking again just before the insert. Narrows the window. Does not close it. A narrower race is a race that is harder to reproduce.

How often does it actually happen?

Worth doing the arithmetic, because the answer surprises people in both directions.

From the estimation post, a regional booking site runs about 0.3 bookings per second at peak. If the window between the check and the insert is 5 milliseconds, the chance that two specific bookings collide seems vanishingly small — and for two random bookings it is.

But collisions do not arrive randomly. They cluster:

   a concert announcement, a sale, a popular listing on a holiday weekend

   normal:  0.3 bookings/sec spread over 2M listings   -> never collides
   spike:   4,000 requests/sec for ONE listing         -> collides constantly

   The rate that matters is not bookings/sec.
   It is requests for THE SAME ROW per second.

That is why estimation does not excuse you here. Throughput can be trivial while contention is severe, because contention is about concentration, not volume. And the moment it matters is always the moment you least want a bug — launch day, a sale, the last seat.

The inverse is also worth saying: this is not a reason to add locking everywhere. Rows nobody contends for need no protection. The question is always which specific row do many writers want at once, and in a booking system there are exactly two — the calendar for a hot listing, and any counter of remaining inventory.

The mechanisms, worst to best

Pessimistic locking

Take a lock, do the work, release it. SELECT ... FOR UPDATE locks rows so nobody else can touch them until you commit.

BEGIN;
SELECT * FROM properties WHERE id = 7 FOR UPDATE;   -- others now wait here
-- check availability, insert the booking
COMMIT;                                             -- lock released

It works, and it is the right tool when contention is genuinely high and conflicts are expensive. Its costs are real: every other request for that property waits, so throughput on a popular listing becomes serial; a long transaction holds the lock for its whole duration; and locks taken in inconsistent orders deadlock.

Note that the lock is on the property, not on the booking — you cannot lock a row that does not exist yet. Locking a parent row to protect inserts of children is a common and slightly awkward pattern, and it serialises far more than you wanted.

Optimistic locking

Assume conflicts are rare. Do not lock; detect.

-- read: note the version
SELECT id, price, version FROM properties WHERE id = 7;    -- version = 4

-- write: only if nobody else changed it
UPDATE properties SET price = 250, version = 5
 WHERE id = 7 AND version = 4;
-- 0 rows updated? Somebody else won. Re-read and retry.

No waiting at all in the common case, which makes it excellent for low-contention updates. The cost is that the caller must handle the retry, and under high contention it degrades badly — everyone keeps retrying and failing, which is worse than queuing politely.

The rule of thumb: optimistic when conflicts are rare, pessimistic when they are common. Editing your own profile is optimistic; the last seat on a flight is pessimistic.

Database constraints

The best answer, when it applies, and the one people think of last: make the bad state impossible to represent. Then no amount of concurrency can produce it, because the database refuses.

StayHub’s calendar rule is a Postgres exclusion constraint:

ALTER TABLE bookings ADD CONSTRAINT bookings_no_overlapping_bookings
EXCLUDE USING gist (
    property_id WITH =,
    daterange(check_in, check_out, '[)') WITH &&
) WHERE (status IN ('PENDING', 'CONFIRMED', 'COMPLETED'));

In English: for any two rows in a blocking status with the same property_id, their date ranges may not overlap. It is a unique constraint generalised from equality to any operator — here &&, “ranges overlap”.

Two details in it are worth stealing. The range is '[)' — inclusive of check-in, exclusive of check-out — so a booking ending on the 5th and one starting on the 5th do not overlap, which is exactly how hotel nights work. And the WHERE clause means cancelled bookings stop blocking dates automatically, with no cleanup job.

Now the race resolves in the only place that can see both writes at once:

   guest A:  check? NO ──> INSERT ──> ✓ committed
   guest B:      check? NO ──> INSERT ──> ✗ rejected by the constraint

   B's application believed the dates were free one millisecond earlier.
   It does not matter. The database is the only thing that gets a vote.

Idempotency keys

The fourth mechanism, for a different flavour of the same problem: not two users racing, but one user whose request happened twice. A double-clicked button, a client retry after a timeout, a proxy replaying a request.

The client sends a key it generates; the server records it and refuses to do the work twice.

CREATE TABLE idempotency_keys (
    key          VARCHAR(64) PRIMARY KEY,   -- the uniqueness IS the enforcement
    response     JSONB       NOT NULL,      -- replay the original answer
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

The important detail: store the response, not just the key. A retry should receive the same answer as the original request, not a 409. From the client’s point of view the operation simply succeeded, which is what makes retrying safe — and safe retries are the only escape from the two generals problem in the consistency post.

This is what payment providers do, and it is why their APIs ask for an idempotency key on every charge.

What if your database cannot do that?

Exclusion constraints are a Postgres speciality. The idea generalises, though — the goal is always to turn the invariant into something the database enforces on a single write.

The portable trick is to make the thing you are protecting into a unique key. For bookings, that means storing one row per night rather than one row per stay:

CREATE TABLE booked_nights (
    property_id BIGINT NOT NULL,
    night       DATE   NOT NULL,
    booking_id  BIGINT NOT NULL REFERENCES bookings(id),
    PRIMARY KEY (property_id, night)      -- the uniqueness IS the guard
);

Inserting three nights for a three-night stay, in the same transaction as the booking, now cannot overlap an existing stay — the primary key forbids it, on any database. The costs are more rows and a multi-row insert, and it makes “which nights are taken” a trivially indexed query rather than a range comparison.

Airlines do the same thing with seats, and inventory systems with reserved units. When you cannot express the rule as a constraint, the move is to reshape the data until you can — and that is usually a better instinct than reaching for a lock.

Keeping the friendly error

A constraint violation is a database error. Shown to a user it is a 500 and a stack trace, which is not acceptable for something that happens legitimately.

So StayHub keeps both checks, and each has a distinct job:

        if self.bookings.overlapping(prop.id, req.check_in, req.check_out):
            raise ConflictException("Those dates are no longer available.")

This is the friendly check. It catches the overwhelmingly common case — dates that were already taken when the request arrived — and produces a good message. It loses the race, and it is not trying to win it. The repository comment says so directly: “This is the friendly check.”

Then the constraint wins the race, and the application translates its verdict:

        except IntegrityError as exc:
            self.db.rollback()
            if _is_overlap_violation(exc):
                raise ConflictException(
                    "Those dates were just booked by someone else."
                ) from exc
            raise

Note that the two messages differ. “No longer available” versus “just booked by someone else” — because they describe genuinely different situations, and the second one is worth knowing about when reading logs.

And note that it does not catch IntegrityError broadly:

def _is_overlap_violation(exc: IntegrityError) -> bool:
    """Was this IntegrityError our exclusion constraint, or something else entirely?

Treating every integrity error as “dates taken” would report a broken foreign key as a booking clash — turning a real bug into a plausible business message that nobody investigates. Checking the constraint name by hand is unglamorous and it is the difference between an error handler and a bug concealer.

Counters, and the same bug in miniature

The smallest instance of check-then-act, and the one most people write without noticing:

count = redis.get("views:42")      # read
redis.set("views:42", count + 1)   # modify, write

Two processes read 100, both write 101, and one view is gone. It is the identical shape, compressed into two lines, and under any real concurrency it loses a substantial fraction of every count.

The fix is that the operation must be one operation:

INCR views:42          # atomic — read, add, write, indivisibly

Same in SQL: UPDATE posts SET views = views + 1 WHERE id = 42 is atomic because the database evaluates views + 1 under a row lock it takes for you. Reading the value into your application and writing it back is not.

The general rule, and it is the through-line of this entire post: if a decision depends on a value, read and write it in a single operation, in the system that owns it. Every mechanism here is a way of achieving that — a constraint does it in the database, an atomic increment does it in Redis, a Lua script does it for logic too complex for one command.

Where the last one is needed

Rate limiting is the canonical case where no single command is enough, because the logic is “refill the bucket for elapsed time, then spend a token if there is one” — three steps that must be indivisible.

tokens = redis.get(key)            # process A reads 1.0    process B reads 1.0
if tokens >= 1: tokens -= 1        # A: allow               B: allow
redis.set(key, tokens)             # A writes 0.0           B writes 0.0

Two requests, one token, both allowed. And this is not a rare interleaving — it is the normal outcome when a client sends requests concurrently, which is exactly what an abusive client does. Under four API workers the effective limit becomes four times what it says, and the limiter passes every sequential test ever written for it.

The answer is to send the logic to where the data is, as one atomic unit. Redis runs a Lua script without interleaving any other command:

local tokens = tonumber(bucket[1])
...
tokens = math.min(capacity, tokens + elapsed * refill)

local allowed = 0
if tokens >= cost then
  tokens = tokens - cost
  allowed = 1
end

Read, decide and write become one operation, which is what the algorithm assumed all along. The next post is about that in full, including the test that proves it: 50 concurrent threads against a 20-token bucket, and exactly 20 get through.

Distributed locks

When the resource is not a database row — a file, an external API, a scheduled job that must run on exactly one server — there is no constraint to lean on, and you need a lock that several machines share.

The naive version has a fatal flaw:

if redis.get("lock:job") is None:      # check
    redis.set("lock:job", "mine")      # ...act
    do_the_work()

Check-then-act again, one level up. Two processes both see no lock and both take it. The atomic version is a single command:

SET lock:job <owner-token> NX PX 30000
#                          │  └─ expire after 30s, so a crashed holder
#                          │     does not lock the resource forever
#                          └──── only if it does not already exist

The NX makes it atomic and PX is the safety net. A lock with no expiry held by a process that dies is a resource nobody can ever have again.

But the expiry creates the problem that makes distributed locks genuinely hard:

   t=0    A takes the lock, TTL 30s
   t=25   A pauses — GC, a slow disk, the VM is descheduled
   t=30   the lock EXPIRES. A does not know.
   t=31   B takes the lock and starts working
   t=35   A wakes up, believing it still holds the lock

          A and B are now both in the critical section.

Nothing detects this. Partial mitigations exist — release with a compare-and-delete so A cannot release B’s lock, extend the TTL with a heartbeat, use fencing tokens so the protected resource rejects the stale holder — but a distributed lock is fundamentally a lease with a timeout, and timeouts can be wrong.

Hence the ordering in this post. A database constraint cannot fail this way, because there is no lease and no timeout: the check and the write are the same operation, inside the system that owns the data. Reach for a distributed lock when there is genuinely no such system.

Deadlocks

Once locks are involved, two transactions can each hold what the other wants.

   transaction A            transaction B

   lock row 1               lock row 2
   ...                      ...
   wants row 2  ──waits──>  holds it
   holds it     <──waits──  wants row 1

   Neither can proceed. Postgres detects the cycle after ~1s
   and kills one of them with a deadlock error.

The database resolving it is the good outcome — the alternative is waiting forever. But one transaction failed, and the fix is not to catch the error. It is to make the cycle impossible.

Lock in a consistent order. If every transaction that touches several rows takes them in ascending id order, no cycle can form: A holds 1 and wants 2, so B cannot be holding 2 and wanting 1, because B would have taken 1 first. This is the whole solution, and it is almost always enough.

Two supporting habits. Keep transactions short — a transaction that opens, calls an external API for two seconds, then writes is holding locks for two seconds. And never call an external service inside a transaction at all; do the call first, or afterwards through the outbox.

Since a deadlock is transient by nature, a retry usually succeeds — so retrying the whole transaction once or twice, with a small delay, is a reasonable belt-and-braces on top of the ordering rule.

How to test for a race

The problem with concurrency bugs is that a normal test cannot see them. Calling the function twice in sequence exercises the code and proves nothing, because sequentially the buggy version is correct.

The test has to be genuinely concurrent, and it has to assert on a total:

        with cf.ThreadPoolExecutor(max_workers=50) as pool:
            allowed = list(pool.map(lambda _: rate_limit.check(rule, "swarm").allowed, range(50)))

        assert sum(allowed) == 20, (
            f"{sum(allowed)} of 50 allowed against a capacity of 20 — the check is not atomic"
        )

Two design decisions in that test are worth copying. It uses far more threads than capacity, so a lost update shows up as a number rather than as a rare flake. And it asserts an exact total, not “at most” — a racy implementation does not fail cleanly, it allows 23 or 27 or 21 depending on timing, and only an exact assertion catches the small overruns.

The equivalent for a database race needs real, separate connections. Two sessions sharing one connection cannot block each other, so a test using a shared transactional fixture will pass against code with no locking whatsoever — which is the worst kind of green, and a trap this project walked into while writing the outbox tests.

Choosing

Holds, and the case a constraint cannot cover

One shape that recurs in ticketing, retail checkout and flights: the resource must be reserved while the user pays, and released if they abandon.

StayHub handles it by making the hold a real row in a blocking status:

            status=BookingStatus.PENDING,

Because PENDING is one of the statuses the exclusion constraint covers, the dates are protected during checkout by exactly the same mechanism that prevents double booking. No separate hold table, no lock, no timer — the constraint does both jobs.

   PENDING    dates held, not yet paid       ─┐
   CONFIRMED  paid                            ├─ block the calendar
   COMPLETED  stay finished                  ─┘
   CANCELLED  ──────────────────────────────── does NOT block

   Cancelling frees the dates automatically. No cleanup job:
   the constraint's WHERE clause stops seeing the row.

That last line is the elegant part. Expressing “which states block” once, in the constraint, means release is a status change rather than a deletion — and the availability query and the constraint cannot disagree, because they are reading the same definition.

What it does not solve is expiry. A guest who opens checkout and closes the tab holds those dates forever. The answer is a scheduled job that cancels stale pending bookings after some minutes — and it has its own concurrency question, because that job must run on exactly one server. Which is precisely the case with no database row to constrain, and therefore one of the few legitimate uses of a distributed lock.

The higher-stakes version of this is a flight seat, where holds are shorter, contention is far worse, and the business deliberately oversells. The airline post takes it further.

The summary

• Check-then-act is always a race. The gap is microseconds, which is why it passes review and fails in production.
• A transaction alone does not fix it at the default isolation level.
• Prefer a constraint. Making the bad state unrepresentable beats coordinating to avoid it.
• Keep the friendly check anyway, for the message — and be explicit that it is not the guard.
• Translate the specific violation, never every integrity error.
• Optimistic for rare conflicts, pessimistic for common ones.
• Distributed locks are leases that can be wrong. Last resort.

One last framing, because it is what makes this topic click. Every mechanism above is an answer to the same question: which single component gets to decide, and can it decide in one step? A constraint says the database decides, in one write. An atomic increment says Redis decides, in one command. A Lua script says Redis decides, in one script. A distributed lock is the weakest of the four precisely because the decider — a lease with a timeout — can be wrong about whether it still holds.

Next: rate limiting — the same atomicity problem, in a system where the answer is a Lua script rather than a constraint.