Synchronization

How does the system keep an index in object storage consistent when nodes join, leave, or fail? Two complementary mechanisms maintain consistency and coordinate writes:

• Leadership table: A shared table that tracks node liveness and assigns index write responsibility.
• Conditional writes: Atomic updates and epoch-scoped storage keys that prevent stale or concurrent writers from corrupting data.

Neither mechanism replaces the other. The leadership table provides liveness and efficient resource use, while conditional writes enforce data safety.

The manifest is what a synchronized index is

A push writes the files of a pinned Lucene commit together with the index definition and a manifest. The manifest lists those files with their sizes and checksums. The system compares manifests to decide whether to push or pull data, rather than comparing files directly or relying on Lucene segment numbers, because an index definition can change without a new Lucene commit.

Replacing the remote manifest requires a conditional write using an If-Match header on the ETag of the manifest the writer last saw, or If-None-Match: * for the initial write. If a push is based on a manifest that the remote storage no longer holds, the write fails instead of overwriting changes pushed by another writer. This conditional check provides safety: a node that erroneously assumes it is still the designated writer can attempt a push, but storage rejects it. At startup, candidate nodes verify that the storage backend enforces conditional writes and refuse to run against storage that does not. Google Cloud Storage states the same condition on the generation of the object instead of on its ETag, so a node pointed at it translates the headers of a write. For the headers themselves, see Object storage requirements.

The registry carries change hints

A node keeps its own copy of the manifest of each open index and of the search settings object of each index it serves. Keeping a copy current means asking storage whether the object changed. Both reads are conditional, but asking about each index separately still costs one request per index per interval per node, so the cost grows with the number of indexes rather than with the number of changes.

To reduce polling requests, the registry includes version hints for each index. For each index, the registry records the version of the most recent manifest push for each generation and the version of the stored settings object. A node skips storage requests if its local copy matches the hinted version, and fetches the object only when the hint changes. This design keeps steady-state request costs proportional to the number of nodes rather than the number of indexes.

The node that updates an object also reports its hint. A manifest push reports the manifest version, and updating settings reports the settings version. Because both document writes and settings updates execute on the designated index writer, the writer always reports its own updates. The writer buffers hints for several seconds and applies them to the registry in a single conditional write. This batching avoids contending for the registry on every push. For entries that predate hints, the index writer reads storage and populates hints over multiple passes, avoiding request bursts during upgrades.

A version reaches a reading node through these steps:

A single read of the index registry serves the whole node. Two parts of a node work from the registry: open indexes (which pull manifests) and the search settings of the indexes the node serves. Each part receives the result of that one read. Each part states how often it wants the registry read, and the node reads at the shortest interval any part requests:

• The open indexes request EXOFIND_INDEXES_REFRESH_INTERVAL (default 30 seconds).
• The search settings request EXOFIND_SETTINGS_REFRESH_INTERVAL (default 10 seconds), and request nothing while the node holds no settings. A node that has served no searches reads at the index interval alone.

Each refresh interval provides two guarantees for the objects it names:

• The node makes no two storage requests for the same index inside the interval. An index written continuously therefore costs a reader at most one manifest request per interval, regardless of how often the registry reports a new version.
• The node considers every index at least that often.

Because a part acts on every read rather than on a schedule of its own, a change usually reaches a node sooner than its own interval promises. An index that has been quiet is pulled on the first read after its version changes.

A part never runs on the reading thread, and a part still working from an earlier read is passed over until it finishes. A pull that takes minutes does not delay search settings from reaching the node.

The read reports whether the registry changed. A node removes the local copies of indexes and generations the deployment no longer holds only when it changed, and once when the node starts. A registry that says the same thing twice costs the node nothing.

Hints stay advisory rather than authoritative state. A node verifies every local copy directly against storage after at most EXOFIND_INDEXES_VERIFY_INTERVAL (default 10 minutes), whatever the hints say. Both refresh intervals are held under the verify interval, so a refresh interval configured above it cannot suppress the verification it promises. If a writer crashes before reporting a hint, or if an older node overwrites the registry without hints, the system experiences temporary staleness or extra requests, but never corrupted state. Like the leadership table, hints optimize performance while conditional reads and writes enforce safety.

Visibility is bounded staleness, not a watch

A node learns of changes by polling the registry at its refresh interval. Storage never notifies a node that an index changed.

S3 has no watch, subscribe, or long-poll API. Reads are pull-only. Storage notifications, such as AWS S3 Event Notifications, MinIO bucket notifications, SeaweedFS metadata subscriptions, and Ceph RGW notifications, are not portable across storage backends and operate on a best-effort basis. A notification can shorten the window between a push and the pull that follows, but it cannot close the window or serve as a source of truth.

The engine defines a staleness bound instead. A local copy is at most one refresh interval behind the registry, and the registry is at most a few seconds behind a push. Because every poll is a conditional request, an up-to-date copy costs one storage request per interval.

A read that cannot accept this bound carries a freshness token naming an index state. The answering node closes the gap for that read alone: a conditional read of the registry for a generation, a conditional read of the settings object for a settings version, and a pull for a commit sequence. The requests this costs grow with the reads that demand freshness, not with the number of indexes. Background polling remains unchanged.

A freshness token cannot make an uncommitted write survive a writer crash. If a writer stops before committing, it loses the write. When a successor node takes over the index, its commits advance the sequence to the number named in the token without the lost write. For more details, see What a write guarantees.

Why writes are scoped to epochs

Lucene names its files by sequential numbering. Two independent writer sessions can both produce a file named _5.cfs. If both sessions uploaded files using that name, a writer that fails the manifest race could overwrite a file referenced by the winning writer’s manifest.

To prevent collisions, object keys are scoped to epochs. A writer session claims an epoch by conditionally updating the manifest, and then uploads all files under e<epoch>/. If storage rejects the epoch claim, the session does not upload any files and does not open its writer. File names remain local, while object keys are remote.

A session makes a conditional manifest write twice: once to claim its epoch, and once for every push:

When the remote holds no manifest, because the index is new or because the manifest was removed, the claim writes nothing. The first push then writes the manifest on the condition that there still is none, so of two sessions that both found the remote empty, only the first to push is accepted. A claim written as a manifest that names no files would make every reader remove its copy and answer with nothing until that push lands.

The claim happens when the writer opens, before the node acknowledges any write, rather than before the first upload. A node that takes an index over acknowledges writes from the moment its writer opens, but pushes only at the next commit interval. Claiming at the first push would leave that whole span with nothing refusing the node the index was taken from: a stale push accepted in it wins the manifest race, and the successor then abandons and pulls over the documents it has already acknowledged.

Epochs keep the uploads of two sessions apart, but not two uploads of one session. Lucene never writes to a name twice, so that is enough for its files. The files the engine keeps beside the segments, such as the definition and the change log, are rewritten in place under the same name. Their keys also carry the checksum of the contents, so a push whose manifest is refused cannot have replaced an object that the accepted manifest still names. A pull verifies the size and checksum of every file it downloads against the manifest, so an object that does not hold what the manifest describes fails the pull instead of entering the index.

Unchanged files retain their keys across epochs, and the manifest records each key alongside its corresponding file name. This mapping avoids re-uploading files that a previous indexer already pushed, making failovers efficient. After adopting a pulled manifest, a session claims a new epoch because the adopted manifest can reference keys from the epoch where it originated.

When a push replaces the manifest, the writer records the objects the new manifest no longer names. The writer deletes them at the first push after a one-hour grace period. This delay exists because a reader that pulled the old manifest can still be downloading an object, and a missing object fails its whole pull. A first pull of a large index takes longer than the commit interval, so a writer that deleted at once could fail every attempt while merges run.

A periodic listing sweep removes unreferenced objects that a push did not clean up. The sweep skips objects younger than the grace period. The sweep also skips objects the writer stopped naming less than a grace period ago, regardless of upload age.

The recorded list lives in memory, so a writer that stops loses it. The sweep of the next writer measures the age of those objects from their upload. A writer restart during a long pull therefore lets that sweep remove an object the reader is still downloading.

A failed pull keeps the files it verified. The retry compares size and checksum against the manifest and downloads only what is missing or differs. The pull always downloads an entry without a checksum, because a file with the same name and size that a writer on the node left behind would otherwise pass.

How merged files leave the bucket

When Lucene merges small segments into larger ones, a commit includes the merged segment and drops the old segment files from the index. The merge deletes nothing from the bucket directly. The push following the commit records the objects the new manifest no longer names, and the writer deletes them at the first push after a one-hour grace period.

Commit triggers determine when the push that records obsolete objects runs:

• EXOFIND_INDEXES_COMMIT_MAX_CHANGES triggers a commit after reaching a change threshold (default 10,000 changes).
• EXOFIND_INDEXES_COMMIT_MAX_INTERVAL triggers a commit after a time threshold (default 5 seconds).

Under a normal write load, the writer deletes obsolete objects at the first push after the grace period, so the bucket holds about an hour of merged segments beside the index. When writes stop, no further pushes occur. Every node that can index therefore also runs the orphan sweep on a timer.

The timer ticks every EXOFIND_INDEXES_REMOVAL_SWEEP_INTERVAL (default: 10m), the same interval the removal sweep uses. On every tick the node asks each open generation it writes to sweep. A generation lists its objects at most once per grace period, whether a push or the timer asks, and a new instance seeds its last sweep at a random point inside the grace period, so a node that opens many generations does not list all of them at once.

A push sweeps right after its conditional manifest write, so it sweeps with the manifest the remote holds. A timer sweep has no such guarantee: the node may have lost the index without knowing it yet, and its successor may name again a checksum-keyed object this node stopped naming. A timer sweep therefore reads the entity tag of the remote manifest first and removes nothing when the tag differs from the one the node last synchronized. A successor replaces the manifest when it claims the index, so a changed tag also catches a claim the node has not learned of.

As a known limit, only open generations are swept. A held index that nothing has opened has no manifest in memory, and opening it costs a writer and a pull, so its remote is left until it opens again.

A push uploads all new files before writing the manifest conditionally. If a push stops halfway and uploads only some of its files, readers remain unharmed because they continue following the previous manifest. The unreferenced uploads sit harmlessly in the bucket until a later sweep removes them after the one-hour grace period. In contrast, publishing a manifest that names a missing object would break readers by causing their pulls to fail. To prevent missing objects, a push uploads all required files before updating the manifest, checks the remote baseline by entity tag before uploading, and treats a missing remote manifest as empty so all files are uploaded again.

A pull replaces the files an open writer holds

Only the node that holds the index claim writes to an index, using a single Lucene IndexWriter over a local directory. Two events can put an index in the needs_pull state while its writer remains open: object storage refuses the conditional manifest write of a push because another node pushed since, causing a conflict, or the node loses the index claim while paused and discovers the loss during the next coordination round. The refresh loop then pulls every index in needs_pull.

An index accepts writes for as long as a push runs, so a write can record its change after the conflict. The index stays in needs_pull and refuses the writes that follow. An index that said it held changes again would never be pulled: it would go on accepting writes, conflict on every commit, and drop everything it accepted when a pull finally came.

Calling IndexWriter.rollback restores the local directory to the last commit that writer made, deleting every segments file and segment file not named in that commit. Because a pull downloads files with those names from the manifest pushed by another node, rolling back after the download deletes the newly pulled files. The directory is left with no commit or with files from two writers. If the node only reads, it answers searches with no hits, and subsequent pulls receive HTTP 304 because the local manifest already matches remote storage. If the node writes, it opens a new empty index over the directory and pushes it using the current entity tag, deleting every segment object named in the previous manifest.

To prevent this corruption, the engine rolls the writer back before the pull starts, while the index is marked as pulling. Uncommitted documents are dropped, which the pull does in either case because the node continues from the copy in storage. The index holds no writer and refuses writes until the pull finishes and opens the next one. If a pull fails, the index remains in needs_pull so that a subsequent pull opens a writer.

Two checks protect a local copy that loses files for another reason. When the local manifest names a commit, the node opens the writer in Lucene append mode, so Lucene reports a missing commit as an error instead of creating an empty index. Local storage mode keeps no manifest and does not use append mode. In addition, a push fails if its files contain no Lucene commit while the last synchronized manifest names one, preventing a manifest with no segments from deleting remote segment objects and making the index unrecoverable.

A close waits for the pull that is running

A node closes an open generation when the cache evicts it, when the node hands the index over, and when it shuts down. The next request opens a new instance over the same local directory, and that instance pulls into it. A close therefore stops further pulls of the instance it closes and waits for the pull that is running. Two pulls of one directory download into the same paths, write their own manifest over each other, and delete the files the other one brought.

The close tells the synchronization to stop before it waits, so the pull returns after the file it is downloading instead of after the whole index. A stopped pull writes no manifest and deletes no local file. It leaves the local copy in the state a pull that failed part way leaves, and the pull of the next instance checks the downloaded files against the manifest it applies.

Leadership is liveness, not safety

The bucket maintains a single leadership table that tracks which node writes to each index. The table contains a claim entry for each index naming its holder, alongside an entry for each candidate node indicating that the candidate is alive. The table is updated as a whole, conditionally based on its version. If two candidates attempt concurrent updates, only one write succeeds.

Every candidate runs a coordination round at an interval equal to one-third of the claim duration (EXOFIND_INDEXER_LEASE_DURATION), measured from the start of the previous round rather than from its end. Three rounds have to fit inside a lease, so an interval that grew with the time storage took would let the claims of a running node lapse. During a round, a candidate performs the following actions:

• Renews its own entries.
• Takes over claims whose holders stopped renewing due to crashes, hangs, or network partitions.
• Rebalances index distribution by claiming unassigned indexes if it holds fewer than its fair share, or handing over an index if it holds more than its share while another candidate holds less.

The candidate selects the most idle index for handover based on a write count that halves every few minutes. This keeps active indexes on writers with warm Lucene state, while quiet indexes absorb the cost of pulling and reopening data. The time required for a claim to lapse determines approximate failover duration.

Equal index counts can still result in uneven load, such as when one node holds all active indexes and another holds only idle ones. To balance load, each claim includes a write load metric equal to the bit length of the decaying write count, which increments when write traffic approximately doubles.

When a node’s load total substantially exceeds that of the least-loaded candidate, the node marks an index claim as offered. An underloaded candidate responds by recording itself as the taker in the claim, and the holder transfers the claim. Only the holder transfers a claim, ensuring an index changes hands only when its writer initiates the transfer. The count-balancing process completes the exchange by moving an idle index back in the opposite direction.

A node offers an index only when the index load metric fits twice into the difference between the two nodes’ totals. Moving the index must narrow the load gap rather than reverse it. This threshold prevents two nodes from repeatedly trading a single active index back and forth.

Index handovers follow a strict order to protect acknowledged writes. A holder releases a claim only after committing and pushing all pending index data:

1. The index stops accepting writes. Incoming writes during this transition are rejected, and the caller retries them.
2. The holder flushes and pushes pending data to storage.
3. In a subsequent round, the holder transfers the claim to the taker, or drops the claim for an under-capacity candidate to acquire.

A failed flush and a lapsed lease each end the handover in a different way:

Because of this order, a successor node always pulls a manifest that includes the flush, preserving acknowledged documents across rebalances. A shutting-down node follows the same order for all held indexes: it flushes first, then removes itself from the table. If a flush exceeds the lease duration, the claims lapse instead of being released mid-flush. When a node loses a claim because the lease lapsed, it pushes nothing, because a successor might already be writing. The expired node drops unpushed data, while conditional writes prevent storage corruption.

A failed flush cancels the handover instead of completing it. The index keeps its writer and the data the flush could not push, the claim stays with the holder, and the index accepts writes here again. A subsequent round starts the handover anew and flushes again. A shutting-down node whose flush fails leaves its claims to lapse rather than releasing them, so a successor waits the lease out instead of pulling a manifest that is missing acknowledged documents.

A node tells the two cases apart by how the claim left it. A claim it still holds while the index drains marks a handover it chose, so the index flushes. A claim that ends up naming another node, or that the round dropped along with a deleted index, marks a loss. Every generation of a lost index then stops pushing at once, including a flush that a handover queued moments earlier and the push that closing an index normally makes.

An unassigned index does not wait for a coordination round. The first candidate node that receives a write claims the index immediately. This ensures newly created indexes acquire writers immediately and routes writes promptly if a holder fails.

The leadership table ensures that at most one node expends effort writing to each index, and informs other nodes where to forward writes. The table does not guarantee data safety. For example, clock drift or a paused process can cause a stale node to attempt writes after its claim has lapsed. Conditional manifest writes and epoch scoping prevent stale writers from corrupting data. The leadership table maintains system liveness, while conditional writes provide safety.

Related

• What a write guarantees - What a client can assume after a write returns, and what a failover does to it.
• Storage layout - The manifests, the leadership table, and the epoch prefixes as objects in the bucket.
• Object storage requirements - The conditional write headers a bucket has to enforce.
• Glossary - Claims, epochs, manifests, and the other terms used here.
• Generations - Why an index is a name with generations under it.
• Running multiple nodes - Configuring candidacy, failover, and write routing.