Replica sets

A replica set is the answer to one blunt question: what happens when the server holding your data stops responding? With a single mongod, the answer is that your application is down until someone fixes the machine. With a replica set, another copy takes over in seconds. A replica set is a group of mongod instances that all hold the same data — one primary that accepts every write, and one or more secondaries that keep their own copies in step with it.

Everything marked 3-node below was run on this Mac on a throwaway replica set of three mongo:8.3.11 containers (rs-1, rs-2, rs-3, replica set name rsb3, mongosh 2.11.1, Node driver 7.7.0 in a fourth container), each capped at 512 MB, all removed afterwards. Fences with a path comment also ran against the course’s one-node replica set. Python/Go/Rust tabs mirror the verified shell/Node call and are not executed by the course’s automated checker.

The lab

Three containers on a private Docker network, one rs.initiate:

scripts/rs-lab.sh

docker network create mongodb-lab
for i in 1 2 3; do
  docker run -d --name rs-$i --hostname rs-$i --network mongodb-lab --memory 512m \
    mongo:8 mongod --replSet rsb3 --bind_ip_all --wiredTigerCacheSizeGB 0.25 --oplogSize 128
done
docker exec rs-1 mongosh --quiet --eval 'rs.initiate({ _id: "rsb3", members: [
  { _id: 0, host: "rs-1:27017" }, { _id: 1, host: "rs-2:27017" }, { _id: 2, host: "rs-3:27017" } ] })'
# clean up when done: docker rm -f rs-1 rs-2 rs-3 && docker network rm mongodb-lab

Primary, secondaries, and the oplog

Only the primary takes writes. Each write it applies is also recorded in a capped collection called the oplog (local.oplog.rs). Every secondary reads the primary’s oplog and replays the same operations, so the copies converge. That makes secondaries eventually consistent: there is a lag between a write on the primary and the same write everywhere else. A real oplog entry for insertOne({ name: "Ada" }):

{
  op: 'i',
  ns: 'lib.members',
  o: { _id: ObjectId('6abccde57c8538514ddd8679'), name: 'Ada' },
  ts: Timestamp({ t: 1790758373, i: 2 }),
  t: Long('1')
}

op: "i" is an insert, o is the document, ts orders it, t is the election term. And what the whole set looks like (3-node):

// mongosh/rs-status.js @skip-verify needs the 3-node replica set (real output below)
const st = rs.status();
console.log(st.members.map((m) => ({ name: m.name, stateStr: m.stateStr, health: m.health })));
const c = rs.conf();
console.log({ heartbeatIntervalMillis: c.settings.heartbeatIntervalMillis, electionTimeoutMillis: c.settings.electionTimeoutMillis, writeConcernMajorityJournalDefault: c.writeConcernMajorityJournalDefault });
const h = db.hello();
console.log({ setName: h.setName, hosts: h.hosts, primary: h.primary, isWritablePrimary: h.isWritablePrimary });

[
  { name: 'rs-1:27017', stateStr: 'PRIMARY', health: 1 },
  { name: 'rs-2:27017', stateStr: 'SECONDARY', health: 1 },
  { name: 'rs-3:27017', stateStr: 'SECONDARY', health: 1 }
]
{
  heartbeatIntervalMillis: 2000,
  electionTimeoutMillis: 10000,
  writeConcernMajorityJournalDefault: true
}
{
  setName: 'rsb3',
  hosts: [ 'rs-1:27017', 'rs-2:27017', 'rs-3:27017' ],
  primary: 'rs-1:27017',
  isWritablePrimary: true
}

The heartbeat interval (2 s) and election timeout (10 s) are the defaults; they are the same on any set, so this fence also runs on the course’s one-node set:

mongosh/rs-settings.js

const s = rs.conf().settings;
console.log({ heartbeatIntervalMillis: s.heartbeatIntervalMillis, electionTimeoutMillis: s.electionTimeoutMillis });
assert.strictEqual(s.heartbeatIntervalMillis, 2000);
assert.strictEqual(s.electionTimeoutMillis, 10000);

{ heartbeatIntervalMillis: 2000, electionTimeoutMillis: 10000 }

flowchart TB
  Client["Application writes"] --> P["Primary"]
  P -- "records every write" --> OL["Oplog (capped collection)"]
  OL -- "tail and replay" --> S1["Secondary 1"]
  OL -- "tail and replay" --> S2["Secondary 2"]
  P -. "heartbeats every 2s" .- S1
  P -. "heartbeats every 2s" .- S2

The primary takes all writes and logs them; secondaries tail the oplog and replay it to stay in sync

The members exchange heartbeats every 2 seconds (heartbeatIntervalMillis); the MongoDB docs add that a member whose heartbeat does not return within 10 seconds is marked inaccessible (Replica set elections, consulted 2026-09-30). Secondaries refuse writes (3-node, connected directly to rs-2):

MongoServerError | 10107 NotWritablePrimary | not primary

The size of the oplog decides how long a secondary (or a change-stream consumer) can be offline and still catch up; the lab set had 128 MB (rs.printReplicationInfo()):

actual oplog size
'128 MB'
---
configured oplog size
'128 MB'
---
log length start to end
'13 secs (0 hrs)'
---
oplog first event time
'Wed Sep 30 2026 08:52:19 GMT+0000 (Coordinated Universal Time)'
---
oplog last event time
'Wed Sep 30 2026 08:52:32 GMT+0000 (Coordinated Universal Time)'
---
now
'Wed Sep 30 2026 08:52:50 GMT+0000 (Coordinated Universal Time)'

Automatic elections — timed

When the primary disappears, the remaining members vote; a candidate that gets a majority becomes primary. How long does that take? The manual says: “The median time before a cluster elects a new primary should not typically exceed 12 seconds, assuming default replica configuration settings. This includes time required to mark the primary as unavailable and call and complete an election.” (Replica set elections). While elections run, the set cannot process writes; reads on secondaries can continue.

Measured, twice. A writer inside the set inserted with w: "majority" every 100 ms (retryWrites is on by default, so an interrupted write is retried once a primary exists), and the writer logged how long each insert took:

// mongosh/rs-writer.js @skip-verify needs the 3-node replica set (run inside a member)
const c = db.getSiblingDB("lib").ticks;
const end = Date.now() + 40000;
let i = 0;
while (Date.now() < end) {
  const t0 = Date.now();
  let r = "ok";
  try { c.insertOne({ i: i++, t: new Date() }, { writeConcern: { w: "majority" } }); } catch (e) { r = "ERR " + e.name + " " + e.code; }
  print(JSON.stringify({ at: t0, ms: Date.now() - t0, r }));
  sleep(100);
}

Run 1 — the primary killed with docker kill (SIGKILL: a crash, no goodbye). Run 2 — the primary stopped with docker stop (SIGTERM: a clean shutdown). The only iterations slower than 300 ms:

{"at":1790758407048,"ms":10773,"r":"ok"}
rs-1:27017 SECONDARY | rs-2:27017 PRIMARY | rs-3:27017 SECONDARY

{"at":1790758473930,"ms":324,"r":"ok"}
rs-1:27017 PRIMARY | rs-2:27017 (not reachable/healthy) | rs-3:27017 SECONDARY

A crash stalled one write for about 10.8 s — the 10 s election timeout plus the election itself, in line with the manual’s “typically … 12 seconds” — and then that write succeeded: the driver waited for the new primary and retried. A clean shutdown lets the primary hand over, and the stall was under half a second. Both leave you with the same lesson: application code must ride out a pause of about ten seconds during a crash, not fail fast. (Atlas documents a failover test button and says an election takes five seconds on average on its clusters — Test Resilience, consulted 2026-09-30.)

Why an odd number of voters

A candidate needs votes from more than half the voting members. Below, two of three members were stopped; the survivor could not win an election and stayed a secondary, so it refused writes:

rs-1:27017 SECONDARY | rs-2:27017 (not reachable/healthy) | rs-3:27017 (not reachable/healthy)
MongoServerError | 10107 NotWritablePrimary | not primary

After the two members were restarted the set healed by itself:

rs-1:27017 PRIMARY | rs-2:27017 SECONDARY | rs-3:27017 SECONDARY

With three voters one failure leaves a majority of two. With two voters one failure leaves one of two — no majority, no primary.

Connecting to a replica set

A driver connects to the set: list a few seeds and name the set with replicaSet=. It discovers the rest of the topology, finds the primary, and follows it across an election. Node driver inside the lab network (3-node output — 20 reads with secondaryPreferred were served by the two secondaries):

mongosh

scripts/rs-connect.sh

mongosh "mongodb://rs-1:27017,rs-2:27017,rs-3:27017/?replicaSet=rsb3"

Node.js

// app/rs-connect.mjs @skip-verify needs the 3-node replica set (run inside the lab network)
import { MongoClient } from "mongodb";

const uri = "mongodb://rs-1:27017,rs-2:27017,rs-3:27017/?replicaSet=rsb3";
const client = new MongoClient(uri);
await client.connect();
const hello = await client.db("admin").command({ hello: 1 });
console.log({ setName: hello.setName, primary: hello.primary, hosts: hello.hosts });

const members = client.db("lib").collection("members", { readPreference: "secondaryPreferred" });
const served = new Set();
for (let i = 0; i < 20; i++) served.add((await members.find({}).explain()).serverInfo.host);
console.log("reads served by:", [...served].sort());
await client.close();

Python

from pymongo import MongoClient

uri = "mongodb://rs-1:27017,rs-2:27017,rs-3:27017/?replicaSet=rsb3"
client = MongoClient(uri)

Go

uri := "mongodb://rs-1:27017,rs-2:27017,rs-3:27017/?replicaSet=rsb3"
client, err := mongo.Connect(options.Client().ApplyURI(uri))
if err != nil {
    return err
}

Rust

let uri = "mongodb://rs-1:27017,rs-2:27017,rs-3:27017/?replicaSet=rsb3";
let client = Client::with_uri_str(uri).await?;

{
  setName: 'rsb3',
  primary: 'rs-1:27017',
  hosts: [ 'rs-1:27017', 'rs-2:27017', 'rs-3:27017' ]
}
reads served by: [ 'rs-2', 'rs-3' ]

By default reads go to the primary, which keeps them strongly consistent. To spread read load you can choose a read preference such as secondaryPreferred — at a price, shown next.

A secondary can be behind

One secondary (rs-2) was frozen with db.fsyncLock() — a stand-in for a lagging member — while an order was written to the primary and read straight back three ways (Node, 3-node):

// app/rs-stale-read.mjs @skip-verify needs the 3-node replica set and a frozen secondary (real output below)
const orders = client.db("shop").collection("orders");
const direct = new MongoClient("mongodb://rs-2:27017/?directConnection=true", { readPreference: "secondary" });
await direct.connect();
await orders.insertOne({ _id: "o-1", total: 42 });
console.log("bug   (read the frozen secondary):", await direct.db("shop").collection("orders").findOne({ _id: "o-1" }));
console.log("fix 1 (read the primary)         :", await orders.findOne({ _id: "o-1" }, { readPreference: "primary" }));
// fix 2: causal consistency - the read waits for the secondary to reach the write
const session = client.startSession({ causalConsistency: true });
await orders.insertOne({ _id: "o-2", total: 7 }, { session, writeConcern: { w: "majority" } });
await orders.findOne({ _id: "o-2" }, { session, readPreference: "secondary", readConcern: { level: "majority" }, maxTimeMS: 2000 });

bug   (read the frozen secondary): null
fix 1 (read the primary)         : { _id: 'o-1', total: 42 }
fix 2 (causal session, secondary) : served by a caught-up secondary in 40 ms

The write was acknowledged (w: "majority" by default — the other secondary and the primary had it) and yet the frozen secondary returned null. A causally consistent session gives a read-your-writes guarantee on secondaries: the read is served only by a member that has reached the write — here the healthy rs-3, in 40 ms — instead of the frozen rs-2.

In Atlas: an Atlas cluster is a replica set (in a sharded cluster, each shard is one — Manage clusters), and Test Failover triggers the election for you.

Tips and gotchas

• Keep an odd number of voting members: three tolerate one failure, five tolerate two.
• An arbiter votes but holds no data; it cannot become primary and adds no durability. It also changes the default write concern (see the read/write concerns lesson).
• Secondaries lag. If your application reads its own writes right after writing, read from the primary or use a causally consistent session (proved above).
• Replication is availability, not backup: a bad deleteMany replicates too.
• A crash is not a clean stop. Budget about ten seconds of write pause for a hard failure.

🍃 Availability through copies and votes

A replica set keeps copies in sync through the oplog and elects a new primary by majority vote. A crashed primary cost one write about ten seconds; a stopped one under half a second; a lost majority cost every write.

Reading AI-generated code

An assistant scales reads by sending them to secondaries and writes a confident comment:

🐛 Spot the bug

// AI comment: "the insert is acknowledged, so reading it back from a secondary is safe"
await orders.insertOne({ _id: "o-1", total: 42 });
const saved = await ordersOnSecondary.findOne({ _id: "o-1" });   // readPreference: "secondary"
if (!saved) throw new Error("order lost");

Show the answer

An acknowledged write is on the primary and a majority — not necessarily on the particular secondary you read. In the lab a frozen secondary returned null for a write that had been acknowledged, and the code above would report a lost order that is not lost. Read your own writes from the primary, or use a causally consistent session (causalConsistency: true). The run above shows the bug, the primary read and the causal read side by side.

Mental Model

A replica set is one authoritative log (the oplog) and several copies replaying it, with a vote to decide who owns the log. Writes go to the log owner; the copies are always slightly behind; the vote needs a strict majority of all voters, alive or not. Failover is not instantaneous because the others must first decide the owner is really gone — that decision is what the ten-second timeout buys.

Trade-offs

Choice	Benefit	Cost
Replica set (vs. a single mongod)	Automatic failover; measured ~10.8 s pause on a crash, 0.3 s on a clean stop	At least three nodes to run; writes still bottleneck on one primary
Reading from secondaries	Spreads read load off the primary	Proven above: an acknowledged write was invisible on a lagging secondary
Reading from the primary only	Always the latest write	The primary carries all the load
Even number of voters	Fewer machines than the next odd number	A single failure can leave no majority

Common Misconceptions

• “Adding secondaries scales writes.” — every write goes to the one primary; secondaries scale reads only. Scaling writes is sharding.
• “Failover takes a second or two.” — a crash paused one write for ~10.8 s (a clean stop 0.3 s); the manual says the median should not typically exceed 12 s.
• “An acknowledged write is visible everywhere.” — visible on the primary and a majority; a specific secondary may be behind (null above).
• “Two nodes are enough for HA.” — with two voters a single failure leaves one of two, and the survivor stays a secondary.
• “Replicas are backups.” — a dropped collection replicates too.
• “All members in one availability zone is fine.” — a replica set protects against a node failing, not against the zone failing; spread members across zones.

Python/Go/Rust tabs mirror the verified shell/Node behaviour; they are not executed by the course’s automated checker.

How do secondaries keep their data in sync with the primary?

They copy the primary's files on a fixed schedule They tail the primary's oplog and replay each operation They forward every read to the primary They re-import the whole collection each minute

You docker kill (SIGKILL) the primary of a 3-node set while a writer inserts with w: "majority". What did the real run show?

The writer failed immediately with an error One insert stalled about 10.8 s and then succeeded on the new primary Nothing: failover is instantaneous Writes were lost silently

Why is an odd number of voting members recommended?

It makes the oplog smaller A single failure still leaves a strict majority; with two voters one failure leaves no majority Drivers only connect to odd-numbered sets Even numbers are not allowed

An insert was acknowledged. You read it from a secondary and get null. Which explanation fits?

The write was lost That secondary has not applied the write yet (replication lag) Secondaries only hold half the documents Secondaries reject reads

What are the default heartbeat interval and election timeout?

2 s and 10 s 10 s and 2 s 30 s and 60 s 1 s and 1 s

Submit

Mark as complete