Pipeline basics - $match and $project

Two stages carry most aggregation work before anything fancy happens: $match throws away documents you do not need, and $project reshapes the ones you keep. The subtle lesson hiding in them is that where you put a stage changes how fast and how correct your pipeline is - and, more surprisingly, that the server already fixes some orderings for you.

The lesson seeds the module’s sales collection (200 000 orders { title, genre, quantity, price, buyer }, index on price) and the aplan() helper described in the module intro. Captured on mongo 8.3.11, mongosh 2.11.1 and Node driver 7.7.0 on one Mac; Python, Go and Rust tabs were each run once in a scratch program.

mongosh/pb-seed.js

const genres = ["fiction", "science", "history", "poetry", "comics"];
globalThis.aplan = (pipeline) => {               // one line per aggregate explain: what the first stage read, then the stages left in the pipeline
  const e = db.sales.explain("executionStats").aggregate(pipeline);
  const cursor = e.stages ? e.stages[0].$cursor : e, rest = e.stages ? e.stages.slice(1).map((s) => Object.keys(s)[0]) : [];
  const chain = (p) => { const out = []; for (p = p.queryPlan ?? p; p; p = p.inputStage) out.push(p.stage + (p.indexName ? "(" + p.indexName + ")" : "")); return out.join(" > "); };
  const s = cursor.executionStats;
  return chain(cursor.queryPlanner.winningPlan) + "  keys=" + s.totalKeysExamined + " docs=" + s.totalDocsExamined + (rest.length ? "  | left in pipeline: " + rest.join(" > ") : "");
};
for (let s = 0; s < 200000; s += 50000) {
  db.sales.insertMany(Array.from({ length: 50000 }, (_, k) => {
    const i = s + k;                                          // 500 titles, 5 genres, quantity 1..5, price 5..25, 1000 buyers (pseudo-random but repeatable)
    return { title: "T" + (i % 500), genre: genres[(i % 500) % 5], quantity: 1 + ((Math.imul(i, 2654435761) >>> 0) % 5), price: 5 + ((Math.imul(i, 40503) >>> 0) % 21), buyer: "b" + (i % 1000) };
  }), { ordered: false });
}
db.sales.createIndex({ price: 1 });
console.log("sales:", db.sales.countDocuments(), "orders");

sales: 200000 orders

$match filters the stream - put it first

$match keeps only documents that satisfy its condition, using the same syntax as find ($gt, $in, $and…). When it is the first stage the engine plans it like a find, so an index can skip non-matching documents instead of streaming the whole collection through later stages:

flowchart LR
  In["all sales"] --> M["$match: price >= 24 (uses index)"]
  M --> P["$project: keep title, compute revenue"]
  P --> Out["slim, reshaped documents"]

Filter first so an index can prune the stream, then reshape what survives

mongosh/pb-match.js

const rows = db.sales.aggregate([{ $match: { price: { $gte: 24 } } }, { $limit: 2 }]).toArray();
console.log(rows);
console.log(aplan([{ $match: { price: { $gte: 24 } } }]));
assert.ok(rows.every((d) => d.price >= 24));

[
  {
    _id: ObjectId('6abcb8d411b1b6194d7740a3'),
    title: 'T46',
    genre: 'science',
    quantity: 2,
    price: 25,
    buyer: 'b46'
  },
  {
    _id: ObjectId('6abcb8d411b1b6194d7740aa'),
    title: 'T53',
    genre: 'poetry',
    quantity: 5,
    price: 25,
    buyer: 'b53'
  }
]
FETCH > IXSCAN(price_1)  keys=13422 docs=13422

13 422 of 200 000 orders match, and the plan read exactly 13 422 index keys and 13 422 documents.

$project reshapes and computes

$project decides which fields leave the stage and builds new fields from expressions. 1 keeps a field, 0 drops it, an expression computes it. Field references inside expressions use a $ prefix: "$price" is the value of the field, "price" is a literal string. Here we keep the title and compute revenue = price x quantity; the Node.js tab runs it on a three-order collection:

mongosh

mongosh/pb-project.js

const rows = db.sales.aggregate([
  { $match: { price: { $gte: 24 } } },
  { $project: { _id: 0, title: 1, revenue: { $multiply: ["$price", "$quantity"] } } },
  { $limit: 3 },
]).toArray();
console.log(rows);
assert.ok(rows.every((d) => d.revenue >= 24 && !("_id" in d)));

Node.js

app/pb-project.mjs

const sales = db.collection("sales");
await sales.insertMany([
  { title: "Dune", genre: "fiction", quantity: 3, price: 12, buyer: "Ada" },
  { title: "Foundation", genre: "fiction", quantity: 1, price: 15, buyer: "Linus" },
  { title: "Cosmos", genre: "science", quantity: 2, price: 6, buyer: "Ada" },
]);
const rows = await sales.aggregate([
  { $match: { price: { $gte: 10 } } },
  { $project: { _id: 0, title: 1, revenue: { $multiply: ["$price", "$quantity"] } } },
]).toArray();
console.log(rows);
if (rows.length !== 2 || rows[0].revenue !== 36) throw new Error("unexpected projection");

Python

rows = list(db.sales.aggregate([
    {"$match": {"price": {"$gte": 10}}},
    {"$project": {"_id": 0, "title": 1, "revenue": {"$multiply": ["$price", "$quantity"]}}},
]))

Go

pipeline := mongo.Pipeline{
    {{Key: "$match", Value: bson.D{{Key: "price", Value: bson.D{{Key: "$gte", Value: 10}}}}}},
    {{Key: "$project", Value: bson.D{
        {Key: "_id", Value: 0},
        {Key: "title", Value: 1},
        {Key: "revenue", Value: bson.D{{Key: "$multiply", Value: bson.A{"$price", "$quantity"}}}},
    }}},
}
cur, err := coll.Aggregate(ctx, pipeline)
if err != nil {
    return err
}
var rows []bson.M
if err := cur.All(ctx, &rows); err != nil {
    return err
}

Rust

let mut cursor = coll
    .aggregate(vec![
        doc! { "$match": { "price": { "$gte": 10 } } },
        doc! { "$project": {
            "_id": 0,
            "title": 1,
            "revenue": { "$multiply": ["$price", "$quantity"] },
        } },
    ])
    .await?;

[
  { title: 'T46', revenue: 50 },
  { title: 'T53', revenue: 125 },
  { title: 'T60', revenue: 75 }
]

[
  { title: 'Dune', revenue: 36 },
  { title: 'Foundation', revenue: 15 }
]

$set, $unset and the inclusion/exclusion rule

$project with a field list is an inclusion projection: everything not listed disappears. $set (alias $addFields) adds or overwrites fields and keeps the rest; $unset removes named fields. A single $project cannot mix inclusion and exclusion except for _id:

mongosh/pb-set.js

const one = (extra) => db.sales.aggregate([{ $match: { title: "T7" } }, { $limit: 1 }, ...extra]).toArray()[0];
console.log("$project keeps only what you list :", Object.keys(one([{ $project: { title: 1, revenue: { $multiply: ["$price", "$quantity"] } } }])).join(", "));
console.log("$set adds, keeps everything else  :", Object.keys(one([{ $set: { revenue: { $multiply: ["$price", "$quantity"] } } }])).join(", "));
console.log("$unset removes named fields       :", Object.keys(one([{ $unset: ["buyer", "genre"] }])).join(", "));
try { db.sales.aggregate([{ $project: { title: 1, buyer: 0 } }]).toArray(); }
catch (e) { console.log("mixing 1 and 0:", e.codeName + " -", e.message); }

$project keeps only what you list : _id, title, revenue
$set adds, keeps everything else  : _id, title, genre, quantity, price, buyer, revenue
$unset removes named fields       : _id, title, quantity, price
mixing 1 and 0: Location31254 - Invalid $project :: caused by :: Cannot do exclusion on field buyer in inclusion projection

In Compass: add a $match stage, then a $project stage in the Aggregations tab. The live preview under each stage shows the count shrinking after $match and the fields changing after $project.

Why stage order matters - and what the optimizer fixes

Older material says a $match after $project or $group can never use an index. That is only half true. MongoDB’s optimizer rewrites the pipeline first: for a projection stage followed by $match, it “moves any filters in the $match stage that do not require values computed in the projection stage to a new $match stage before the projection” (pipeline optimization, consulted 2026-09-30). Five orderings of the same kind of question, with the plan each actually got:

mongosh/pb-order.js

const rev = { $multiply: ["$price", "$quantity"] };
console.log("A $match first                 :", aplan([{ $match: { price: { $gte: 24 } } }, { $project: { revenue: rev } }]));
console.log("B $match AFTER $project        :", aplan([{ $project: { revenue: rev, price: 1 } }, { $match: { price: { $gte: 24 } } }]));
console.log("C $match on a computed field   :", aplan([{ $project: { revenue: rev } }, { $match: { revenue: { $gte: 100 } } }]));
console.log("D $match after $group (on _id) :", aplan([{ $group: { _id: "$price", n: { $sum: 1 } } }, { $match: { _id: { $gte: 24 } } }]));
console.log("E $match on a $group total     :", aplan([{ $group: { _id: "$genre", n: { $sum: 1 } } }, { $match: { n: { $gt: 5 } } }]));
const e = db.sales.explain().aggregate([{ $project: { revenue: rev, price: 1 } }, { $match: { price: { $gte: 24 } } }]);
console.log("B optimizedPipeline:", e.queryPlanner.optimizedPipeline, "| parsedQuery:", JSON.stringify(e.queryPlanner.parsedQuery));

A $match first                 : PROJECTION_DEFAULT > FETCH > IXSCAN(price_1)  keys=13422 docs=13422
B $match AFTER $project        : PROJECTION_DEFAULT > FETCH > IXSCAN(price_1)  keys=13422 docs=13422
C $match on a computed field   : PROJECTION_DEFAULT > COLLSCAN  keys=0 docs=200000  | left in pipeline: $match
D $match after $group (on _id) : GROUP > PROJECTION_COVERED > IXSCAN(price_1)  keys=13422 docs=0
E $match on a $group total     : GROUP > COLLSCAN  keys=0 docs=200000  | left in pipeline: $match
B optimizedPipeline: true | parsedQuery: {"price":{"$gte":24}}

• A, B - same plan: the filter on the stored price was moved in front of the $project (optimizedPipeline: true, parsedQuery is the price filter).
• C - the filter needs revenue, a value that only exists after $project, so it stays behind: COLLSCAN over 200 000 documents and $match left in the pipeline.
• D - a $match on the $group key (_id) is moved before the $group too, and the plan is even covered (docs=0).
• E - a filter on an accumulator (n) can only run after the group exists: COLLSCAN and $match left behind.

So the real rule is not “always write $match first” (though it is the clearest style) but “a filter is only stuck late when it depends on a computed field or an aggregate value”.

Memory: 100 MB per stage, disk by default

Blocking stages ($sort, $group) keep their working set in memory up to 100 MB. Since MongoDB 6.0 allowDiskUseByDefault is true, so a stage that exceeds the limit spills to disk instead of failing; pass { allowDiskUse: false } to get the old behaviour. A 130 MB collection makes the difference visible:

mongosh/pb-memory.js

console.log(db.adminCommand({ getParameter: 1, allowDiskUseByDefault: 1, internalQueryMaxBlockingSortMemoryUsageBytes: 1, internalDocumentSourceGroupMaxMemoryBytes: 1 }));
for (let s = 0; s < 130000; s += 10000) db.big.insertMany(Array.from({ length: 10000 }, (_, k) => ({ n: s + k, r: Math.random(), pad: "x".repeat(1000) })), { ordered: false });
console.log("big collection MB:", +(db.big.stats().size / 1048576).toFixed(0), "(over the 100 MB per-stage limit)");
let t = Date.now();
console.log("default            ->", db.big.aggregate([{ $sort: { r: 1 } }, { $project: { n: 1 } }]).toArray().length, "rows in", Date.now() - t, "ms");
try { db.big.aggregate([{ $sort: { r: 1 } }, { $project: { n: 1 } }], { allowDiskUse: false }).toArray(); }
catch (e) { console.log("allowDiskUse:false ->", e.codeName + ":", e.message.slice(0, 130)); }

{
  allowDiskUseByDefault: true,
  internalDocumentSourceGroupMaxMemoryBytes: Long('104857600'),
  internalQueryMaxBlockingSortMemoryUsageBytes: Long('104857600'),
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1790752983, i: 53993 }),
    signature: {
      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
      keyId: Long('0')
    }
  },
  operationTime: Timestamp({ t: 1790752983, i: 53993 })
}
big collection MB: 130 (over the 100 MB per-stage limit)
default            -> 130000 rows in 444 ms
allowDiskUse:false -> QueryExceededMemoryLimitNoDiskUseAllowed: Executor error during aggregate command on namespace: b2_bbc59d.big :: caused by :: Sort exceeded memory limit of 104857600 bytes,

Tips and gotchas

• An early $match on stored fields can use an index; the optimizer will hoist it past $project/$set and past a $group on the key, but never past a computed field or an accumulator.
• Mixing inclusion (1) and exclusion (0) in one $project is an error, except for _id.
• "$price" is a field value, "price" a literal string.
• Use explain (aplan above) to see what was hoisted: left in pipeline lists what the first stage could not absorb.

🍃 Filter early, reshape late

The first instinct for any pipeline: can I $match sooner? Often the optimizer does it for you, but a filter on a computed value or a group total cannot move - store and index the value, or pre-filter on an indexed field.

Reading AI-generated code

Asked for “orders worth at least 100 with an index-friendly pipeline”, an assistant writes:

🐛 Spot the bug

// AI comment: "price is indexed, and $match is the only filter - the index handles it"
db.sales.aggregate([
{ $project: { title: 1, revenue: { $multiply: ["$price", "$quantity"] } } },
{ $match: { revenue: { $gte: 100 } } },
]);

Show the answer

revenue only exists after $project, so the optimizer cannot hoist the filter: plan C above scans all 200 000 documents, and no index on price can answer a test on price x quantity. Two fixes, both measured below: add a coarse pre-filter the index can serve (revenue >= 100 implies price >= 20 because quantity <= 5, cutting the read to 57 143 documents), or store revenue in the document, index it and match on the stored field (14 111 keys, exactly the answer):

mongosh/pb-bug.js

const rev = { $multiply: ["$price", "$quantity"] };
const buggy = [{ $project: { title: 1, revenue: rev } }, { $match: { revenue: { $gte: 100 } } }];            // filter on a computed field
const coarse = [{ $match: { price: { $gte: 20 } } }, ...buggy];                                                 // revenue >= 100 implies price >= 20 (quantity <= 5)
console.log("buggy   (filter on a computed field):", aplan(buggy));
console.log("coarse pre-filter on the index      :", aplan(coarse));
db.sales.updateMany({}, [{ $set: { revenue: rev } }]);                                                            // store the value once ...
db.sales.createIndex({ revenue: 1 });                                                                             // ... and index it
const fixed = [{ $match: { revenue: { $gte: 100 } } }, { $project: { title: 1, revenue: 1 } }];
console.log("stored + indexed revenue             :", aplan(fixed));
const n = (p) => db.sales.aggregate(p).toArray().length;
console.log("rows returned by all three:", n(buggy), n(coarse), n(fixed));
assert.ok(n(buggy) === n(coarse) && n(coarse) === n(fixed));

buggy   (filter on a computed field): PROJECTION_DEFAULT > COLLSCAN  keys=0 docs=200000  | left in pipeline: $match
coarse pre-filter on the index      : PROJECTION_DEFAULT > FETCH > IXSCAN(price_1)  keys=57143 docs=57143  | left in pipeline: $match
stored + indexed revenue             : PROJECTION_SIMPLE > FETCH > IXSCAN(revenue_1)  keys=14111 docs=14111
rows returned by all three: 14111 14111 14111

Mental Model

Think of the optimizer as a helpful colleague who moves your filter earlier whenever it can prove the filter does not need anything produced in between. You control the rest: filters that depend on computed values or group totals must stay where the value exists, so either accept the scan, pre-filter on something stored, or store the value.

Trade-offs

Choice	Benefit	Cost
$match first on stored fields	Uses an index: 13 422 keys, not 200 000 documents	Conditions must target stored fields
$match on a computed field	Expresses the real question directly	COLLSCAN of 200 000 documents
Store + index the computed value	14 111 keys; exact	Write-time upkeep, extra index
$project with explicit fields	Precise output shape	New fields must be added everywhere

Common Misconceptions

• “A $match after $project cannot use an index.” - plan B used price_1 (13 422 keys).
• “A $match after $group is always late.” - a filter on the group key was hoisted (plan D, covered).
• “So stage order never matters.” - filters on revenue or n stayed late and scanned 200 000 documents.
• “A big $sort fails at 100 MB.” - it spilled to disk and finished (130 000 rows in 449 ms) unless allowDiskUse: false.

Real-world example

The MongoDB manual documents the optimizer behaviour measured here as a named rule (“$match … moves any filters … that do not require values computed in the projection stage”) together with the reverse case, where computed values pin a filter in place (pipeline optimization, consulted 2026-09-30). Keller Williams uses this style of pipeline over its listing data on Atlas (mongodb.com/customers/keller-williams).

Python/Go/Rust tabs mirror the verified shell/Node behaviour; each was run once in a scratch program against the same server (the match-project pipeline returned the same revenue rows) but they are not executed by the course’s automated checker.

Why is it best to place $match as the first stage of a pipeline?

It is the only stage allowed to run first So the engine can use an index and read fewer documents before later work Because $match cannot appear anywhere else It automatically sorts the results

In a $project stage, what does "$price" refer to?

The literal text price The value stored in the price field of each document A new field named price set to zero The price of the whole collection

Which $project pattern is NOT allowed in a single stage?

Including several fields with 1 Excluding several fields with 0 Mixing 1 and 0 for non-_id fields Computing a field with an expression

A pipeline is [ $project (adds revenue, keeps price), then $match { price >= 24 } ]. Does the $match still use the price index?

No, a $match after $project can never use an index Yes: the optimizer moves the part of the $match that does not need computed values in front of the $project Only with allowDiskUse Only on a replica set

How do you fix a slow $match on a computed value like price x quantity?

Use $sort first Store the computed value in the document, index it, and $match on the stored field (14 111 keys instead of 200 000 documents) Move $match after $group It cannot be sped up

A 130 MB $sort runs with default settings on MongoDB 8. What happens?

It fails at 100 MB It spills to disk and succeeds; it fails only with allowDiskUse: false It is silently truncated It waits for more RAM

Submit

Mark as complete