events → ranked list · where each guarantee is enforced

A measuring instrument for recommendation and search.

RankForge is an offline ML engineering framework for two-stage systems — retrieve candidates, then rank them. It is not a recommender product and has no serving layer. Its whole job is to answer one question reliably: does this model actually rank better than the simple thing?

Two properties carry the design. Nothing may see the future. And the simple thing is always on the leaderboard.

44 modules 335 tests 43 point-in-time features MIT licensed Python ≥ 3.10

00Install & run

Python 3.10 or newer. The core needs only numpy, pandas, scipy, scikit-learn and pydantic — the heavy pieces are optional extras, so CI can run the whole core suite without compiling a native dependency.

Clone and install

trees is LightGBM, neural is torch, ann is faiss-cpu. [all] takes everything.

git clone https://github.com/Gariyuuu/rankforge && cd rankforge
uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python -e ".[all]"

Run the quickstart

Events through split, candidates, negatives, features, ranker and evaluation — with the baselines on the leaderboard whether you asked for them or not.

rankforge run configs/quickstart.yaml
Replay it from the artifact

The written config is the resolved one, so a replay reproduces the run even if the library's defaults change later.
```
rankforge run artifacts/quickstart/config.yaml \
  --run-name replay
```

01The map

One pipeline, seven stages. The two shaded stages are where a leak would enter if nothing stopped it, so those are the two that carry hard runtime guarantees rather than conventions.

01Events

A validated interaction log: user, item, timestamp, label. Frame-level contracts, checked vectorised.

02 · gateSplit

Chronological and user-aware. One global boundary, or each user's own tail. Ties never straddle it.

03Candidates

Popularity, item-item CF, two-tower retrieval. Merged by reciprocal rank fusion, not raw score.

04Negatives

Random, popularity-aware and hard, in a seeded mixture. Every negative keeps its provenance.

05 · gateFeatures

Point-in-time only. Each row is answered from events strictly before its own cutoff instant.

06Ranker

Pointwise, pairwise, LambdaMART, listwise neural — one training table, one feature matrix.

07Evaluation

Recall, precision, NDCG, MRR, MAP, with bootstrap intervals and cold-start slices.

pipeline stage enforced boundary strict timestamp < cutoff — never <=

02What each stage ships

Every stage is usable on its own. The metrics, the point-in-time index and the negative samplers have no dependency on the pipeline that orchestrates them.

Components, by stage.
Stage	Ships	Notable
splitting	global-time · per-user holdout	boundaries snap to real timestamps, so no instant straddles two splits
retrieval	popularity · item-kNN · two-tower · random floor	FAISS exact / HNSW with an exact numpy fallback returning identical sets
sampling	random · popularity-aware · hard · mixture	seeded per query, so filtering the query set perturbs nothing
features	user · item · affinity · recency · content	43 features, all answered by one sorted-prefix index
ranking	pointwise · pairwise · LambdaMART · listwise neural	chosen so a comparison varies objective or capacity, never both
evaluation	Recall · Precision · NDCG · MRR · MAP · hit-rate	two implementations — a readable reference and a vectorised one — asserted equal

03The cutoff

An offline recommender that leaks does not crash. It reports a better number. So leakage is enforced as a runtime invariant — a leak fails the run, not just the test suite.

The split refuses to overlap.Every training timestamp is strictly below every evaluation timestamp. A quantile cut on row index can put two events at the same instant on opposite sides; the boundary is snapped to a real timestamp value so it cannot.
The store drops the future physically.Each split's feature store is built with an explicit maximum cutoff and discards every event at or after it. Data that is not in the store cannot leak out of it.
Every lookup is strict per row.Features are read through a point-in-time index with <, not <=. An event at the scoring instant is future information for a system that has to act at that instant.
The same rule covers everything else.Two-tower input history, item-kNN profiles, popularity counts, and even which items to exclude as already-seen — all point-in-time questions, all answered the same way.

Is the check sensitive?

A leakage suite that only ever passes on correct code proves nothing about its own sensitivity. So the suite was mutation-tested: ten single-token defects, injected one at a time.

All ten were caught.

The injected defects: < weakened to <= in the index · the construction filter removed · the cutoff bound check removed · the split boundary loosened · the temporal assertion disabled · test events admitted into their own fit set · popularity computed over the whole log · the seen-set built over the whole window · a two-tower positive admitted into its own history · relevance duplicates left uncollapsed.

04What actually won

One benchmark run, seed 17, on a synthetic log of 193,188 events across 2,903 users and 2,000 items; 2,817 test queries; top-200 candidates; 43 features. Confidence intervals are 95% percentile bootstrap over queries.

Ranking stage, test split — NDCG@10.
Model	NDCG@10	95% CI
neural (ListNet)	0.1528	0.1465 – 0.1592
pointwise logistic	0.1506	0.1446 – 0.1570
pairwise RankNet	0.1485	0.1424 – 0.1551
LambdaMART	0.1435	0.1373 – 0.1495
pointwise GBDT	0.1303	0.1248 – 0.1363
retrieval order — no reranking	0.1195	0.1138 – 0.1245

The reading

Reranking is worth +27.9% NDCG@10 over serving the retrieval order unchanged. No ranker separated from plain logistic regression.

The neural ranker's +1.5% lead sits inside its own confidence interval, so this run does not resolve it. Calling that "the neural ranker won" would be fabricating a result. The stage pays for itself; the choice of ranker does not, on this data.

Two results cut against expectation and are reported as they came out. LambdaMART lost to logistic regression despite being the NDCG-optimising listwise model. Pointwise GBDT lost badly on identical features and an identical objective — so this is capacity hurting, not the objective.

Retrieval stage, test split — the candidate pool before any reranking.
Retriever	NDCG@10	95% CI	Recall@50
item-kNN	0.1362	0.1297 – 0.1423	0.3252
popularity	0.1346	0.1278 – 0.1406	0.3241
two-tower	0.1230	0.1170 – 0.1288	0.3195
merged pool (RRF)	0.1195	0.1138 – 0.1245	0.3207
random floor	0.0047	0.0036 – 0.0057	0.0258

No retriever separated from popularity either — and the two-tower was worse than it on both splits. Everything sits far above the random floor, which is the check that the task is learnable at all: a leaderboard where popularity wins only means something next to one where random loses badly.

05What the runtime measurements found

Single-threaded, 64-dimensional vectors, 1,000 queries, k=100. Index construction is timed separately from search, because they have different lifetimes — a build happens once per model refresh, a query happens per request.

Nearest-neighbour search: exact numpy against exact FAISS.
Catalogue	numpy query	FAISS query	Speedup	Result overlap
1,000	0.015 s	0.024 s	0.64×	1.0000
10,000	0.106 s	0.082 s	1.28×	1.0000
100,000	0.999 s	0.747 s	1.34×	1.0000
500,000	4.930 s	2.775 s	1.78×	1.0000

Approximate search (HNSW) at 100,000 items — build cost 12.03 s, once.
efSearch	Recall	Query speedup	Queries to repay the build
200	0.812	2.09×	~23,000
400	0.938	1.03×	~414,000
800	0.989	0.44×	never

Why the default is exact

Measuring the build cost is what makes that table decisive rather than suggestive. At the lowest recall most people would accept, the approximate index must serve roughly 414,000 queries before it breaks even against exact search. Offline evaluation runs a few thousand.

So automatic index selection never chooses the approximate path. Silently trading retrieval recall for latency, inside a framework whose job is measuring retrieval recall, would be the wrong default.

A bug this benchmark caught: with the original settings, HNSW ran with an exploration budget smaller than the number of neighbours requested and returned 0.50 recall. An index asked for 100 neighbours while exploring 64 candidates cannot answer — and it fails silently rather than erroring. The budget is now floored at k and warns once.

06A number without a manifest is an anecdote

Every run writes a manifest recording everything that can change a result, and two fingerprints: one over the configuration, one over everything that determines the numbers.

Recorded per run.
Field	Captures
dataset	content hash of the whole event log, plus loader provenance and preprocessing thresholds
split	strategy, boundaries, per-split counts, and the full leakage audit
seed	the root seed; every component derives its own from it by label, never sequentially
features	the exact ordered feature-name list
generators & rankers	all hyperparameters, plus fitted artefacts such as the selected boosting iteration
environment	platform, library versions, git commit, whether the tree was dirty, thread-pinning state

Replay

rankforge run configs/quickstart.yaml
rankforge run artifacts/quickstart/config.yaml \
  --run-name replay

The written config is the resolved one, with every default expanded, so a replay reproduces the run even if the library's defaults change later. The result fingerprints differ if and only if something that can change a number changed — the run name and output directory are excluded on purpose.

Seeds are derived by hashing a root seed with a stable label, so adding a component cannot shift an existing one's random stream, and a stage re-run alone reproduces exactly what the whole pipeline produced.

07Status, and what it has not done

The framework is complete and frozen at v0.1.0. Ruff, mypy and a 335-test suite gate every change; the core suite passes with no native ML wheels installed at all, so the metrics, splitting, leakage and sampling layers stay usable without compiling anything.

It is a library and a CLI. There is no serving layer, no dashboard, and no online component — those are deliberately out of scope.

pip install -e ".[all]"

rankforge validate  configs/quickstart.yaml
rankforge audit     configs/quickstart.yaml
rankforge run       configs/quickstart.yaml

audit emits the leakage report as JSON on stdout and nothing else, so it composes. run writes a Markdown report, a JSON report, the manifest, the resolved config, the candidate tables and the fitted models.

Limitations, stated plainly

No real-dataset validation.Every number here is from the synthetic generator. The MovieLens loader is implemented and licence-correct, but the host's TLS certificate is expired, so the archive cannot be fetched over a verified channel. The loader fails with instructions rather than disabling certificate verification, and its opt-in override requires a checksum obtained independently — the publisher's own sidecar travels over the same broken channel, so it is not a check.
One seed, one machine.The ranking results are a single seed with no multi-seed variance study, and all timings are single-threaded on one laptop CPU. Nothing here says how these models rank on real interaction data.
A thread pin is in effect on macOS.Torch, LightGBM and FAISS each bundle their own OpenMP runtime, and every load order fails — one segfaults, the other hangs. Pinning thread counts before import is the only fix that holds, and it is recorded in the manifest because it changes timings.