Raw results, binaries and harness behind STP's benchmark pages.
This site exists to be fetched rather than read: the pages in STP's manual load their figures from the JSON below. What is here that the pages do not show is the per-run data, the retained solver output, and the exact binary each campaign was run with.
| Path | What |
|---|---|
data/campaigns.json |
One entry per campaign: provenance and headline figures. Start here — it names the rest. |
data/summary/<campaign>.json |
Per-logic and per-family aggregates. |
data/failures/<campaign>.json |
Every instance the campaign did not solve, and why. |
data/detail/<campaign>.json.gz |
Per-file class, wall time and peak RSS. |
data/runs/<campaign>.jsonl.gz |
Every column of every run — the archival form. |
data/corpus.jsonl.gz |
The benchmark index the runs are keyed to: sha256, size, query count, declared status. |
outputs/<campaign>.jsonl.gz |
Each run's stdout and stderr. |
binaries/<sha256>.json |
Provenance of the binary that produced a campaign — STP commit, compiler, linked SAT solvers — and the download_url to fetch it from. |
The binaries themselves are
releases, one per
distinct binary, tagged binary-<first 12 of its sha256>.
Check what you download against that hash before trusting a measurement made
with it.
The .jsonl.gz files are a header line naming the columns, then
one compact JSON array per row.
github.com/stp/benchmarks-data
holds all of this plus scripts/, the harness that runs a campaign
and publishes it. Its README covers how a campaign is reproduced, and what is
excluded from publication and why.