SPARQLoscope DBLP Benchmark Results

We ran the SPARQLoscope DBLP evaluation — 105 SPARQL 1.1 queries against the DBLP computer science bibliography — with all seven engines on the same machine, on identical data and hardware. Fluree (v4.0.6) leads every aggregate metric and is one of only two engines to answer all 105 queries.

Unlike our earlier write-up, which placed Fluree alongside the University of Freiburg's published numbers for other engines, this run is purely engine-vs-engine: every engine was installed natively (no Docker) on the same AWS m7a.4xlarge (16 cores / 64 GB) and measured against the same DBLP-core dataset, so the comparison reflects the engines, not the hardware.

Everything here is reproducible. Datasets, queries, per-engine setup notes, raw per-query results, and the report generators all live in github.com/fluree/benchmark-db.

The numbers below: full DBLP-core report · per-engine raw TSVs

---

Speed Rankings

The full SPARQLoscope suite over DBLP-core (~561.5M triples), all seven engines on the same box. Times in milliseconds (lower is faster).

Geometric Mean (P=2)

This is the SPARQLoscope paper's official aggregate: a failed or timed-out query counts as 2× the 180s timeout, so every engine is scored on the same 105 queries.

Rank	Engine	Geo Mean	vs Fluree	Completed
1	Fluree	19.4 ms	—	105/105
2	QLever	202 ms	10.4× slower	105/105
3	Virtuoso	300 ms	15.4× slower	103/105
4	MillenniumDB	1,664 ms	86× slower	103/105
5	Jena	67.7 s	3,487× slower	34/105
6	Oxigraph	87.0 s	4,486× slower	39/105
7	Blazegraph	333 s	17,158× slower	3/105

Fluree is 10.4× faster than the next-fastest engine (QLever) — the second-place finisher in this benchmark.

Arithmetic Mean (completed queries)

The arithmetic mean reflects total workload time. (Computed over each engine's completed queries, so it flatters engines that failed many.)

Rank	Engine	Arith Mean	vs Fluree
1	Fluree	251 ms	—
2	QLever	1,904 ms	7.6× slower
3	Virtuoso	8.0 s	32× slower
4	MillenniumDB	12.3 s	49× slower
5	Blazegraph	23.0 s	91.6× slower
6	Jena	31.0 s	124× slower
7	Oxigraph	36.8 s	147× slower

Fluree is the only engine with a sub-second arithmetic mean.

Median (completed queries)

Rank	Engine	Median
1	Fluree	41 ms
2	QLever	310 ms
3	Virtuoso	326 ms
4	MillenniumDB	3,894 ms
5	Oxigraph	5,090 ms
6	Jena	6,033 ms
7	Blazegraph	23,041 ms

Query Completion

Fluree and QLever are the only two engines to complete all 105 queries within the 180s timeout. Virtuoso and MillenniumDB each timed out on 2; Jena completed 34, Oxigraph 39, and Blazegraph just 3.

---

Results by Category

Fluree leads 12 of the 13 SPARQLoscope categories — every category except result-size/export, where MillenniumDB's result streaming edges ahead. The per-query tables below report median-of-3 times in milliseconds; — means the engine timed out or errored on that query.

Joins

Query	Fluree	QLever	Virtuoso	MillenniumDB	Jena
3-chain (largest)	228	2,587	7,458	27,614	—
3-star (largest)	130	2,282	233	24,863	—
2-large-large (large result)	25	414	853	9,036	93,497
multicolumn-join-large	1,121	4,891	569	16,526	69,778

Aggregations (GROUP BY)

Query	Fluree	QLever	Virtuoso	MillenniumDB
complex aggregate	375	1,962	39,342	18,675
GROUP_CONCAT	49	26,789	161,898	58,578
distinct-count, low multiplicity	1	12,907	36,078	78,634
implicit numeric AVG	59	85	193	255

Graph Patterns (OPTIONAL, MINUS, EXISTS, UNION)

Query	Fluree	QLever	Virtuoso	MillenniumDB
OPTIONAL 3-chain-1	228	2,278	2,878	27,708
MINUS 3-chain-1	221	1,407	2,880	20,481
EXISTS 3-chain-1	225	3,693	11,977	21,380
UNION large-join	208	764	1,804	16,409

String Operations and Regex

Query	Fluree	QLever	Virtuoso	MillenniumDB
regex-3	482	7,381	9,067	21,136
strafter	129	10,927	1,703	34,461
strstarts	42	6,700	1,553	2,925
regex-prefix-1	155	7	1,794	19,565

Fluree now leads the string/regex category overall (98 ms geo mean vs QLever's 957 ms). QLever still wins the regex-prefix family, where it can use a specialized prefix index.

Transitive Paths

Query	Fluree	QLever	Virtuoso	MillenniumDB
large path+join	1	682	137	2,897
plus	1	1	—	1
plus, fixed subject	1	1	1	1

Date and Numeric Operations

Query	Fluree	QLever	Virtuoso	Blazegraph
date-day	2	235	96	22,803
numeric baseline	43	85	125	—
numeric filter (50/50 bin)	1	31	14	—

Dataset Statistics (metadata)

Query	Fluree	QLever	Virtuoso	MillenniumDB
number-of-triples	1	1	5,004	13,363
number-of-literals	1	3,484	5,915	13,689
number-of-subjects	1	1	177,869	—
number-of-blank-nodes	41	5,164	1,383	14,445

Fluree v4.0.6 answers all six metadata queries in single-digit milliseconds via metadata-driven fast paths. Several engines time out entirely on the full distinct-count scans.

---

Beyond DBLP — billion-triple scale

The DBLP-core run is the head-to-head reference, but the same suite scales far past it:

• Wikidata-truthy (8.19B triples), 5 engines: Fluree posts a 363 ms penalized geo mean — 10.5× faster than QLever — and is the only engine to answer all 105 queries at this scale.
• WGPB (21.5B-triple full Wikidata all-dump), Fluree: the separate 850-query Wikidata Graph Pattern Benchmark — 850/850 passed, 43 ms geo mean, 0 timeouts on a 794 GB index.

And Fluree barely slows down on smaller hardware: re-running the DBLP-core suite from the full 16c/64GB box down to a quarter-sized box (4c/16GB), the geo mean moves only 19 → 20 → 25 ms and the median 41 → 44 → 49 ms, with all 105 queries still passing. The quarter-box result is still 8.1× faster than QLever's full-box geo mean.

Reports: Wikidata-truthy · WGPB · resource-scaling.

---

About this benchmark

What SPARQLoscope is

SPARQLoscope is a SPARQL evaluation framework developed at the University of Freiburg (the team behind QLever), presented at ISWC 2025. The suite runs 105 queries covering joins, aggregations, filters, graph patterns (OPTIONAL, MINUS, EXISTS, UNION), string operations, regex, transitive paths, date/numeric operations, and dataset-statistics queries — against the DBLP computer science bibliography, real-world data, not synthetic.

What we ran

All seven engines, natively (no Docker), on dedicated AWS m7a.4xlarge boxes (16 cores / 64 GB), against the DBLP-core archive of 2026-06-01 (574.2M raw N-Triples lines, ~561.5M distinct triples after dedup, 90 predicates). Each engine's result cache was disabled or cleared so every timed run re-executes — stricter than the paper's warm-cache protocol.

Engine versions: Fluree v4.0.6, QLever (git 621cf31, native), Oxigraph 0.5.8, Virtuoso 7.2.5.1, MillenniumDB v1.0.0, Apache Jena / Fuseki 6.1.0, Blazegraph 2.1.6-RC. Per-engine install, load, and serve notes are in common/engine-setup/.

Metrics

• Geometric mean (P=2): timed-out queries (180s) penalized at 360s, geometric mean across all 105 queries. The paper's headline metric.
• Arithmetic mean / median: computed over completed queries; sensitive to slow outliers, and they flatter engines with many failures.
• Completed: queries that returned within the 180s timeout.

---

Caveats

• This is a single-user, read-only query benchmark. It does not test write throughput, concurrent load, or mixed workloads.
• Absolute times are specific to this box (m7a.4xlarge, 16c/64GB) and not bit-comparable to the published SPARQLoscope table, which used different DBLP dumps (2024/2025).
• Jena's completion count is cache-state sensitive: Fuseki was started cold, and a cold 54 GB TDB2 index on a 64 GB box times out on many heavy queries (34/105). In an earlier pass with a warm OS page cache, Jena completed 69/105.
• Oxigraph cannot cancel a running query, so its sweep used a documented deviation (1 run, memory-capped, restart-on-failure); its times are not directly comparable to the warmup+median-of-3 engines.
• This run compares query completion and latency, not result-set equivalence across engines.
• Several products are not in this run (Neptune, Stardog, MarkLogic, AllegroGraph, RDFox, GraphDB). We don't make speed claims about engines we haven't measured.
• No benchmark replaces testing on your own data and queries.

---

Beyond query speed

This benchmark tests SPARQL query execution only. Fluree also provides capabilities that the benchmark does not measure and that most engines in the comparison do not have — immutable append-only storage with time-travel queries, git-like branching and merging, content-addressed commits with W3C Verifiable Credentials, built-in HNSW vector search, BM25 full-text search, triple-level access control, AES-256-GCM encryption at rest, Apache Iceberg integration, and GeoSPARQL with S2 spatial indexing.

These are documented separately. See the Fluree developer documentation for details.

---

Replicate this yourself

Everything is in github.com/fluree/benchmark-db. Datasets are pinned and published to s3://fluree-benchmark-data/ so you don't have to re-derive them.

# 1. Install Fluree (official v4.0.6 release — native binary, no Docker)
curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/fluree/db/releases/latest/download/fluree-db-cli-installer.sh | sh

# 2. Load a dataset, start the server, then run the suite
common/run_benchmark.sh \
  --endpoint http://localhost:8090/v1/fluree/query/dblp:main \
  -r 3 -w 1 -t 180 \
  -o benchmarks/sparqloscope/reports/dblp-core/engines/fluree.tsv

# 3. (Re)generate the report and headline charts
python3 common/generate_report.py benchmarks/sparqloscope/reports/dblp-core/
python3 common/make_charts.py

Native setup notes for every engine are under common/engine-setup/, and the query runner (common/run_benchmark.sh) does warmup + median-of-N with a per-query timeout. On equivalent hardware, expect ±10–20% variance on individual queries and ±5% on aggregate metrics.