ParetoPilot | Arm64 inference results

00 · Measured configurations

Four configurations, compared step by step.

Each row shows one tested configuration, what changed, and the largest measured differences from the previous row.

Evidence statusVerified Arm64 measurements

Decision tolerance

Tolerance cutoff: ≤ 2,254.3 msp95 end-to-end latency
Selected configuration: Q8 generic reference22.3 ms inside the current cutoff
Closest option outside the cutoff: Q4 with KleidiAI45.2 ms outside the current cutoff

01
Reference
Q8 generic reference
q8-generic
Reference configuration
- Quantization Q8_0
- KleidiAI off
Selected
p95 end-to-end latency2,231.9 ms
Baseline for the following comparisons.
02
Quantization
Q4 generic quantization
q4-generic
Configuration change
- Quantization Q8_0 → Q4_0
Eligible Pareto-frontier option
p95 end-to-end latency2,311.1 ms
- p95 end-to-end latency3.55% higherMeasured tradeoff
- model size43.72% lowerMeasured improvement
- peak resident memory42.80% lowerMeasured improvement
03
KleidiAI build
Q4 with KleidiAI
q4-kleidiai
Configuration change
- KleidiAI off → on
Closest option outside cutoff
p95 end-to-end latency2,299.5 ms
- p95 end-to-end latency0.50% lowerMeasured improvement
- p95 time to first token2.63% lowerMeasured improvement
- generation throughput1.04% higherMeasured improvement
04
Runtime tuning
Q4 with KleidiAI and 512-token micro-batch
q4-kleidiai-tuned
Configuration change
- Micro-batch 128 → 512
Eligible Pareto-frontier option
p95 end-to-end latency2,307.7 ms
- p95 end-to-end latency0.36% higherMeasured tradeoff
- prompt processing throughput14.86% higherMeasured improvement
- generation throughput0.79% lowerMeasured tradeoff

Scope: These comparisons use the supplied measurements and do not change the selected configuration. Results apply only to this runner, model, and workload.

Capacity study

Selected capacity point: 4 server slots / 4 clients

Both configurations passed every limit at 4 server slots and 4 concurrent clients. The stated tiebreakers selected this point. This study compares the measured server settings for each candidate.

At the selected 4-slot / 4-client point: compared with Q8, Q4 with KleidiAI and 512-token micro-batch measured 6.74% more generation throughput and 41.09% less peak RSS.

Operating points: 18
Measured requests: 288
Repeat passes: 2
Blocked points: 6

Reference configuration

Q8 generic reference

Selected point: 4 slots / 4 clients
Peak RSS: 3,448.1 MiB
Quality: 21/24
Passing points: 6/9

Swipe to inspect every client level →

Q8 generic reference serving-capacity matrix
Slots × clients	ClientsC1	ClientsC2	ClientsC4
Server slotsP1	PassReference 35.10tok/s E2E 1,883 ms TTFT 225 ms	Blocked 35.26tok/s E2E 3,678 ms TTFT 2,049 ms TTFT	Blocked 35.14tok/s E2E 7,348 ms TTFT 5,685 ms TTFT + E2E
Server slotsP2	Pass 35.36tok/s E2E 1,864 ms TTFT 227 ms	Pass 38.85tok/s E2E 3,347 ms TTFT 390 ms	Blocked 38.65tok/s E2E 6,670 ms TTFT 3,700 ms TTFT + E2E
Server slotsP4	Pass 35.42tok/s E2E 1,857 ms TTFT 221 ms	Pass 37.09tok/s E2E 3,595 ms TTFT 416 ms	Selected 84.94tok/s E2E 3,057 ms TTFT 757 ms

Why operating points were blocked

P1 / C2The observed p95 time to first token exceeded the declared limit.
P1 / C4The observed p95 time to first token exceeded the declared limit. The observed p95 end-to-end latency exceeded the declared limit.
P2 / C4The observed p95 time to first token exceeded the declared limit. The observed p95 end-to-end latency exceeded the declared limit.

Resource alternative

Q4 with KleidiAI and 512-token micro-batch

Selected point: 4 slots / 4 clients
Peak RSS: 2,031.3 MiB
Quality: 20/24
Passing points: 6/9

Swipe to inspect every client level →

Q4 with KleidiAI and 512-token micro-batch serving-capacity matrix
Slots × clients	ClientsC1	ClientsC2	ClientsC4
Server slotsP1	PassReference 32.82tok/s E2E 2,090 ms TTFT 170 ms	Blocked 33.24tok/s E2E 3,889 ms TTFT 2,101 ms TTFT	Blocked 32.97tok/s E2E 7,804 ms TTFT 6,010 ms TTFT + E2E
Server slotsP2	Pass 33.11tok/s E2E 1,964 ms TTFT 171 ms	Pass 43.73tok/s E2E 2,967 ms TTFT 315 ms	Blocked 43.54tok/s E2E 6,024 ms TTFT 3,232 ms TTFT
Server slotsP4	Pass 33.15tok/s E2E 1,974 ms TTFT 170 ms	Pass 37.69tok/s E2E 3,881 ms TTFT 325 ms	Selected 90.67tok/s E2E 2,853 ms TTFT 640 ms

Why operating points were blocked

P1 / C2The observed p95 time to first token exceeded the declared limit.
P1 / C4The observed p95 time to first token exceeded the declared limit. The observed p95 end-to-end latency exceeded the declared limit.
P2 / C4The observed p95 time to first token exceeded the declared limit.

TTFT p95: ≤ 2,000 ms
E2E p95: ≤ 6,500 ms
Peak RSS: ≤ 4,096 MiB
Completion: ≥ 100%
Quality: ≥ 80% · ≥ 95% of reference
Quality outcomes: Same result at every server-slot setting
Generation stability: ≤ 15.0%
E2E stability: ≤ 20.0%

Verification: 121 files matched their checksums · study 3bc4aeae1d63… · archive a73d801bc399…

Study limits: This is a bounded, closed-loop study on one native Arm64 runner. Each displayed p95 is the median of two pass-level p95 values. Each pass contains eight requests, so its p95 is the observed maximum. The KleidiAI log marker confirms the enabled model-buffer path; it does not prove universal microkernel use.

01 · Decision rule

Why this configuration was selected

The objective tolerance and preference order were defined before selection.

Decision tolerance

1.00% decision tolerance

Lower is better. The tolerance cutoff was 2,254.3 ms for p95 end-to-end latency.

Q8 generic referenceWithin tolerance · selected
2,231.9 ms
Q4 with KleidiAIOutside tolerance
2,299.5 ms
Q4 with KleidiAI and 512-token micro-batchOutside tolerance
2,307.7 ms
Q4 generic quantizationOutside tolerance
2,311.1 ms

Markers show each measured p95 end-to-end latency on the same scale. Exact values are in the table below.

Tolerance defined before the run

1.00%

The best measured result was Q8 generic reference at 2,231.9329 ms. The tolerance range included values at or below 2,254.2522 ms.

The best measured result also ranked first under the stated preference order.

Measured values and tolerance results
Candidate	End-to-end latency (p95)	Decision role
Q4 generic quantization	2,311.1251 ms	Outside tolerance
Q4 with KleidiAI	2,299.4543 ms	Outside tolerance
Q4 with KleidiAI and 512-token micro-batch	2,307.7153 ms	Outside tolerance
Q8 generic reference	2,231.9329 ms	Best measured result · Within tolerance · Selected

02 · Tradeoffs

Measured tradeoffs

The measured baseline is compared with Q4 with KleidiAI and 512-token micro-batch. Labels follow the stated better-or-worse direction for each metric.

End-to-end latency (p95)

Baseline2,231.9329 ms

Alternative2,307.7153 ms

Tradeoff · 3.40% higher

Time to first token (p95)

Baseline545.3739 ms

Alternative469.9681 ms

Better · 13.83% lower

Prompt processing throughput

Baseline102.6185 tok/s

Alternative131.4565 tok/s

Changed · 28.10% higher

Generation throughput

Baseline38.7264 tok/s

Alternative35.0959 tok/s

Tradeoff · 9.37% lower

Peak resident memory

Baseline3,437.5977 MiB

Alternative1,966.4805 MiB

Better · 42.79% lower

Model size

Baseline1,806.7666 MiB

Alternative1,016.8335 MiB

Better · 43.72% lower

Quality score

Baseline0.875

Alternative0.8333

Tradeoff · 4.76% lower

03 · Policy comparison

All deployment priorities

Each priority applies a different objective to the same validated measurements.

Swipe to inspect every deployment priority →

Published latency result

Q8 generic reference

Uses p95 end-to-end latency, a 1% practical tolerance, and the declared preference order.

Objective: End-to-end latency (p95)
Direction: Minimize

The best measured result also ranked first under the stated preference order.

Published latency decision.

End-to-end latency (p95)2,231.9329 ms
Time to first token (p95)545.3739 ms
Prompt processing throughput102.6185 tok/s
Generation throughput38.7264 tok/s
Peak resident memory3,437.5977 MiB

04 · Load test

Measured load behavior

Charts use only supplied, validated measurements; missing values are left blank.

Measured takeawayEvery candidate’s highest measured all-gates passing concurrency was C1. At C4, every candidate crossed the declared 6,500 ms p95 end-to-end ceiling.

Chart legend

Q8 generic referenceSelected · Baseline · Frontier
Q4 generic quantizationFrontier
Q4 with KleidiAIFrontier
Q4 with KleidiAI and 512-token micro-batchFrontier

Measured completed requests per second as concurrent request count increases.

Request throughput by concurrency data
Candidate	Concurrency	Request throughput
Q4 generic quantization	1	0.5036 req/s
Q4 generic quantization	2	0.5089 req/s
Q4 generic quantization	4	0.5087 req/s
Q4 with KleidiAI	1	0.5126 req/s
Q4 with KleidiAI	2	0.5107 req/s
Q4 with KleidiAI	4	0.5139 req/s
Q4 with KleidiAI and 512-token micro-batch	1	0.5127 req/s
Q4 with KleidiAI and 512-token micro-batch	2	0.5055 req/s
Q4 with KleidiAI and 512-token micro-batch	4	0.5151 req/s
Q8 generic reference	1	0.5457 req/s
Q8 generic reference	2	0.5492 req/s
Q8 generic reference	4	0.5391 req/s

Measured generated tokens per second as concurrent request count increases.

Generated-token throughput by concurrency data
Candidate	Concurrency	Generated-token throughput
Q4 generic quantization	1	32.2286 tok/s
Q4 generic quantization	2	32.5665 tok/s
Q4 generic quantization	4	32.5585 tok/s
Q4 with KleidiAI	1	32.8078 tok/s
Q4 with KleidiAI	2	32.682 tok/s
Q4 with KleidiAI	4	32.8879 tok/s
Q4 with KleidiAI and 512-token micro-batch	1	32.8119 tok/s
Q4 with KleidiAI and 512-token micro-batch	2	32.3519 tok/s
Q4 with KleidiAI and 512-token micro-batch	4	32.9648 tok/s
Q8 generic reference	1	34.9234 tok/s
Q8 generic reference	2	35.1494 tok/s
Q8 generic reference	4	34.5001 tok/s

Measured p95 end-to-end response latency as concurrent request count increases. The amber line marks the declared 6,500 ms latency ceiling; a passing level must also satisfy the TTFT and completion-rate gates.

p95 end-to-end latency by concurrency data
Candidate	Concurrency	End-to-end latency (p95)
Q4 generic quantization	1	2,044.6926 ms
Q4 generic quantization	2	3,978.9609 ms
Q4 generic quantization	4	7,972.9287 ms
Q4 with KleidiAI	1	1,986.324 ms
Q4 with KleidiAI	2	3,968.8569 ms
Q4 with KleidiAI	4	7,807.2995 ms
Q4 with KleidiAI and 512-token micro-batch	1	1,984.003 ms
Q4 with KleidiAI and 512-token micro-batch	2	4,150.2731 ms
Q4 with KleidiAI and 512-token micro-batch	4	7,792.2621 ms
Q8 generic reference	1	1,915.4125 ms
Q8 generic reference	2	3,699.7596 ms
Q8 generic reference	4	7,512.7399 ms

View load-test settings and prompts

Verified test commands

Test command match: Each load test used the same settings as its recorded server command; only the host or port could differ.

Load plan SHA-256: b10f61b7093f58ad6d142deafd7d6befe600d1d54a433b59c101131dd8c2d9bb

View exact commands used

Recorded load and server commands
Candidate	Request endpoint	Recorded parallel	Allowed differences used	Load command SHA-256	Recorded command SHA-256
Q4 generic quantization	`http://127.0.0.1:18182`	1	--port	`8557a6aca70a361ce08b2e3fa4406d71f2765d7a2d7b5ae6b61095da7b4fae03`	`0c74feec5d6df4d2070cf37c73c6199893ceb562634655282814b731bce720c9`
Q4 with KleidiAI	`http://127.0.0.1:18183`	1	--port	`f79679a2ab642d4fd14cfe9c74da40c8fef69e27e1dbde5602d50a38cc71e113`	`93bde6fa12bfc3a74719cac2a98c3f0afd447c79ba7e6ba6a29902bedc00ad87`
Q4 with KleidiAI and 512-token micro-batch	`http://127.0.0.1:18184`	1	--port	`cedca7ce6bbcd754b1c3278f605da7886be2ebf886ee2a400b67184c42ae1aa1`	`4e09666b2e6542694e0a96f3e20f27697a7f9202155f7b9c80658fd51ccc3487`
Q8 generic reference	`http://127.0.0.1:18181`	1	--port	`9a2302850253f4b86fd9b74de5cfb20afb8822683698f604d2a8de79c581cd49`	`8b5040317a38b52930ab87494bb4d02aaccf696ebe7d0826df33816a3045d74f`

Highest concurrency that met every limit

Highest measured concurrency meeting all declared SLO gates
Field	Value
Q4 generic quantization	`1`
Q4 with kleidiai	`1`
Q4 with kleidiai and 512-token micro-batch	`1`
Q8 generic reference	`1`

Declared service-level objective

Load sweep service-level objective
Field	Value
Max e2e latency ms p95	`6500.0`
Max ttft ms p95	`2000.0`
Min completion rate	`1.0`

Measured methodology

Load sweep methodology
Field	Value
Client scheduler	`"bounded thread pool"`
Concurrency levels	`[1, 2, 4]`
Measured requests per level	`8`
Output tokens	`64`
Prompt schedule	`"round-robin"`
Prompts	[{"index": 1, "sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "text": "An engineering team is choosing an Arm64 inference configuration. Explain in two concise sentences why measured latency and memory evidence should be reviewed together."}, {"index": 2, "sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "text": "Summarize this deployment note in two concise sentences: the service must preserve response quality, keep tail latency within its declared limit, and leave enough memory for other workloads."}, {"index": 3, "sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "text": "Describe in two concise sentences how a reproducible benchmark helps a student developer make an honest deployment decision."}]
Warmup requests per level	`4`

View all measured load results

Validated measurements at each concurrency level
Candidate	Concurrency	Measured requests	Completed requests	Request throughput	Generated-token throughput	Time to first token (p50)	Time to first token (p95)	End-to-end latency (p50)	End-to-end latency (p95)	Completion rate	Error rate	SLO status	Samples
Q4 generic quantization	1	8	8	0.5036 req/s	32.2286 tok/s	146.8574 ms	190.4137 ms	1,984.5072 ms	2,044.6926 ms	100.00%	0.00%	Met	View Q4 generic quantization samples at concurrency 1 [{"completed": true, "e2e_latency_ms": 1949.6210849999898, "error": null, "finished_at_seconds": 1710.667977062, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1708.718355977, "ttft_ms": 146.68788899984975}, {"completed": true, "e2e_latency_ms": 2000.5232860000888, "error": null, "finished_at_seconds": 1712.668628779, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1710.668105493, "ttft_ms": 190.41365199996108}, {"completed": true, "e2e_latency_ms": 1928.5806919999686, "error": null, "finished_at_seconds": 1714.59731592, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1712.668735228, "ttft_ms": 104.59789899982752}, {"completed": true, "e2e_latency_ms": 2044.6926259999145, "error": null, "finished_at_seconds": 1716.642113771, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1714.597421145, "ttft_ms": 146.07854900009443}, {"completed": true, "e2e_latency_ms": 2019.6252349999213, "error": null, "finished_at_seconds": 1718.661853439, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1716.642228204, "ttft_ms": 189.46377399993253}, {"completed": true, "e2e_latency_ms": 1938.6470189999727, "error": null, "finished_at_seconds": 1720.600609379, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1718.66196236, "ttft_ms": 105.64129800013689}, {"completed": true, "e2e_latency_ms": 1968.4910539999692, "error": null, "finished_at_seconds": 1722.569210682, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1720.600719628, "ttft_ms": 147.0269989999906}, {"completed": true, "e2e_latency_ms": 2035.5482820000361, "error": null, "finished_at_seconds": 1724.604870101, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1722.569321819, "ttft_ms": 189.65193799999724}]
Q4 generic quantization	2	8	8	0.5089 req/s	32.5665 tok/s	2,108.0442 ms	2,178.0842 ms	3,924.859 ms	3,978.9609 ms	100.00%	0.00%	Not met	View Q4 generic quantization samples at concurrency 2 [{"completed": true, "e2e_latency_ms": 1950.265088000151, "error": null, "finished_at_seconds": 1734.378775629, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1732.428510541, "ttft_ms": 145.67688200008888}, {"completed": true, "e2e_latency_ms": 3948.6531970001124, "error": null, "finished_at_seconds": 1736.377427479, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1732.428774282, "ttft_ms": 2139.0681750001477}, {"completed": true, "e2e_latency_ms": 3921.9944689998556, "error": null, "finished_at_seconds": 1738.300870459, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1734.37887599, "ttft_ms": 2101.0757259998627}, {"completed": true, "e2e_latency_ms": 3863.157734999959, "error": null, "finished_at_seconds": 1740.243284453, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1736.380126718, "ttft_ms": 2065.0054479999653}, {"completed": true, "e2e_latency_ms": 3953.858683000135, "error": null, "finished_at_seconds": 1742.254827016, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1738.300968333, "ttft_ms": 2130.7733920000373}, {"completed": true, "e2e_latency_ms": 3927.7236240000093, "error": null, "finished_at_seconds": 1744.171115198, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1740.243391574, "ttft_ms": 2115.0126459999683}, {"completed": true, "e2e_latency_ms": 3905.9285629998612, "error": null, "finished_at_seconds": 1746.160865572, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1742.254937009, "ttft_ms": 2062.1976549998635}, {"completed": true, "e2e_latency_ms": 3978.9609189999737, "error": null, "finished_at_seconds": 1748.150175231, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1744.171214312, "ttft_ms": 2178.084228999978}]
Q4 generic quantization	4	8	8	0.5087 req/s	32.5585 tok/s	6,017.3187 ms	6,167.0907 ms	7,823.7666 ms	7,972.9287 ms	100.00%	0.00%	Not met	View Q4 generic quantization samples at concurrency 4 [{"completed": true, "e2e_latency_ms": 3842.7767540001696, "error": null, "finished_at_seconds": 1759.804218509, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1755.961441755, "ttft_ms": 2046.001441000044}, {"completed": true, "e2e_latency_ms": 5926.847155000132, "error": null, "finished_at_seconds": 1761.88853919, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1755.961692035, "ttft_ms": 4031.4800959999957}, {"completed": true, "e2e_latency_ms": 1900.5368069999804, "error": null, "finished_at_seconds": 1757.862476634, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1755.961939827, "ttft_ms": 103.90798900016307}, {"completed": true, "e2e_latency_ms": 7879.752792999852, "error": null, "finished_at_seconds": 1763.842073635, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1755.962320842, "ttft_ms": 6071.819995999931}, {"completed": true, "e2e_latency_ms": 7972.928658000001, "error": null, "finished_at_seconds": 1765.835494734, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1757.862566076, "ttft_ms": 6167.090718000054}, {"completed": true, "e2e_latency_ms": 7932.512668000072, "error": null, "finished_at_seconds": 1767.73683681, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1759.804324142, "ttft_ms": 6133.627446999981}, {"completed": true, "e2e_latency_ms": 7802.7348919999895, "error": null, "finished_at_seconds": 1769.691405769, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1761.888670877, "ttft_ms": 5997.132936999833}, {"completed": true, "e2e_latency_ms": 7844.798350000019, "error": null, "finished_at_seconds": 1771.686979882, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1763.842181532, "ttft_ms": 6037.504558999899}]
Q4 with KleidiAI	1	8	8	0.5126 req/s	32.8078 tok/s	143.6856 ms	170.5038 ms	1,951.2007 ms	1,986.324 ms	100.00%	0.00%	Met	View Q4 with KleidiAI samples at concurrency 1 [{"completed": true, "e2e_latency_ms": 1946.666162999918, "error": null, "finished_at_seconds": 1784.656093164, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1782.709427001, "ttft_ms": 144.07943500009424}, {"completed": true, "e2e_latency_ms": 1977.7977620001366, "error": null, "finished_at_seconds": 1786.633995175, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1784.656197413, "ttft_ms": 169.0082729999176}, {"completed": true, "e2e_latency_ms": 1900.2986949999467, "error": null, "finished_at_seconds": 1788.534426324, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1786.634127629, "ttft_ms": 101.44893699998647}, {"completed": true, "e2e_latency_ms": 1953.4156979998443, "error": null, "finished_at_seconds": 1790.487949318, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1788.53453362, "ttft_ms": 143.2917649999581}, {"completed": true, "e2e_latency_ms": 1986.3239559999784, "error": null, "finished_at_seconds": 1792.474412104, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1790.488088148, "ttft_ms": 169.7093440000117}, {"completed": true, "e2e_latency_ms": 1907.9663110001093, "error": null, "finished_at_seconds": 1794.382485416, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1792.474519105, "ttft_ms": 101.35752800010778}, {"completed": true, "e2e_latency_ms": 1948.985666999988, "error": null, "finished_at_seconds": 1796.331576508, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1794.382590841, "ttft_ms": 143.19726400003674}, {"completed": true, "e2e_latency_ms": 1983.8007589999052, "error": null, "finished_at_seconds": 1798.315483996, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1796.331683237, "ttft_ms": 170.50377199984723}]
Q4 with KleidiAI	2	8	8	0.5107 req/s	32.682 tok/s	2,097.2113 ms	2,133.1307 ms	3,914.6173 ms	3,968.8569 ms	100.00%	0.00%	Not met	View Q4 with KleidiAI samples at concurrency 2 [{"completed": true, "e2e_latency_ms": 1950.6665710000561, "error": null, "finished_at_seconds": 1808.07427089, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1806.123604319, "ttft_ms": 143.95071500007361}, {"completed": true, "e2e_latency_ms": 3939.669130999846, "error": null, "finished_at_seconds": 1810.063492956, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1806.123823825, "ttft_ms": 2118.404326000018}, {"completed": true, "e2e_latency_ms": 3932.931126000085, "error": null, "finished_at_seconds": 1812.007297258, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1808.074366132, "ttft_ms": 2097.8106229999867}, {"completed": true, "e2e_latency_ms": 3905.763395999884, "error": null, "finished_at_seconds": 1813.969350962, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1810.063587566, "ttft_ms": 2087.500675000001}, {"completed": true, "e2e_latency_ms": 3968.8569159998224, "error": null, "finished_at_seconds": 1815.976263463, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1812.007406547, "ttft_ms": 2133.1307010000273}, {"completed": true, "e2e_latency_ms": 3923.4711300000527, "error": null, "finished_at_seconds": 1817.892927477, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1813.969456347, "ttft_ms": 2111.798550000003}, {"completed": true, "e2e_latency_ms": 3845.3242040000077, "error": null, "finished_at_seconds": 1819.821679909, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1815.976355705, "ttft_ms": 2059.3017769999733}, {"completed": true, "e2e_latency_ms": 3896.660265999799, "error": null, "finished_at_seconds": 1821.78970384, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1817.893043574, "ttft_ms": 2096.611907999886}]
Q4 with KleidiAI	4	8	8	0.5139 req/s	32.8879 tok/s	5,945.6177 ms	6,008.9782 ms	7,745.128 ms	7,807.2995 ms	100.00%	0.00%	Not met	View Q4 with KleidiAI samples at concurrency 4 [{"completed": true, "e2e_latency_ms": 1944.0086070001144, "error": null, "finished_at_seconds": 1831.503282523, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1829.559273916, "ttft_ms": 143.28228199997284}, {"completed": true, "e2e_latency_ms": 3925.212963000149, "error": null, "finished_at_seconds": 1833.484754301, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1829.559541338, "ttft_ms": 2111.392037000087}, {"completed": true, "e2e_latency_ms": 5830.627899999854, "error": null, "finished_at_seconds": 1835.390466914, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1829.559839014, "ttft_ms": 4025.7414399998197}, {"completed": true, "e2e_latency_ms": 7783.589126999914, "error": null, "finished_at_seconds": 1837.343858705, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1829.560269578, "ttft_ms": 5972.488849000001}, {"completed": true, "e2e_latency_ms": 7807.299460999957, "error": null, "finished_at_seconds": 1839.310690585, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1831.503391124, "ttft_ms": 6008.978203999959}, {"completed": true, "e2e_latency_ms": 7729.2094389999875, "error": null, "finished_at_seconds": 1841.214060798, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1833.484851359, "ttft_ms": 5926.030214000093}, {"completed": true, "e2e_latency_ms": 7761.0466339999675, "error": null, "finished_at_seconds": 1843.151608638, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1835.390562004, "ttft_ms": 5965.2050980000695}, {"completed": true, "e2e_latency_ms": 7783.327761999999, "error": null, "finished_at_seconds": 1845.127286876, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1837.343959114, "ttft_ms": 5975.1547749999645}]
Q4 with KleidiAI and 512-token micro-batch	1	8	8	0.5127 req/s	32.8119 tok/s	157.2909 ms	176.5833 ms	1,958.5087 ms	1,984.003 ms	100.00%	0.00%	Met	View Q4 with KleidiAI and 512-token micro-batch samples at concurrency 1 [{"completed": true, "e2e_latency_ms": 1943.1996179998805, "error": null, "finished_at_seconds": 1858.047414535, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1856.104214917, "ttft_ms": 143.389556999864}, {"completed": true, "e2e_latency_ms": 1971.211435999976, "error": null, "finished_at_seconds": 1860.018723052, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1858.047511616, "ttft_ms": 170.8221119999962}, {"completed": true, "e2e_latency_ms": 1913.7540710000849, "error": null, "finished_at_seconds": 1861.932586228, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1860.018832157, "ttft_ms": 102.20275300002868}, {"completed": true, "e2e_latency_ms": 1978.3992650000073, "error": null, "finished_at_seconds": 1863.911094118, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1861.932694853, "ttft_ms": 176.58332800010612}, {"completed": true, "e2e_latency_ms": 1977.528083000152, "error": null, "finished_at_seconds": 1865.888726458, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1863.911198375, "ttft_ms": 169.7822090000045}, {"completed": true, "e2e_latency_ms": 1889.4534899998234, "error": null, "finished_at_seconds": 1867.778284245, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1865.888830755, "ttft_ms": 101.47703300003741}, {"completed": true, "e2e_latency_ms": 1945.8059020000746, "error": null, "finished_at_seconds": 1869.724194668, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1867.778388766, "ttft_ms": 145.37016700001004}, {"completed": true, "e2e_latency_ms": 1984.0029950000826, "error": null, "finished_at_seconds": 1871.708301248, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1869.724298253, "ttft_ms": 169.21172800016393}]
Q4 with KleidiAI and 512-token micro-batch	2	8	8	0.5055 req/s	32.3519 tok/s	2,068.0193 ms	2,351.7369 ms	3,878.7578 ms	4,150.2731 ms	100.00%	0.00%	Not met	View Q4 with KleidiAI and 512-token micro-batch samples at concurrency 2 [{"completed": true, "e2e_latency_ms": 4150.273077999827, "error": null, "finished_at_seconds": 1883.871618085, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1879.721345007, "ttft_ms": 2351.7368509999415}, {"completed": true, "e2e_latency_ms": 2208.4207119999064, "error": null, "finished_at_seconds": 1881.93004894, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1879.721628228, "ttft_ms": 214.0422839997882}, {"completed": true, "e2e_latency_ms": 3843.9485330000025, "error": null, "finished_at_seconds": 1885.774090843, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1881.93014231, "ttft_ms": 2045.7916169998498}, {"completed": true, "e2e_latency_ms": 3866.1812479999753, "error": null, "finished_at_seconds": 1887.737901062, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1883.871719814, "ttft_ms": 2046.5239150000798}, {"completed": true, "e2e_latency_ms": 3941.512565000039, "error": null, "finished_at_seconds": 1889.715711857, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1885.774199292, "ttft_ms": 2132.2770019999098}, {"completed": true, "e2e_latency_ms": 3891.3342719999946, "error": null, "finished_at_seconds": 1891.629362302, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1887.73802803, "ttft_ms": 2079.6453520001705}, {"completed": true, "e2e_latency_ms": 3860.6337339999754, "error": null, "finished_at_seconds": 1893.576447296, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1889.715813562, "ttft_ms": 2056.3933369999177}, {"completed": true, "e2e_latency_ms": 3917.8510120000283, "error": null, "finished_at_seconds": 1895.547310028, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1891.629459016, "ttft_ms": 2115.715507999994}]
Q4 with KleidiAI and 512-token micro-batch	4	8	8	0.5151 req/s	32.9648 tok/s	5,909.923 ms	5,987.6053 ms	7,704.7339 ms	7,792.2621 ms	100.00%	0.00%	Not met	View Q4 with KleidiAI and 512-token micro-batch samples at concurrency 4 [{"completed": true, "e2e_latency_ms": 1938.3449540000584, "error": null, "finished_at_seconds": 1905.246908952, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1903.308563998, "ttft_ms": 144.07620500014673}, {"completed": true, "e2e_latency_ms": 3919.9429020000025, "error": null, "finished_at_seconds": 1907.228757324, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1903.308814422, "ttft_ms": 2106.1574959999234}, {"completed": true, "e2e_latency_ms": 7756.85716199996, "error": null, "finished_at_seconds": 1911.065921199, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1903.309064037, "ttft_ms": 5957.664638999859}, {"completed": true, "e2e_latency_ms": 5856.797578999931, "error": null, "finished_at_seconds": 1909.166313538, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1903.309515959, "ttft_ms": 4061.5395810000336}, {"completed": true, "e2e_latency_ms": 7792.262099000027, "error": null, "finished_at_seconds": 1913.039261013, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1905.246998914, "ttft_ms": 5987.605336000115}, {"completed": true, "e2e_latency_ms": 7704.093109999803, "error": null, "finished_at_seconds": 1914.932956212, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1907.228863102, "ttft_ms": 5911.0863269997935}, {"completed": true, "e2e_latency_ms": 7705.3746330000195, "error": null, "finished_at_seconds": 1916.871776174, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1909.166401541, "ttft_ms": 5908.759608000082}, {"completed": true, "e2e_latency_ms": 7774.239778000037, "error": null, "finished_at_seconds": 1918.840284033, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1911.066044255, "ttft_ms": 5973.124458000029}]
Q8 generic reference	1	8	8	0.5457 req/s	34.9234 tok/s	166.3753 ms	221.5149 ms	1,828.9081 ms	1,915.4125 ms	100.00%	0.00%	Met	View Q8 generic reference samples at concurrency 1 [{"completed": true, "e2e_latency_ms": 1798.8625559999036, "error": null, "finished_at_seconds": 1640.790112086, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1638.99124953, "ttft_ms": 166.21581999993396}, {"completed": true, "e2e_latency_ms": 1865.428042999838, "error": null, "finished_at_seconds": 1642.655649482, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1640.790221439, "ttft_ms": 221.32033099978798}, {"completed": true, "e2e_latency_ms": 1760.4547210000874, "error": null, "finished_at_seconds": 1644.416210996, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1642.655756275, "ttft_ms": 124.19632600017394}, {"completed": true, "e2e_latency_ms": 1827.8209700001753, "error": null, "finished_at_seconds": 1646.244138551, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1644.416317581, "ttft_ms": 166.53473900009885}, {"completed": true, "e2e_latency_ms": 1880.6345320001583, "error": null, "finished_at_seconds": 1648.124881764, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1646.244247232, "ttft_ms": 221.51485600011256}, {"completed": true, "e2e_latency_ms": 1781.2798869999824, "error": null, "finished_at_seconds": 1649.90627082, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1648.124990933, "ttft_ms": 124.80181499995524}, {"completed": true, "e2e_latency_ms": 1829.995161999932, "error": null, "finished_at_seconds": 1651.736374055, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1649.906378893, "ttft_ms": 165.9717830000318}, {"completed": true, "e2e_latency_ms": 1915.4125049999493, "error": null, "finished_at_seconds": 1653.651897337, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1651.736484832, "ttft_ms": 220.88710899993202}]
Q8 generic reference	2	8	8	0.5492 req/s	35.1494 tok/s	1,997.303 ms	2,037.5929 ms	3,645.7782 ms	3,699.7596 ms	100.00%	0.00%	Not met	View Q8 generic reference samples at concurrency 2 [{"completed": true, "e2e_latency_ms": 1817.6788560001569, "error": null, "finished_at_seconds": 1662.766442885, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1660.948764029, "ttft_ms": 166.29236800008584}, {"completed": true, "e2e_latency_ms": 3699.7595810000803, "error": null, "finished_at_seconds": 1664.648756963, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1660.948997382, "ttft_ms": 2037.5929140000153}, {"completed": true, "e2e_latency_ms": 3655.853397000101, "error": null, "finished_at_seconds": 1666.422398915, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1662.766545518, "ttft_ms": 2007.629131000158}, {"completed": true, "e2e_latency_ms": 3572.888002999889, "error": null, "finished_at_seconds": 1668.221760639, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1664.648872636, "ttft_ms": 1937.6681319999989}, {"completed": true, "e2e_latency_ms": 3657.94876699988, "error": null, "finished_at_seconds": 1670.083062646, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1666.425113879, "ttft_ms": 2016.2819259999196}, {"completed": true, "e2e_latency_ms": 3635.7030509998367, "error": null, "finished_at_seconds": 1671.857567891, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1668.22186484, "ttft_ms": 1986.976850000019}, {"completed": true, "e2e_latency_ms": 3577.882627000008, "error": null, "finished_at_seconds": 1673.661046802, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1670.083164175, "ttft_ms": 1939.8738609997963}, {"completed": true, "e2e_latency_ms": 3657.471274000045, "error": null, "finished_at_seconds": 1675.515145646, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1671.857674372, "ttft_ms": 2028.2735000000685}]
Q8 generic reference	4	8	8	0.5391 req/s	34.5001 tok/s	5,675.5386 ms	5,824.2042 ms	7,373.7089 ms	7,512.7399 ms	100.00%	0.00%	Not met	View Q8 generic reference samples at concurrency 4 [{"completed": true, "e2e_latency_ms": 1803.7393029999294, "error": null, "finished_at_seconds": 1684.617693484, "generated_tokens": 64, "index": 1, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1682.813954181, "ttft_ms": 167.27661400000216}, {"completed": true, "e2e_latency_ms": 3674.8212630000126, "error": null, "finished_at_seconds": 1686.489043627, "generated_tokens": 64, "index": 2, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1682.814222364, "ttft_ms": 2024.137742999983}, {"completed": true, "e2e_latency_ms": 5463.084967999976, "error": null, "finished_at_seconds": 1688.277522758, "generated_tokens": 64, "index": 3, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1682.81443779, "ttft_ms": 3798.668943999928}, {"completed": true, "e2e_latency_ms": 7326.948791999939, "error": null, "finished_at_seconds": 1690.141647045, "generated_tokens": 64, "index": 4, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1682.814698253, "ttft_ms": 5629.245366000077}, {"completed": true, "e2e_latency_ms": 7466.097500999922, "error": null, "finished_at_seconds": 1692.083893306, "generated_tokens": 64, "index": 5, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1684.617795805, "ttft_ms": 5746.490197999947}, {"completed": true, "e2e_latency_ms": 7420.468996999944, "error": null, "finished_at_seconds": 1693.909616441, "generated_tokens": 64, "index": 6, "prompt_index": 3, "prompt_sha256": "a48be792aff96c3b81cc8be1bc60faa0a072f1f106cf956261339b33d0d18d04", "started_at_seconds": 1686.489147444, "ttft_ms": 5721.831898999881}, {"completed": true, "e2e_latency_ms": 7466.920954999978, "error": null, "finished_at_seconds": 1695.744549106, "generated_tokens": 64, "index": 7, "prompt_index": 1, "prompt_sha256": "549face5a90122a765c1cbefe79cff73ed227a759e36590082aefbe44dcb5291", "started_at_seconds": 1688.277628151, "ttft_ms": 5797.618667000052}, {"completed": true, "e2e_latency_ms": 7512.7398800000265, "error": null, "finished_at_seconds": 1697.654487854, "generated_tokens": 64, "index": 8, "prompt_index": 2, "prompt_sha256": "5fd318ce3ceb729df9ee61f9e38c59fcd89cb885b01f4f0c8025053abb67571c", "started_at_seconds": 1690.141747974, "ttft_ms": 5824.204237999993}]

05 · Repeatability

Repeatability across two passes

Validated pass-level values show whether each measured direction repeated and how widely the candidate values spread.

Observed passes: pass-1, pass-2

Method: observed pass direction and relative spread; no significance claim.

How to read the deltas: positive means improvement after applying each metric’s declared direction; negative means regression. “Consistent” means the two observed passes had the same comparison direction—it is not a statistical-significance threshold.

Inspect pass-level values and deltas

Observed pass directions, relative deltas, and relative spread
Candidate	Metric	Direction	Pass values and deltas	Relative spread	Consistency	Overall direction
Q4 generic quantization	End-to-end latency (p95)	Lower is better	pass-12,311.1251 ms vs 2,231.9329 ms-3.55% · regressed pass-22,315.0595 ms vs 2,271.4757 ms-1.92% · regressed	0.17%	consistent	regressed
Q4 generic quantization	Generation throughput	Higher is better	pass-134.7858 tok/s vs 39.2861 tok/s-11.46% · regressed pass-235.3714 tok/s vs 38.5038 tok/s-8.14% · regressed	1.67%	consistent	regressed
Q4 generic quantization	Peak resident memory	Lower is better	pass-11,966.4609 MiB vs 3,437.5938 MiB+42.80% · improved pass-21,966.4727 MiB vs 3,437.5977 MiB+42.80% · improved	0.00%	consistent	improved
Q4 generic quantization	Prompt processing throughput	Higher is better	pass-1113.7295 tok/s vs 102.7365 tok/s+10.70% · improved pass-2113.874 tok/s vs 102.519 tok/s+11.08% · improved	0.13%	consistent	improved
Q4 generic quantization	Quality score	Higher is better	pass-10.8333 vs 0.875-4.76% · regressed pass-20.8333 vs 0.875-4.76% · regressed	0.00%	consistent	regressed
Q4 generic quantization	Time to first token (p95)	Lower is better	pass-1481.3993 ms vs 546.4947 ms+11.91% · improved pass-2483.8053 ms vs 545.3739 ms+11.29% · improved	0.50%	consistent	improved
Q4 with KleidiAI	End-to-end latency (p95)	Lower is better	pass-12,295.5086 ms vs 2,231.9329 ms-2.85% · regressed pass-22,306.7364 ms vs 2,271.4757 ms-1.55% · regressed	0.49%	consistent	regressed
Q4 with KleidiAI	Generation throughput	Higher is better	pass-135.5067 tok/s vs 39.2861 tok/s-9.62% · regressed pass-235.0159 tok/s vs 38.5038 tok/s-9.06% · regressed	1.39%	consistent	regressed
Q4 with KleidiAI	Peak resident memory	Lower is better	pass-11,966.4766 MiB vs 3,437.5938 MiB+42.79% · improved pass-21,966.4844 MiB vs 3,437.5977 MiB+42.79% · improved	0.00%	consistent	improved
Q4 with KleidiAI	Prompt processing throughput	Higher is better	pass-1114.532 tok/s vs 102.7365 tok/s+11.48% · improved pass-2114.425 tok/s vs 102.519 tok/s+11.61% · improved	0.09%	consistent	improved
Q4 with KleidiAI	Quality score	Higher is better	pass-10.8333 vs 0.875-4.76% · regressed pass-20.8333 vs 0.875-4.76% · regressed	0.00%	consistent	regressed
Q4 with KleidiAI	Time to first token (p95)	Lower is better	pass-1467.1517 ms vs 546.4947 ms+14.52% · improved pass-2471.8408 ms vs 545.3739 ms+13.48% · improved	1.00%	consistent	improved
Q4 with KleidiAI and 512-token micro-batch	End-to-end latency (p95)	Lower is better	pass-12,312.7413 ms vs 2,231.9329 ms-3.62% · regressed pass-22,307.7153 ms vs 2,271.4757 ms-1.60% · regressed	0.22%	consistent	regressed
Q4 with KleidiAI and 512-token micro-batch	Generation throughput	Higher is better	pass-134.9091 tok/s vs 39.2861 tok/s-11.14% · regressed pass-235.1905 tok/s vs 38.5038 tok/s-8.61% · regressed	0.80%	consistent	regressed
Q4 with KleidiAI and 512-token micro-batch	Peak resident memory	Lower is better	pass-11,966.4805 MiB vs 3,437.5938 MiB+42.79% · improved pass-21,966.4727 MiB vs 3,437.5977 MiB+42.80% · improved	0.00%	consistent	improved
Q4 with KleidiAI and 512-token micro-batch	Prompt processing throughput	Higher is better	pass-1131.516 tok/s vs 102.7365 tok/s+28.01% · improved pass-2131.361 tok/s vs 102.519 tok/s+28.13% · improved	0.12%	consistent	improved
Q4 with KleidiAI and 512-token micro-batch	Quality score	Higher is better	pass-10.8333 vs 0.875-4.76% · regressed pass-20.8333 vs 0.875-4.76% · regressed	0.00%	consistent	regressed
Q4 with KleidiAI and 512-token micro-batch	Time to first token (p95)	Lower is better	pass-1467.41 ms vs 546.4947 ms+14.47% · improved pass-2470.2895 ms vs 545.3739 ms+13.77% · improved	0.61%	consistent	improved
Q8 generic reference	End-to-end latency (p95)	Lower is better	pass-12,231.9329 ms vs 2,231.9329 ms+0.00% · no change pass-22,271.4757 ms vs 2,271.4757 ms+0.00% · no change	1.76%	no change	no change
Q8 generic reference	Generation throughput	Higher is better	pass-139.2861 tok/s vs 39.2861 tok/s+0.00% · no change pass-238.5038 tok/s vs 38.5038 tok/s+0.00% · no change	2.01%	no change	no change
Q8 generic reference	Peak resident memory	Lower is better	pass-13,437.5938 MiB vs 3,437.5938 MiB+0.00% · no change pass-23,437.5977 MiB vs 3,437.5977 MiB+0.00% · no change	0.00%	no change	no change
Q8 generic reference	Prompt processing throughput	Higher is better	pass-1102.7365 tok/s vs 102.7365 tok/s+0.00% · no change pass-2102.519 tok/s vs 102.519 tok/s+0.00% · no change	0.21%	no change	no change
Q8 generic reference	Quality score	Higher is better	pass-10.875 vs 0.875+0.00% · no change pass-20.875 vs 0.875+0.00% · no change	0.00%	no change	no change
Q8 generic reference	Time to first token (p95)	Lower is better	pass-1546.4947 ms vs 546.4947 ms+0.00% · no change pass-2545.3739 ms vs 545.3739 ms+0.00% · no change	0.21%	no change	no change

Interpretation limit: consistency and relative spread describe these observed passes only. They are not a statistical-significance claim.

06 · Latency and throughput

Latency versus generation throughput

This chart shows latency and generation throughput only. The full decision also considers memory, model size, quality, and constraints.

Chart legend

Q8 generic referenceSelected · Baseline · Frontier
Q4 generic quantizationFrontier
Q4 with KleidiAIFrontier
Q4 with KleidiAI and 512-token micro-batchFrontier

Each point is one measured candidate; the candidate legend carries the full names. Left is lower p95 end-to-end latency; higher is greater generation throughput. This is a two-metric view, not the complete multi-objective frontier.

p95 end-to-end latency vs generation throughput data
Candidate	p95 end-to-end latency	Generation throughput	Decision role
Q4 generic quantization	2,311.1251 ms	35.0123 tok/s	Declared frontier
Q4 with KleidiAI	2,299.4543 ms	35.3764 tok/s	Declared frontier
Q4 with KleidiAI and 512-token micro-batch	2,307.7153 ms	35.0959 tok/s	Declared frontier
Q8 generic reference	2,231.9329 ms	38.7264 tok/s	Baseline · Selected · Declared frontier

07 · Evidence matrix

Full evidence table

Unmeasured values are labeled as missing. Rejection reasons come from the recommendation.

Scroll the table horizontally.

All candidate measurements, parameters, statuses, and rejection reasons
Candidate	Decision status	Parameters	End-to-end latency (p95)	Time to first token (p95)	Prompt processing throughput	Generation throughput	Peak resident memory	Model size	Quality score	Constraint result
Q4 generic quantization`q4-generic`	Declared frontier · Eligible	View configuration for Q4 generic quantization {"configuration": {"attribution_stage": "quantization", "batch_size": 512, "context_size": 2048, "kleidiai": false, "runtime_tuned": false, "threads": 4, "ubatch_size": 128}, "deployment_argv": ["./.candidate-build/generic/bin/llama-server", "--model", "./.candidate-models/qwen2.5-1.5b-instruct-q4_0.gguf", "--threads", "4", "--threads-batch", "4", "--batch-size", "512", "--ubatch-size", "128", "--ctx-size", "2048", "--parallel", "1", "--n-gpu-layers", "0", "-lv", "4", "--host", "127.0.0.1", "--port", "18082"], "model": {"family": "Qwen2.5-1.5B-Instruct", "filename": "qwen2.5-1.5b-instruct-q4_0.gguf", "quantization": "Q4_0", "repository": "Qwen/Qwen2.5-1.5B-Instruct-GGUF", "revision": "91cad51170dc346986eccefdc2dd33a9da36ead9", "sha256": "dcd819ff094852c38faba6873d8ff0c9d51eadb2844539e52042ae5d647bbfdb", "size_bytes": 1066227232}, "runtime": {"name": "llama.cpp", "repository": "https://github.com/ggml-org/llama.cpp", "revision": "67b9b0e7f6ce45d929a4411907d3c48ec719e81c"}}	2,311.1251 ms	483.1131 ms	113.821 tok/s	35.0123 tok/s	1,966.4727 MiB	1,016.8335 MiB	0.8333	No declared constraint violations.
Q4 with KleidiAI`q4-kleidiai`	Declared frontier · Eligible	View configuration for Q4 with KleidiAI {"configuration": {"attribution_stage": "arm-kernel", "batch_size": 512, "context_size": 2048, "kleidiai": true, "runtime_tuned": false, "threads": 4, "ubatch_size": 128}, "deployment_argv": ["./.candidate-build/kleidiai/bin/llama-server", "--model", "./.candidate-models/qwen2.5-1.5b-instruct-q4_0.gguf", "--threads", "4", "--threads-batch", "4", "--batch-size", "512", "--ubatch-size", "128", "--ctx-size", "2048", "--parallel", "1", "--n-gpu-layers", "0", "-lv", "4", "--host", "127.0.0.1", "--port", "18083"], "model": {"family": "Qwen2.5-1.5B-Instruct", "filename": "qwen2.5-1.5b-instruct-q4_0.gguf", "quantization": "Q4_0", "repository": "Qwen/Qwen2.5-1.5B-Instruct-GGUF", "revision": "91cad51170dc346986eccefdc2dd33a9da36ead9", "sha256": "dcd819ff094852c38faba6873d8ff0c9d51eadb2844539e52042ae5d647bbfdb", "size_bytes": 1066227232}, "runtime": {"name": "llama.cpp", "repository": "https://github.com/ggml-org/llama.cpp", "revision": "67b9b0e7f6ce45d929a4411907d3c48ec719e81c"}}	2,299.4543 ms	470.4023 ms	114.448 tok/s	35.3764 tok/s	1,966.4844 MiB	1,016.8335 MiB	0.8333	No declared constraint violations.
Q4 with KleidiAI and 512-token micro-batch`q4-kleidiai-tuned`	Declared frontier · Eligible	View configuration for Q4 with KleidiAI and 512-token micro-batch {"configuration": {"attribution_stage": "runtime-tuning", "batch_size": 512, "context_size": 2048, "kleidiai": true, "runtime_tuned": true, "threads": 4, "ubatch_size": 512}, "deployment_argv": ["./.candidate-build/kleidiai/bin/llama-server", "--model", "./.candidate-models/qwen2.5-1.5b-instruct-q4_0.gguf", "--threads", "4", "--threads-batch", "4", "--batch-size", "512", "--ubatch-size", "512", "--ctx-size", "2048", "--parallel", "1", "--n-gpu-layers", "0", "-lv", "4", "--host", "127.0.0.1", "--port", "18084"], "model": {"family": "Qwen2.5-1.5B-Instruct", "filename": "qwen2.5-1.5b-instruct-q4_0.gguf", "quantization": "Q4_0", "repository": "Qwen/Qwen2.5-1.5B-Instruct-GGUF", "revision": "91cad51170dc346986eccefdc2dd33a9da36ead9", "sha256": "dcd819ff094852c38faba6873d8ff0c9d51eadb2844539e52042ae5d647bbfdb", "size_bytes": 1066227232}, "runtime": {"name": "llama.cpp", "repository": "https://github.com/ggml-org/llama.cpp", "revision": "67b9b0e7f6ce45d929a4411907d3c48ec719e81c"}}	2,307.7153 ms	469.9681 ms	131.4565 tok/s	35.0959 tok/s	1,966.4805 MiB	1,016.8335 MiB	0.8333	No declared constraint violations.
Q8 generic reference`q8-generic`	Selected · Baseline · Declared frontier · Eligible	View configuration for Q8 generic reference {"configuration": {"attribution_stage": "reference", "batch_size": 512, "context_size": 2048, "kleidiai": false, "runtime_tuned": false, "threads": 4, "ubatch_size": 128}, "deployment_argv": ["./.candidate-build/generic/bin/llama-server", "--model", "./.candidate-models/qwen2.5-1.5b-instruct-q8_0.gguf", "--threads", "4", "--threads-batch", "4", "--batch-size", "512", "--ubatch-size", "128", "--ctx-size", "2048", "--parallel", "1", "--n-gpu-layers", "0", "-lv", "4", "--host", "127.0.0.1", "--port", "18081"], "model": {"family": "Qwen2.5-1.5B-Instruct", "filename": "qwen2.5-1.5b-instruct-q8_0.gguf", "quantization": "Q8_0", "repository": "Qwen/Qwen2.5-1.5B-Instruct-GGUF", "revision": "91cad51170dc346986eccefdc2dd33a9da36ead9", "sha256": "d7efb072e7724d25048a4fda0a3e10b04bdef5d06b1403a1c93bd9f1240a63c8", "size_bytes": 1894532128}, "runtime": {"name": "llama.cpp", "repository": "https://github.com/ggml-org/llama.cpp", "revision": "67b9b0e7f6ce45d929a4411907d3c48ec719e81c"}}	2,231.9329 ms	545.3739 ms	102.6185 tok/s	38.7264 tok/s	3,437.5977 MiB	1,806.7666 MiB	0.875	No declared constraint violations.

08 · Reproduction

Verification and reproduction

Every result is tied to source files and SHA-256 hashes. The page has no generated timestamp and loads no external assets.

Verification details

Benchmark SHA-256: b95f2a2801069d1e1428faa7a3809583b26245ee7eb2e797d338cd7262d5267b
Recommendation SHA-256: e23b50ddd02ed72200417ddb146215c3aa995fb9c8c1a98faa64ee6ba925c7a4
Policy profiles SHA-256: 6fe9d423a5c41bd48c74fa255c19fb174c8d09e718ecd45d5ab9ff8d23508c37
Load sweep SHA-256: 923b3222440314644c5e87b326d7d7fb1a394b9d688262ffa636d6ef275e9dfd
Stability summary SHA-256: e3d1ee557005efb076ed1a34ba38fb52e3378b1eead8a8f745a85f25773712a3
Published report SHA-256: e2cce9ce8ed56f98c40996476a35289229574e01f154a4089a3531aa64ee5399
Evidence archive SHA-256: b5586878ccd214667911390f417db0417111ac2c31d163a2f5f55c4469aefeb2
Checksum manifest SHA-256: b9c7789e804d4228ac016aa9ebee3726776a668491b6384a242c13284e720a4f
Benchmark schema: 1.0
Report version: 1.1.0
Baseline ID: q8-generic
Selected ID: q8-generic

Selected server command

The command below reproduces only the selected server configuration. It was recorded from validated input; no flags or paths were inferred.

./.candidate-build/generic/bin/llama-server --model ./.candidate-models/qwen2.5-1.5b-instruct-q8_0.gguf --threads 4 --threads-batch 4 --batch-size 512 --ubatch-size 128 --ctx-size 2048 --parallel 1 --n-gpu-layers 0 -lv 4 --host 127.0.0.1 --port 18081

View full source metadata

Benchmark source details
Field	Value
Candidate evidence	{"q4-generic": {"artifacts": {"llama_summary": {"path": "candidates/q4-generic/llama-summary.json", "sha256": "04e8ad1043a3b155a42ac32a90a7dcde434bd6aac829cb540e883c8084831c42"}, "resource_measurement": {"path": "candidates/q4-generic/resource-measurement.json", "sha256": "3e87fd5ba9ffef213df355d185955b47d9907c59e6479fbedca375ceb73ce1e6"}, "server_evaluation": {"path": "candidates/q4-generic/server-evaluation.json", "sha256": "2f390312f2d0c8079ca6e86d6f74507a45648018cd023f1d50e271f7fec8e789"}, "server_evaluation_pass_1": {"path": "candidates/q4-generic/server-evaluation-pass-1.json", "sha256": "20b90cf334e211b2de4faaee98fe304f7a601e90026bd778d13249e20b3ec0b8"}, "server_evaluation_pass_2": {"path": "candidates/q4-generic/server-evaluation-pass-2.json", "sha256": "cdd29a077d46c05c26cade5dd1e74c4a8fc1db2d49a64d8f82dcda4a9fecaf6a"}, "server_stderr_pass_1": {"path": "candidates/q4-generic/server-pass-1.stderr.log", "sha256": "35f467ec737f5860c02b5a51e1b640bd34fc90fb8bf56cee6e4b94b844eb3baf"}, "server_stderr_pass_2": {"path": "candidates/q4-generic/server-pass-2.stderr.log", "sha256": "debbf7fe3683871cfbaa437c6dc6272746943105f43e24ba1b78aea98984f72b"}, "server_time_pass_1": {"path": "candidates/q4-generic/server-time-pass-1.txt", "sha256": "3cde7534471d26a13600a0d6a68799eeef3fcff6d3c5129083f52c9a6c4ca2d9"}, "server_time_pass_2": {"path": "candidates/q4-generic/server-time-pass-2.txt", "sha256": "d112bd9f197fcde159204138e132c3aca696e75643c53d3d148aa5a2e80df4a0"}, "throughput_command": {"path": "candidates/q4-generic/throughput-command.json", "sha256": "fdd7e4954c2c3d3e924dcc51b950bedcda74da8c15855d9d83a94b2301e449d5"}, "throughput_pass_1": {"path": "candidates/q4-generic/raw/pass-1.jsonl", "sha256": "9d545db9509886dd94a78de0421826b087eb3f61ca966fb2ff3fdc7919a56c48"}, "throughput_pass_2": {"path": "candidates/q4-generic/raw/pass-2.jsonl", "sha256": "5fca70848fc11e151dd486b61d90e6c89fffea7cb9f83fe680a012c07d3d765e"}, "throughput_settings": {"path": "candidates/q4-generic/settings.json", "sha256": "a0e58888c94cc186216af765357eca03d8a3753b92216efc1321a1985129bc18"}, "throughput_stderr_pass_1": {"path": "candidates/q4-generic/raw/pass-1.stderr.log", "sha256": "98b3e0afd84fa5c8a8d251891a3a2ed0441bd5ad54518e92023632faf93e4055"}, "throughput_stderr_pass_2": {"path": "candidates/q4-generic/raw/pass-2.stderr.log", "sha256": "492ccbb636b2eb243e0c1525c499fcf67a97ebaf87757375698f1e1e3bb96ec9"}}, "metric_sources": {"e2e_latency_ms_p95": {"artifact": "server_evaluation", "json_pointer": "/latency/e2e_latency_ms_p95"}, "generation_tps": {"artifact": "llama_summary", "json_pointer": "/tests/tg/tokens_per_second/median"}, "model_size_mib": {"artifact": "manifest", "json_pointer": "/candidates//model/size_bytes (divided by 1048576)"}, "peak_rss_mib": {"artifact": "resource_measurement", "json_pointer": "/maximum_resident_set_kbytes (divided by 1024)"}, "prompt_tps": {"artifact": "llama_summary", "json_pointer": "/tests/pp/tokens_per_second/median"}, "quality_score": {"artifact": "server_evaluation", "json_pointer": "/quality/score"}, "ttft_ms_p95": {"artifact": "server_evaluation", "json_pointer": "/latency/ttft_ms_p95"}}, "model": {"family": "Qwen2.5-1.5B-Instruct", "filename": "qwen2.5-1.5b-instruct-q4_0.gguf", "quantization": "Q4_0", "repository": "Qwen/Qwen2.5-1.5B-Instruct-GGUF", "revision": "91cad51170dc346986eccefdc2dd33a9da36ead9", "sha256": "dcd819ff094852c38faba6873d8ff0c9d51eadb2844539e52042ae5d647bbfdb", "size_bytes": 1066227232}}, "q4-kleidiai": {"artifacts": {"llama_summary": {"path": "candidates/q4-kleidiai/llama-summary.json", "sha256": "b95143c12f9ef2622eef31b041a0334c8f1461bc9fd76dc32b403f531f8a3460"}, "resource_measurement": {"path": "candidates/q4-kleidiai/resource-measurement.json", "sha256": "4b6e97b9f695fc5bc6a4b2e44560c19ae1c88169c718783474e67714edccea09"}, "server_evaluation": {"path": "candidates/q4-kleidiai/server-evaluation.json", "sha256": "7ff8571dcbd437d763cc196244a5b8df5e1c49ca6824898d8ebb548ae8a093df"}, "server_evaluation_pass_1": {"path": "candidates/q4-kleidiai/server-evaluation-pass-1.json", "sha256": "8571713dd511b4f40379dd38d77d8ee4a3993348382edac0e34a3ae679b0292e"}, "server_evaluation_pass_2": {"path": "candidates/q4-kleidiai/server-evaluation-pass-2.json", "sha256": "1290479aec1da03b4272f5b9eadfa277818c9b8b5ff45396cfa41cd5afccf67c"}, "server_stderr_pass_1": {"path": "candidates/q4-kleidiai/server-pass-1.stderr.log", "sha256": "7338bd441a8ba9afc7c1693f188e1e885bedb7becbf4ab5863ba8eee60514b8c"}, "server_stderr_pass_2": {"path": "candidates/q4-kleidiai/server-pass-2.stderr.log", "sha256": "9c9ebfbdd26452e35d6c55d2d08fb2c49be5f51be0d42ab1fb95576f5c58ea08"}, "server_time_pass_1": {"path": "candidates/q4-kleidiai/server-time-pass-1.txt", "sha256": "ebba9031a349f4094d32a428b853fcae56dbbcb2643b7a55ba68713a7410815a"}, "server_time_pass_2": {"path": "candidates/q4-kleidiai/server-time-pass-2.txt", "sha256": "c570ff52671d5d30fd4d5b1d5bc08b5e1baab01b98e79cb6458b5ba11588cfef"}, "throughput_command": {"path": "candidates/q4-kleidiai/throughput-command.json", "sha256": "5bd7daafc6e8c2da09ddbc3921ec7a76703b0c92e3bea0da3d048673dbf7a3f5"}, "throughput_pass_1": {"path": "candidates/q4-kleidiai/raw/pass-1.jsonl", "sha256": "1da5bf0b3f3cca835634619a3ea14a3d3b6dd8e5974ec6a8dfb2049121e4002e"}, "throughput_pass_2": {"path": "candidates/q4-kleidiai/raw/pass-2.jsonl", "sha256": "4061217d3f69ddf91852a7fcc84931f8b098dd758030692abd7087069c2345e4"}, "throughput_settings": {"path": "candidates/q4-kleidiai/settings.json", "sha256": "69c9e507a996bf87c8c7dd4203462598498edc12525b5695b02ea434e3931ccf"}, "throughput_stderr_pass_1": {"path": "candidates/q4-kleidiai/raw/pass-1.stderr.log", "sha256": "8d9e4b8f7efbbdf087e7d5d55a82eb94eafe78dbef9fe66de402dc524c0e6bf6"}, "throughput_stderr_pass_2": {"path": "candidates/q4-kleidiai/raw/pass-2.stderr.log", "sha256": "80cd4d7bce8238b1a0439131dcab2b2419928462b2cca4e1b4a0975efc39d1e5"}}, "metric_sources": {"e2e_latency_ms_p95": {"artifact": "server_evaluation", "json_pointer": "/latency/e2e_latency_ms_p95"}, "generation_tps": {"artifact": "llama_summary", "json_pointer": "/tests/tg/tokens_per_second/median"}, "model_size_mib": {"artifact": "manifest", "json_pointer": "/candidates//model/size_bytes (divided by 1048576)"}, "peak_rss_mib": {"artifact": "resource_measurement", "json_pointer": "/maximum_resident_set_kbytes (divided by 1024)"}, "prompt_tps": {"artifact": "llama_summary", "json_pointer": "/tests/pp/tokens_per_second/median"}, "quality_score": {"artifact": "server_evaluation", "json_pointer": "/quality/score"}, "ttft_ms_p95": {"artifact": "server_evaluation", "json_pointer": "/latency/ttft_ms_p95"}}, "model": {"family": "Qwen2.5-1.5B-Instruct", "filename": "qwen2.5-1.5b-instruct-q4_0.gguf", "quantization": "Q4_0", "repository": "Qwen/Qwen2.5-1.5B-Instruct-GGUF", "revision": "91cad51170dc346986eccefdc2dd33a9da36ead9", "sha256": "dcd819ff094852c38faba6873d8ff0c9d51eadb2844539e52042ae5d647bbfdb", "size_bytes": 1066227232}}, "q4-kleidiai-tuned": {"artifacts": {"llama_summary": {"path": "candidates/q4-kleidiai-tuned/llama-summary.json", "sha256": "05ec5eb7ab35fcbc5efff2f3843d4050c64074ba5bf9380c3eee1d0b1bd9fb03"}, "resource_measurement": {"path": "candidates/q4-kleidiai-tuned/resource-measurement.json", "sha256": "75e17b27f624f00c0ec597d23e74a7429294bc4e4ada57fc2df37c2edba6b910"}, "server_evaluation": {"path": "candidates/q4-kleidiai-tuned/server-evaluation.json", "sha256": "b240649cf666dcf2e5e8a121806282a96b77460c914868a3ddca99f7a658c4f3"}, "server_evaluation_pass_1": {"path": "candidates/q4-kleidiai-tuned/server-evaluation-pass-1.json", "sha256": "3a1e1559d30946d6d1edbb99edf6af716414cfcd1c4a320fefe0352e274ed61c"}, "server_evaluation_pass_2": {"path": "candidates/q4-kleidiai-tuned/server-evaluation-pass-2.json", "sha256": "0c8683dae11abe889caefff3687caa2b2000f95d6af5fb7c3a7c278feb27da1b"}, "server_stderr_pass_1": {"path": "candidates/q4-kleidiai-tuned/server-pass-1.stderr.log", "sha256": "0eea701e6ae7a432fdd901af2a6f6988e68558a99ec2dc3da7bc90e68c997c69"}, "server_stderr_pass_2": {"path": "candidates/q4-kleidiai-tuned/server-pass-2.stderr.log", "sha256": "b1e740f0c8df535772504ebf797f75b4ce09b718e5c3a9e5c11c9ec998a96403"}, "server_time_pass_1": {"path": "candidates/q4-kleidiai-tuned/server-time-pass-1.txt", "sha256": "0822190409d7b22a529d2ae770e70d680b158812508d15045edff07f27f0649a"}, "server_time_pass_2": {"path": "candidates/q4-kleidiai-tuned/server-time-pass-2.txt", "sha256": "8a5d84c7035c05c368a9eff00b686bd57489b36581b30e57ea2d44c19ec8b2bf"}, "throughput_command": {"path": "candidates/q4-kleidiai-tuned/throughput-command.json", "sha256": "f44706e79d6532205f5fd53abf00479995cab9abd86f0b6c6e4e7309bb481b12"}, "throughput_pass_1": {"path": "candidates/q4-kleidiai-tuned/raw/pass-1.jsonl", "sha256": "c4c36b7900ad4d5cefb4ed2e09109eaef51a60a9bdefbeb2a78df3a0e2c60853"}, "throughput_pass_2": {"path": "candidates/q4-kleidiai-tuned/raw/pass-2.jsonl", "sha256": "21bce7cdcbcc1cb8ec9599b0fbe76a62642dfbf72b3fcd97dadd7203a5a6dabc"}, "throughput_settings": {"path": "candidates/q4-kleidiai-tuned/settings.json", "sha256": "06698a361f4877d74c3ace0d1c33ad70a130525abd7b005d4b80a4b0287d844f"}, "throughput_stderr_pass_1": {"path": "candidates/q4-kleidiai-tuned/raw/pass-1.stderr.log", "sha256": "50b3cb922958d95b2322301eb0b83c6f93333365cb40fcc7b7884d3e11217637"}, "throughput_stderr_pass_2": {"path": "candidates/q4-kleidiai-tuned/raw/pass-2.stderr.log", "sha256": "e7702c6b5ac5a25fffb390a99571f46d5b4a4202a95f3fa35598634199371af5"}}, "metric_sources": {"e2e_latency_ms_p95": {"artifact": "server_evaluation", "json_pointer": "/latency/e2e_latency_ms_p95"}, "generation_tps": {"artifact": "llama_summary", "json_pointer": "/tests/tg/tokens_per_second/median"}, "model_size_mib": {"artifact": "manifest", "json_pointer": "/candidates//model/size_bytes (divided by 1048576)"}, "peak_rss_mib": {"artifact": "resource_measurement", "json_pointer": "/maximum_resident_set_kbytes (divided by 1024)"}, "prompt_tps": {"artifact": "llama_summary", "json_pointer": "/tests/pp/tokens_per_second/median"}, "quality_score": {"artifact": "server_evaluation", "json_pointer": "/quality/score"}, "ttft_ms_p95": {"artifact": "server_evaluation", "json_pointer": "/latency/ttft_ms_p95"}}, "model": {"family": "Qwen2.5-1.5B-Instruct", "filename": "qwen2.5-1.5b-instruct-q4_0.gguf", "quantization": "Q4_0", "repository": "Qwen/Qwen2.5-1.5B-Instruct-GGUF", "revision": "91cad51170dc346986eccefdc2dd33a9da36ead9", "sha256": "dcd819ff094852c38faba6873d8ff0c9d51eadb2844539e52042ae5d647bbfdb", "size_bytes": 1066227232}}, "q8-generic": {"artifacts": {"llama_summary": {"path": "candidates/q8-generic/llama-summary.json", "sha256": "5a64bedf1b1d295b1a56bf19cb57786282375cb51ab939805be4188720f74219"}, "resource_measurement": {"path": "candidates/q8-generic/resource-measurement.json", "sha256": "d184d590679da493d3ddd1156faafba286e8705697957994ed411bada765adea"}, "server_evaluation": {"path": "candidates/q8-generic/server-evaluation.json", "sha256": "e9b8d1359b838edc4621499c4fc506efaeecac64a39688052d4ac274322040c1"}, "server_evaluation_pass_1": {"path": "candidates/q8-generic/server-evaluation-pass-1.json", "sha256": "a30ef003fe8bdc0dc92db4f0ffac21653fd853bd0d689c315828efc5190731f1"}, "server_evaluation_pass_2": {"path": "candidates/q8-generic/server-evaluation-pass-2.json", "sha256": "629be14c1a7307d8bee12816d03a35002d75a495d4276845f2c99581abcc1fa1"}, "server_stderr_pass_1": {"path": "candidates/q8-generic/server-pass-1.stderr.log", "sha256": "8676225e38654ca474cb59c737b198823fc916a41df7fb1c7b8be1bfde0fd0af"}, "server_stderr_pass_2": {"path": "candidates/q8-generic/server-pass-2.stderr.log", "sha256": "3ea47bca3d9497c0d1f7e628e70d134b01b29da03f99f4305cf0c522979a75f0"}, "server_time_pass_1": {"path": "candidates/q8-generic/server-time-pass-1.txt", "sha256": "1d54a5a189731eff0880929d1434b1771f60eb6adb47f36e5cd079d948404d78"}, "server_time_pass_2": {"path": "candidates/q8-generic/server-time-pass-2.txt", "sha256": "ad5d7e91fea9e0839e1de3239bdba20d50e097024d44b8bfc221f6a418632513"}, "throughput_command": {"path": "candidates/q8-generic/throughput-command.json", "sha256": "3e34dc7419db28305e96ae686a16c9db5583e864e2cc5fd37e5d85d356eb1504"}, "throughput_pass_1": {"path": "candidates/q8-generic/raw/pass-1.jsonl", "sha256": "c270df39f12aef4e0643520ae3587e1e52e131358298a3229086bc4d44f53bf7"}, "throughput_pass_2": {"path": "candidates/q8-generic/raw/pass-2.jsonl", "sha256": "2f34053759527077229f1c06765e0cbea87147174a3b084e69893c4902000bca"}, "throughput_settings": {"path": "candidates/q8-generic/settings.json", "sha256": "5c344fb0afaa527bf17447eef11362d27188a791b861fff72c135ddef17a571e"}, "throughput_stderr_pass_1": {"path": "candidates/q8-generic/raw/pass-1.stderr.log", "sha256": "310fb48add49717b7a11b4cc84291f9d83ba65c3b253ede2163de5306c7a813e"}, "throughput_stderr_pass_2": {"path": "candidates/q8-generic/raw/pass-2.stderr.log", "sha256": "667ecb7ef19c636126b2b11544bc06a83500cae9295b7f7f5bbefff5ecab6a64"}}, "metric_sources": {"e2e_latency_ms_p95": {"artifact": "server_evaluation", "json_pointer": "/latency/e2e_latency_ms_p95"}, "generation_tps": {"artifact": "llama_summary", "json_pointer": "/tests/tg/tokens_per_second/median"}, "model_size_mib": {"artifact": "manifest", "json_pointer": "/candidates//model/size_bytes (divided by 1048576)"}, "peak_rss_mib": {"artifact": "resource_measurement", "json_pointer": "/maximum_resident_set_kbytes (divided by 1024)"}, "prompt_tps": {"artifact": "llama_summary", "json_pointer": "/tests/pp/tokens_per_second/median"}, "quality_score": {"artifact": "server_evaluation", "json_pointer": "/quality/score"}, "ttft_ms_p95": {"artifact": "server_evaluation", "json_pointer": "/latency/ttft_ms_p95"}}, "model": {"family": "Qwen2.5-1.5B-Instruct", "filename": "qwen2.5-1.5b-instruct-q8_0.gguf", "quantization": "Q8_0", "repository": "Qwen/Qwen2.5-1.5B-Instruct-GGUF", "revision": "91cad51170dc346986eccefdc2dd33a9da36ead9", "sha256": "d7efb072e7724d25048a4fda0a3e10b04bdef5d06b1403a1c93bd9f1240a63c8", "size_bytes": 1894532128}}}
Classification	`"canonical"`
Evaluation suite	`{"id": "paretopilot-qwen-behavior-v2", "sha256": "e49c16fba32fd65c947264aef4141026ab68b1fd415ef09eeea6e8ade9a545c7"}`
Experiment id	`"paretopilot-arm64-candidate-study-v1"`
Manifest	`{"filename": "manifest.json", "sha256": "7a8760e2950b7cc0c34c9377a57bee8dc41b58a229866f07e5cf55ffc1c91c82"}`
Model family	`{"name": "Qwen2.5-1.5B-Instruct", "repository": "Qwen/Qwen2.5-1.5B-Instruct-GGUF", "revision": "91cad51170dc346986eccefdc2dd33a9da36ead9"}`
Optimization library	`{"name": "KleidiAI", "repository": "https://github.com/ARM-software/kleidiai", "source_archive_sha256": "9348b969e042d8890a54b01a463dbe71f5a4c074b5329e9c26a85ef3b68aa19b", "source_archive_size_bytes": 2466038, "version": "v1.24.0"}`
Runtime	`{"name": "llama.cpp", "repository": "https://github.com/ggml-org/llama.cpp", "revision": "67b9b0e7f6ce45d929a4411907d3c48ec719e81c"}`
Source	`{"generated_at_utc": "2026-07-24T00:46:28Z", "repository": "agrovr/ParetoPilot", "revision": "8a9ddce0afa2272c4a4097fe87ef6f06cb7689a9", "run_attempt": 1, "run_id": "30055662526", "runner": {"architecture": "arm64", "cpu": "Neoverse-N2", "cpu_count": 4, "os": "Ubuntu 24.04"}, "workflow": ".github/workflows/candidate-study-arm64.yml"}`

How to reproduce this result

Verify the evidence archive and recorded source hashes.
Rebuild the benchmark data and published recommendation from the verified inputs.
Generate a new report and compare its SHA-256 hash.

The v1.1 report adds policy, load, and repeatability views while keeping the v1.0 replay available.

Compare measured Arm64 inference configurations.

Latency choice and serving capacity

Compare deployment priorities

Q8 generic reference

Q4 generic quantization

Q4 with KleidiAI and 512-token micro-batch

Q8 generic reference

Q4 with KleidiAI and 512-token micro-batch

Four configurations, compared step by step.

Q8 generic reference

Q4 generic quantization

Q4 with KleidiAI

Q4 with KleidiAI and 512-token micro-batch

Selected capacity point: 4 server slots / 4 clients

Q8 generic reference

Q4 with KleidiAI and 512-token micro-batch

Why this configuration was selected

1.00% decision tolerance

Measured tradeoffs

All deployment priorities

Q8 generic reference

Q4 generic quantization

Q4 with KleidiAI and 512-token micro-batch

Q4 with KleidiAI and 512-token micro-batch

Q8 generic reference

Measured load behavior

Verified test commands

Highest concurrency that met every limit

Declared service-level objective

Measured methodology

Repeatability across two passes

Latency versus generation throughput

Full evidence table

Verification and reproduction

Verification details

Selected server command

How to reproduce this result