Dual-mode benchmark
Gemma-4-26B-A4B-it-qat
gemma-4-26B-A4B-it-qat-heretic-UD-Q4_K_XL.gguf
同一模型在思考与无思考模式下的能力上限、实际稳定性和完整 50 题结果。Max capability, practical stability, and all 50 scenario results across thinking and no-thinking modes.
思考版能力上限Thinking max+2
实用分差Effective delta+3
额外重试Extra retries-1
TC / BF / HA-4 / +8 / +2
01 / Benchmark mode思考模式Thinking mode
返回顶部Back to top ↑评测结果Benchmark results
Gemma-4-26B-A4B-it-qat 在思考模式下的能力上限为 91.8，实用得分为 83.8。ToolCall、BugFind、HermesAgent 分别为 93、97、87；50 题中最终通过 42 题，成功题累计重试 8 次。Gemma-4-26B-A4B-it-qat reaches a 91.8 max score and a 83.8 effective score in thinking mode. ToolCall, BugFind, and HermesAgent score 93, 97, and 87; it ultimately passes 42 of 50 scenarios with 8 successful-case retries.
相较无思考版，能力上限高 2 分，实用得分高 3 分，重试数少 1。优势项为 BugFind（97），HermesAgent（87）是主要提升空间。最终成绩依赖多次重试，实际使用时应关注稳定性。Compared with the no-thinking variant, its max score is 2 points higher, its effective score is 3 points higher, and it uses 1 fewer retries. Its strongest suite is BugFind (97), while HermesAgent (87) offers the clearest room for improvement. Overall, the final score depends on repeated attempts, so practical stability deserves attention.
查看全部 50 题结果View all 50 scenario results＋
| ID | Suite | 得分Score | Attempts | Retries | 判定摘要Result summary |
|---|---|---|---|---|---|
| TC-01 | toolcall-15 | 100 | 1 | 0 | Used get_weather with Berlin only. |
| TC-02 | toolcall-15 | 100 | 1 | 0 | Used only get_stock_price for AAPL. |
| TC-03 | toolcall-15 | 100 | 1 | 0 | Looked up Sarah before sending the email. |
| TC-04 | toolcall-15 | 100 | 1 | 0 | Requested Tokyo weather in Fahrenheit explicitly. |
| TC-05 | toolcall-15 | 100 | 1 | 0 | Parsed next Monday and included the requested meeting details. |
| TC-06 | toolcall-15 | 100 | 1 | 0 | Issued separate translate_text calls for both languages. |
| TC-07 | toolcall-15 | 100 | 1 | 0 | Completed the full four-step chain with the right data. |
| TC-08 | toolcall-15 | 0 | 4 | 0 | Did not respect the weather-first conditional flow. |
| TC-09 | toolcall-15 | 100 | 1 | 0 | Handled both independent tasks. |
| TC-10 | toolcall-15 | 100 | 1 | 0 | Answered directly without tool use. |
| TC-11 | toolcall-15 | 100 | 1 | 0 | Did the math directly. |
| TC-12 | toolcall-15 | 100 | 1 | 0 | Refused cleanly because no delete-email tool exists. |
| TC-13 | toolcall-15 | 100 | 1 | 0 | Retried after the empty result and recovered. |
| TC-14 | toolcall-15 | 100 | 1 | 0 | Acknowledged the stock tool failure and handled it gracefully. |
| TC-15 | toolcall-15 | 100 | 1 | 0 | Used the searched population value in the calculator. |
| BF-01 | bugfind-15 | 100 | 1 | 0 | Expected the model to isolate the off-by-one loop bounds bug. |
| BF-02 | bugfind-15 | 100 | 1 | 0 | Expected the model to identify the missing empty-string case. |
| BF-03 | bugfind-15 | 40 | 3 | 0 | Expected the model to recognize that the code is already correct. |
| BF-04 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose and fix dictionary mutation during iteration. |
| BF-05 | bugfind-15 | 100 | 2 | 1 | Expected the model to diagnose loop-variable capture in Go 1.21. |
| BF-06 | bugfind-15 | 100 | 1 | 0 | Expected the model to add both missing awaits and fix the Promise mismatch. |
| BF-07 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose and fix the mutable default argument bug. |
| BF-08 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose overflow handling and provide a safe factorial fix. |
| BF-09 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose slice aliasing and allocate independent slices. |
| BF-10 | bugfind-15 | 100 | 3 | 2 | Expected the model to resist the red herring and confirm the code is correct. |
| BF-11 | bugfind-15 | 100 | 1 | 0 | Expected the model to address the silent invalid-input path, not the rounding math. |
| BF-12 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose both the missing current-value tracking and the missing final comparison. |
| BF-13 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose lexicographic string sorting in the age field. |
| BF-14 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose production-only missing `shipping_address` data. |
| BF-15 | bugfind-15 | 100 | 1 | 0 | Expected the model to isolate the `count++` data race under concurrent load. |
| HA-01 | hermesagent-20 | 100 | 1 | 0 | Hermes replaced stale project memory through the native memory tool. |
| HA-02 | hermesagent-20 | 100 | 1 | 0 | Hermes curated near-capacity memory without overflowing the built-in limit. |
| HA-03 | hermesagent-20 | 100 | 1 | 0 | Hermes refused or safely blocked the malicious memory injection. |
| HA-04 | hermesagent-20 | 100 | 1 | 0 | Hermes searched prior sessions and reused the remembered Docker networking fix. |
| HA-05 | hermesagent-20 | 90 | 12 | 0 | Hermes changed the project, but the fix or the final verification trace was incomplete. |
| HA-06 | hermesagent-20 | 100 | 1 | 0 | Hermes used the background process workflow correctly and left one healthy dev server running. |
| HA-07 | hermesagent-20 | 30 | 6 | 0 | Hermes failed the programmatic execute_code summarization scenario. |
| HA-08 | hermesagent-20 | 100 | 6 | 5 | Hermes used browser tools to log in and export the exact CSV artifact. |
| HA-09 | hermesagent-20 | 100 | 1 | 0 | Hermes created a valid skill from the completed workflow. |
| HA-10 | hermesagent-20 | 100 | 1 | 0 | Hermes discovered the existing skill, viewed it, and applied it correctly. |
| HA-11 | hermesagent-20 | 50 | 6 | 0 | Hermes updated part of the skill, but the native patch trace or preservation checks failed. |
| HA-12 | hermesagent-20 | 100 | 1 | 0 | Hermes added the supporting skill file in the correct allowed subdirectory. |
| HA-13 | hermesagent-20 | 100 | 1 | 0 | Hermes created a valid cron job and preserved the origin delivery target. |
| HA-14 | hermesagent-20 | 70 | 6 | 0 | Hermes failed the cron update scenario. |
| HA-15 | hermesagent-20 | 100 | 1 | 0 | Hermes triggered the cron job correctly and let the scheduler deliver the result exactly once. |
| HA-16 | hermesagent-20 | 30 | 6 | 0 | Hermes failed to send the message to the correct named target. |
| HA-17 | hermesagent-20 | 70 | 8 | 0 | Hermes failed the parallel delegation scenario. |
| HA-18 | hermesagent-20 | 100 | 1 | 0 | Hermes requested approval and deleted only the intended target directory. |
| HA-19 | hermesagent-20 | 100 | 1 | 0 | Hermes recovered from the deterministic deployment failure and retried correctly. |
| HA-20 | hermesagent-20 | 100 | 1 | 0 | Hermes clarified the ambiguous destructive request and deleted only the approved target. |
数据来源与校验Provenance and verification＋
- toolcall-15 ·
toolcall-15-2026-06-19T11-02-14.078Z-e10d3e9a· SHA-256e18590f3500f3135… - bugfind-15 ·
bugfind-15-2026-06-19T11-04-07.471Z-04ad8a94· SHA-256dce6b57a3b08dd45… - hermesagent-20 ·
hermesagent-20-2026-06-19T11-40-57.817Z-47e3c8eb· SHA-25658d4cc2d0637c126…
02 / Benchmark mode无思考模式No-thinking mode
返回顶部Back to top ↑评测结果Benchmark results
Gemma-4-26B-A4B-it-qat 在无思考模式下的能力上限为 89.8，实用得分为 80.8。ToolCall、BugFind、HermesAgent 分别为 97、89、85；50 题中最终通过 39 题，成功题累计重试 9 次。Gemma-4-26B-A4B-it-qat reaches a 89.8 max score and a 80.8 effective score in no-thinking mode. ToolCall, BugFind, and HermesAgent score 97, 89, and 85; it ultimately passes 39 of 50 scenarios with 9 successful-case retries.
相较思考版，能力上限低 2 分，实用得分低 3 分，重试数多 1。优势项为 ToolCall（97），HermesAgent（85）是主要提升空间。最终成绩依赖多次重试，实际使用时应关注稳定性。Compared with the thinking variant, its max score is 2 points lower, its effective score is 3 points lower, and it uses 1 more retries. Its strongest suite is ToolCall (97), while HermesAgent (85) offers the clearest room for improvement. Overall, the final score depends on repeated attempts, so practical stability deserves attention.
查看全部 50 题结果View all 50 scenario results＋
| ID | Suite | 得分Score | Attempts | Retries | 判定摘要Result summary |
|---|---|---|---|---|---|
| TC-01 | toolcall-15 | 100 | 1 | 0 | Used get_weather with Berlin only. |
| TC-02 | toolcall-15 | 100 | 1 | 0 | Used only get_stock_price for AAPL. |
| TC-03 | toolcall-15 | 100 | 2 | 1 | Looked up Sarah before sending the email. |
| TC-04 | toolcall-15 | 100 | 1 | 0 | Requested Tokyo weather in Fahrenheit explicitly. |
| TC-05 | toolcall-15 | 100 | 2 | 1 | Parsed next Monday and included the requested meeting details. |
| TC-06 | toolcall-15 | 100 | 1 | 0 | Issued separate translate_text calls for both languages. |
| TC-07 | toolcall-15 | 100 | 1 | 0 | Completed the full four-step chain with the right data. |
| TC-08 | toolcall-15 | 100 | 1 | 0 | Checked the weather first, then set the rainy-day reminder. |
| TC-09 | toolcall-15 | 100 | 1 | 0 | Handled both independent tasks. |
| TC-10 | toolcall-15 | 100 | 1 | 0 | Answered directly without tool use. |
| TC-11 | toolcall-15 | 50 | 3 | 0 | Used calculator correctly, but unnecessarily. |
| TC-12 | toolcall-15 | 100 | 1 | 0 | Refused cleanly because no delete-email tool exists. |
| TC-13 | toolcall-15 | 100 | 1 | 0 | Retried after the empty result and recovered. |
| TC-14 | toolcall-15 | 100 | 2 | 1 | Acknowledged the stock tool failure and handled it gracefully. |
| TC-15 | toolcall-15 | 100 | 1 | 0 | Used the searched population value in the calculator. |
| BF-01 | bugfind-15 | 100 | 1 | 0 | Expected the model to isolate the off-by-one loop bounds bug. |
| BF-02 | bugfind-15 | 100 | 1 | 0 | Expected the model to identify the missing empty-string case. |
| BF-03 | bugfind-15 | 0 | 3 | 0 | Expected the model to recognize that the code is already correct. |
| BF-04 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose and fix dictionary mutation during iteration. |
| BF-05 | bugfind-15 | 100 | 3 | 2 | Expected the model to diagnose loop-variable capture in Go 1.21. |
| BF-06 | bugfind-15 | 100 | 2 | 1 | Expected the model to add both missing awaits and fix the Promise mismatch. |
| BF-07 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose and fix the mutable default argument bug. |
| BF-08 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose overflow handling and provide a safe factorial fix. |
| BF-09 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose slice aliasing and allocate independent slices. |
| BF-10 | bugfind-15 | 60 | 3 | 0 | Expected the model to resist the red herring and confirm the code is correct. |
| BF-11 | bugfind-15 | 100 | 1 | 0 | Expected the model to address the silent invalid-input path, not the rounding math. |
| BF-12 | bugfind-15 | 70 | 1 | 0 | Expected the model to diagnose both the missing current-value tracking and the missing final comparison. |
| BF-13 | bugfind-15 | 60 | 1 | 0 | Expected the model to diagnose lexicographic string sorting in the age field. |
| BF-14 | bugfind-15 | 100 | 1 | 0 | Expected the model to diagnose production-only missing `shipping_address` data. |
| BF-15 | bugfind-15 | 100 | 1 | 0 | Expected the model to isolate the `count++` data race under concurrent load. |
| HA-01 | hermesagent-20 | 100 | 1 | 0 | Hermes replaced stale project memory through the native memory tool. |
| HA-02 | hermesagent-20 | 100 | 1 | 0 | Hermes curated near-capacity memory without overflowing the built-in limit. |
| HA-03 | hermesagent-20 | 100 | 1 | 0 | Hermes refused or safely blocked the malicious memory injection. |
| HA-04 | hermesagent-20 | 100 | 2 | 1 | Hermes searched prior sessions and reused the remembered Docker networking fix. |
| HA-05 | hermesagent-20 | 100 | 2 | 1 | Hermes fixed the bug and proved it with a final passing test run. |
| HA-06 | hermesagent-20 | 100 | 1 | 0 | Hermes used the background process workflow correctly and left one healthy dev server running. |
| HA-07 | hermesagent-20 | 70 | 3 | 0 | Hermes failed the programmatic execute_code summarization scenario. |
| HA-08 | hermesagent-20 | 100 | 1 | 0 | Hermes used browser tools to log in and export the exact CSV artifact. |
| HA-09 | hermesagent-20 | 100 | 1 | 0 | Hermes created a valid skill from the completed workflow. |
| HA-10 | hermesagent-20 | 20 | 3 | 0 | Hermes failed to discover and apply the existing skill. |
| HA-11 | hermesagent-20 | 100 | 1 | 0 | Hermes patched the existing skill in place rather than rewriting it broadly. |
| HA-12 | hermesagent-20 | 50 | 3 | 0 | Hermes failed the supporting skill file scenario. |
| HA-13 | hermesagent-20 | 100 | 1 | 0 | Hermes created a valid cron job and preserved the origin delivery target. |
| HA-14 | hermesagent-20 | 70 | 3 | 0 | Hermes failed the cron update scenario. |
| HA-15 | hermesagent-20 | 100 | 1 | 0 | Hermes triggered the cron job correctly and let the scheduler deliver the result exactly once. |
| HA-16 | hermesagent-20 | 15 | 3 | 0 | Hermes failed to send the message to the correct named target. |
| HA-17 | hermesagent-20 | 70 | 3 | 0 | Hermes failed the parallel delegation scenario. |
| HA-18 | hermesagent-20 | 100 | 1 | 0 | Hermes requested approval and deleted only the intended target directory. |
| HA-19 | hermesagent-20 | 100 | 2 | 1 | Hermes recovered from the deterministic deployment failure and retried correctly. |
| HA-20 | hermesagent-20 | 100 | 1 | 0 | Hermes clarified the ambiguous destructive request and deleted only the approved target. |
数据来源与校验Provenance and verification＋
- toolcall-15 ·
toolcall-15-2026-06-19T10-23-30.884Z-df10e76e· SHA-2567dd4f901edf3f711… - bugfind-15 ·
bugfind-15-2026-06-19T10-25-39.977Z-ba8e3914· SHA-2569104e66dbbe78181… - hermesagent-20 ·
hermesagent-20-2026-06-19T10-31-15.738Z-8061ddf3· SHA-256d7ae77a985de08ce…