Gemma-4-12B-it-heretic-QAT (思考)(Thinking)

gemma-4-12B-it-heretic-QAT-Q4_0.gguf · Q4_0 · 6.4 GB
🧠 MoE (4B active)📦 6.4 GB⚡ In 1522 t/s · Out 126.3 t/s🎮 RTX 5070 Ti 16GB + 128GB RAM📅 2026-06-21
在 HuggingFace 查看模型 →View on HuggingFace →
加权总分Weighted Total
90.3
加权原始分 = TC×0.3 + BF×0.3 + HA×0.4Weighted raw score = TC×0.3 + BF×0.3 + HA×0.4
实用得分Effective Score
85.3
−5 重试扣分−5 retry penalty
ToolCall-15
97
15/15 通过 · 100% 🏆15/15 passed · 100% 🏆15/15 passed · 100% 🏆 · 重试Retry -4
BugFind-15
88
13/15 通过 · 87%13/15 passed · 87% · 重试Retry -1
HermesAgent-20
87
17/20 通过 · 85%17/20 passed · 85%
📊 计分规则Scoring Rules能力上限Max Score = 加权原始分 = ToolCall×0.3 + BugFind×0.3 + HermesAgent×0.4= Weighted raw score = ToolCall×0.3 + BugFind×0.3 + HermesAgent×0.4
实用得分Effective Score = 能力上限 − 重试扣分(每题首次通过不扣分,重试后通过每重试1次扣1分,重试后仍失败不参与扣分)。Max Score − retry penalty (first pass = no penalty, each retry = +1pt, failed retries excluded).

📋 全部测试结果All Test Results

全部 (50)All (50)
ToolCall (15)
BugFind (15)
HermesAgent (20)
❌ 失败 (12)❌ Failed (12)
#题目Question测试包Pack难度Difficulty结果Result得分Score重试Retrysandbox耗时Time失败类型Failure Type点评Comment

🔍 错题分析Error Analysis

🌊 📊 模型评估总结Model Evaluation Summary

优势Strengths

  • HermesAgent 84.8 分全场最高之一,Agent 场景能力突出HermesAgent 84.8, one of the highest scores; outstanding Agent scenario capabilities.
  • ToolCall 96.7 分,仅差 1 题满分,工具调用能力接近顶尖ToolCall 96.7, only 1 question away from full marks; tool calling capability near top-tier.
  • 唯一通过 HA-04(Docker 修复回忆)的之一,记忆召回能力独特One of the few to pass HA-04 (Docker fix recall), with unique memory retrieval capabilities

⚠️ 不足Weaknesses

  • Trap 题(BF-03)仍然失败,BF-10 仅 partial(60分)Trap question (BF-03) still failed; BF-10 only partial (60 points).
  • BugFind 86 分,低于 Qwen3.6-27B 无思考版(84.2)和思考版(93.3)BugFind 86, lower than Qwen3.6-27B non-thinking version (84.2) and thinking version (93.3).
  • HA-10 技能发现、HA-16 消息投递均失败HA-10 skill discovery and HA-16 message delivery both failed.

📋 测试环境Test Environment

  • 硬件Hardware — RTX 5070 Ti 16GB + 128GB RAM,MoE模型Models部分专家层offload到CPUexpert layers partially offloaded to CPU
  • 推理后端Inference Backend — llama.cpp
  • 测试包Pack — ToolCall-15 / BugFind-15 / HermesAgent-20(共 50 题)— ToolCall-15 / BugFind-15 / HermesAgent-20 (total 50 questions)
  • 模型下载Model DownloadHF: SC117

Gemma-4-12B 90.3 分, 85.3 分( 5 次)。12B 稠密,6.4GB 体积,126.3 t/s 输出速度——Gemma-4 系列中最轻量。QAT 量化技术让 12B 参数在 Q4_0 精度下仍保持不错性能。TC=97、BF=88、HA=87 三项均衡,仅 5 次。Gemma-4-12B: 90.3, 85.3 (5 runs). 12B dense, 6.4GB size, 126.3 t/s output speed—the lightest in the Gemma-4 series. QAT quantization maintains solid performance for the 12B parameters at Q4_0 precision. Balanced TC=97, BF=88, HA=87 with only 5 runs.

一句话评价 TL;DR:Gemma-4 家族最轻量成员——6.4GB、126t/s、85.3 分,QAT 量化的性价比之选。TL;DR: The lightest member of the Gemma-4 family—6.4GB, 126t/s, 85.3 points, a cost-effective choice with QAT quantization.