Mellum2.1 12B A2.5B Thinking

Solving coding problems. Version 6 only; published pass rates can vary with generation settings.

Benchmark efficiency: 10.159 published score points per GB of selected weights. This ratio is not an inference-speed measure; check the test settings to see whether the score covers the selected quantization.

Test settings: JetBrains publisher evaluation of the official BF16 source checkpoint on LiveCodeBench v6 in thinking mode with greedy decoding.

Publisher-run source-checkpoint pass@1, not an independent result and not a benchmark of the selected Q4_K_M file. JetBrains states that all comparison models used the same internal evaluation pipeline, but the card does not pin that pipeline's code revision. The 8.071 GB file excludes runtime, KV cache and context memory and does not establish device speed or compatibility.

Choosing and calling functions. Published BFCL v4 scores; tool environments and settings differ.

Benchmark efficiency: 7.719 published score points per GB of selected weights. This ratio is not an inference-speed measure; check the test settings to see whether the score covers the selected quantization.

Test settings: JetBrains publisher evaluation of the official BF16 source checkpoint in thinking mode. The reported BFCL v4 value is the macro-average of the v1, v2, v3, web-search and memory subtasks.

Publisher-run source-checkpoint result, not an independent evaluation and not a benchmark of the selected Q4_K_M file. Tool schemas, parser behavior and runtime configuration can change function-calling results.

Fixing real software issues with tools. Agent setup and external tools affect these scores. File size covers model weights only.

Benchmark efficiency: 5.823 published score points per GB of selected weights. This ratio is not an inference-speed measure; check the test settings to see whether the score covers the selected quantization.

Test settings: JetBrains publisher evaluation of the official BF16 source checkpoint on SWE-bench Verified with Pi v0.73.1, shell and file tools, Mellum2.1's default temperature 1.0, a 114K-token context and up to 16K generated tokens per turn.

Publisher-run source-checkpoint resolved rate, not an independent result and not a benchmark of the selected Q4_K_M file. Agent scaffold, tools, timeouts, sampling and context budget materially affect this score. The Q4 file size alone does not establish compatible hardware or local-agent speed.

Agentic terminal-use task success. Publisher-reported percentages are retained, but agent scaffold, tools, timeouts and reasoning budget can materially affect results.

Benchmark efficiency: 2.156 published score points per GB of selected weights. This ratio is not an inference-speed measure; check the test settings to see whether the score covers the selected quantization.

Test settings: JetBrains publisher evaluation of the official BF16 source checkpoint on Terminal-Bench 2.1 with Pi v0.73.1, shell and file tools, Mellum2.1's default temperature 1.0, a 114K-token context and up to 16K generated tokens per turn.

Publisher-run source-checkpoint resolved rate, not an independent result and not a benchmark of the selected Q4_K_M file. The published percentage is specific to the stated agent harness and token budget; no local Q4 speed result is claimed.