MiniMax M2.5: 80.2% SWE-bench Verified and How to Actually Test It
The Number That Stopped Us Mid-Sprint
80.2% on SWE-bench Verified. That is the score MiniMax posted for M2.5, and it landed on Hacker News with enough signal that we pulled it into our model evaluation queue the same day.
SWE-bench Verified is not a soft benchmark. It runs real GitHub issues against real codebases, and the "Verified" qualifier means the task set has been manually confirmed to have unambiguous correct answers. Getting above 80% puts a model in the same conversation as frontier closed models. That is worth taking seriously.
What we will not do here is invent specs that have not been confirmed. Context size, parameter count, and pricing for M2.5 are not in the sources we have reviewed. We will say plainly what is confirmed and focus on how to evaluate the model on your own workload, which is the only number that actually matters for a deployment decision.
The M2.x Family Progression
MiniMax has been shipping fast. M2.1 positioned itself around complex task handling and multi-language code generation. M2.5 raises the SWE-bench anchor to 80.2%. M2.7 is confirmed open-source.
That progression matters for infrastructure decisions. If M2.7 is open-weight and performs comparably on your tasks, you may not need to route traffic through a proprietary API at all. But open-source licensing for M2.5 specifically has not been confirmed in what we have reviewed. Plan for API access on M2.5 until the licensing is clear.
One pattern we have seen across Chinese-origin model families: the open-weight release lags the API release by one or two versions. MiniMax appears to be following it. Verify before you build a self-hosted pipeline around M2.5.
What SWE-bench Verified Actually Tests
SWE-bench Verified gives a model a GitHub issue and surrounding repo context, then asks it to generate a patch that passes the test suite. It measures autonomous code repair across real open-source Python projects.
What it does not measure:
- Multi-file edits across a large, unfamiliar monorepo
- Code generation in languages outside the Python-heavy task set
- Long-context retrieval when the relevant function sits 50,000 tokens away
- Instruction following when requirements shift mid-task
- Security posture: the benchmark does not penalize a model for generating a correct-but-vulnerable patch
For agentic coding pipelines at production scale, SWE-bench is a useful signal, not a complete picture. A model that scores 80% there may still struggle on your TypeScript monorepo with 400 files, your internal DSL, or anything requiring tool-use chaining across more than three steps.
Running Your Own Evaluation Before You Commit
Before routing production agentic tasks to any new model, we run a small harness against three task categories: pure code generation, bug fix, and multi-step tool-use planning. Here is a minimal Python harness that targets an OpenAI-compatible endpoint:
import os, json, time
from openai import OpenAI
# MiniMax exposes an OpenAI-compatible endpoint.
# Confirm the base_url and model id from their API docs.
client = OpenAI(
base_url=os.environ["MINIMAX_BASE_URL"],
api_key=os.environ["MINIMAX_API_KEY"],
)
TASKS = [
{
"id": "codegen_typescript",
"prompt": (
"Write a TypeScript function that accepts an array of "
"{id: string, score: number} and returns the top-N by score, "
"typed correctly, with a unit test using vitest."
),
"eval_hint": "must compile; must include vitest test",
},
{
"id": "bug_fix_python",
"prompt": (
"Find and fix the bug:\n\n"
"def merge_sorted(a, b):\n"
" result = []\n"
" i, j = 0, 0\n"
" while i < len(a) and j < len(b):\n"
" if a[i] < b[j]:\n"
" result.append(a[i]); i += 1\n"
" else:\n"
" result.append(b[j]); j += 1\n"
" return result # drops the tail\n"
),
"eval_hint": "must append remaining elements from a and b",
},
{
"id": "tool_use_planning",
"prompt": (
"You have a tool search_repo(query: str) -> list[str]. "
"Plan, in pseudocode, the steps an agent should take to find all "
"usages of deprecated function get_user_by_email and replace them "
"with fetch_user(email=...)."
),
"eval_hint": "should include search, read, edit, verify steps in order",
},
]
results = []
for task in TASKS:
start = time.monotonic()
resp = client.chat.completions.create(
model="minimax-m2.5", # confirm exact model id in their docs
messages=[{"role": "user", "content": task["prompt"]}],
temperature=0.2,
)
elapsed = time.monotonic() - start
results.append({
"id": task["id"],
"latency_s": round(elapsed, 2),
"tokens_in": resp.usage.prompt_tokens,
"tokens_out": resp.usage.completion_tokens,
"output_preview": resp.choices[0].message.content[:500],
"eval_hint": task["eval_hint"],
})
print(f"[{task['id']}] {elapsed:.1f}s | {resp.usage.completion_tokens} tok out")
with open("m25_eval_results.json", "w") as f:
json.dump(results, f, indent=2)
Run this against M2.5 and at least one model you already trust in production. Compare latency, token counts, and output quality on tasks that match your actual workload. The model that wins on your tasks is the one to deploy, regardless of the benchmark headline.
Fitting M2.5 Into an Agentic Coding Pipeline
For AI agent workloads, the question is not just quality but where the model fits in the task graph.
High-SWE-bench models are strong candidates for the "execute" node: the step that actually writes or edits code after a planner has scoped the task. They are not automatically good at planning, tool selection, or context compression. Those roles may suit a model with stronger instruction following or more predictable structured output.
A pattern we have used: a fast, cheap model at the planner layer that decomposes a feature request into discrete file-level tasks, then a high-SWE-bench model at the executor layer. Planner output is typically 200 to 400 tokens. Executor handles 1,000 to 4,000 token code blocks. That split aligns cost and capability without routing everything through the expensive model.
Watch for these failure modes at production depth:
- Hallucinated imports: syntactically correct code that references a library not in the project
- Over-confidence on ambiguous specs: models trained heavily on code tend to pick an interpretation and run, rather than surfacing ambiguity
- Context drop on long chains: per-step pass rates often degrade by task four or five in a multi-step sequence; monitor those individually, not just end-to-end success
MiniMax M2.5 vs. GLM 4.7 and Other Open-Weight Competitors
Hacker News put MiniMax M2.5 and GLM 4.7 in the same conversation. GLM 4.7 comes from Tsinghua's THUDM group. Both are serious engineering efforts.
We do not have confirmed head-to-head benchmark numbers for GLM 4.7 vs. M2.5 from the sources we have reviewed, so we will not fabricate a comparison table. What we can say: GLM 4.7 has been positioned as a strong multi-language open-weight model. If self-hosted deployment is the requirement, GLM 4.7 may have clearer licensing at the performance tier you need, since MiniMax's confirmed open-weight release is M2.7, not M2.5.
The practical decision tree:
- Need open weights now for self-hosting: look at GLM 4.7 and MiniMax M2.7, not M2.5
- Willing to use a hosted API and want the highest confirmed SWE-bench score from this family: M2.5 is the current anchor
- Evaluating for non-Python codebases: run both models against your actual language stack; SWE-bench is Python-heavy and scores do not transfer cleanly to TypeScript, Go, or Rust workloads
One thing worth tracking: the gap between M2.5 and M2.7 in terms of capability. If MiniMax follows the pattern where open-weight releases trail proprietary ones by meaningful margin, M2.7 may underperform M2.5 on SWE-bench. If the gap is small, the case for self-hosting M2.7 gets much stronger.
Next Steps
If you want help integrating MiniMax M2.5 or any open-weight coding model into a production agentic pipeline, including evaluation harness design, latency benchmarking, and cost modeling against your task volume, reach out to us at KeMeT Tech.