Coding agents with an engine map add phone LLM support more reliably

A preprint describes EngineAgent, a set of AI agents that adapts on-device inference engines to new language models. It passed 9 of 12 tasks against 4 for the best general coding agent. The benchmark is small and the authors built it themselves.

7 min readIIBenchmarkedPreprint, not peer reviewed
Save
Figure 9 from the paper "EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities"
Table 3 (overall results) and Figure 9 (failure breakdown across the 12 tasks) together show the headline gap. The paper is CC BY, so reuse is allowed with credit. Figure 9 from Zhengxiang Huang et al., 2026, arXiv:2611.14356. CC BY.

What the study shows

An on-device inference engine is the software that runs a language model on a phone. Examples include MNN, llama.cpp, MLC-LLM and ExecuTorch. Each new model family tends to bring new parts, such as a different attention mechanism, and each part has to be wired through the engine and written again for each type of chip: the main processor (CPU), the graphics chip (GPU) and the AI accelerator (NPU). The authors call this work model enablement. They report that adding Qwen3.5-2B to MNN touched 42 files and more than 4,200 lines of code. They also report that two of the four engines they surveyed did not support Gemma4-E2B-it, and three did not support MiniCPM-V 4.6.

The paper is a preprint and has not been peer reviewed. The authors are at Shanghai Jiao Tong University and Alibaba Group. Alibaba develops MNN, one of the engines tested.

EngineAgent prepares two things before any new model arrives. The first is an index of the engine's code, sorted into nine layers. The layers run from chat templates and tokenisation (splitting text into units the model reads) down to hardware-specific kernels, the low-level routines that do the arithmetic on a given chip. The second is a set of build, test and debug procedures for each hardware backend. These are checked by building and running models the engine already supports on a real phone. When a new model arrives, three agents take over. A builder compares the new model with similar supported ones and makes changes layer by layer. A debugger diagnoses failures. An auditor runs layer-by-layer tests and then a full test on the device.

The benchmark has 12 tasks: three models (Qwen3.5-2B, Gemma4-E2B-it and MiniCPM-V 4.6) on four engine and hardware pairs. The pairs are MNN on CPU, llama.cpp on a Vulkan GPU, MLC-LLM on an OpenCL GPU and ExecuTorch on a Qualcomm NPU. A task passed only if the code compiled, ran on a phone, and produced output scores (logits) that matched a PyTorch reference on a desktop computer within a set tolerance. Where the engine already supported the model, the authors reset the engine to the version just before support was added. They blocked access to later versions and to web search.

The authors report that EngineAgent passed 9 of 12 tasks with either DeepSeek-V4-Flash or GPT-5.6 Sol as its underlying model. Codex with GPT-5.6 Sol passed 4. Three baselines using DeepSeek-V4-Flash did worse: OpenCode passed 1, OpenCode with added code-graph and failure-tracing tools passed 1, and SWE-Agent passed none. Each run was capped at five hours.

With DeepSeek-V4-Flash, EngineAgent averaged 3.82 hours and $1.53 in API cost per task. The comparable baselines took 4.78 to 4.91 hours and cost $2.11 to $2.44. With GPT-5.6 Sol, EngineAgent averaged 4.16 hours and $11.29, against 4.82 hours and $14.90 for Codex.

The authors also removed each component in turn. Without the code index, success fell from 9 to 3 of 12, and exploration turns rose from 42 to 271. Without the hardware procedures, success fell to 5 of 12, and compilation and debugging failures rose from 11 to 83. Building the index and procedures cost $0.66 to $2.68 per engine. Updating them averaged about $0.07 per code commit across five MNN commits.

Our reading

In our reading, the main lesson is about preparation. Neither of EngineAgent's two components is clever on its own. One is a map of where things live in the engine. The other is a set of build-and-test recipes known to work on a real phone. Both are built before the hard task begins. The removal tests suggest each accounts for a large share of the gain. The failure breakdown fits this: EngineAgent had no missing-implementation failures and no deployment failures, which are the two kinds of error the components target.

The gap between 9 of 12 and 4 of 12 is large, but it rests on 12 tasks and, as far as we can tell, one run each. A difference of a few tasks could change with another run. The comparison with matched underlying models helps. It suggests the scaffolding around the model, not a stronger model, explains most of the difference. Even so, the baselines were general-purpose tools that were not tuned for this job.

On our four-level readiness scale we rate this level II, Benchmarked, and only just. The tests run on real phones against named baselines, but the benchmark is the authors' own and has not been published. We found no released code, index, procedures or benchmark, so level III is not met.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

II
Benchmarked
LevelStatusEvidence
I  Lab onlyMetA working prototype system is described and tested end-to-end on real phones.
II  BenchmarkedMetEvaluated on a 12-task benchmark against four coding-agent baselines with matched underlying models: 9/12 against 4/12 for the best baseline, plus ablations. However, the benchmark is the authors' own, not a public community benchmark, so this is a weak pass.
III  Open sourceNot yetThe paper gives no link to code, prompts, the index, the skills, the benchmark tasks or any weights. No repository is cited for EngineAgent.
IV  In productionNot yetNo documented use by a named organisation in real engine maintenance. All evidence is from the authors' benchmark.

What would move it up
Release the code, the prebuilt index and skills, and the 12-task benchmark with its evaluation scripts (level III). For level IV, show that maintainers of a named engine such as MNN used it to land real model-support patches, with merge outcomes. More tasks, repeated runs with variance, and independent replication would also strengthen the benchmark claim.

Checked automatically every week, last on 16 November.

The study

Zhengxiang Huang, Yujie Sun, Shengheng Chen et al.. EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities. arXiv, 10 November 2026. doi:10.48550/arxiv.2611.14356

Preprint, not peer reviewed · Licence CC BY · Shanghai Jiao Tong University, Shanghai · Alibaba Group (China), Hangzhou · Apple (United States), Cupertino

Notes

  1. Huang, Z. et al. EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities. arXiv preprint 2611.14356, 10 November 2026. Licence CC BY. Source
  2. No code, benchmark or prebuilt artefacts for EngineAgent are linked in the paper.
  3. Readiness: level II, Benchmarked (weak pass, authors' own benchmark). Level III not met, as nothing has been released.

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Agents