# Coding agents with an engine map add phone LLM support more reliably

A preprint describes EngineAgent, a set of AI agents that adapts on-device inference engines to new language models. It passed 9 of 12 tasks against 4 for the best general coding agent. The benchmark is small and the authors built it themselves.

- Category: Agents
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-12
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/coding-agents-with-an-engine-map-add-phone-llm-support-more-reliably

## The study

EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities (preprint, not peer reviewed). DOI 10.48550/arxiv.2611.14356, arXiv 2611.14356. https://arxiv.org/abs/2611.14356

## At a glance

- What it shows: On 12 tasks that each add a new model to a phone inference engine, EngineAgent passed 9 (75%) with either of two underlying models. The best baseline, Codex, passed 4 (33.3%). EngineAgent also used less time and API spend.
- What it does not show: It does not show that the patches would be accepted by engine maintainers, or how fast the enabled models run. Results appear to come from one run per task on a benchmark the authors designed.
- Why it matters: Supporting each new small model on phones takes weeks of specialist work across many files and hardware types. Moving part of that work to agents could shorten the delay between a model's release and its use on devices.

## What the study shows

An on-device inference engine is the software that runs a language model on a phone. Examples include MNN, llama.cpp, MLC-LLM and ExecuTorch. Each new model family tends to bring new parts, such as a different attention mechanism, and each part has to be wired through the engine and written again for each type of chip: the main processor (CPU), the graphics chip (GPU) and the AI accelerator (NPU). The authors call this work model enablement. They report that adding Qwen3.5-2B to MNN touched 42 files and more than 4,200 lines of code. They also report that two of the four engines they surveyed did not support Gemma4-E2B-it, and three did not support MiniCPM-V 4.6.

The paper is a preprint and has not been peer reviewed. The authors are at Shanghai Jiao Tong University and Alibaba Group. Alibaba develops MNN, one of the engines tested.

EngineAgent prepares two things before any new model arrives. The first is an index of the engine's code, sorted into nine layers. The layers run from chat templates and tokenisation (splitting text into units the model reads) down to hardware-specific kernels, the low-level routines that do the arithmetic on a given chip. The second is a set of build, test and debug procedures for each hardware backend. These are checked by building and running models the engine already supports on a real phone. When a new model arrives, three agents take over. A builder compares the new model with similar supported ones and makes changes layer by layer. A debugger diagnoses failures. An auditor runs layer-by-layer tests and then a full test on the device.

The benchmark has 12 tasks: three models (Qwen3.5-2B, Gemma4-E2B-it and MiniCPM-V 4.6) on four engine and hardware pairs. The pairs are MNN on CPU, llama.cpp on a Vulkan GPU, MLC-LLM on an OpenCL GPU and ExecuTorch on a Qualcomm NPU. A task passed only if the code compiled, ran on a phone, and produced output scores (logits) that matched a PyTorch reference on a desktop computer within a set tolerance. Where the engine already supported the model, the authors reset the engine to the version just before support was added. They blocked access to later versions and to web search.

The authors report that EngineAgent passed 9 of 12 tasks with either DeepSeek-V4-Flash or GPT-5.6 Sol as its underlying model. Codex with GPT-5.6 Sol passed 4. Three baselines using DeepSeek-V4-Flash did worse: OpenCode passed 1, OpenCode with added code-graph and failure-tracing tools passed 1, and SWE-Agent passed none. Each run was capped at five hours.

With DeepSeek-V4-Flash, EngineAgent averaged 3.82 hours and $1.53 in API cost per task. The comparable baselines took 4.78 to 4.91 hours and cost $2.11 to $2.44. With GPT-5.6 Sol, EngineAgent averaged 4.16 hours and $11.29, against 4.82 hours and $14.90 for Codex.

The authors also removed each component in turn. Without the code index, success fell from 9 to 3 of 12, and exploration turns rose from 42 to 271. Without the hardware procedures, success fell to 5 of 12, and compilation and debugging failures rose from 11 to 83. Building the index and procedures cost $0.66 to $2.68 per engine. Updating them averaged about $0.07 per code commit across five MNN commits.

## Our reading

In our reading, the main lesson is about preparation. Neither of EngineAgent's two components is clever on its own. One is a map of where things live in the engine. The other is a set of build-and-test recipes known to work on a real phone. Both are built before the hard task begins. The removal tests suggest each accounts for a large share of the gain. The failure breakdown fits this: EngineAgent had no missing-implementation failures and no deployment failures, which are the two kinds of error the components target.

The gap between 9 of 12 and 4 of 12 is large, but it rests on 12 tasks and, as far as we can tell, one run each. A difference of a few tasks could change with another run. The comparison with matched underlying models helps. It suggests the scaffolding around the model, not a stronger model, explains most of the difference. Even so, the baselines were general-purpose tools that were not tuned for this job.

On our four-level readiness scale we rate this level II, Benchmarked, and only just. The tests run on real phones against named baselines, but the benchmark is the authors' own and has not been published. We found no released code, index, procedures or benchmark, so level III is not met.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/coding-agents-with-an-engine-map-add-phone-llm-support-more-reliably

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
