Android Bench benchmark: Google adds new LLMs, Gemini slips

Google updated Android Bench with eight new large language models and a new testing sandbox, and the refreshed leaderboard shows Gemini 3.1 Pro slipping to fifth while Claude Fable 5 leads the pack at 84.5 percent accuracy. The benchmark still runs the same suite of 100 Android development tasks across 10 runs, and Google re-ran prior tests with the new Harbor framework to produce the current standings.

What did Google change in Android Bench?

Google added eight new models to the Android Bench leaderboard and switched the testing stack to the Harbor framework to make running and sharing tests easier: the new models are Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus, and Qwen 3.7 Max. Google also introduced new metrics such as cost and efficiency and re-ran previous tests under Harbor to establish an updated baseline, while keeping the original 100-task, 10-run workload and preserving historical results in an archive.

How did the updated leaderboard shake out?

The updated rankings place Gemini 3.1 Pro in fifth, behind GPT 5.4, Claude Sonnet 5, and Claude Fable 5, with Fable 5 achieving 84.5 percent accuracy on the Android Bench tasks. OpenAI models remain near the top: GPT 5.4 is ahead of Gemini 3.1 Pro, and OpenAI’s latest LLMs were slightly in the lead in the initial Android Bench release. Cost and runtime now factor into comparisons: Claude Fable 5 and GPT 5.5 consumed more than $130 in tokens for the 100-problem, 10-run benchmark, while Gemini 3.1 Pro cost $87 to run the same workload.

What do cost and runtime reveal about model selection?

Cost and runtime now meaningfully change the ranking calculus because some models that score highly are expensive or slow to run: for example, Gemini 3.5 Flash, touted as cheaper, ended up with the highest cost on the leaderboard at $165 per run and a 28-hour runtime because it took much longer to complete the benchmark. By contrast, Gemini 3.1 Pro cost $87 for the full benchmark despite scoring lower than several competitors. Google highlights these metrics to help developers weigh accuracy against operating cost and latency when choosing agents for Android development tasks.

Why it matters

Android Bench is explicitly designed to show which LLMs perform on Android development tasks, and the new Harbor framework plus added metrics make those comparisons more actionable for developers deciding which agents to integrate into workflows. Google’s own models performing behind competitors matters because the company is shifting projects toward agentic development and would prefer Android developers adopt its tools, yet the current results show third-party models leading on coding accuracy and, in some cases, cost-efficiency.

How can developers contribute and reproduce results?

Google updated the Android Bench GitHub with the new dataset and Harbor-based instructions so developers can run their own development tasks against the benchmark and submit results for possible inclusion. The Harbor testing sandbox is intended to simplify running, evaluating, and sharing benchmarks; Google re-ran its earlier tests with Harbor to create the new baseline while keeping historical data available in an archive.

What to watch

Look for community-submitted tasks and benchmarks on the Android Bench GitHub and for any future leaderboard changes after independent developers run tests under Harbor, particularly any shifts that close the gap between Google’s Gemini family and the top-performing models. Also watch for updates that expand cost or efficiency metrics or add new models to the public leaderboard.