# Google graph model cuts datacenter repair time on H100s and TPUs

In a live test at Google, a graph neural network diagnosed broken AI machines faster than a long-maintained rulebook. It cut mean repair time by about 16 to 18%. The paper is a preprint written by Google staff about Google's own system.

- Category: In industry
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-19
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/google-graph-model-cuts-datacenter-repair-time-on-h100s-and-tpus

## The study

HardwareDx: A Foundational Graph Model System for Datacenter Machine Repairs (preprint, not peer reviewed). arXiv 2611.23398. https://arxiv.org/abs/2611.23398

## At a glance

- What it shows: Google replaced part of its hand-written repair rules with an AI model that reads each machine as a map of connected parts. In a live trial, machines diagnosed by the model went back into service about a sixth faster.
- What it does not show: It does not show that the system works outside Google, or what the saved time is worth in money. No code or data is released, so nobody else can check the results.
- Why it matters: When one machine in a large AI training cluster fails, thousands of others can sit idle. Faster repairs mean less wasted computing, and new hardware can be put to work sooner.

## What the study shows

Large AI models are trained on thousands of linked machines. If one breaks, the whole job can stall. Each machine holds over 100 parts that can be swapped, such as chips, cables, memory and power supplies. Finding the faulty one quickly is hard, especially when several faults appear at once.

Until now, Google diagnosed these failures with a rules system. Engineers write hundreds of rules that add 'badness' scores to each part, and the part with the highest score gets repaired first. Every new hardware generation needs a new set of rules.

The authors built a graph neural network, a model that learns from data shaped as nodes and links. Each machine becomes a graph of its parts, cables and test results. Text from error logs is turned into numbers by Gecko, a Google text-embedding model, so a symptom that is renamed on a new platform still looks similar to the model.

The authors report a live A/B test, in which machines were randomly sent to either the new model or the old rules. On Nvidia H100 GPU machines, mean time to repair fell by 18.3%. On a newly introduced TPU platform it fell by 16.1%. The share of cases diagnosed without a human rose by about 10 percentage points on both.

Offline tests on Google's own held-out data back up a second claim. A model pretrained on H100 data and then tuned on the newer H200 reached the same accuracy with about 10 times fewer labelled examples. The authors say this let the system be ready before the H200 went on general sale, instead of about four months after.

## Our reading

In our reading, the strongest evidence here is the live A/B test. Random assignment against a working production system is rarer than an offline benchmark score, and it measures what operators care about: hours a machine spends out of service.

The TPU result is the firmer one. It rests on 2,567 repair windows and a narrow error band. The H100 result rests on only 225 windows, and its error band is plus or minus 7.6 points, so the true gain could be much smaller than 18%.

The design idea is easy to state. The model sees where a fault sits in the machine, not just which alarms fired. One example from the paper: if two GPUs on the same PCIe chain, the data link that connects them, vanish together, the model points at the shared chain. If they sit on different chains, it points at the GPUs. Fixed rules handle this kind of case poorly. On our readiness scale we rate this work II, Benchmarked, and only loosely: results are reported on private data, not a public benchmark, and nothing is released.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/google-graph-model-cuts-datacenter-repair-time-on-h100s-and-tpus

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
