Maintenance, failure rates and how many spares to hold

Ask an operator what their downtime costs and they will usually quote the revenue of the dead machines. The real number is larger, because a unit waiting twelve weeks for a part is not just idle, it is idle during the only period its efficiency class was still worth running. Maintenance planning is mostly about shortening that wait.

What breaks, roughly in order

• Fans. Cheapest, most frequent, easiest. On air cooled fleets they are a consumable, not a fault.
• Power supplies. The most common component failure that actually stops a machine. Causes and prevention in ASIC power supplies.
• Hashboards. Fewer failures, much bigger consequence. A machine with one of three boards down still hashes, at two thirds output, which is why it hides on a dashboard that only shows online and offline.
• Control boards. Rare, but a dead control board takes the whole unit off the network and the part is often the hardest to source.
• Cabling, connectors and PDUs. Not the machine's fault, frequently the actual cause, and almost always found last.

The failure curve

Hardware failures cluster at the beginning and at the end. Early failures show up in the first days, which is exactly why the whole consignment should be powered on inside the claim window rather than only the units you need this week. That is the argument in the first 72 hours, and it applies with more force to used stock, where the cover is a short dead on arrival window rather than a manufacturer warranty.

Then a long quiet period. Then a rising tail as the fleet ages, which usually arrives around the same time the machines are approaching their breakeven anyway, and the sensible answer stops being repair and starts being replacement.

How many spares

There is no universal ratio, but the shape of the answer is stable. Hold whole spare machines at a low single digit percentage of the fleet, more for used stock and for models with a thin parts market. Hold power supplies deepest, because they fail most and are model specific. Hold fans generously, they are cheap and always needed. Hold a small number of hashboards for your main model only, because board stock ties up capital and ages alongside the fleet.

The test is not a percentage, it is a question: if five machines fail next Tuesday, how long until they are hashing again? If the honest answer is measured in months, the spares position is wrong regardless of what the ratio says.

Buy the spares with the batch

This is the single most valuable habit in the whole subject. Spares bought at the same time as the fleet match the part numbers, come from the same production run, ship on the same freight and clear on the same customs entry. Spares bought eighteen months later are a separate order, a separate import, a different revision, and sometimes a discontinued model. The premium for buying early is small and the premium for buying late is enormous.

The routine that prevents most of it

• Filter changes and dust removal on a fixed calendar, not when someone notices.
• Thermal imaging or a spot check of connectors quarterly. Loose connections warm before they fail.
• Water quality checks and filter changes on hydro loops, on the schedule the manufacturer states rather than the one that is convenient.
• Hashrate monitoring per board, not just machine up or down, so a machine running on two of three boards is caught in a day rather than a quarter.
• A written record of which unit has which serial, which profile and which repair history. In eighteen months nobody remembers.

Repair or replace

Board level repair is a real trade and worth using on machines of the current generation with a decent remaining life. On a machine already near the end of its economic life, repair costs more than the machine will earn afterwards. Work that out from the same numbers as everything else: what does this unit contribute per day at today's power price, and how many days does it need to run to pay for the repair and the freight both ways.