Three things about running other people's pods on Akash that cost us money

A lease closes itself when the escrow runs dry, with no notification. Pausing a pod did not pause the bill. A failed deploy can still leave you an active, paid lease.

Disclosure first: we host agents on Akash and sell that, so we have an interest. That is also why we found these — all three cost us real money in one evening, and none of them is written down anywhere we could find. Every number below is from our own deployments, with the dseq, so you can argue with it.

1. A lease closes itself when the escrow runs dry, and nothing tells you

One of our customers' agents vanished. On chain, lease 28337339: state: closed, escrow state: overdrawn, funds: -143 uakt. No event, no callback, nothing. Our database still said the agent was running, and our own status page showed a green dot next to the word "Good". The customer found out by clicking a button that did nothing.

If you run pods on behalf of other people, watching the escrow balance is your job. There is no push. We now poll and alert below a threshold, because the alternative is hearing about it from the customer.

2. Pausing a pod does not pause the bill

We had a Pause button. It scaled the deployment to zero replicas. Our own UI said, in English, that this "stops compute charges". It does not. The lease stays open, and the provider bills the escrow for the reserved capacity per block, whether or not anything is running in it.

The mechanism is visible in the provider's own code: for as long as the lease is open the meter runs against the reserved capacity, however many replicas are actually running in it, so scaling to zero saves nothing. On chain it ends like this: lease 28331717 closed on insufficient_funds, escrow overdrawn at −146 uakt. One caveat for accuracy: that particular pod was running, not paused, when the deposit ran out — it shows where an open lease ends up, not that pausing fails to help. The only thing that actually stops the meter is closing the lease, which also returns the remaining deposit.

So we rebuilt pause as: snapshot, then close the lease. Resume deploys a fresh pod and restores the snapshot. The ordering is the whole safety of it — if the snapshot fails, the lease stays open.

3. A deploy that fails can still leave you a lease

Twice in one evening our /deploy call did not return, while the deployment itself went ahead and created the lease. Our side recorded a failure and stored no dseq — so a pod existed, money was flowing, and nothing in our system pointed at it.

Cause: Node's fetch (via undici) defaults headersTimeout to exactly 300000 ms. A deploy legitimately takes minutes, so the client gave up while the provider kept working. We also found a separate lease, 28370359, in a third state — active on chain with leases: [] — where 500000 uakt sat locked to a lease that never actually formed.

The general lesson is dull and worth saying anyway: reconcile the provider's list of leases against your own database on a schedule. Anything unclaimed for half an hour is either a leak or your own infrastructure, and you should be able to say which by name.

What we do not know

How long a deposit lasts at a given plan size — depends on provider pricing and pod size, not measured properly. Whether Akash offers any way to subscribe to escrow events — we could not find one and now poll. All of this is one provider, our own; behaviour elsewhere may differ.