A trading bot that misses a transaction loses one trade. A protocol that misses a transaction has a user asking why their position was not closed.
The difference is not technical. It is that you own the outcome — and that changes which failures you can tolerate.
Where Protocols Submit Transactions
Most Solana protocols end up with a backend that sends transactions, even when the design started as fully user-driven:
Keepers. Liquidations, expiries, and threshold triggers that must happen without a user present.
Crank turns. Order matching, reward distribution, state advancement — anything requiring someone to pay for the next step.
Settlement. Batched outcomes, epoch rollovers, oracle-dependent updates.
Assisted user actions. The user signs, your backend submits and retries.
★All four share one property: nobody is watching in real time.★ A user retries when a swap fails. A keeper that silently stops running produces a bad debt event nobody notices until it is on chain.
The Failure That Matters Most
For a protocol, the expensive failure is not a slow transaction. It is a transaction that did not land and was not noticed.
// ★This is the shape of the bug.★
async function runKeeper() {
const positions = await findLiquidatable();
for (const p of positions) {
const tx = buildLiquidation(p);
await connection.sendRawTransaction(tx.serialize());
// Returned a signature. Nothing checked whether it landed.
}
}
sendRawTransaction returning a signature means the RPC accepted the bytes. It does not mean anything landed. A keeper written this way reports complete success while the position stays open.
The fix is structural rather than clever: every submission needs a terminal state.
const outcome = await submitAndResolve(tx, lastValidBlockHeight);
switch (outcome.status) {
case "success": await markDone(p, outcome.slot); break;
case "reverted": await recordRevert(p, outcome.err); break; // ★logic problem★
case "expired": await requeue(p); break; // ★timing problem★
}
★Three terminal states, and they need different handling.★ A revert means your instruction was wrong for the state it hit, so requeuing the identical transaction will revert again. An expiry means it never executed, so requeuing with a fresh blockhash is exactly right.
If your keeper is correct and the expiry rate is still your problem, a free BoltTx key is one line to test against.
Idempotency Is Not Optional
A retried keeper action must not double-execute. On Solana you get part of this for free and have to build the rest.
Free: identical signed bytes produce an identical signature, and a signature is included at most once. Resending the same transaction is safe.
Not free: rebuilding a transaction produces a different signature. If the first one landed and you did not observe it, the rebuilt one can execute a second time.
// ★Resolve the old signature before rebuilding.★
const prior = await connection.getSignatureStatus(lastSig,
{ searchTransactionHistory: true }); // ★may be older than the status cache★
if (prior.value && !prior.value.err) return; // already done
const rebuilt = await buildWithFreshBlockhash(action);
The stronger version puts the guard on chain: the program itself rejects a second execution, by checking a state flag, a sequence number, or a per-epoch marker. Client-side checks race; on-chain checks do not.
★For anything that moves user funds, the on-chain guard is worth the extra account.★
Keeper Competition
If liquidations are permissionless, you are racing other keepers, and losing that race is not neutral — the opportunity is gone and you spent the fee.
// Same target, several keepers. Assume you will sometimes lose.
const result = await submitAndResolve(tx, lastValidBlockHeight);
if (result.status === "reverted" && isAlreadyLiquidated(result.err)) {
// ★Someone else won. Expected, not an incident.★
metrics.lostRace.inc();
return;
}
★Separate "lost the race" from "our keeper is broken" in your metrics.★ A rising lost-race rate means competitors are landing faster. A rising expiry rate means your submission path is the problem. Collapsing both into one error count hides which one you have.
Fee Policy for Automated Submission
A user picks a fee once. A keeper picks one thousands of times per day, so the policy compounds.
Do not hardcode. A fixed priority fee is either wasteful in calm conditions or insufficient in the exact moments that matter.
Derive from contested accounts. Fee pressure is per-account, not global:
const fees = await connection.getRecentPrioritizationFees({
lockedWritableAccounts: writableAccounts,
});
const sorted = fees.map((f) => f.prioritizationFee).sort((a, b) => a - b);
const median = sorted[Math.floor(sorted.length / 2)] ?? 0;
Scale to the value at stake. A liquidation protecting a large position justifies more than a routine crank turn. ★A protocol that pays the same fee for both is overpaying on one and underpaying on the other.★
Cap it. An unbounded fee policy in a fee spike is an unbounded expense. Bound it, and alert when you hit the bound rather than silently paying.
Observability That Answers the Right Question
Protocol dashboards frequently track submission counts, which tells you almost nothing.
★What to track instead:★
| Metric | Why |
|---|---|
| Landed / attempted, per action type | The actual success rate |
| Expiry rate | Isolates the submission path |
| Revert rate by error code | Isolates instruction logic |
| ★Slot distance, as a distribution★ | ★Where the tail is★ |
| Lost-race rate | Competitive, not a fault |
| Fee spend per action | Whether policy is sane |
★Distribution, not average.★ A keeper landing within two slots on the median and twelve at the tail has a real problem that an average hides completely — and the tail is exactly when the network is congested, which is exactly when liquidations arrive.
Degraded Mode
Protocols need an answer for the case where submission is failing broadly, and it should be decided in advance:
Do not retry forever. Past lastValidBlockHeight, rebuild. A retry loop with no exit consumes fees on transactions that cannot land.
Prioritise. Under load, the largest at-risk positions come first. ★A FIFO queue during congestion processes a dust liquidation ahead of a solvency-threatening one.★
Alert on rate, not on instances. Individual failures are normal. A rising rate is the signal.
Have a manual path. When automation cannot land, someone needs the ability to submit directly.
What Landing Looks Like
Real transactions through our delivery nodes: median confirmation 336ms — under one slot.
★For a protocol, the value is in it being predictable rather than in it being fast.★ Predictability is what lets you set timeouts, size your keeper fleet, and tell users what to expect.
Where BoltTx Fits
We handle submission. Not custody, not execution logic, not your keeper design.
Submissions route through our own delivery nodes in four regions with stake-weighted routing and no public mempool exposure, so a liquidation is not observable in transit before it lands — which matters when other keepers are watching for exactly that.
Your keeper signs locally with its own key. We never hold funds, never sign, and never modify transaction contents. The tip travels inside the transaction, paid on chain from the keeper wallet, and reverts with the transaction if it fails, because that is how Solana handles atomic transactions. You pay only on transactions that reach the chain.
Get a free API key. No monthly fee:
const connection = new Connection("https://la.bolttx.io/?api-key=YOUR_KEY");
FAQ
How should a DeFi protocol handle transaction failures? Resolve every submission to one of three terminal states: landed successfully, landed and reverted, or expired. Reverts indicate instruction logic, expiries indicate the submission path, and they need different responses.
Why does my keeper report success when nothing happened?
Because sendRawTransaction returning a signature only means the RPC accepted the bytes. It says nothing about landing. Poll getSignatureStatuses until the transaction resolves or the blockhash expires.
How do I make keeper actions idempotent on Solana? Resending identical bytes is safe, since one signature is included at most once. Rebuilding is not, so check the prior signature first — and for anything moving user funds, enforce the guard in the program itself.
What fee should a Solana keeper pay? Derived from recent fees on the accounts it will write to, scaled to the value at stake, and capped. A hardcoded fee is wasteful in calm conditions and insufficient during the congestion when liquidations arrive.
How do I tell if my keeper is losing races or broken? Track them separately. Reverts that indicate the action was already performed are lost races. A rising expiry rate means your transactions are not landing at all, which is a submission problem.
Should a protocol retry a reverted transaction? Not identically. A revert means a program check rejected it against the state it hit, so the same transaction will revert again. Rebuild against current state, or record it and move on.
How long should a keeper retry before giving up?
Until the blockhash expires — compare getBlockHeight() against lastValidBlockHeight. Past that the transaction is permanently invalid, so continuing to resend spends fees on something that cannot land.
What should a protocol monitor for transaction health? Landed-versus-attempted per action type, expiry rate, revert rate by error code, slot distance as a distribution, and fee spend. Submission counts alone tell you nothing about outcomes.
How do I prioritise keeper actions during congestion? By value at risk rather than arrival order. A FIFO queue under load processes trivial actions ahead of solvency-threatening ones, which is exactly backwards when capacity is scarce.
Can a protocol batch keeper transactions? When the actions are independent, yes. Anything with a required order cannot be batched, since separately submitted transactions have no guaranteed ordering even from the same signer.
Why do liquidations fail more often during volatility? Because that is when the network is congested and when every other keeper is submitting the same action. Both your competition and your latency worsen at the same moment.
Should keeper transactions use skipPreflight? Generally yes. Preflight costs a round trip and simulates against the current slot rather than the one you will land in. Simulate during development and when validating new instruction paths.
How do I prevent double liquidation? Enforce it on chain. The program should reject a second execution against the same position by checking state, since client-side deduplication races against your own retries and against other keepers.
What is the right alerting threshold for keeper failures? Alert on a rising rate rather than on individual failures. Some failures are normal — lost races and reverts against changed state — so instance-level alerts train people to ignore them.
Does a protocol need its own RPC infrastructure? Not necessarily, but it needs a submission path whose behaviour under congestion it understands. Shared public endpoints degrade precisely when keeper actions become urgent.
How do I test keeper reliability before mainnet? Measure the slot distribution rather than a pass or fail result, and test under simulated contention with several keepers competing. A keeper that works when uncontested tells you little about how it behaves during a liquidation cascade.
Related Reading
- Solana Liquidation Bot Infrastructure
- Solana Bot Monitoring and Alerting
- Solana Transaction Retry Patterns
- Solana Compute Unit Pricing
- Solana Transaction Landing