The Two Generals’ Problem

You can’t guarantee a state change across an unreliable network with 100% certainty. You can only increase the probability of success through acknowledgments and retries. This is why idempotent operations are the bedrock of reliable systems—if you don’t know if the command worked, you must be able to send it again without side effects.

Backpressure

If a consumer can’t keep up, the producer must slow down or the system will fail. Don’t hide the failure; surface it. In a microservices architecture, this often manifests as a “thundering herd” if not handled correctly. Using bounded queues and explicit rejection (HTTP 429 or 503) is often better than letting requests sit in a queue until they time out.

Failure Domains

Assume every component will fail. Design so that a failure in one region or one service doesn’t cascade. This means graceful degradation. If the search service is down, the user should still be able to view their profile. If the primary database is read-only, the UI should reflect that state instead of just spinning.

Fallacies of Distributed Computing

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn’t change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

Never optimize based on these assumptions. They are the most common source of “impossible” production bugs.

• Cold Starts: Edge functions with embedded SQLite have near-zero database cold start overhead compared to TCP-based connections.
• Write Latency: Propagating writes to the primary region is the main bottleneck. Group writes whenever possible.

Moving critical paths (parsers, schedulers, state machines) to Rust has reduced our P99 latency significantly.

Key wins:

1. Zero-cost abstractions.
2. Memory safety without a GC pause.
3. Explicit error handling (Result/Option) forces better edge-case thinking.

A content-first site gives the project a fast feedback loop. Publish, learn what resonates, and only then add heavier app-like infrastructure.