Distribution turns process state into architecture
A crawler, media processor or synchronisation system may begin as one scheduled command. As volume and external dependencies grow, work is split across hosts and processes. The original sequence is then no longer guaranteed by one call stack.
The architecture must represent what work exists, who currently owns it, which prerequisites are complete, what result was produced and how an abandoned step is recovered. Without explicit state, operators infer progress from process lists and fragmented logs.
Distribution is useful when it isolates workloads or adds capacity. It is harmful when it merely multiplies places where an unknown state can hide.
Separate orchestration from execution
The orchestrator decides what should happen and when. A worker performs one bounded task. Keeping these concerns separate prevents workers from embedding an untraceable chain of further work and makes capacity easier to control.
The orchestrator does not need to be one large central service. A durable database, queue and scheduled reconciliation process can share the role. The important point is that state transitions and prerequisites have one consistent model.
Typical responsibilities
- Discover or schedule units of work
- Record prerequisites and dependencies
- Assign priority and availability time
- Grant and renew temporary ownership
- Accept results and create follow-up work
- Detect stalled or inconsistent state
Use leases and fencing for temporary ownership
A host can pause, lose network access or restart while holding work. A lease allows ownership to expire so that another process can continue. Heartbeats extend the lease while useful work is still progressing.
A late worker may resume after its lease has expired. A fencing token or monotonically increasing attempt number lets result storage reject writes from the old owner. Expiry alone is insufficient if stale processes can still commit.
Lease duration should reflect real task time and heartbeat behaviour. A lease that is too short creates duplicate work under temporary load; one that is too long delays recovery.
Model state transitions and invariants
A free-form status string is not a complete state model. Each transition should define who may perform it, what data must already exist and whether the transition can be repeated.
For example, a successful crawl may create media-analysis jobs only after the page result is committed. A failed analysis may return to pending with a later availability time, while an invalid input moves to a permanent correction state. These distinctions support automation without hiding failure.
Reconciliation checks for missing or inconsistent transitions independently.
Make result writes idempotent
Retries and expired leases mean that two attempts may compute the same result. Storage should use a stable work identifier and processing version so that the business outcome is committed once.
If result storage and follow-up scheduling occur in separate systems, an outbox or reconciliation step prevents the chain from stopping between them. A transaction can guarantee local state, but the wider workflow still needs recovery from partial completion.
Plan rechecks as first-class work
Crawled pages change, external APIs recover and models are updated. Rechecks should not be an improvised “run everything again” command. They need scope, reason, processing version, priority and a way to avoid overwhelming normal traffic.
A recheck can target stale records, a failed provider window, a new model version or a random quality sample. Recording the reason allows later analysis of cost and effectiveness.
Monitor end-to-end progress, not individual processes alone
Host CPU, memory and process uptime are necessary signals, but a green host can still run a stalled business process. Monitoring should connect discovery rate, queue age, lease expiry, result completion, retry volume and downstream publication.
Age is often more useful than count. Ten tasks waiting for seconds may be normal, while one critical task waiting for a day may require intervention. Metrics should be split by task type, priority and processing version where those dimensions change expectations.
Operational questions a dashboard should answer
- Is new work still being discovered?
- How old is the oldest eligible task?
- Are leases expiring more often than usual?
- Which error classes consume retry capacity?
- Do committed results create all expected follow-up work?
- Which processing versions remain active?
Use reconciliation to repair silent gaps
Events and queues can be reliable without being infallible. Configuration errors, manual data changes and defects can produce a state that no normal handler expects. A reconciliation process scans durable records for violated invariants.
It may find completed crawl records without analysis jobs, expired leases still marked active or results without publication state. Repairs should be idempotent and logged as their own operations, not applied as invisible database edits.
Design shutdown and deployment behaviour
Workers should stop accepting new work, finish or safely release current tasks and expose when they are ready to terminate. Killing every process at once can create avoidable retries and duplicate expensive computation.
During rolling deployment, old and new versions may run together. Job contracts and result storage need an explicit compatibility policy. If a task requires one model or code version, that requirement belongs in the work record rather than in an operator assumption.