Why image analysis should leave the web request
Model inference, image decoding, feature extraction and comparison can take far longer than a normal interactive request. Keeping the browser connection open couples user experience to CPU or GPU capacity, provider latency and temporary failures.
A queue separates acceptance from processing. The application validates the request, creates a durable job and returns an identifier. Workers consume that job according to available capacity, while the caller receives status through polling, events or a later result view.
This separation does not remove complexity. It moves complexity into explicit state, which can be monitored, retried and recovered instead of being hidden behind a request timeout.
Define a durable job contract
A job should contain enough information to process the requested work without depending on temporary web-session state. It normally references durable input, a processing version, options, priority and a correlation identifier. Large binary payloads are better stored in object or file storage and referenced by an immutable key.
The contract also needs an answer for changed inputs. If an image can be replaced while a job is waiting, the worker may analyse a different asset from the one the user submitted. A content hash, immutable object key or explicit version prevents that ambiguity.
Useful job fields
- Stable job and correlation identifiers
- Immutable input reference and checksum
- Model or algorithm version
- Requested processing options
- Creation time, priority and maximum age
- Attempt count and current state
- Expected result location or callback
Use leases instead of assuming workers never disappear
A worker can terminate after taking a job because of a process crash, resource limit or host restart. If ownership is permanent, the job remains stuck. If the message is acknowledged too early, the work may be lost.
A lease grants temporary ownership. The worker renews it while progress continues, and another worker can claim the job after expiry. Lease duration must exceed normal heartbeat delays without leaving abandoned work invisible for too long.
The state transition that claims the job should be atomic. Two workers reading “pending” at the same time must not both obtain ownership simply because the update was not guarded.
An expired lease returns the job to a controlled retry path.
Make processing idempotent
At-least-once delivery is common because it protects against lost work. It also means that the same logical job may run more than once. Writing duplicate detections, sending repeated notifications or charging twice is unacceptable.
The result should be keyed by job and processing version, with a uniqueness rule or transactional check. Side effects can be emitted through an outbox after the result is committed. A repeated attempt then finds the existing result or safely completes the missing final step.
Idempotency does not require every CPU cycle to be avoided. It requires repeated execution to produce one coherent business outcome.
Bound concurrency to the real bottleneck
Starting more workers can reduce throughput when they compete for memory, GPU capacity, disk bandwidth or a rate-limited model endpoint. Concurrency should be measured against the constrained resource and the shape of the workload.
Separate worker pools can protect different resource classes. Thumbnail preprocessing may be CPU and I/O heavy, model inference may require a GPU, and database comparison may depend on memory and indexes. One global worker count cannot represent all three.
Capacity signals
- Queue age rather than queue length alone
- Processing time by job type and model version
- Memory, CPU and GPU utilisation
- External-provider quota and latency
- Database write and query time
- Failure and timeout rate under load
Use priorities without starving normal work
Interactive jobs may deserve faster service than large background rechecks. A priority queue can express that difference, but permanent high-priority traffic can starve the normal queue.
Weighted scheduling, separate capacity reservations or an age-based promotion rule provide a fairer model. Maximum job age also prevents a stale request from consuming expensive compute after the result is no longer useful.
Classify failure and retain useful evidence
A corrupt image, unsupported format or missing source object is a data failure. A temporary model endpoint outage may be retriable. An out-of-memory process can indicate that the job needs a different pool or size limit. These cases require different responses.
The final failure record should preserve the error class, safe diagnostic context, attempt history and processing version. Moving a job to a dead-letter state without enough evidence only transfers the uncertainty to an operator.
Monitor the pipeline as a business process
Infrastructure metrics are necessary but incomplete. Operators also need to know how many assets are waiting, how old the oldest request is, which processing version produced the results and whether downstream publication completed.
Dashboards should connect queue state with result state. A queue can be empty because work completed successfully, because producers stopped or because jobs were discarded. Counts and age must be interpreted with the originating workload.