Design Patterns: Competing Consumer Pattern

Building on the Queue-Based Load Leveling Pattern, the Competing Consumer Pattern enables a system to process multiple messages concurrently to optimize throughput, to improve scalability and availability, and to balance the workload.

During the lifetime of an application, the number of requests may vary significantly over time. Using a single instance of the consumer service might cause that instance to become flooded with requests.

Competing Consumers

To handle this fluctuating workload, the system can run multiple instances of the consumer service. However these consumers must be coordinated to ensure that each message is only delivered to a single consumer. The workload also needs to be load balanced across consumers to prevent an instance from becoming a bottleneck.

Using a Message Queue to Distribute Work to a Pool of Consumer Services

Benefits

  • Scalability: Depending on the backlog of the Queue, consumers can be increased or decreased on the fly.
  • Load-Levelling: Since this Design Pattern, inherently uses the Queue-Based Load Leveling Pattern, all of the Benefits applied there apply here as well.
  • Reliability: If all the consumers are unresponsive or overloaded, the system can still continue working and no messages would be lost.
  • Guaranteed Delivery: At least once
  • Resiliency: If the consumer fails mid-task, the message is not lost, but rather returned to the queue, to be picked up by another consumer.

Considerations

  • Message Ordering: Once you introduce multiple consumer instances, you are no longer guaranteed the order in which messages are received and the order in which they were pushed to the queue. Systems that use the Competing Consumer Pattern should ensure that message processing is idempotent.
  • Poison Messages: A malformed message, or any message that a service cannot process, may be returned to the queue crashing other services and creating and infinite loop processing these messages.
  • One-Way: As we discussed in the Queue-Based Load Leveling Pattern, this Pattern is fully decoupled, so if the service generates a result, it must be passed back to the application logic and must be stored in a location that is accessible to both.