Conquering the Queue: Lessons from processing one billion Celery tasks

Day 2: Saturday

10:30 am

At Userlike, Celery plays a critical role as the backbone of our Django-based SaaS application, orchestrating over 100 million tasks per month with speed, reliability, and precision. In this talk, I’ll share the lessons we’ve learned while scaling Celery to handle massive workloads and support the needs of a growing user base. From optimizing performance and avoiding common pitfalls to handling failures gracefully and ensuring a resilient architecture, this session will provide actionable insights for developers and architects working with distributed task queues.

Whether you’re just starting with Celery or looking to scale an established system, you’ll walk away with practical tips, battle-tested strategies, and a deeper understanding of how to harness Celery’s full potential in real-world scenarios.

Outline:

• Introduction: Why Userlike needs a task queue, and why you need one too
• Fundamental concepts: latency, throughput, failure modes
• Optimizing Performance: Strategies for faster and more efficient task execution
• Avoiding Pitfalls: Common mistakes and how to mitigate them
• Handling Failures: Building fault-tolerant workflows and monitoring systems
• Resilient Architecture: Designing for reliability and scalability
• Key Takeaways: Practical tips for implementing and scaling Celery in your own projects

This talk is designed to be technical, engaging, and packed with real-world experiences to help you conquer the queue in your own applications.