Amazon Prime Video is one of the most popular streaming services in the world, offering thousands of movies and TV shows to millions of customers. To ensure the quality of the video streams, the Prime Video team needed a monitoring tool that could detect and report any issues in real time.
The team initially built a solution using a microservices and serverless architecture, leveraging AWS services such as Step Functions, Lambda, and S3. However, they soon realised that this approach had several drawbacks:
- Scalability: The service performed multiple state transitions for every second of the stream, which quickly reached the account limits of Step Functions. Moreover, the service had to handle thousands of concurrent streams, which required a lot of computing resources and network bandwidth.
- Cost: Step Functions charged users per state transition, which added up to a significant amount for the high volume of streams. Additionally, the service made a lot of calls to S3 for temporary storage of video frames, which also incurred costs.
- Complexity: The service had many distributed components that needed to be orchestrated and coordinated. This increased the development and maintenance effort, as well as the risk of errors and failures.
The team decided to rethink their architecture and move to a monolithic approach. They packed all the components into a single process and implemented a lightweight orchestration layer to distribute customer requests. They also eliminated the need for S3 by storing the video frames in memory. They deployed their service on EC2 and ECS, using EC2 savings plans to reduce costs.
The results were impressive:
- Scalability: The service was able to handle 20 times more load than before, without hitting any limits or bottlenecks. The service also became more resilient and fault-tolerant, as it could recover from failures faster and easier.
- Cost: The service reduced its infrastructure cost by over 90%, thanks to the elimination of Step Functions and S3 charges, as well as the optimisation of EC2 usage.
- Simplicity: The service became easier to develop, test, deploy, and monitor, as it had fewer moving parts and dependencies. The team also gained more control and visibility over their service, as they could tune and tweak it according to their needs.
The team concluded that microservices and serverless are not always the best choice for every use case, and that sometimes a monolithic architecture can offer more benefits in terms of performance, cost, and simplicity.
They also learned that it is important to evaluate the trade-offs of different architectural options and choose the one that best suits the problem at hand.
Check out Prime Video Tech blog – https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90
Like, Comment or WordPress Reblog the post and Subscribe to IT Service Guru for future blog posts.
