Designing A Video Search Service with AWS – AWS Cloud Computing Architecture
Hello there, how are you doing?
Welcome to the system design & architecture series of scaleyourapp.com
In this write-up, we will be designing A Scalable Video Search Service with AWS Amazon Web Services Platform.
So, without any further ado. Let’s have a look at the use case.
1. USE CASE
The service should enable a user to run a search, with parameters such as geographical locations, important people, entities or anything specific, based on the context, on the data stored in the system in the video format.
Once a user runs the search, the system should locate the video containing the searched content & the time of appearance of the content in the video.
2. AWS SERVICES FOR THE JOB & ARCHITECTURE
We will first begin with the basic architectural diagram. Naturally, it would be a client server architecture.
For this scenario, we will pick the serverless service of AWS, AWS Lambda, for the backend processing.
The client will interact with the backend via a REST-based API enabled by the Amazon API Gateway service
The core algorithm for data search in the videos, based on the meta tags will be handled by the Amazon Rekognition service
For storing the data, we will use the Amazon S3 service
For storing the metadata for search we’ll use the AWS Elastic Search service
I’ll talk about every service up ahead in the article before that let’s have a look at the architecture & the use case flow.
3. AWS CLOUD ARCHITECTURE
Here is the architectural design of the service.
4. Use CASE FLOW
1. Video data is stored in Amazon S3 which ensures scalability, availability, security & performance of the data.
2. User uploads the video via the mobile app. The app client interacts with the backend system via a REST-based API powered by AWS Gateway API.
3. The Gateway API process the video upload request triggers an event on the AWS Lambda serverless service, runs the business logic & streams the data to the Amazon Rekognition Machine Learning module.
4. The machine learning service Rekognition automatically generates the metadata from the video, such as the location, people, objects, entities in the video.
5. All the metadata generated is indexed in Amazon Elasticsearch service for faster querying. Elasticsearch is the most popular open source search framework in the industry today, having Lucene at its core.
The flow above processes, indexes & setups up the data for the search queries to be run on.
1. Now that the data is setup with the metadata indexed in the AWS Elasticsearch indexes.
2. The user communicates with the backend system via the same flow.
3. For searching for a particular video, the user enters the search query in the client. The client hits the backend REST API with the query string. AWS Lambda is triggered with scans the Elastic indexes against the query string.
4. Once there is a hit, the video pointer is fetched from the Elasticsearch services & the data is fetched from the S3 storage & streamed back to the client.
5. Running a Video Search Based on the Meta Data
Every video file has some form of internal meta-data, like title, description, coding quality & other specifications.
The external metadata contains the position of text, speech, entities etc. in the video.
Speech recognition, text recognition & frame analysis etc. are run on the video to locate the queried content.
For more information on this read video content analysis.
6. A Brief on AWS Services Involved
S3 is an object storage service by AWS, with several storage classes, applicable for storing data for any kind of use case be it a mobile app, website or archiving massive amount of enterprise data & running big data analytics on it.
The batch operations with S3 lets’ us manage billions of objects on the fly via its API or the management interface.
This is a fully managed Elasticsearch service hosted on Amazon. AWS offers the ELK Elastic Logstash Kibana to businesses without any kind of operational overhead.
This is the serverless backend services offered by Amazon. Be worry free of provision servers & other server management tasks. Pay only for the instance compute time, not the idle instance time.
AWS Gateway API
A fully managed REST-based API service with WebSockets. Enables us to handle a massive number of concurrent API calls, authorization & other stuff.
Add video & image analysis to your application via the AWS machine learning Rekognition library. It enables us to detect faces, locations, text, scenes, activities etc. from video content. The service is based on the deep learning technology built by Amazon.
More On the Blog
Well, Guys, this is pretty much it on designing the video search services with AWS. Don’t forget to let me know your thoughts in the comments.
Share it with your folks if you liked the article.
Follow 8bitmen on social media to stay notified on the new developments in software engineering, distributed systems, system design, real-life architecture space.
Our Twitter Handle
I’ll see you in the next write-up.
> Spotify Engineering: From Live to Recording
> Ingesting LIVE video streams at a global scale at Twitch
> $64,944 spent on AWS, to support 25,000 customers, in August by ConvertKit.
> Read how Storytel engineering computes customer consumption of books transitioning from batch processing to streaming bookmarks data with Apache Beam and Google Cloud.
> How Pokemon Go scales to millions of requests per second?
> Insight into how Grab built a high-performance ad server.
SUBSCRIBE TO MY NEWSLETTER to be notified of new additions to the list. Fortnight/monthly emails.
Looking for developer, software architect jobs? Try Jooble. Jooble is a job search engine created for a single purpose: To help you find the job of your dreams!!
- Web Application Architecture Explained With Designing a Real-World Service
- Wide-column, Column-oriented and Column Family Databases – A Deep Dive with Bigtable and Cassandra
- Design For Scale and High Availability – What Does 100 Million Users On A Google Service Mean?
- How Razorpay handled significant transaction bursts during events like IPL
- Facebook’s Photo Storage Architecture