00:00You have seen this before: hundreds of players trying to survive a deadly game

00:04One wrong move and you are out, but what if I told you we could create this iconic scene using AI? In this video, I'll show you how to design the actual red light, green light game of Squid Game using computer vision, pose estimation, and real-time inference

00:18So, welcome to Freeird Stew, and let's design an ML-based surveillance system that can detect the movement violations for you

00:25So first, we have this raw problem statement given to us by the commander: in the red light, green light game, players are allowed to move during the Green light

00:36During red light, any detectable body movement leads to elimination

00:38The challenge is to build an AI system that can watch 100 plus players in real time and instantly detect if anyone moves during the red light phase

00:48The system must be accurate, fast, and robust under varying conditions like occlusion, lighting, or player diversity

00:55So, as per our ML system design framework that I already discussed in our previous videos, you can watch that first, and the link is in the I button or video description

01:07So, as our first step is problem formulation

01:10So, in this first step, you set your objective

01:13That is, classify whether a tracked person has moved significantly between the frames during red light, and real-time movement detection based on the object detection, that is, the finding people

01:24Then pose estimation, tracking the body joints, then the motion analysis that can detect the movement from pose over the time deltas

01:34Okay, then we have to define our task

01:36Our task is simple

01:39We have to do binary classification here to predict whether the person moved or not

01:41Okay, and our input is a sequence of 2D pose key points for each player recent frames, and the output is a binary classifier, that is, one

01:51Or zero, and the features we need for this problem statement are like joint positions of each player, pose deltas that a player moves from a last frame to this frame, and the velocity or acceleration of the limbs in that point frame as well

02:06Okay, and the edges involved occur, jittery cameras, uh, partial key points, or the false movements like wind twitch, eyes twitching, like these kind of movements, we should not consider as the violation

02:22Okay, then ask about the data sources to your interviewer

02:25If the interviewer has the data sources, ask what kind of data they have and where the data is placed so that it

02:32Can help in building the data pipelines for us, and then ask about your assumptions

02:37In that case, I assume that the game has static wide-angle cameras installed and stable, players remain stationary during the red light, and players are fully visible and upright during most of the game

02:52Environment conditions like lighting, wind, and shadows are manageable

02:54Okay, and to clear why we need ML for this problem statement

03:00So, because we need ML as a hybrid approach here, because the pose estimation, movement classification, player tracking can be done with great accuracy with ML only, and For the rule-based approach, we can use thresholds for movement violation detections as well

03:09Okay

03:17After all this is done, ask and clarify with the interviewer about the functional and the non-functional requirements that helps you clear your scope much better

03:26So I created these functional requirements that show what the system must do, and what I have to do are detect human players in the camera frame first, and then estimate the 2D pose key points for each player

03:41Track each player uniquely across the video frames

03:43Detect the red light and green light phases during the red light

03:48Detect the movement

03:50Violations using the pose delta analysis, raise an alert or flag if the person moves during the red light, provide visual feedbacks like rounding boxes as well, and log all the events like player IDs, timestamp, movement, and decisions as well, and support a configurable movement threshold and post smoothly, and handle multiple players around the field as well

04:13And similarly, for the non-functional requirements, that means how well it does in the real-time conditions, that's what I have to take care about, is the real-time latency, it should be fast, and the accuracy, that it should work on a 95%

04:30Precision and recall for the movement detections as well

04:32It is scalable to the 100 plus players in the multiple camera streams as well

04:36It is robust that it can handle the different lighting conditions, different uh camera angles, and the wind conditions as well

04:46It is fault tolerance and logs and retries on drop frame and tracking errors as well

04:51It has the security and privacy

04:53It doesn't store or leak the raw footage unnecessarily

04:56It is transparent that it can uh log why it flags some player or why not with the threshold values as well

05:04So when this is done and finalized with your interviewer, then Talk about the matrices, offline and online matrices both, because offline matrices ensure your system cache violation without any falsely accusing innocence, while maintaining the clean, stable key points for downstream logic, and the online matrices ensure that the system should stay fast, stable, and trustworthy while deployed, especially when stakes are too high, like in the Squid Game

05:33So, offline metrics that I think best are the precision recall, key point stability score that measures the jitter across the frames when the uh player is stationary, and the tracking consistency as the

05:46Percentage of the frame when the player ID is maintained correctly and the frames drop rate as the percentage of frames skipped due to inference failure or the bottlenecks as well

05:57These are the like basic offline matrices that we should uh take care and to maintain as well

06:02We should skip the matrices like F1 score as it's a balance between the precision and the recall

06:10It is critical in the edge cases like a subtle movement, but not as actionable as pre precision and recall individually

06:18And we can also not use the pose estimation accuracy here because as the key point wise accuracy of the pose model

06:24It is not meaningful unless training your own pose model here at the first, okay

06:30And similarly for the online matrices, I think the best ones are the inference latency as it's a real-time system

06:37It takes 200 milliseconds, less time as the movement go undetected or feel laggy

06:43And then we use the violation detection rate that it is calculated as the movement violation movement violation detected versus the total red light events that can help us to see that how many actual violations it can detect in the red light phase, okay

07:02So after you have the requirements, you have the matrices, you have the problem

07:05Statement formula very nicely, then you can move to the high-level system design that I created here as a hybrid of ML and the rule-based approach

07:15So as you see in my design, how the flow goes from the video input to pose detection to pose tracking, along with the feature extraction, then all features go to the hybrid model and decision phase, and the output we have is the red or the green signal

07:30And for the model-based approach, I have to do pose estimation, player tracking, and optionally, we can use the movement classification or a binary ML model as well

07:41And then for the rule-based approach, I have to do the final

07:43Movement violation decisions use of the delta thresholds or the smoothing conditions, and apply a confidence score to say this person is violated or not

07:54And this trade-off of the hybrid decision system, uh, ensures we have a high precision and recall

08:02The system meets a very low latency requirement, and it remained explainable and trustworthy in the real world games

08:09And here is the pro tip for you: start always simple and iterate

08:15Always began with the rule-based threshold to, uh, to detect the movements as false positive or edge cases appear, incrementally introduce the lightweight ML models like Uh, binary classification models, uh, logic regressions, or the tree-based models to handle the ambiguity

08:31Now the question occurs, why this approach is best, why we use the hybrid approach

08:36Let me clear this out as well

08:39The first benefit of this approach is the problem is structured and well constrained because we have the input that is clear and standardized

08:48We have the output that is also cleared and standardized as well

08:52Environmental conditions like, uh, static cameras, winds, upright players, they can be controlled, and the conclusions, we don't need a deep learning end-to-end model here

09:02Instead, we can leverage the existing computer vision pipelines and make smart decisions on it

09:07And the next benefit is pose estimation, that we use the MediaPipe uh framework by Google here, that helps in 2D joint estimation, that converts the pixel to a structured XY points per player as well

09:21And there is no need of training from scratch

09:24It can work really better on the 50 to 100 players in the frame as well

09:29And it is easy to pipe into the next stages like tracking and the decision logic as well

09:37Pose estimation is a solved problem, that is, uh pose estimation is a solved problem in a more structured way, we

09:43Don't need to reinvent this, and why I use MediaPipe over OpenPose here, because MediaPipe for real-time production environments with 50 to 100 plus players are very good

09:56You, uh, but we can only use the OpenPose if I need a maximum accuracy, I have very powerful GPUs as well, and I can afford the latency as well, but here we don't afford the latency

10:09We, we, uh, we need a very fast system

10:12We need a very fast system here, that's why we use the MediaPipe

10:14And the third benefit is player tracking using the DeepSORT, because now the MediaPipe only helps us to detect the joints, but it can't identify each player uniquely

10:26So that's where we need the deep sort

10:28It helps in the maintain the consistent unique IDs across the frames

10:34It handles dozens of players per frame efficiently

10:37It can recover from the partial occurence or the dropped detections as well

10:41It works on the top of uh bounding boxes from the pose models as well, okay? Because we don't use a track by detection because, as I already said, the MediaPipe can lose player identities or uniqueness or swap them that leads to false positive or negatives as well

10:59But there is another tool that we can use that is called as FairMOT, but here we don't use the FairMOT

11:06Because FairMOT is more accurate on tracking benchmarks, but it is very heavy, and it is very hard to plug into the existing pose detectors, and it is overkill here

11:16So that's why we use this smart approach to use the DeepSORT here

11:21So, for a very start, we can use the logistic regression or the shallow MLP, because the ML classification is optional

11:28It is only used to assist in the ambiguous cases

11:32Okay

11:35So we can start with the logistic regression to support or smooth the rule-based systems, not to replace it entirely

11:41Okay

11:47And then the final question, why rule-based for the final movement deduction? Because movement is deterministic only during the red light, so if any joint moving is greater than the threshold, we call it a violation

11:54But how much movement is subjective? So that's where the ML helps before this rule is applied

11:59So here is the summary of the ML-based, rule-based, or the hybrid-based approach that gives you a clear view why we use the hybrid approach, because it has everything better and balanced for us

12:10Next, let's move to the data pipeline creation, our next step in the ML design framework

12:16So as we have input from the RTSP IP cameras or the local webcam footages with a frame rate of 50 to 30 frames per second and the resolution is 720 to 180p, okay

12:28And this problem statement, that we are uh doing a red light, green light game of Squid Game, can have multiple cameras supported via a wide coverage as well

12:38And for each frame, we do the frame processing, that is, we can resize the frame, we can normalize the pixels

12:45We can convert the BGR to the RGB using the MediaPipe, or one optional thing that we can do is background subtraction or the stabilization as well if the camera moves

12:58And then the next thing is feature extraction for the pose estimation

13:02We, uh, we can use a MediaPipe here that can use multiple

13:06Models like BlazePose or the Pose Landmark to give us 2D key points for each person, just like an ID, or the key point of the nose, left wrist, waist, all the like body points that it has, and for each key point, it includes the XY pixels coordinate with the confidence score

13:24And then we use the DeepSORT to create unique identities of each of the player for each input

13:30We detect the pose and/or the bounding box, and for each output, we got a unique player ID per frame across the time, and then maintain the player's identity, uh, despite any kind of occlusion or jitter as well

13:46And to detect the violation, we have these

13:49Features about the pose history buffer, that for each frame, while the light is red, we have to maintain a sliding window of the end frames of key points per player, that means during the red light time only, because it enables detection over the time and filter out the jitter via smoothing or multiple other techniques that we can discuss further

13:59Okay

14:09So that's how you, you can like only build these kind of features while the light is red

14:18Okay

14:28And for the light red only, we can create features such as the pose deltas, joint velocities or accelerations, normalized motion features because whenever a player Moves from the last frame to this frame, we have to calculate the pose deltas as well

14:37So if they, there is a pose delta, we have to calculate that feature, and we have to pass that feature to our ML model, and that ML model will decide that how much confidence it has that the player violation occurs or not, okay

14:52And then we have to pass all these features into our loss regression model, and that will return whether a player moved or not with the help of zero and one

15:02And for the role based approach, we can apply a hard threshold, uh, on movement matrices

15:08For example, if any limb moved that is by, uh, x pixels consider a

15:15Pose confidence and the consistency across the frame with the smooth joint signals, and the output is movement deducted, true or false

15:21Okay

15:24So whenever a person just shifts his arm, leg, or any kind of body part, and that changes from the last frame, that's considered as a violation

15:32Okay

15:40So all these features are created only in the red light phase only, because we can manual toggle our model to run via the control panel programmatic approach, because we need only flag violation during the red light

15:48So as you can see, this is the data sample here with all the feature and the label along with the red light game

15:53phase as well

15:58So here are the variable descriptions as well, for your help to know which kind of features are the best for the red light post violation detection model as well

16:03Now the part comes here as a model training

16:08So now we have the data covers with us from the uh, very traditional approach

16:14Now you want to create a actual ML based model out of it

16:16So here we are using a hybrid approach

16:19We have the pose estimation, we have the tracking, we have the feature extraction, we have the ML model, and then a rule-based approach to suppress the false positives for our training set, train data for a 70% data

16:25Our 15%

16:32data is validation, and the 15% data is test

16:38So to create our baseline ML model, our goal is, if we give some features from the joint motions to predict whether a player violate the rules or not, it can use a simple logistic regression or the tree-based model as a baseline, and uh, it can uh get all the numerical features from us and give us the binary output

17:00And for the rule-based model, we have the rule that means violation is equal to any joint movement that is greater than some threshold, for that can be called as a violation, and for this pros are that it is transparent, it is fast, it is deterministic, but the cons are, it is sensitive to jitters, so that's why we need an ML model here

17:22That ML model can work with the rule-based model to, uh, not give us the false positives examples here

17:31And the question is, how do you choose a good test data? Because for that, we have to select the clips or the frame with different players, lightings, and a collision that include a few of the true positives or a few of the true negatives as well, edges like minor jitters, or the background movement sensor, and that helps in, uh, test our hybrid approach very well

17:53And now comes part of hyperparameter tuning, and for that, we use stratified k-fold, uh, Hyperparameter tuning to tune our logistic regression model

18:03And now the question comes, why, why only these tools or models we use, why not other tools? So let me clear that as well

18:12We use the logistic regression here because it is very simple, interpretable, and it is fast to train and inference as well

18:20It works really well on numeric data and very easy to optimize as well

18:24On the other hand, the tree-based models are hard to interpret, risk of overfitting, need to fine-tune, and not good for real-time inferences, but we can use the tree-based model if our logistic regression is under fit, you

18:40Have a large amount of data and you want a better recall on the edge cases as well, and you are post-processing offline, not on the real time

18:46Okay

18:53Then why we use the stratified K-fold here, not the other K-fold methods, because the problem we have is binary classification with the class imbalance

19:00In the most of the red light frames, players do not move

19:02So if we use a regular K-fold, uh, then the random split of data happens, and some folds might contain only zeros and mostly contain ones, lead to unstable model training, and bias performance estimates as well

19:19But the stratified K-fold ensures that each fold has Same class distribution as the fold data set, and it preserves the movement and the no movement ratio in every fold as well

19:30So that's why I choose these over others for faster model and fulfill our non-functional requirements as well

19:38So now let's move to the online inference

19:43So whenever a video input got fitted to our pipeline, we perform data prep-processing and the verification using the following tools

19:50We use a MediaPipe for realtime 2D post key points

19:55We use the DeepSORT for player ID tracking

19:58We use the OpenCV to frame grabbing and the video decoding and the image deprocessing

20:02We use the Numpy and Pandas for the feature generation and processing

20:06We use Pydantic for the input and output schema validation

20:09We use all these, and I already clarified why these are the best in the previous slides as well, got it? And then, create some features using this, and while training our model, and for deployment, we can use FastAPI that is very lightweight, asynchronous, and Python native, and perfect for wrapping the model in the API

20:34We can also use Flask, that is okay, but has slow and synchronous functions that can be only used for the demos

20:42Another one is Django, that is a too

20:44Heavyweight and overkill for this serving predictions

20:50So, but and uh, for the servers we can use uh, Uvicorn, that is a synchronous server gateway inference server, the for the Fast API, that that is fast, scalable for our use case

21:02Other one is Unicorn, that is slower for asynchronous, but better for Flask, and it is only uh, used with Flask if needed

21:10So, at last, we choose the best offers, that is Fast API with the Uvicorn, that the Fast API with the Uvicorn is the best solution for fast and scalable deployment tools, and for the uh, deployment style, we can use Docker and the Kubernetes here, uh, as Docker helps us in

21:30Reproducible uh deployment strategies and Kubernetes helps us in scaling to more cameras and the larger crowds as well

21:37So at a production level, it works like this

21:40The uh the fast API app runs and predict endpoints receives pose and the tracking data, calls the classifier, validate it with the rule-based logic, and return the JSON that is violation confidence reason and the frame ID or the player ID as well

21:59And now the point comes for scaling and updating the pipeline or the model that is more crucial in the system design part

22:06The the problem we have is as the game size grows, so that the cameras grow, so That the players grow, so that the ML inference per second time also grows, because at a production level, our system should have many cameras, a real time, uh, latency should be very, very less, and continuous movements and continuous improvements, fail safe and monitoring as well

22:31So for scaling increase in demand, we can use parallel pipelines, video level, or the player level

22:38Video level parallelism helps us to process multiple videos or cameras in parallel and scale horizontally across the cameras

22:46A player level parallelism helps us ensure that within one video, we split the post tracking or Movement checking per player ID, granular CPU GPU parallelism as well

22:58But at this level, video level parallelism here, because that helps us to get a viewpoint from all kind of the cameras as well

23:04And when the size of the data and the model increases, we use the data parallelism here, because for data parallelism, that is recommended for this kind of use case

23:18We can split the data by players or the frames

23:20We can run pose estimation or tracking per player or each batch of frames as well

23:25We can send to multiple CPU cores

23:28We can cue based streaming by using Kafkas and Rabbit MQ as well

23:33And why we use data Parallelism here? Because each player movement decision is independent per frame, ideal for the real-time inference and easy to batch and retry field segments as well

23:47And why not the model parallelism here? Because model parallelism actually works for larger models like LLMs or ResNets, and here we are using very lightweight models like, uh, MediaPipe, DeepSORT, or a binary classifier as well, so there is no need for the model parallelism here

24:05And for the A/B testing and deployment, we need it as we try new movement logics at different thresholds or smoothing parameters

24:15We measure the impact of False positive and the negatives as well

24:19We compare the model one with the model two before the roll out as well

24:24And for the model monitoring, we can use MLflow, as it has versioning of models, deployment experiments, and it is tailored for the ML and great logging capability it has, and we can use Optuna, as it has hyperparameters tuning experiments, and it has very lightweight features, and other tools that might help us in that are for model health and drift detection, we can use Evidently AI, it create lightweight drift dashboards, and for live alerts, we can use Prometheus or Graphfana that can

24:59Help us in real time metric and alerting, and for the model failure logging, we can use Sentry Loki that helps in log exceptions and the data snapshots

25:08And for the inference time tracking, we can use Prometheus or StatsD that helps in catch the bottlenecks per model stage

25:15So for our whole project, we use this architecture, and all steps are done now

25:20So for this, interviewer might ask you some more questions about your choices, and uh, you should be ready why you use some tools or method, why not others with all the trade-offs as so for your help, I create some interview questions that you might study and check

25:37Based on the knowledge that we use to create our design as of now, the questions are like, how would you ensure that the violation detection pipeline consistently operates under the low latency with 100 plus players on screen and multiple cameras as well? Why did you choose MediaPipe over OpenPose or YOLOv8 pose, and how would you switch if real-time constraints fail? Describe how you evaluate the tracking consistency of DeepSORT, and what failure models you expect in covered scenes

26:10How would you mitigate them? Design architecture that can scale horizontally for 10 camera streams each

26:17Detecting up to 50 players in real time

26:20What components need to be stateless here, and why is precision more important than recall in this system? And would your answers change if deployed in a real tournament? So all these questions help you get a good grip of this project

26:33So just be ready and read all these question and answer that I already given here very carefully, and if you have any question, just ask in the comment section

26:44Just be with me, and in our next video, I will teach you how you can design the Spotify music recommendation system

26:50Just be with me, and if you want to learn about generative AI, Agent, AI, prompted engineering, or data science and machine learning, you can watch my videos on YouTube and also read my blogs on Medium

26:55We'll meet in the next video

26:59Thank you guys

27:03Thank you so much.