00:00So, there is a new company called Physical Intelligence, and they are really innovating the game of robotics

00:06In this video, I'll dive into exactly what they're doing and what makes them so special

00:11On their blog post, it starts by saying that we are living through an AI revolution

00:15The past decade witnessed practically useful AI assistants and AI systems that can generate photorealistic images and videos, and even video models that can predict the structure of proteins

00:26But in spite of all these advances, human intelligence dramatically outpaces AI when it comes to the physical world

00:32And to paraphrase Moravec's Paradox, winning a game of chess or discovering a new drug represent easy problems for AI to solve, but folding a shirt or Cleaning up a table requires solving some of the most difficult engineering problems ever conceived

00:47To build AI systems that have the kind of physically situated versatility that humans possess, we need a new approach, and we need to make systems embodied so that they can acquire physical intelligence

01:00Over the past 8 months, we've developed a general-purpose robot foundation model that we call Pi Z, and we believe that this is the first step towards our long-term goal of developing artificial physical intelligence, so that users can simply ask robots to perform any tasks they want, just like they can ask any LLM and chatbot assistance

01:20Like LLMs, our model is trained on broad and diverse data and can follow various text

01:27instructions

01:34And unlike LLMs, it spans images, text, and actions, and acquires physical intelligence by training on embodied experience from robots, learning to directly output low-level motor commands via a novel architecture

01:42And it can control a variety of different robots and can either be prompted to carry out the desired task or fine-tuned to specialize it to challenging application scenarios

01:52So this is where they talk about the promise of generalist robot policies

01:54It says today's robots are narrow specialists

01:58Industrial robots are programmed for repetitive motions in choreographed settings, repeatedly making the same world in the same spot on an assembly line or dropping the same item into the same box

02:09Even such simple behaviors require extensive manual engineering, and more complex behaviors in messy real-world environments, such as homes, are simply infeasible

02:19They state that if we could train a single generalist robot policy that can perform a wide range of different skills and control a wide range of different robots, we could overcome this challenge

02:30And such a model would only need only a little bit of data from each robot and each application, just as a person learns a new skill quickly by drawing on a lifetime's worth of experience

02:41Such a generalist robot policy could be specialized to new tasks with only modest amounts of data, and this would not be the first time that a generalist model beat a specialist at these

02:50specialist's own tasks

02:58Language models have superseded more specialized language processing systems because they can better solve these downstream specialist tasks by drawing on their diverse and general-purpose pre-training

03:06In the same way that LLMs provide a foundation model for language, these generalist robot policies will provide a robot foundation model for physical intelligence

03:16Now, to get there, they're going to need to solve major technical challenges

03:20They say that their first step is PiZ, which is a prototype model that combines large-scale multitask and multi-root data collection with a new network architecture that enables the most capable and dexterous generalist robot policy

03:34To date, and while they believe that this is only a small, early step towards developing truly general-purpose robot models, they think it represents an exciting step that provides a glimpse of what is to come

03:47And I personally do believe that this company is doing something absolutely incredible

03:51Taking a look at the majority of these demos in real time, I can see that the tasks that these robots are faced with are genuinely really hard

03:59How do you manage to train a robot on how to fold a shirt when there's a million different ways to fold the shirt? When you're picking up the shirt, it could crumble, it could fall this way, it could fall that way

04:09There's a million different scenarios that these robots do have to face, and watching These robots perform these actions and complete them successfully time and time again

04:17Shows me that physical intelligence, their first generalist policy, Pi Zero, is steps ahead on the other robot demos that I've seen, especially some of the ones that have broken the internet

04:28What you need to understand is that this is something that right now is fully autonomous

04:31None of the demos that you're currently seeing are teleoperated

04:35It's all fully autonomous, being done by their one single generalist policy

04:40Here, you can see their cross-embodiment training mixture, which is where Pi Zero, the brain, uses internet-scale vision-language pre-training, open-source robot manipulation datasets, and their own datasets consisting of

04:53Dexterous tasks from eight distinct robots, and then the model can perform a variety of tasks via either zero-shot prompting or fine-tuning

05:04The entire dataset contains diverse tasks, with each task exhibiting a wide variety of motion primitives, many different objects, and various scenes

05:11And the tasks in this dataset exercise different dimensions of robot dexterity while covering the range of real tasks that these robots might be asked to perform, from bussing dishes to packaging items into envelopes, to folding clothing, to routing cables, assembling boxes, plugging in power plugs, packaging food into go-boxes, and picking up and throwing out the trash

05:35And their goal in selecting these tasks is not to solve any particular

05:39application, but to provide their model with a general understanding of physical interactions, which is basically the initial foundation for physical intelligence, basically stating that look, we're not training it for anything specifically

05:53We're going to train it so that it understands a huge variety of different tasks, and that way it becomes smarter overall

05:59Inheriting internet-scale semantic understanding says that beyond training on many different robots, Piero inherits semantic knowledge and visual understanding from internet-scale pre-training by starting from a pre-trained vision language model

06:12And VMS are trained to model text and images on the web

06:17Widely used VMS include GPT for vision and Gemini, and we use a smaller 3 billion parameter VLM as a starting point and adapt it for real-time dexterous robot control

06:29Now, VLMs effectively transfer semantic knowledge from the web, but they are trained to output only discrete language tokens

06:36And dexterous robot manipulation requires PiZ to output motor commands at a high frequency, up to 50 times per second

06:44And to provide this level of dexterity, they state that they've developed a novel method to augment pre-trained VLMs with continuous action outputs via flow matching, a variant of diffusion models

06:59Starting from a diverse robot data and a VLM pre-trained on internet-scale data, we train our vision, language, action flow matching model, which you can then post-train

07:08On high-quality robot data to solve a range of downstream

07:10So, next, we have the post-training for dextrous manipulation, which is basically where they just want to train the model in a very specific way for certain tasks that are really difficult

07:21They basically state that fine-tuning the model with high-quality data for challenging tasks like folding laundry is quite similar with how post-training processes are employed by the LLM designers

07:32And pre-training teaches the model about the physical world, while fine-tuning for robots, per basically just makes it perform at a particular task really, really well

07:43Now, they did one task, which is, of course, laundry, and they fine-tuned Pi Z to fold laundry using either a mobile

07:50robot or a fixed pair of arms, and the goal was to get it to get the clothing into a neat stack

07:55This task is exceptionally difficult for robots, and of course, some humans

08:00While a single t-shirt laid on the flat table can sometimes be folded by repeating a prescripted set of motions, a pile of tangled laundry can be crumpled in many different ways

08:12So it's not enough to just simply, you know, move the arms through the same kind of motion

08:15And to their knowledge, okay, no prior robot system has been demonstrated to perform this task at this level of complexity

08:25Basically, what they're saying is here is that, like, look, this robot is currently state-of-the-art

08:28Robots can be folded in a variety of different ways, and ensuring that you have Repeatable success, which is what they're currently demonstrating here, is something that we simply haven't seen before, and it's particularly hard to do

08:41Now, they also talk about table bussing

08:43They said, "We also fine-tuned the model to bus a table." This requires the robot to pick up the dishes and trash on the table, putting any dishes, cutlery, or cups into a bussing bin and putting trash into the trash bin

08:57Now, this task requires the robot to handle a dizzying variety of items

09:01One of the exciting consequences of training Pi Zero on a large and diverse data set was the range of emerging strategies that the robot employed

09:11Instead of simply grasping each item in turn, the model could stack multiple dishes to put them into the bin

09:19together or shake off trash from a plate into the garbage before placing the plate into the busing bin

09:27This is rather fascinating

09:35We often talk about how robots and models have these emerging capabilities, but having this robot being able to stack multiple dishes, to put them in the bin together, or shake off trash from a plate to put it into the garbage before placing the plate into the busing bin is actually pretty cool

09:42It's quite similar to what humans would do in that scenario

09:44Now, of course, this is where they have something that seems absolutely incredible, which is where they got this robot to assemble a box

09:53So it says here, the robot has to take a flattened cardboard box and build it, fold in these sides, and then tucking in the flaps, and this is very difficult because each fold and tuck might fail in unexpected ways

10:04The robot needs to watch its progress and adjust as it goes

10:06It also needs to brace the box with both arms, even using the table, so that the partially folded box doesn't come apart, which is pretty incredible when you see exactly how this robot is able to do that

10:21There's a lot of subtle information that goes into training these robots, so seeing them do this firsthand is truly rather fascinating

10:28I know a few people that struggle to fold certain boxes

10:33Now, of course, they evaluate this model, so it says, "We compared the PiZ to other robot foundation models that have been proposed in the academic literature on their tasks." They've got OpenVA7

10:44billion parameter VA model that uses discretized actions and Oto, a 93 million parameter model that uses diffusion outputs

10:51And these tasks are very difficult compared to those that are used in academic experiments

10:55For example, the tasks used in the open VA evaluation typically consist of single-stage behaviors, which is, you know, putting an egg, a pot into a pot

11:05Whereas our simplest busting tasks, which is pretty difficult because it consists of multiple objects and sorting those into either a garbage bin or a busting bin

11:14And their more complex tasks might require multiple stages, manipulation of deformable objects, and the ability to deploy one of many possible strategies given the current configuration of the

11:25environment, and these tasks are evaluated according to a scoring rubric that assigns a score of one for a fully successful completion, with partial credit for partially correct execution

11:37For example, busting half the objects leads to a score of 0.5

11:42Now, the average scores across five zero-shot evaluation tasks are shown in the graph that you can see on screen, comparing the full PiZero pre-trained model to a small PiZero, small, which is a 470 parameter model that does not use VM pre-training

11:57So, OpenVA and and Oto, although OpenVA and Oto can attain nonzero performance on the easiest of tasks, PiZero is by far the best performing model across all of the tasks, and the small version of PiZero attains the second best performance, but There is more than a two times improvement in performance from using the full-size architecture with VM pre-chaining

12:17Now, they also state where they go from here

12:21They state that the mission of physical intelligence is to develop foundation models that can control any robot to perform any tasks, and their experiments show, and their experiments so far show, that models can control a variety of robots and perform tasks that no prior robot learning system has done successfully, such as folding laundry from a hamper or assembling a cardboard box

12:40But generalist robot policies are still in their infancy, and we have a long way to go

12:44And the frontiers of robot foundation model research include long-horizon reasoning and planning, autonomous self-improvement, robustness, and safety

12:53And they expect that the coming year, you'll see major advances along all of these directions

12:57But the initial results paint a promising picture for the future of robot foundation models, highly capable generalist policies that inherit semantic understanding from internet-scale pre-training

13:08They also incorporate data from many tasks and robot platforms and enable unprecedented dexterity and physical capability

13:14And they also think that succeeding at this will require not only new technologies but a ton more data, and of course, a collective effort involving the entire robotics community

13:24They've already got collaborations underway with a number of different companies and robotics labs, both to refine hardware designs for teleoperation and autonomy, and incorporate data from their partners into their pre-trained models, so that they can provide access to models adapted to their specific platforms

13:40Overall, I think Physical Intelligence is absolutely incredible

13:42They've managed to demonstrate that their models are able to successfully perform a variety of different tasks that we previously would have thought would be teleoperated, but now we can see entire operations from start to finish, completely autonomous in a zero-shot setting

13:56So, it will be interesting to see how this company develops over time.