00:00I think back to when I first started learning coding

00:02I was the kid that sat in the back of math class in middle school, and I had my little TI-83 plus calculator, and I would just program it with basic, because at some point I realized that I can actually program the answers of the for the math test into the calculator

00:19[Music] Hey, I'm Alex

00:24I lead cloud relations here at Anthropic

00:26Today we're going to be talking about cloud code and the future of software engineering, and I'm joined by my colleague Boris

00:30I'm Boris

00:34I'm member of technical staff here at Anthropic and creator of cloud code

00:38A lot has happened in The past 12 months, and things are moving very, very fast, especially in the coding domain

00:42For folks that maybe aren't following the news every single day, or even staying on top of the latest, and I have trouble myself sometimes, can you catch us up here on what's happened, and where are we standing currently? A year ago, coding was totally different than what it is today

00:59A year ago, if you want to write code, you have an IDE

01:02You have sort autocomplete in the IDE, and then there's chat app, and you might copy and paste code back and forth a little bit, and that was the state-of-the-art

01:10That was AI in coding

01:12And I think maybe sometime around a year ago, we started to see agents appear as a thing that people earnestly use in coding

01:21It's a part of the workflow

01:23It's not a gimmick or a prototype

01:25It's actually part of the inner loop when you're doing dev

01:28And I think this is the thing that's changed the most in the last year, is now when you code, you use an agent

01:37You don't directly manipulate text in ID anymore

01:39It's not just about tab, it's about the model writing code for you

01:44And I think what we've started to see is the shift from directly manipulating text to having the model do the text manipulation for

01:48You

01:51And I think projecting it out, this is the trajectory that we're on

01:55Is this continuing into the future? I see

01:59So we've gone from it all being within a web app where you're copy-pasting the code out and you're making very targeted edits, almost to just being a lot more hands-off and telling an agent what you want it to do and then trusting it to go make tons of edits and create whole apps sometimes even by itself

02:08Exactly

02:18And this was something that I think the reason we couldn't do it a year ago, and people have tried to make AI do coding for the longest time and to automate more and more of coding in

02:28various ways, and it hasn't really worked

02:29I think probably for a couple reasons

02:31One is the models weren't really good enough, and the second one is that the scaffolding, the thing on top of the model, wasn't good enough

02:36And when we initially launched Claude Code, the very, very first versions late last year, I think this is still using Sonnet 3.5, which wasn't even 3.6 or whatever we call this, the new Sonnet 3, upgraded Sonnet

02:50It wasn't even this, and it worked

02:52I used it for maybe 10% of my code or something, but even then, I remember when we launched it, we gave it to the core team, and it was just me and a few

03:00other people on the team at the time

03:01And I remember walking in one morning, and on the way to my desk, there were a few engineers sitting there, and one of them was Robert, and there were a couple other engineers, and I just walked in, and I saw Claude Code on their screen the first time, and I just gave this to them the day before, and they're already using it, and it was just the craziest thing, and the model wasn't very good

03:19The harness wasn't very good, but even in this early version, it was already a little bit useful

03:24And I think that over the last year, what's happened is the model has gotten way better at agent

03:29Coding, and that's happened with 3.7, and now 4.0 and Opus 4.1, and the harness has also gotten a lot better

03:38And obviously, the harness is quad code because the way you interact with the model, you can't just directly use the model

03:43You have to use a harness

03:45It's sort of like if you're riding a horse, you need a saddle

03:48The saddle makes a giant difference when you're riding a horse

03:49I'm not a horse rider

03:54I like that analogy, though

03:59I mean, it is like Claude is the horse, and as the engineer, you're trying to get it to go in a certain direction, and you're trying to guide it, and you need scaffolding around it to

04:04Be able to steer it correctly, and the harness in this case, just so we're on the same page, is everything from the tools we're giving it to how we handle context and everything for the model

04:08Exactly, exactly

04:13It's all of quad code

04:18The model is the thing behind the API, and then quad code is the, it's the system prompt, it's context management, it's tools, it's the ability to plug in MCP servers, settings, permissions, all this stuff, all this interfaces with the model, and the model sees all the context, all the output from the stuff, and it makes a giant difference in the way that it performs, and I, I think

04:36Over the last year, we've learned how exactly we build for the model, and the model has, with not just quad code, but all these different products that are using Anthropic models to build agent coding tools

04:42Maybe let's speak more on that

04:49When you say it's pretty organic, like at Anthropic, everyone uses Claude Code, that includes the researchers

04:56And so every day, the people building the models are using the model in order to do their job

05:00And I think as part of that, you see these natural limits that you hit with a model

05:06So, as an example, maybe the model's really bad at doing certain edits, and sometimes

05:10When you use quad code, you see, or another example, maybe something higher level is if you just let the model cook for 30 minutes

05:16With 3.5, it could do it for a little bit, maybe for a minute or something, it would stay on track

05:23And then with newer models, it gets longer and longer

05:26This amount of time the model can operate autonomously, and I think this is really based on experience because you use the model, you see where as a human you have to course correct and steer it, and then we've learned from that, and we can incorporate that into the model and teach it better to do this itself

05:40When you're Evaluating a new model, do you have a vibe check set of tests that you run, or if it's a new feature that we're rolling out to make something better in the harness? How do you personally evaluate if the performance is getting better? I just do my work that day

05:57Interesting

05:58Yeah

05:59My perfect day is I'm coding all day

06:00Mhm

06:01And whatever the model is, whatever is the new thing we're testing, I'll just code using that and see what the pipe is

06:06There, there's a specific thing I do, right? You just see how does it actually work for me in my day-to-day

06:08Yeah, exactly, exactly

06:10And in day-to-day work, you do all sorts

06:13of stuff

06:14You're writing new code, you're maybe fixing bugs, you're maybe reading Slack messages or GitHub issues to respond to feedback, and I think more and more the model is able to do more and more of this

06:24So actually, in a way, if you had maybe one thing that you always used the model for, you would miss out on some of these newer capabilities, like pulling in context through MCP, like reading your Slack messages, or automatically debugging stuff, 'cause you can pull in Sentry logs automatically

06:32Yeah

06:40So the best eval, in some sense, is the one that most looks like real life, and in that case, just using it gives You get the best result

06:48We, we tried really hard when building Claude Code to build product evals

06:53Yeah, to have some sort of benchmark, like when we change a system prompt or whatever, is the model getting better? And we have a little bit of this, but honestly, it's so hard to build evals, and by far the biggest signal is just the vibes, does it feel smarter? Because there, there's such a broad range of tasks they use it for

07:08Yeah, that's actually a question I hear from developers all the time, is they would appreciate more guidance on how we go about prompt testing and iterating

07:16I know for different products, we have various sorts of

07:19Evals that we've tried to create, but for Claude Code, it really is a tight feedback loop that almost gives us more immediate signal than any hard-coded set of evals

07:28I wonder if people want to hear a better answer from A, but yeah man, it's all vibes

07:33Yeah, I think at this point we're, the models are doing so good on evals like Swebench

07:37We're just trying to find these harder evals, and now there's like T-bench, which is a little bit less kind of saturated, but I think it's just really hard to find synthetic evals that capture all the complexity in software engineering

07:42Right

07:48Right

07:53Do you think there's something? We did uniquely to set up that feedback loop internally, because I feel Claude Code has the best dog fooding cycle I've seen of any type of product

07:59Initially, I built it the way that I do any other product, which is just listen to users and make it as easy as possible to listen to users

08:08And I think one part of it is, when we built Quad Code, there was a single feedback channel in Slack, and anytime anyone had feedback, I would just direct them to that

08:15And I think one of the things that we did really was, from the beginning, whenever someone gave feedback, I would try to fix it as fast as I can

08:22And sometimes I Would go into the office and then just spend 3 hours or 2 hours or whatever, just go through as many bugs as I can and fix them as fast as I can, and then every time comment back and tell people it's fixed, right? And this encourages them to keep giving feedback and, and to this day, the quad code feedback channel internally is just this fire hose, just non-stop

08:35Oh, totally

08:48I, I remember on those early days and still, still do, dropping in there, posting something and immediately you're emoji reacting or you're, you're asking for more clarification and more questions and you do feel like, oh, okay, my Feedback's being heard, and then you're able to actually be incentivized to go post more feedback in the future

09:00Yeah, because honestly, I don't know what I'm doing

09:02No one really knows what they're doing with AI

09:03We're discovering this thing as we build it

09:05Yeah, and the best indicator is what the users want, right? So you just, you got to listen, right? Switching gears slightly, what is like the current state of Cloud Code as a product? What are the latest features? What are you excited about? Some things that you're seeing folks do with it right now

09:20Cloud Code from the start was built to be the simplest

09:22thing it can and to be as hackable as possible

09:26And I think the hackability is something that we've been developing a lot, and that's something I'm really excited about

09:31So originally, the way to hack Quad Code is adding to its Quad, that was the original extension point, and QuadMD, as you know, is like this file you can put it in the root directory, you can put it in child directories, there's different places you can put it, and it, it's just additional context to give Quad Code, and it goes through your repo, you often check it into your codebase, so it's a little bit more information about the code, but over time

09:55We've added a lot more extension points, so now there's a very sophisticated setting system and permission system

10:00Mhm

10:02There's hooks now, which Dixon built

10:04Dixon's an engineer on our team, and he just saw all these different user apps coming in for, I want to hook into this, hook into this

10:11MCP, obviously, this is a really great extension point

10:13Um, and now there's slash commands and sub-agents, and user-defined slash commands is something we've invested in a lot

10:19And the idea is it's just a workflow

10:21It's like a markdown file

10:23You put it in your code, and it's something that you can reuse a lot

10:26So, for example, I Have a slash command for making commits, and I have some instructions in there

10:32It's like a slash command, but it has a forked context window

10:38And so you can think of agents and slash commands as two sides of the same thing, and this is also very exciting

10:41It's just another way to extend quad code

10:45And so when I look at the future, I think a lot of it is just about how do we extend quad code more? How do we make it easier for other people to build on top? How do we make the SDK more useful for people? So, it's useful for code if you want to build a coding agent, but also you can use it for other stuff, anything that you

10:59Need an agent for, you can just use the SDK for

11:03Yeah

11:03And I think these are the things that I'm the most excited about

11:05And obviously, all this benefits from all the other work we're doing to make the model more autonomous, to make it work for longer periods of time, to make it better adhere to instructions, to make it remember things better

11:16And so, everything along the way benefits

11:17So, I'm using cloud code or whatever form of it in six to 12 months

11:22What does my work actually look like? Am I reviewing PRs all day, or what, what does it day-to-day breakdown to? Yeah, I think there's going to be a mix of more

11:31Hands-on coding, I, I don't think that's going away

11:34Yeah, and maybe it'll look different though

11:37So, maybe hands-on coding today is directly manipulating text, but in the future, it might be using quad to manipulate the text for you

11:44Okay, and then I think there's going to be this other bucket of maybe less direct coding where Claude proactively does something, and maybe Claude even reviewed it, and it's your job to decide if this is a change that you want or not

11:59And I think maybe 12 or 24 months from now, we're going to start seeing quad that's more about goals and more about these higher-level things that

12:06It needs to do and less about the specific tasks that go into it, in the same way that as an engineer, I think about what is it that I want to do over the next month, and I make small changes to work towards that

12:19Maybe Claude will go through the same thing, right? So we're moving up and up the stack to some degree of these like abstraction levels of getting Claude to make individual changes to files, to getting Claude to make changes to a whole PR, to getting Claude to think about a goal of building an app or whatever else it is

12:36Yeah, okay, that's interesting

12:38If I'm an engineer and I'm hearing that, it seems Like there's going to be a lot changing in a very short amount of time, especially with my role and what I should be doing

12:48What's your advice for folks out there that are looking to prepare themselves and adapt to this world about what they should be learning or what skills they should be developing? I think back to when I first started learning coding

12:57I was the kid that sat in the back of math class in middle school, and I had my little TI-83 plus calculator

13:04It was a transparent gray one, and you can see the circuit, and I would just program it with BASIC, because at some point I realized that I can actually

13:15Program the answers of the for the math test into the calculator and you can get better grades that way

13:22And there's just something about this visceral feeling of being able to hack and having this idea of maybe there's this one program I can make and just I go into my calculator and I code it and then I can just restart and use it really quick, this kind of feedback cycle that was really amazing and it made it possible for me to build stuff that I never could have before and it was just so easy to get started

13:43And I think about the difference between that world and the world before agent coding where stacks just

13:48Got way too complicated

13:52If I wanted to make a JavaScript website, I had to learn about React and maybe Next.js JS, and then three different build systems and a deploy system, and it was just so complicated

14:04And I think one really cool thing about agents is that they're changing this

14:06So with coding agents, it makes it really easy to get started, and if you have an idea, you can just build it

14:13And it's a lot more about the idea now than it is about the details, because just like Claude Code, you can rewrite the code over and over

14:21Claude Code itself, we rewrite all the time, and I think this is just something that a Coding agents enable, the code itself is no longer precious, and there's still an art to writing it, and also code by hand sometimes

14:31And one of the engineers on the team, Lena, she was talking about how on the weekends she still sometimes writes C++ by hand just 'cause it's fun, and as a coder, this is a, this is, it can be a really joyous thing to do this

14:43But I think more and more it's going to be about the thing you make and not about the process of making it as much, and I think, I think my advice for people learning to code today is you still have to learn the craft

14:56So you still have to learn to code, learn languages, learn

15:00Compilers, runtimes, how to build web apps, how to build programs, system design

15:05You still have to know all the stuff, but also just start to get more creative, right? And if you have an idea for a startup or an idea for a product, you can just build it now in a way that you just couldn't before

15:16And we don't really understand what this means, but there's just so much potential that's about to be unlocked because of it

15:18Yeah, I love that

15:21I think that's great advice, too

15:24The ideas suddenly become something you can action on in a span of a few minutes almost, whereas before it could be just in your backlog

15:32Forever, yeah

15:35Before we wrap, I want to ask you, as a creator of Claude Code, what are your best practices for using Claude Code, and any tips or tricks? Yeah

15:44I think the biggest thing that I recommend, okay, maybe two, maybe two tricks

15:49Yeah

15:53So, one thing I recommend is that if you're brand new to Quad Code and you haven't used it before, don't use it to write code

15:58And instead, I know it's, I know it sounds crazy, explain, but you got to stop yourself

16:02Don't use it to write code yet

16:03The thing to start with is use it to ask questions about the codebase

16:08So, you can ask, why is this function designed the way that it is? Claude Code can go in, and it can look through Git history, and it can answer this stuff for you

16:16So I think ask Claude Code questions about the codebase and just don't code yet

16:19And then once you feel comfortable with using Claude Code this way, and you get comfortable with this idea of an agent that's doing this research for you, then start to use it to code

16:26I think the second thing is, when you are using Claude Code to write code, think about what work do you want to do and how big is the task

16:36So for something that's really easy, in my mind I have these three categories: easy, medium, and hard, very roughly

16:40And so easy task

16:44Are something that Claude can write in one shot, one prompt, it'll get it pretty much right

16:48And nowadays, I'll just go to GitHub and I'll tag, add Claude on an issue and just have Claude write the PR for me

16:52And this is how I do easy tasks, 'cause that frees up my terminal

16:55I don't have to spend it on this

16:56Medium tasks, I'll start in the terminal and I'll start in plan mode

17:00So just shift tab into plan and I'll align on a plan with quad first, and then once I feel good about the plan, I'll go into auto accept and I'll have it implemented

17:09And then for really hard tasks, I'm still the one driving and quad is more of a tool and I'm Pairing with it, but really, I'm the one in the driver's seat, not Claude, for this

17:16Um, and so I'll use Claude maybe to do some code-based research, maybe prototype a few ideas, maybe I'll just code a few options and to understand the boundaries of the system and what works well, but I'll still mostly implement it myself, and maybe Claude write the unit tests, but it's still mostly me doing the coding

17:35So, I think that'll be the second advice, is just think about what's the task that you're doing and what's the right way to use Claude Code to do it

17:38Those are great tips

17:40Really, really appreciate the time, Boris

17:43This has been Awesome, thank you

17:46Yeah, thanks, Alex.