Video notes · 2 talks · 1 h 45 min · 31+29 highlights · ~72 min to watch them all
How two engineers run more agents with less of themselves. Lauren Tan builds the checks that let her agents merge their own code. Felix Rieseberg stops being the conveyor belt between Claude and the work. Every point links to its moment in the video. Press a timestamp to play it in the player.
Lauren Tan
Engineer at Cursor, working on the agents window
SpaceX AI engineer, Lauren Tan
Ai insider · 55:30 · 2026-09-26
Felix Rieseberg
Leads engineering for Claude at Anthropic
Claude Engineering Lead: “Stop Using Claude Like It’s 2024”
Silicon Valley Girl · 49:23 · 2026-10-20
Both talks describe the same path. You cannot hand over work you cannot check. So the job is to build checks that do not need you, then widen what the agents may do alone. Lauren draws it as a curve: the more you trust the agents, the more of them can run at once.
Watch every output1–3 local agents
2:01–3:23Agents verify locallyYou watch them check
5:19–6:22Cloud agentsPick up bug reports alone
24:52–26:14Auto-mergeYou review on main
3:23–5:19An agent that can run the app, take traces and see the screen closes its own loop. Without that, you are the verifier, and you are the bottleneck. Lauren calls this the most important skill to have. It does not make the code good, but it makes the code correct.
You already have the parts: FlowDeck and sim-navigate for iOS, claude-in-chrome for the web. The gap is a one-command verification skill per app that a cloud agent can run with nobody watching.
A control skill is useless if the agent cannot find the feature. Lauren’s feature map lists every feature with the path a user takes to reach it, its shortcuts and its selectors. With it, an agent can turn a vague bug report or a bare screenshot into a reproduction.
Write a feature map for each live app: each screen, how to reach it from launch, and the accessibility IDs to tap. Keep it beside the verification skill and update it in every feature PR.
Lauren built her plugin one failure at a time. She read the tool calls, saw where the agent guessed, and wrote a skill against it. Every skill change then runs an eval. A coordinator writes a rubric, sub-agents run the task in neutrally named folders, and a judge on a different model checks the score. A loop repeats until every case scores 10 out of 10.
Your improvement logs already capture the failures. Add the missing half: a small eval per skill, run with /loop, so a skill change has to prove itself.
Felix warns that skills often encode advice for the last model generation. Sometimes removing a skill gives better output. Think about what the model needs, not what you would tell a junior.
When you eval a skill, also run the same task with the skill removed. If the skill does not win, delete it.
A session should rarely judge its own work. Fan out several sessions whose only job is to prove the output wrong. Then have one more session decide which criticisms hold up, and send those back to the maker. When the leftover criticism is weak, the work is good. Critique is easier than creation, so critics can run on a cheaper model. Hard outside evidence such as tests or metrics beats any judge.
Add an adversarial pass to any long agent run: several attackers, one judge, one round of fixes.
You stay the bottleneck while every output must pass your own judgment. Felix’s fix has three steps. Collect all the feedback you have given. Have Claude condense it into principles. Then ask which principles you hold without knowing it. Use that list as the brief for review sessions, and run the Claude review side by side with yours for a few rounds before you hand over.
Your improvement logs and past corrections are this raw feedback. Distil them into one principles file that every review agent reads.
If you enforce a rule by writing a review comment, treat that as a smell. Turn it into a lint rule or a CI failure, or remove the whole class of problem. Soft layers such as rules files, skills and style guides get forgotten. Hard layers such as architecture, the compiler and CI do not. Strict CI annoys humans, but agents absorb the annoyance.
Take the three corrections you repeat most often and make each one a hook, a lint rule or a test.
Agents copy existing patterns and take the shortest path. So make the shortest path the correct one: one folder per feature, import boundaries checked in CI, banned patterns. A vibe-coded greenfield app is the riskiest place, because it has no guardrails for the agent to copy.
For any prototype you plan to keep, add the constraints before the codebase grows, not after.
Felix used to pick a crash, paste it into Claude and ask for a fix. Then he saw that he was only the conveyor belt. Give the agent the source itself, such as all the crash data, the feedback channels and the business data, through well-described tools. Let it decide what matters. Lauren’s agent Benny does this for bug reports. It boots its own copy of the app in the cloud, tries to reproduce each report and reports back.
Point a scheduled agent at TestFlight crashes and user feedback. Its job: reproduce, check whether main already fixed it, then write a report. Allow PRs only once you trust the reports.
Six small moves, each tied to the moment that suggests it. Ticks are saved in this browser only.
Pick the app you test most. For each screen, write how to reach it from launch and which IDs to tap. Add it to that app’s verification skill.
Lauren 11:27–12:52Search the improvement logs for corrections you made more than once. Make each one a hook, a lint rule or a test that fails.
Lauren 46:22–47:20Write a rubric. Run three sub-agents in neutrally named folders. Score them with a judge on a different model. Then run the same cases with the skill removed.
Lauren 18:24–19:44A scheduled agent that reads new crash reports or feedback, tries to reproduce each one, and writes a short report. Start with reports only. Allow PRs later.
Lauren 24:52–26:14“Fire off five threads. Task each with proving this is wrong.” Then have one more thread judge which points hold.
Felix 13:57–14:33Write your own feedback first, then compare it with Claude’s, using your principles file. Note where they differ.
Felix 30:17–30:51The two speakers mostly agree. These are the three places where they pull in different directions.
Reconciled: Both measure. Keep a skill only while it beats a run without it.
We over-rotate on agents. Don’t invent a cast of characters. Split by work item, not role.
17:32–18:45Reconciled: Split agents by work item (an account, an app, a queue), never by role-play.
Reconciled: Give wide read access through tools. Put hard limits on what can break.
How she went from watching every output to waking up to 20 merged PRs: verification skills, evals, and a codebase that enforces its own rules.
Agents that guess, and announce a “smoking gun” for the hundredth time, lose your trust. With low trust you micromanage, like a manager who has to watch every report’s work.
“When you don’t have much trust in your agents, I feel like you really can’t get the most out of them.”
A year ago she watched every output of a handful of agents. Parallel work was impossible, because you cannot spawn 100 agents when you do not trust the output of one.
“You can’t go to 100 agents when you don’t even trust the output of one agent.”
Her agents now merge PRs on their own. She woke up to 20 merged PRs and reviewed them on main. She landed about 1,000 PRs last month, and almost 800 by the 12th of this month.
Let the agent meet the code the way users do: run the app, take CPU traces and heap snapshots, open the iOS simulator. This does not guarantee good code, but it gets you correct code.
“That’s the thing that really closes the loop.”
You prompt, open the dev build, and copy screenshots and console errors back to the agent. You stay in every loop, so nothing runs in parallel.
Teach the agent the Chrome DevTools Protocol for Electron and web apps, or Apple’s simulator tools for iOS, so it can drive the app and take traces. The agent can write this skill for you.
Her first control skill could launch the app, but it flailed when told “the left sidebar is laggy”. It did not know where anything was. A feature map file tells the agent how to reach every feature.
Reports are often a screenshot and “???”. With the map, the agent knows each feature’s user path, keyboard shortcuts and DOM selectors. PStack’s create-verification skill explores the code to draft the map. Its maintain-verification skill keeps it current.
She saw the agent name a cause without reading the code that mattered. The tool calls showed her exactly where to step in.
Treat the agent like a strong new hire with no business context. Skills are the onboarding. Her PStack plugin grew this way. Each failure she saw became a skill, such as “stop guessing, look up the code, use sub-agents”.
A coordinator agent writes a rubric and spawns sub-agents. Each works in a folder with a neutral name, because agents change their behavior when they know they are being tested. Run the eval across several models, and run it on every skill change.
Maintaining skills takes taste. Early on, read the tool calls and thinking blocks, and ask why the agent did what it did, as you would in pair programming.
The eval produces a score. A judge on a different model cross-checks it for bias. Then /loop runs until every case scores 10 out of 10. She built her control skill’s CLI almost hands-off this way.
You no longer cook the food. You design the kitchen, set up the stations and hand out the tasks.
Build verification locally, where you can watch the agent use the app. Once it works, cloud agents multiply it for the whole team, not only for you.
Benny takes each bug report and starts its own cloud computer running the app. Then it uses the same control skills to reproduce the report. In one example it reproduced a bug and found it already fixed on main, so the only action left was a release.
Jumping to thousands of cloud agents before you trust one wastes tokens and money.
First verification, then trust built locally, then cloud agents that pick up signals such as bug reports. Last comes auto-merge with review on main. There is no shortcut, because your own trust has to grow.
Large codebases are built for the least capable engineer, with frameworks, conventions and restricted credentials. The same guardrails stop agents from causing havoc.
“Before AI slop, we had human slop.”
With no guardrails, agents solve each task the most convenient way. The architecture grows “organically” until nobody understands it.
She spent 600+ PRs moving Grokbot to a strict architecture. Now she does not look at the code. Designers, PMs and go-to-market staff add features, and she does not lie awake fearing a performance regression.
A CI that checks everything is painful for humans to write against. Agents do not mind.
There is no hard size limit, but her agents split work into several PRs. The git history becomes rich context, and reverting a bug is easy.
In the “Dune” architecture, CI fails on useEffect. Code comments are banned too, because agents fill them with stale history such as “Lauren said never do this”.
“Everything you can imagine that the agents are bad at, we ban.”
Electron main-process code and renderer code live in separate folders. CI checks the dependency graph, so heavy code cannot slip onto the thread that draws the UI.
Agents copy patterns and take shortcuts. So each feature lives in one folder with a standard shape, and the quickest way to solve a task is also the right way.
Architecture, lints and compiler diagnostics turn CI red. Rules files, skills, Bugbot and style guides are soft, and agents forget them. Use both, but never rely on the soft layers alone.
With a strict compiler such as Rust’s, code that compiles probably works. You have less to check yourself.
Each time you enforce a rule by commenting on a PR, ask how to make it a lint rule or a CI failure, or how to remove the problem class entirely.
Refactoring and adding constraints cost many tokens up front. Compare that with hiring. After the investment, even small models write good code, and non-engineers contribute safely.
Grokbot looks like iMessage. Each agent has a name, for example one agent per customer account, or one that summarizes what Lauren shipped overnight. PMs now send fixes that she approves as they are.
Underlined passages are cited in the highlights. The line that is playing turns red. Turn on “Follow along” in the player to keep it in view.
0:00So, the first thing I want to talk about, it's pretty pretty serious. Okay? I need all of you to take this seriously. Put your phones down. Put your laptop down. Put your water bottles down. We're done hydrating. Are you all ready? Very excited to announce.
0:23[applause] Uh, you can't npm insult yet. We'll we'll release that later today. So don't don't take your laptops out. Um but yes, very exciting milestone and after I do the release, you can get started your applications by installing Babel plugin react as well as the new eslint plugin. Uh okay, so I assume you can see my screen.
0:44>> Yes. Yeah, we're good. [clears throat] >> So yeah, today Yeah, I think I think the big theme for me as I've been using agents to write code and I'm sure a lot of you have had the same experience as well is how do you trust it? You know, especially if you are an engineer that's been writing code for a very long time, you have a lot of opinions and lessons that you've learned about doing good engineering.
1:09And when you see agents just, you know, winging it and, you know, guessing, hallucinating, uh, you know, confidently stating that they found the smoking gun, uh, for the hundth time, uh, but it's actually not the real problem. You lose a lot of trust. And when you lose when you don't have much trust in your agents, I feel like you you really can't get the most out of them.
1:32And for me, the parallel is like with management. Uh so if I'm an a manager, an engineering manager of a team and I have a bunch of, you know, I have a team of engineers uh on my team and I don't trust them, then the mode of operation I'm going to be in is going to be like micromanagement, right? I'll have to spend a lot of time looking over my reports shoulders and checking that they're doing their work well, you know, that they're not shipping bugs to production.
2:01And so I drew this chart because uh it's not it's not a very scientific chart, but like this is how I imagine myself and my journey through using agents. So, you know, like fast forward or back forward uh or fast back uh fast backwards like a year or so when you know nobody was or not many people were using agents to code.
2:26Uh I think you uh you know get into this mode where you are in very heavily in the loop with one or several like a handful of agents and you find yourself just constantly fig uh you know trying to understand what your agents are doing uh and you're very very in loop. You're watching every single output.
2:48you are sitting there prompting um and you really can't parallelize beyond that because you don't again you don't have that trust right you can't go to a 100 agents uh like spawn 100 agents when you don't even trust the output of one agent so over the past 5 months I feel like I've really been able to uh like ascend this trust curve and now I'm at the point where uh I actually have this sounds kind kind of scary to say this and it makes me sound like a slop artist, but I I promise I'm not, but I actually have my agents
3:23now um automerging PRs for me. Uh which is like a wild thing to say, but um like I woke up today and there were like 20 PRs landed and I just reviewed them on main like they were already landed and they were good. Uh so how did I get to that point is basically what I wanted to talk about today. Uh and again like yeah feel free to jump in if you have questions Colin.
3:49Um but uh oh yeah of course I got to show this this chart. Uh where uh no do not trust to someone requested to control my computer. Uh probably won't do that. Uh but yeah, so this chart I think I I'm I'm sharing this chart not to kind of like flex but to kind of show like the journey like so you can see like the curve like it sort of like inversely matches the contributions I've been able to land at cursor.
4:22So I joined five months ago and five months ago like I you know my first month I was like not very productive because I was ob you know I was learning the codebase didn't know what the heck was going on and as I got more confident in in my agents uh I've really been able to kind of ramp up my productivity uh and again like yeah like last month I shipped a thousand PRs which is ridiculous.
4:47Uh, and then this month we're only on the 12th. I'm already at like almost 800 PRs landed. Uh, so the velocity is definitely high and you you I'm sure a lot of you will definitely be questioning like how how much of this code is actually good. Um, and I think yeah, like that's definitely fair to question. Um, but uh, yeah, I think I think if you set up your agents well, you can definitely get to a very similar level.
5:19Um, and so I'm going to talk about how we do that. Uh so for me I think I'm curious like I guess call in your experience as well but uh for me I think the most important skill that you should have in your toolbox when you work with agents is verification. Uh and by verification I mean the ability for an agent to actually run the code uh or take CPU traces or heap snapshots or uh you know open an iOS simulator whatever
5:55you know however your application is exposed to your users it can do the same thing and uh run it for real and actually test and verify it don't work because that's the thing that really closes the loop. uh it doesn't guarantee your agent writes good code uh but it allows them to at least write correct code uh which is a big a really big step forward for being able to trust your agents.
6:22Um I will I can share one example that we have uh within cursor. Uh oops where let me open this. Let's make this a uh make full screen.
6:46There you go. Uh so for the for cursor's agent window uh so this is actually an interesting story but uh when I joined cursor five months ago uh they're actually uh well I was supposed to join a different team I was supposed to join like the cloud agents team uh but then since I have a lot of experience working on react and agents window is a react application uh I was uh I was asked to basically help out with uh the agent window work.
7:20Uh but um there wasn't really a lot of like skills to help me. So I just found myself like okay uh agents window is going to launch in like a week, right? We have a really tight deadline and um there was uh you know I was just sitting there like okay I'm going to open up the performance the the Chrome dev tools and just like take a trace look at it myself and try to make sense of this flame graph.
7:44And keep in mind I was just like in my first week. So I had no idea what I was looking at. No idea where you know I mean I had some idea but you know the codebase was completely fresh to me. Uh and I realized like my agent had no idea either you know like I would take a screenshot of the tree. I download trades. I would send it to it and it be like yeah it kind of looks like this you know uh and it would like confidently state like it's this thing and then I try to fix that and turns out that's not the actual thing.
8:13So this was a very very slow process and if you've ever done any like performance work yourself or you know just even development with an agent where you don't have a verification skill you are the verifier right you you're the bottleneck you you you tell your agent to do something and then it goes off and write some code then you open up your you know local dev build and then you start to say oh you know doesn't work and you got to copy paste screen uh you know screenshots or console errors or whatever uh and then your agent like slowly kind of like uh you know works with that and then tries to understand it and um fix
8:49the thing but then you're constantly just in the loop and and being a bottleneck. So there's really no way to parallelize. So the control glass skill is like one of the first skills I built uh for cursor uh and glass by the way is the code name for agents window that we use internally but it's just cursor I guess. Um, and so this skill uh is I guess the the the code itself is not super interesting.
9:14Your agent can very easily make one for you. uh where if if you're building an Electron app or a web app or even iOS uh applications uh you can teach your agent how to use like the ChromeDev tools protocol or through uh Apple has some utilities as well for running the simulator and taking traces and controlling programmatic control as well.
9:39Uh so that's really useful. Uh but one thing I want to talk about is uh the this thing uh where is the read me? Uh so this skill comes with this very unique feature called or not feature uh unique file called a feature map.
10:00And so the story then is like I built this skill and so now the agent was able to uh actually run the agent window and take traces and whatnot. Uh but it had no idea what what the agents window was. So um you know like someone would say like oh the the left sidebar is like laggy or something like that or you know the right side the the PR tab is not working and the agent would just be like kind of flailing around.
10:29It would spend a lot of time trying to like look up the code and you know where is this feature? How do I actually get to it on the UI which made it basically completely useless. Uh you know like we would I would run the skill locally and you know it would spawn a dev build uh but then it just be turnurning like I just try to click here it wouldn't know how to get to things um and it was just an awful experience uh so was putting arrows on my screen um so uh yeah this this feature map has been really useful uh because
11:04it teaches the agent how to get to all of the features that you have um and in PAC the plugin that I I've made uh if you search for PAC cursor on Google you you'll find it uh but there is a create verification skill in that plugin where it actually helps you set up something like this for yourself um including the feature map.
11:27So it will actually explore the code and build up this initial feature map that tells your agent how to get to all of the different features that you have. Uh and this is extremely powerful because now like you have these user reports that come in uh you you can actually map even like a vague report or even a screenshot. So we have this uh internally at cursor where uh we have a slack channel with you know lots of people giving us feedback on the agents window and rockbot and whatnot.
11:58Uh, and often times the report is very bad, like very low quality, like someone will just put very often we get like a screenshot like and then someone just says question mark question mark question mark like what is this? And you know like the without this your agent like I have no clue, right? But with a feature map like this, it has a lot more context and understanding of how to actually navigate, how to get to all the different features.
12:24Uh so like you know example like I guess like the sidebar like what is the sidebar uh you know like all the different sub features that are present in it um like from the user point of view here's where how do I get to it all the different keyboard shortcuts uh even like the the what do you call it the DOM elements or yeah like the attributes that you use for selecting things through the CDP uh are all there.
12:52So uh again yeah this is like really really powerful uh for for agents >> uh and pstack ships uh that create verification skill but also a maintain verification skill uh so you can keep this up to date. >> Cool. Yeah, I was just going to ask how you created that. So do you mind sharing a little bit more about um that that process in the context of Pstack and maybe just what Pstack is for the folks who aren't familiar?
13:17Yeah. So, P stack is pretty interesting because uh well, first of all, the name is kind of goofy. like the P the P in P stack is like potato potato stack because I um so uh uh there is a pretty uh famous person Gary Tan who is the CEO of Y Combinator and he's come up with this plugin called GStack uh Gary Stack and uh funnily enough we share the last name we have no relations uh but I thought it would be funny to kind of you know poke fun at Gary and make Pstack my
13:53version of of of of his plugin. Uh but kind of tailor it to my own set of p engineering practices. Uh but I honestly actually never set out to build PAC. Uh it just started with a bunch of skills, right? Like I started with that control glass skill and then I started with another skill like called how which I also noticed through like observing agents.
14:18Um, so like you know in the early days of me, you know, trying to climb this ladder, I was like super in the loop and I was basically nitpicking my agents to an extreme degree, I was like I would tell it um, you know, this feature has stopped working. Here's a bug report. Like why isn't it working? And very often the agent would just like confidently state like, "Oh, it has to be this, right?
14:42It has to be this thing." And I noticed like when I looked at the actual tool calls, I noticed it wasn't actually reading the code that I thought should be affected. And that made me just extremely suspicious. And at that point, I was like, I'm not going to I can't trust any this agent anymore cuz it's just it's just completely hallucinating. And I think I think it's very easy to just you know like build up that distrust and not and kind of feel helpless like you know you don't know how to help your agents succeed.
15:14But like again I think the the the management analogy is super helpful because like imagine if you were a manager of an engineering team and you had an engineer on your team who was a really good coder. No business context whatsoever. you know they they just you just hired them and they they onboarded you know like five seconds ago. Uh and so how do you actually teach that person to be effective?
15:38So how you do that is through a skill. uh a skill being just you know it's just markdown right but you know it encodes a lot of information instructions a lot of uh you can really draw out a lot of intelligence from an agent by well some people on Twitter call it like you know pull the agent to a different latent space which is kind of like a fancy way of just saying like since uh you know LLMs are sort of like they predict the next token uh when you give it some high quality tokens uh to begin with then you know it it can kind of pattern match on like a higher space
16:13that's you know smarter. Um so that's like a very interesting model there. But yeah I built Pstack very very incrementally. Uh so uh started with just really observing how agents you know all the fail different failure modes of of that agents were having and every time I saw that I just okay I'm just going to make that a skill right like stop hallucinating actually go and search up look up the code use a lot of sub agents uh and yeah stop guessing >> yeah that makes sense one one
16:48kind of followup question here uh both from myself and from a bunch of people in the chat so >> I I guess it's two two parts. So one is like how do you maintain these skills? So like the product changes over time. Obviously there's a lot of people shipping against the codebase. So how do these skills get maintained? Uh and then second to that is like how do you know when your verification is is good enough? Uh like and you know you can trust that the ver verification loops that you've built are going to I guess you trust that the outputs uh when they're done.
17:17>> Uh yeah maybe I'll talk about um I think somehat related. Maybe I'll start with this one first. So like how do I maintain these skills? So um if you're not familiar with this concept, an eval [clears throat] is essentially like a way to uh well I the mental model I have is like it's like a unit test for an agent.
17:39Um and uh you can actually make your own eval. You don't need like a special framework for them. You can build you can you can build one depending on like you know how scientific and how rigorous you want to be. Uh my screen is red. >> Yeah, there's a little button. Um sorry. Do >> you mind like disabling the drawing or something?
18:01I I can't see my screen. >> Yeah, sorry if you guys could not draw on the screen that'd be great. But um there's a little button in the >> I got a troll. >> Yeah, the little drop down. >> How do I clear? >> Yeah, you got it. Perfect. >> Um, >> yeah. So, eval unit test your skills basically.
18:24And actually in Pstack, we ship uh under potato mode there's a playbook if you search for it called Eval playbook. Um, and it's uh uh it's like not it's actually pretty pretty rigorous the way it's done. Uh but um essentially what I do is I spawn a lot of different sub agents. I have like my main coordinator agent uh come up with a rubric for uh what I want the skill to do.
18:52Um and then it spawns all these sub aents and it's it creates individual directories for them uh which are cleverly named to not let the sub agent know that it's being evaluated because uh agents can actually tell and when they do they change their behavior. Uh, but it does a bunch of stuff like that to um essentially yeah like test whether or not the skill I'm making or changing is actually doing what I think it does.
19:22Um, and one of the really nice things about cursor is that we are we we support so many different models. So you can actually eval your skill across all sorts of different models. Um, and you know get a sense of how well it performs across that different matrix. um especially for the models that you use. Uh so I do this a lot.
19:44Every time I I modify a skill, I will run one of these uh like the EBAL playbook uh and make sure that you know it's actually leading to a result I want. Uh but I will say like maintaining skills is actually pretty hard. Uh it requires I think a lot of taste and observation. So you kind of need to be very good at being a backseat driver.
20:08You know what I mean? Like if you do pair if you ever done pair programming for example uh and you watch a coworker code and you just like you you could probably do this better. You know you could do you know like why did you not do this right? You you ask a lot of questions to your coworker. And it's kind of a similar thing here. You like you don't want to just be a passive observer of your agent. You want to be very in the driver seat in the initial stages when you're building up your own set of skills.
20:32uh you know obviously you can use something like PAC but if you're building your own set of skills it's very I think you know opening up the all the tool calls and like reading the code and reading all the uh the agent behavior and their thinking blocks is a really great way to see where they they fail right like what what you know where are they being done and then you can go and build a skill for that and then with verification how you trust it is it's I think it's also very similar iteration loop uh where you know like I actually did the same process for verifying the verification
21:08skill where I actually get um so one thing that's interesting about eval is that you can sort of hill climb them meaning that uh your eval can produce a score right uh a score that you can get your coordinator to produce uh but also comp uh you can have a judge agent of a different model to uh kind of cross reference and make sure that the first model is not being biased, right?
21:35The model that's judging all of the sub aents that are running the thing. Uh but you can also like hill climb. So meaning that you can you can use like /loop in cursor and you can say okay keep looping on this eval right until everything is 10 out of 10 as an example. Uh and I did the same the basically the same approach with the control skill.
21:56And so I kind of it was very it was very hands-off actually. Uh so you know I uh I kind of built I built that skill that way like the CLI in that skill. Um and over time it's gotten really good. Uh but yeah it was definitely not super smooth at the beginning. It required a lot of iteration and I think there's an analogy here for me which is um well I make this analogy later in a different slide on my drawing here.
22:24Uh but I think of it like uh you know as a as a engineer now you're sort of more like you uh like maybe a manager or the analogy I like is like you're like a a chef in a restaurant. Uh you know you're the head chef. Uh you're not cooking all the food yourself anymore. You have a team of cooks, right? You have line cooks, you have a sue chef, you have you know all these different stations.
22:49Um and it's your job to really design the environment. you know, you you're in charge of setting up the kitchen. You're in charge of, you know, like giving tasks to different people. So, um yeah, it's a very interesting way of working. Uh but yeah, that's that's how I've basically built uh these verification skills.
23:11>> Yeah, just just one thought there on like to go try to go one layer deeper. So, are you let's say we wanted to build um an eval or a skill for for something and we wanted to kind of get better on its own, which is is what I think you're suggesting. Uh are you doing that in like a work tree kind of isolated with like the sub agents and and then the reviewer agent and and all that?
23:33Is it happening like in some type of cloud hosted environment? Like what's the the more the practical steps? If I wanted to go do this uh and like set up a verification system for something, what would I what would I do or where would I start? Um I think that uh the best place to start is local because you can observe you can definitely observe what your agents are doing.
23:55So uh if you're building a verification skill for yourself uh I would definitely start local and just have your agent bring up the application whether it's like a CLI or a desktop app or whatever. And so you can actually observe right you can see how the agent is interacting with the the application. you can see it, you know, how it calls like the different APIs that that allow it to interact with the uh the application.
24:23Um, but uh for me personally, uh I have basically been kind of all in mostly all in on cloud agents because they're extremely powerful. Uh and the really powerful thing about cursor is the the cloud agents actually where if you spend a little bit of time setting up your environment these control skills these verification skills pay a huge amount of dividends because it's not just something that makes you as a single engineer better.
24:52It actually levels up your whole team uh and even your whole company because uh you can actually start thinking about cloud agents. change that thing about automations that automatically do things like uh I I get I kind of talk about this a bit later but I'll just kind of get into it. uh where where you know for example like I talk a lot about this agent we have called Benny right who who uh you know takes all of the bug reports that we get and it automatically goes off in the cloud opens up a cloud uh it's you know its desktop it runs cursor
25:28in its own computer and it uses the same control skills to interact with the application and try to reproduce the bug uh or the user report right and this is so so powerful because at I can immediately I I get so much information from this automatically like here in this example you can see that uh the Benny actually reproduced the bug uh but it's already fixed on main.
25:52So it actually confirms that we fixed this problem already and all I need to do is just release another build of of cursor. Uh so that's like huge information there that I didn't have to go off and sit with an agent, you know, and spend an hour trying to figure like is this fixed, is this not fixed. So you you you gain back so much time. Uh but you know everybody on my team benefits from this.
26:14Everybody at the company benefits from this. Uh so definitely think that uh you know keeping these using cloud agents is super powerful. Uh but yeah it's like a journey. You have to trust it first right before you you get to this point. And that's it goes back to what I was saying here where you know it's very hard. is it's almost impossible and I would definitely encourage you not to try to jump from, you know, like if you're still in this zone, you don't want to jump to like I'm going to spawn hundred of thousand or thousands of cloud agents right now because you're just going to waste a
26:49lot of tokens. Um, and it's going to be extremely expensive. >> Yeah. So just to kind of recap so far, basically the if we wanted to go on the journey that you've kind of gone on it, it would be to start with verification, building some some skills and some some ways of determining that the agents are producing at least like correct code. Whether like you said, whether it's good code or not is maybe a separate question, but like it's it's technically solving the problem by looking at, you know, stack traces, looking at, you know, the the actual behavior in the app and so on.
27:18Um, and then once we trust it locally, then we can start to think about scaling into the cloud and running more agents that are picking up signals, I guess, on their own, right? So whether that's like a bug report that comes in or something, they can go and pick it up and solve the problem and and give us back a PR. And then maybe the last step is like automerging the PRs, which uh is where you're at, maybe not where everyone want >> um and then and reviewing them on main, but um is that is that about right?
27:43>> Yeah, exactly. I think yeah, that's why I drew this this uh this this curve, right? because that this this basically describes my journey of you know when I started barely could use a couple agents and I was just observing every single thing. I think there's really no shortcut for going from here to there because this is really about your personal level of trust in agents, right?
28:04Um obviously, you know, as a as engineer, you don't want to just slop code into production. So, how do you actually build up that trust takes um a lot of uh I guess taste and judgment. Um but, uh you know, like I think plugins like Pstack definitely kind of help you uh get up to speed much quicker. Uh and so I guess it's like if you trust me and you trust PAC then in by extension you can maybe trust your agents but if you don't trust me and I I definitely would not encourage people to blindly trust me uh uh
28:40you know if you build up your own set of skills that you can obviously you know take a look at PAC and kind of fork it make it your own improve the skills definitely encourage that uh but for me it's really all about it just keeps coming back to trust you Every one of us here in this chat have a different standard for engineering. Uh and there are different things that are important for us in our codebase and uh when you are able to encode all of that into skills and you can verify that your agent is actually doing them that allows you to really kind of ascend this curve and um uh
29:15you know start automating things. Uh there's another piece I wanted to talk about. Um if there's more >> Yeah, go for it. I'll I'll pick up more questions as I go. But >> yeah, I think there's a third part to this which I haven't talked about yet, which is kind of an interesting one which is like refactoring and rewriting. Like one of the uh I guess most controversial one of the most controversial topics in the industry I think is like should you rewrite your app or not?
29:42Um because I think engineers are very prone to this where especially when you join a company you come in and you see like the codebase and you're like oh man this is like who wrote this code you know it's terrible I want to rewrite the whole thing there is a very common inclination and I think a lot of you know before agents um and I guess arguably even now people will definitely discourage you from re rewriting stuff but I'm actually here to make a case for why you might want to consider Um because I think it really depends.
30:16Uh you know uh brownfield applications I think are actually in a pretty good spot especially if they're set up well already. Uh and like recently I've been talking to some people but uh you know I I was just observing I I just noticed this parallel which is that a lot of big tech company problems are now everybody's problems.
30:38Um, and the big tech company problem, you know, like when I was working at Meta, like we had this giant monor repo, we had like, I don't know, tens of thousands of engineers just, you know, like banging on their keyboards and and shipping code and a lot of really great engineers at Meta. Uh, but, uh, I'll say like, you know, you'll be surprised that the code quality is actually not that good.
31:02Um, and so I often joke that like, you know, before AI slop, we had human sloth. Um and so uh you know I think a lot of big tech infra like uh like what Meta has or Google you know you know really big tech companies are actually designed for that where you're you're sort of like you're catering to the the you know like uh this sounds so bad to say but like the the least capable engineer on your team right you build you build frameworks you build conventions you build guard rails you know you restrict credentials so that you know your intern doesn't wipe your production database.
31:37Um there's uh you know if you have that level of infra already I think your agents can actually already do a very solid job right because they have the the guard rails are already in place for agents to not cause havoc or not cause too much havoc uh in your codebase um and you can always add more you know guardrails.
32:00Uh but I think like green field applications especially are you know like the brand new applications are like the biggest risk in my opinion. Uh and also the greatest opportunity because you know if you vibe code a project uh a prototype um like we did for Grockbot you know Grockbot was spun up very very very quickly. Um and if you if you haven't heard of of Grockbot it's like our a new application we just launched yesterday.
32:25Uh it's it's really cool. uh lets you orchestrate your create like individual agents that have their own identity and you can kind of orchestrate them. It's super cool. Definitely check it out. U but yeah, that was it's like a very it was a very green field application like most prototypes are so like vibe coded very quickly. Humans were not reading the code at all.
32:46And uh I had this tweet recently uh where I said something about organic architecture. Um maybe I'll find it. Uh but the idea is that uh when you have a completely vibe coded application, you essentially have no guardrails whatsoever. So uh your agents when you give them a task, they will just solve it in whatever method is the most convenient.
33:15And over time you get into this uh situation where you have a code base that is spiraling out of control because you don't understand it. Uh your agents understand it I guess in a way but like they've built something that is you know optimized for short for shortcuts. Uh and uh you know it will you will suffer you'll have a lot of of issues with that application.
33:40Uh so I think starting your codebase with uh like very strong constraints is very much needed uh because like when you have a code base that you can trust right when you have guardrails that actually help you uh uh help your agents write good code you can get into the you know like into this part of the curve where I I where like I I said you know I woke up today and I had like 20 PRs merged u by my agents and And that's because I invested a lot a lot of time uh over 600 PRs I I I calculated
34:16yesterday uh when I refactored all of Grockbot to this new architecture that I've been building. Um and yeah, I've gotten to a point where I I don't really look I really don't look at the code anymore. And um I say that not just, you know, to sell you tokens, but because I, you know, it it took a lot of work to get to that point.
34:39I spent a lot of tokens to get the codebase to this point where I no longer have to look at it. Uh but I'm very excited because you know of the potential where you know it's not just this doesn't just benefit me. It benefits everyone contributing to Grockbot and it also empowers you know designers and product managers and you know pe uh even GTM people to add features to Grockbot and I don't have to worry you know I don't have to to wake up at night in in the middle of the night and worry like oh someone's just merged a perf regression right I have a ton of constraints and CI
35:14like it's actually very annoying to write code in in graph web but like agents absorb all of that annoyance Um but yeah, I'm happy to talk about what exactly that is. Um >> yeah, I think one question um >> yeah, >> before we get into the this part here is just around that element of like what your your your CI looks like or maybe some of the constraints and then also like the average PR size.
35:38I saw a question about that earlier just to give people you know kind of a a glance. It doesn't have to be like mathematically average but just a you know like what generally the size of the a PR is. um if it's only a couple lines of code or you know um yeah >> uh um I think it depends uh let me >> I'm trying to do this in a way where I'm not going to like >> you yeah you don't have to share the actual number like an actual average >> I think this is fine >> benchmark >> but like we have so okay this is not that interesting but uh well fun fact is that virtualization in
36:14grobot and in uh cursor is actually powered by uh pretext uh which is a sort of new library that someone's built. Um that's really interesting. You should you should check it out. But that's not really that important. Uh I think the average PR size I actually don't know. I I don't know if I want to click on these.
36:35Uh I probably can, but I would say [clears throat] like they can range anywhere from a few hundred lines or 50 lines to like a thousand depending on what the thing is doing. Uh, so like here I'm actually like deleting a bunch of files. So I expect that it's just this like mostly deletion. Uh, but yeah, it kind of varies. >> There's no like Yeah, >> there's no like hard cap or hard limit.
36:58Are there all like 50 line PR? >> There's no hard cap. Yeah, there's definitely no hard cap. But I I do encourage my agents to split up their work into multiple PRs. Uh I do that mostly because uh I like I like the idea of the I guess maybe this is much harder to do now as in the world of agents and you have like so many commits but I like the idea that you know the git history is a very rich source of context.
37:21Uh, and I like the I like each PR to sort of atomically describe what that small piece of thing is doing, which also makes it easier for me to revert changes and like figure out, you know, oh, I shipped a bug and it's just it's here, right? It's not in this 40,000 line PR where you who knows what landed in there. Uh, but I don't have a hard cap on PR size.
37:48>> Cool. And then um yeah, also quick question on like CI. So again, you don't have to go into like uh the screen share of like your CI does, but just generally would you describe what the CI kind of looks like uh or how strict it is? >> Uh yeah. So uh well specifically for Grockbot. So Dune is the is the sort of cheeky code code name for the architecture that we've built for Grockbot.
38:13Um the CI looks pretty annoying because there's checks for everything. So like literally I have um uh well if you've written any react for example you know you know that one of the biggest foot guns in react is use effect. Uh so in uh Dune and in graphbot we've banned use effect.
38:34So Dune is just you can the the mental model of what Dune is uh you can kind of think of it as like Nex.js JS for uh electron apps and it's designed for agents to write uh and it's like custom for you know our agent powered applications. Um so the CI checks are very like specific to that like you know don't use use effect. It's it's it's banned like CI will fail uh and yell at you.
38:59We have like some of the more interesting ones that people might raise eyebrows is like I actually ban code comments as well uh which is very interesting. Uh, but I've noticed that 99% of the time agents just write code comments that kind of describe some historical thing that is actually totally irrelevant to the code.
39:20Um, like it will often say like, you know, oh, Lauren said you should never do this and it's now in in a code comment. I'm like, what? Like why what that was? I didn't say that as like a durable, you know, global rule. I just meant like your this PR sucks and you should change that part. agents don't really understand us that well surprisingly uh and or they kind of assume too much and they kind of do things in like very stupid ways.
39:46So like yeah we just ban everything everything you can imagine like the agents are bad at we ban. Uh so one example that we actually suffer a lot in the agents window is we have uh you know you if you've used the agents window you've definitely seen performance issues and you know we're constantly trying to fix them. Uh but it's like a it's a never- ending struggle because there's so many pull requests that get merged.
40:12Every any one of them could just regress performance or stability or reliability. Uh you know the agents window doesn't have this architecture yet. I plan to do bring this learning back there and kind of refactor everything there. Uh but uh it just regresses super often. uh because uh there's just one example is like we have very poor um isolation between processes.
40:37So like on you know on on electron you have a renderer thread that renders your UI but you also have like a main thread that you can run other code that you know doesn't need to block the renderer. Um, but we do a poor job of separating those things. And so oftentimes you just accidentally have code that gets pulled into running on the renderer thread and then all of a sudden you're competing with the the renderer that you know that has a very if you want like 60 fps you have to every frame that gets drawn has to be done in 16 milliseconds.
41:09So very very small you know deadline per frame uh if you want you know a very smooth product. Uh and when you start building bringing in accidentally bringing in you know things that are like very computationally heavy or they have a lot of I/IO uh then you just get into like a lot of jank right your your FPS really drops you start uh you know losing frames you get long tasks that take more than 16 milliseconds and you just get this really choppy experience.
41:37So all of those patterns that we've learned basically building electron apps we've encoded into this framework and it becomes like a hard failure. So I literally in in grabbot we literally have a directory called electron main electron renderer and we have uh import uh CI guess where we actually check the dependency graph to make sure you're not accidentally importing code from one directory to another.
42:03Uh so that's enforced by CI. um as well as bug bots uh which is our which cursors um like code review tool that runs on CI uh you know in our agents MD it's everywhere like so I I I I um I have this thing here where I I talk about like um you know like there are multiple layers I think for building a good codebase uh obviously the codebase is one where uh if you have an architecture like this where it's extremely strict uh you know the the the the way to build features is very conventional.
42:39That's like the strongest strongest level of enforcement because agents just love to copy existing patterns. So, uh one example of this in Rockbot is like we have this these concepts called like a feature and we have entry points and transcript cards like oh you know the cards that you see in the chat. These are all like like nouns I guess in in the framework and so there's a very conventional way of creating them and so like a feature is all in in a single directory as an example and so all of the code that contributes to that feature lives in one directory so it's all
43:14collocated in one place makes it super easy you know agents don't have to like uh grap around and try to figure out like where all the things are it just looks at the feature and like oh okay I'm working on the onboarding feature in Grockba. Uh I'm just going to work in this directory and for 80% of the work it's mostly just very encapsulated there [clears throat] but uh like it's like very it's like designed again for you know like the dumbest agent like you don't have to think right the the the one of the key principles I have for this framework is like the shortest the shortest path
43:50is the best path. So uh because that plays exactly to how agents love to write code is like they like to take shortcuts really you know they they find the quickest way to solve the problem. So why not make that the best way to solve the problem. Uh so I I probably won't get into all the specific details um and uh the this framework is really more of a collection of ideas and principles rather than something that will open source.
44:18Uh you can you can you know screenshot this I guess if you want and uh tell your agent to uh do some build build something like this for you too. Um yeah but it's really all about the layers uh you know like the the codebase is one part with features uh and directories and you know import or blocking import dependencies uh that shouldn't be imported uh but and and it all enforces that and static analysis.
44:46So like uh there's CI checks. We have a lot of lints for bad patterns that we observe. Uh compiler diagnostics. Uh there's also rules and bugbot which are um I think like three, four, five are more soft, right? These two actually make make CI red, right?
45:06So that you know there's a hard constraint where the agent can't just write crappy code for rules and skills and bogbot. Your agents can still forget, right? You can still or it may not always consistently apply them. So I like to layer them, but I don't I don't like to rely on them as the only source of enforcement because it's very very soft, right?
45:32And if you if you only have rules and bug bar and skills and a style guide for your code, you will it's only a matter of time before your codebase looks like complete trash. I'm sorry to say that, but uh I definitely recommend yeah like you know investing in you know things that can be hard and forced right and this is why you know maybe the choice of tech stack that you use is also very important.
45:56Um like I think for example Rust is sort of making you know it's like getting super popular again uh because the compiler is so strict right the compiler enforces so many different things you know there's a borrow checker that you have to appease and if as long as you make sure your agents don't write unsafe code blocks uh you can more or less feel somewhat confident that if the code compiles it probably works and it's good.
46:22Uh but you see it gives you that level of trust and confidence that you as a human engineer no longer need to go and check it yourself. You know you you rely on code and static analysis to actually make that uh a lot smoother. Um, and I I guess the worst part, the worst place to be in is if you are stuck in code review land where you actually enforce all of the constraints, the invariance in your codebase by literally the human person saying, you know, reading the code and okay, you
46:57should not do this, right? Every time you have to do that, you should consider that as a code smell, like a anti- pattern. And you should say, okay, instead of me commenting on the PR, how do I turn this into a hard rule, right? How do I turn this into a lint rule, how do I turn this into a CI failure? Or how do I even categorically eliminate this problem uh entirely?
47:20Uh I I can talk about another migration I've done, but I'll probably pause here. >> Sure. >> Yeah. I feel like that's that's where I am to be honest is is what you're describing right now which is that like I don't have all of these rules. So I have some things to go do after this session in terms of being able to scale my agents. I'm I'm definitely on like the uh you know maybe a couple of parallel ones locally stage.
47:42So like two to three locally and I'm sure most people here are on the same. So uh yeah I know we only have a couple minutes left. Lauren, was there anything else that you wanted to to highlight? I obviously there's lots of questions so I can grab more but I want to give you a few minutes if there's anything else you want to talk about. >> Well I think I've been yapping for quite a lot so I'm maybe let's just do questions. >> Okay cool. Uh one question that had uh a couple of uh came up a couple times was just around like token usage.
48:07>> So the the question is like is is what you're describing a realistic thing for people who are on you know uh a normal set of token usage. They don't have you know basically unlimited tokens uh to work with. I think that's a really good point. I mean, like obviously, you know, I work at a AI lab where we have unlimited tokens. So, uh I definitely cannot say that, you know, this is something everyone should do in the exact same way that I did it.
48:36I think it's possible to get to this point without, you know, breaking the bank. But, you know, if you're like an engineering leader or, you know, you're you you have a startup that you lead, um I think to me it's a question of ROI. Um, and it's like, uh, yes, you spend a lot of money on tokens in the upfront stage, you know, like refactoring a code base is going to take a lot of tokens.
49:00Uh, adding all these things uh, is going to take a bunch of tokens. But if we're heading to a world where agents are writing all the code and, you know, you want to be very lean, right? You don't want to have to hire, you don't want to be, you don't want to become like meta, right? Like I mean like in terms of you don't want to become a 10,000 person engineering org because I mean that's a cool problem to have but also you have so much overhead.
49:26There's like planning you know like you it's it's a personally I I wouldn't uh it it's not super fun but um I think you want to stay very nimble right and you want to you want to be like agents are all about allowing you to do things that you couldn't do before. That's really to me like the value of agents, you know, it's not just storing tokens on every single little thing.
49:49But um to me like the thing I couldn't do before is like enforce this level of constraints in a codebase by myself, right? Like I'm just a single person, you know? Uh it would have taken me years to build this framework uh and do all the refactoring and test everything myself and verify, you know, like run.
50:11Imagine if there it was just me, right? No, in in pre- agent era just like running, you know, by it would take me so long, right? And my salary is pretty high, right? Like so you know the the question I think an engineering leader might have is just then you know like what is there's a trade-off of do you hire someone to do this or do you spend the tokens to set up a codebase so that even the the most naive right the dumbest agents can do a good job.
50:40And when you actually get to this point like even agents that are not you know fable size do an excellent job of writing code and this pays a lot of dividends as well for me personally where I've empowered not just myself but again like PMS designers engineers who are not familiar with Grockbot to just contribute in a way that is sustainable.
51:05So, I think yeah, it's definitely like a trade-off for sure. You know, like nothing is like free for sure. Uh, and tokens are pretty expensive. Uh, but oh, actually, uh, I I I don't know how many of you have seen this, but we actually announced Grock 4.6 today. So, very exciting. Finally out. Um, so yeah, Groth 4.6 would be like a great it was very very smart.
51:26Uh, it's really good on the on the benchmarks. Uh, and it's the same the tokens. Uh well uh I hopefully I'm not saying this incorrectly but uh I believe the cost per token is the same as 4.5. So you're actually getting more intelligence for the same cost. Uh I think this is an area that cursor tries to cursor and SpaceX AI try to really optimize for like that heredto frontier of you know cost versus intelligence.
51:56uh you know we don't necessarily want to build the biggest model ever because that is extremely expensive to run. It's really about like how do you find that sweet spot right? You don't you don't need a giant model but it's just super smart right and it's not very expensive for inference. Uh but um yeah, I think to kind of round it up, um I think it's like a it's it's there's a if you do your own analysis, I feel like it's pretty positive.
52:22It it'll be pretty positive that the ROI you get from investing in stuff like this, uh just empowers not just yourself, but your whole team to be so much more productive, right? Like imagine if you have an army of engineers like me who are shipping so much improvements and and bug fixes. uh you know every day, right?
52:43Like that is pretty exciting. >> Cool. Uh one last question before we wrap up. This one is for the people in product on the on the call. >> So let's say we do have an army of engineers who are shipping like Lauren. I'm just curious like how is the product team or other functions of your company keeping up given that like if you're shipping so quickly, have are they using AI more to do their jobs?
53:05like as much as you can speak to that obviously you don't have like you're not in that role but just curious about how that works. Um I think this is where Grothbot has been actually exceedingly powerful. Uh where so before grabbot like you know uh obviously cursor only had cursor like we only had agents window we had a CLI we had an IDE and these are really like power user tools right like de they're designed for developers so it's very very developerentric you can do knowledge work in them but it like the UI is not really optimized for that.
53:39So we actually didn't really have uh well I think like a lot of people like you know GTM product like they might have used cursor uh to do their work but it definitely wasn't like a delightful experience for them. Um I think now with Grogbot uh it's become Grogbot is basically like the cusher moment for people who are not in tech in my opinion like it's like it's like a very very accessible way to use agents in a very comfortable very familiar interface.
54:10It looks like iMessage. Um and it's very fun to you know you can give your agent a fun name. uh you can have you can kind of do orchestration with in a very like natural way where you can sort of you know each agent is like a person right now you got a team of agents like working on you have one one agent per account that you manage as an example or if you're a PM you have you know you can have an agent that summarizes all the work that Lauren did last night and then now you know what I did right so I think our PMs are leveraging that a lot and they're shipping code too uh so you know like often times they will just say oh here's a bug I fixed it
54:45can you look at it and then I'll go review it and actually it's just perfect. I'm like okay stamp. Uh so uh that I think that shows that you know the the Dune architecture is holding up right the all the the really strict constraints allow people who are not experts in engineering to contribute at a high level. Uh so I'm I feel like I'm already seeing that pay off a lot where uh you know designers and PMs are just able to to to ship features directly.
55:14Um, and that just makes the Grockbot team super fast, right? Where we can ship so quickly. Um, and we have a lot planned. So, I'm very excited uh to, you know, uh to to ship more ship more
How the person who leads engineering for Claude works: prototypes instead of code, adversarial reviews, and agents that get the raw data instead of hand-picked tasks.
Agents brief him in the morning. He works at a higher altitude, part PM, part engineering manager, part data scientist. Most of the day is prototyping. Claude builds every variation, then the team critiques them.
Teams used to decide what to build from cost estimates. Now you skip to a prototype. The limit is human attention: he can build a thousand variants but can show you only one a day.
His signal: put it in front of a few people and watch whether it clicks quickly for them.
Automation removes tasks that are costly or keep you from work you enjoy. Augmentation lets you do what you could not do before. The most impressive teams change their workflow, not only their tools.
To really use AI, decide the outcome you want and how you will judge a result. People who reflect well on their own workflows get the most from it.
If an agent guesses the temperature and is right, that should fail. If it finds the location and reputable sources but is off by half a degree, that should pass.
A good writer prompts Claude to magnify the argument or find the strongest counterargument. The rule: ask Claude to take the same steps a professional in the field would take. In 3D, the big jump came from telling Claude to use Blender.
Give one Claude the task. Then spin off ten more whose only job is to prove it wrong, like red teaming or short selling. Finding flaws is easier than creating, so the critics can use a cheaper model.
He asks Claude to fire off five threads. Each one must prove the post wrong: wrong framing, wrong facts or a misunderstanding.
“Please fire off five threads and task each of them with proving that this post is wrong.”
Another Claude decides which criticisms are valid. Valid feedback goes back to the Claude that made the work, and that Claude may still push back. When the leftover feedback is weak, the output is good.
If you cannot tell good from bad, add more tokens: reviews and judges. If you can judge the result, act like a client and skip the process. He rebuilt his website with Blender assets and never learned Blender.
One thread and one instruction: find a primary source for every number, and check that it is a good source. He trusts the result 100%.
Generalist models keep beating specialist ones. A custom math model won silver at the Math Olympiad, and generalist models won gold months later. Inventing a cast of characters can limit what the model does.
Do not hand each agent a slice of the data. A good employee needs the full view: code, customer calls, usage and business data. Give tools, and document what data each tool holds so the agent pulls only what it needs.
Models now know when and how to use a tool without getting stuck in loops. So he no longer decides which data each thread gets, and he can run more threads at once.
Skills often encode advice for the last model generation. Sometimes removing a skill gives better output. This is easiest to see if you have an eval.
Pretend you are the model. You would want the raw data, a computer, and the freedom to install software and make your own decisions.
He used to pick a crash, drop it in front of Claude and ask for a fix. Now Claude gets all the crash data and decides what to fix. The setup costs more up front and opens many doors.
“I’m just a conveyor belt for Claude.”
Let Claude use your computer and log into your tools in its built-in browser. Once a workflow proves itself, give Claude a proper connector.
“Act like a great accountant” gets you a costume and vocabulary. Instead, ask what tools a top professional would use, and tell Claude to use them.
The things he will not trust to a single session outnumber the things he will not trust to Claude. Look for outside signal. Another session works in a pinch. Hard evidence such as metrics is better.
First, gather all the feedback you have given, from Slack, GitHub or chat. Second, have Claude condense it into rules. Third, ask which principles you hold without knowing it. Use the list as input for review sessions.
If you are the bottleneck, write your own feedback and compare it with Claude’s for a few rounds. See where they differ, and sharpen the workflow.
Claude Tag lets several people talk to the same Claude, which suits a shared feedback bot for a team.
Execution is now cheap. What has not changed is a founder’s ability to see what a user will pay for.
Hotel check-in desks, airports and dental offices still run awful software. Building a replacement is fast now. The relationships are the hard part.
Most people ask a “data center of geniuses” for the capital of France. What we believe is possible limits us more than the models do.
Ask what outcome you hope for next, and give the model that task. Keep raising it until you hit the limit. Then you are at the frontier, and you gain from each new model.
He expects all of it to need a revision in about ten weeks.
Underlined passages are cited in the highlights. Sponsor segments are greyed out.
0:00If all we're doing is like summarize this email for me, you're not really benefiting from the new intelligence. >> This is Felix Reeseberg. His team built Claude co-work in about a week and a half. Now he leads engineering for Claude at Anthropic and he doesn't write code anymore. >> I'm just a conveyor belt for Claude. Claude needs to have access to all the crash data and then needs to like figure out itself what it needs to fix, what it doesn't need to fix.
0:21>> Is there still anything you would not trust an agent with? I rarely want the current session to judge itself. That I think is still really iffy. >> How do I protect myself from AI just confidently presenting mistakes to me? >> It happens to a lot of us. I have like a very practical easy answer. >> Felix, thank you so much.
0:42I am very excited to have you. As a user of Claude, like an avid user, we're using Claude for all the operations on this channel. Every team member has their own project for the things they're responsible for. And as someone who runs, let's it's a pretty big team for a creator. I have 35 people and I want them to be as efficient as possible.
1:05I want to produce as much as possible. But also something that I've noticed this year, we were able to launch I think three new social media channels. So we launched a newsletter. We launched a LinkedIn a year and a half ago because you know it's text based. So AI has enabled me in many many ways. And the goal of this conversation is to make me even more advanced and all the viewers and all the listeners.
1:27>> I mean, from what what I've seen you do, it seems like you're pretty advanced as it is. >> But there is no borderline, right, to to being as true as possible. >> That's true. Yeah. [laughter] I think I'm discovering that about myself every single day. Like I'm sort of one of the most fun things of working on claw by far is this notion of I built this product and in comparison to my other products, with my other products, it was pretty clear to me what you're going to use this for, right?
1:48Like the example I always go with is I worked on Slack and like I kind of have a feeling of like what are you going to do with this? The thing that's magical about working on Claude is that I see people really fly and do amazing things that I didn't consider possible and that's like a that's like for a product builder that's such a treat, you know, to see people fly with this thing that you made. >> Is there something you've seen that's super ambitious and you're like, "Oh my god, I want to copy this." >> I'm not sure about copying, but two days ago um there's there's I think he's actually a product manager by trade.
2:15His name is Pavo and he pointed Claude at a large amount of NASA data >> and asked Claude to like find them a new planet and it did. So apparently it is not confirmed yet. NASA is working on actually confirming it. I'm on a little shaky ground here because I'm not an astrophysicist of any kind. But apparently Claude may have found a new planet we didn't know of before.
2:36>> This is fascinating. Okay. And it's the kind of thing that I would have never considered. But if you hear the story of I think for us who work maybe on more knowledgework things, right? The idea of like point AI at a large amount of data and you're going to find interesting things that in like a in like an in isolation seems pretty reasonable, right? Like something we do every single day. But the idea of like looking at all of the public data we have about space and considering at a similar problem of like go and see if you find anything in here we haven't noticed before.
3:03So beautiful. >> Can you talk to me about your day? You help build claude co-work but you said you're not coding anymore. So what happens in the morning? What do you do? Is that like your agents are briefing you on what's coming or what is it? >> That is true. My agents do brief me a little bit on what's coming for the day. I've really leveled up in sort of altitude on the problems that I'm thinking about.
3:26Right? Like sometimes sometimes I feel like these days I'm I'm much more concerned with human emotion, empathy, trying to understand how people work in a way that I think I was less so as a as a classic engineering manager a couple of years ago. So I think for me what I'm really discovering is that the amount of times I have to reach out to other professions to get their help has really dwindled down. I'm now this weird mix between like a product manager, an engineering manager, just a traditional people manager, a data scientist.
3:52Right? To answer your question very specifically, most days I'm I'm busy with a lot of prototyping, right? We just try a lot of things. We sort of like think about stuff. We sleep over it for a couple of nights, try to figure out if it's a good idea or a bad idea, and then we just go tell Claw to like build all the various variations and ideas we had. And then we need to do critique, try to figure out which ones of these have good ideas, which ones are bad.
4:13But I think prototyping is really what I spend most of my day on. >> How do you decide what is a good idea? Like what are the signals you're looking at? This has been difficult for people for a long time now and I'm not sure that I have all the answers to like what is a good idea, what is a bad idea. I think generally speaking, the way we look at good or bad ideas, the calculus has changed, right? Like if you think about products we've built in the past for the last 1015 years, we we always made implicit cost decisions, right?
4:40Like we sort of had to figure out, okay, especially for software, can I build this? What is the maintenance cost? What is the ongoing cost? Um how much how much time do I invest in this? What is the opportunity cost? It's all these decisions that engineering managers make to figure out, do I go and build this? Yes or no? And I think now we can fast forward to the end state much quicker and be like, all right, just build me a prototype. Let me play with this. Let's try it out. I think human attention is still rather limited.
5:02Let's say you, for instance, you have a lot of taste in AI, right? Like I can probably show you maybe one thing a day. I can easily build a thousand. I can't show you a thousand. Even if you're willing to sit down with me for all day, like at some point your eyes are going to glaze over and you're like, "Okay, I'm kind of done with this." So, what what makes a good product? I think especially now that we live in this AI world, there's this old saying that like perfect is sort of when you can't take anything away anymore and less so about like building a product that has every single feature under the sun.
5:27Like we can make as many buttons as you want us to, but I think good products actually have fewer buttons. I kind of know whether or not a product really works if we put it in front of a few people and you see things click for them rather quickly and they can do things that they couldn't do before. That that's really the signal I'm looking for. >> And how do you think about the underlying problem that you're solving? So for example, if you're came to my company now, how would you think about the tasks that you would fully automate with AI versus just leave it to humans versus maybe just automating parts of it?
5:56>> The first thing I want to say is that I think about automation and augmentation in slightly different ways. Automation can be really empowering if you automate away tasks that sort of keep you from doing work that that you enjoy or if it's like expensive and automating it a way like saves you a lot of saves you a lot of money or time. In those cases, automation is is useful. And I think it's really where AI currently probably provides most of the economic value, right?
6:21Like when we evaluate, is this worth it or not? We usually look at automation. The area I find most interesting though and I think where people really do things that they weren't possible with AI before is augmentation. It's this idea of AI is helping you do things that you couldn't do by hand because maybe you were lacking skill or techniques or um just the raw knowledge to do something. And when I go to company, the thing that is often most impressive to me is less so what the tool does and more so how you change your workflow.
6:50>> Does that make any sense? Yeah. >> AI is an interesting product because to really harness it, you kind of have to rethink how you do things. You have to like think about your outcomes. You have to figure out what does good mean, what does bad mean, like how do I evaluate whether or not this thing that I'm asking for, like whether the result is good. And the people who impress me the most are usually the ones who are very good at coming up with like very good at reflecting on their own workflows.
7:13>> Yeah. But it's so hard when I'm thinking about my work for example and a lot of people use AI for drafting texts like how do you define a good text like is it still possible for like knowledge work cuz for coding I guess it's possible right cuz you know what you want you don't want any bugs but when it comes to knowledge work and like the result of human creativity how do you even define what's good for the agent?
7:35Yeah, it's such a tricky problem. Like the the example I often go with is let's say your job is to like report on the weather, right? And you're like, "Agent, please figure out what the temperature is outside." There's like two ways to look at that because you have to rethink what good means, right? Like if the model just guesses a temperature and is correct, that should not be a passing grade. That should be bad. We should like rethink what we're doing here. But if the model is actually off by like half a degree or something, but it like um really figured out the right location, figured out some reputable data sources and like sort of the the the way it got there was reputable and good, that should actually be better than like a gas good result, if that makes any sense.
8:10But for writing and like all the other outputs where you feel like a lot of taste is involved and it's like a little hard to like prove or verify whether or not the result is like better than another. It is tricky, but I think I really enjoy talking to very good writers who are also a little AI friendly. >> What do they say? >> It is it is fascinating to watch them work.
8:30My friend Adam Davidson, who was a staff writer at a number of like pretty good publications, he has all of these various prompts for Claude where he's like, "Okay, like take my argument and put it to like magnify it by 100. Um, just mess with my text a little bit. Like what are the things you disagree with the most? Like what is the strongest counterargument?" It's interesting because I think to me as someone who's not a writer I think the naive approach would be given a text claude make this good and I think what I'm discovering more and more is that for knowledge work the secret sauce is like less and asking for a good output and sort of doing one step in between which is the
9:06best in the field how do they work and can we automate those steps >> this would kind of mimic their workflow but still the taste isn't there that's why I think you still need a lot of >> yeah I think That's true. I think that's true. But the interesting thing is that if you talk to >> really good writers, most of them would disagree that the text is really good just because they have like the magic touch.
9:27They would often say, "No, no, no. This is the result of hard work." >> Interesting. >> Right. Like it's this idea of like most artists have a process >> and many of them would actually probably find it I don't want to say insulting but like if you the more I talk to artists and the more I explore this idea of like is your stuff just good because it comes natural to you and it just like happens to be good or do you have a process that you like refined over the years a lot of them are quite proud of their process and like this is very very hard work like there's a lot like a lot of like manual individual steps go into making this good.
9:58It's not like I sit down and suddenly there's a book that's great. And if you go to like writer studios, right, there's like a lot of white boards and like a lot of information. And I'm I'm noticing this across a number of different domains where Claude is quite receptive to being told not I want a result that looks like huh, but Claude, I want you to take the same steps that a professional in this field would take.
10:19This is like very high manage. So, let me bring this down a little bit. Like I think we're currently in the public seeing this quite a bit with like motion, video, and 3D where I think Opus 55 has made a pretty big jump. But the big jump that the public has made is to just tell Claw to use Blender, right? Just like Claude, please use the same tools that professional animators use. And now the results are quite good. >> The unlock really is here in the media and the technique and the tools and is less so like raw understanding of what makes a good animated movie.
10:46SponsorLet me pause here for a second because a lot of the work behind this podcast happens long before we hit record. For every guest, we already have a pretty heavy research process. Our internal clot agent does the first pass, past interviews, articles, public posts, and the topics the guest always gets asked about and the angles people usually miss. Then my producer goes deeper.
11:09SponsorShe watches full interviews, reads industry pieces, fact checkcks everything, and builds the first version of the episode. Then it comes to me. I do my own research from my point of view and usually we go through the questions a few times before we record. That process gives us a lot of information what people are actually saying about the topic right now. So recently we added dig AI to that workflow.
11:30SponsorIt gives us a layer we didn't really have before. The real human reaction around a topic what people are asking arguing about and still trying to understand. Before an interview, we can look across similar creators and ask, "What questions keep showing up in comments but still aren't being answered? What parts of the conversation are suddenly growing? What are people confused about that nobody has explained well yet?" And then we can go straight to the source.
11:55SponsorAsk Dex scans through billions of videos monthly and sources the actual videos, the actual comments, and the language people are using so we can see the pattern for ourselves instead of relying on another summary. That helps us sharpen the questions before we record and avoid asking the same things every other podcast has already covered. So now we're not only looking at what the guest has said before, we also know what the audience still wants answered.
12:19If you do content marketing, research, PR, and honestly anything where you need to understand what people are actually talking about, ask Dig is worth trying. The link is in the description. And we're back to the video. But I wonder for people like you mentioned this prompt act as if you're a professional in this field right and it's going to go on the internet and research but a lot of information is just not there.
12:41How do you trust the output? Like for example if I want to start another medium [snorts] or like do discover stars right and I have no idea about that but I just tell it act like an asset researcher. How do I protect myself from AI just confidently presenting mistakes to me? >> Yeah that's a good point. I think it still happens to a lot of us. Um I I have I have like a very practical easy answer which is adversarial reviews um which has proven quite effective in coding math and like various other areas where we can give you numerical results on like how effective it is.
13:13Adversarial reviews basically boil down to the idea of you've given given one claw a task spin off another 10 that try to like break this down. Their whole task is to attack this and prove to you why it's wrong. It's essentially the same concept as red teaming or short selling. Are you using the same model for that or are you want to use another model? >> Kind of depends on the task. This is a little context dependent. Um but usually the adversarial review.
13:36It is it is sometimes easier to find flaws than to create new things. Much like as it is with humans, right? It's much easier for me to say whether or not the new phone is good. It's much easier for me to do that than to design a new phone from scratch. So usually you can like take a step down in intelligence and still get very good results. That's the very practical advice. >> How would you prompt that? Like if there is a process already, what's the additional prompt that you add? So you have some kind of output, right?
13:57Like let's say it's a slide deck or a document. Um or in my case, it's you talked about this blog post that I wrote this morning. I just tell Claude very I'm I'm using Claude Projects, which is this new product that we have that allows you to do work in multiple threads with a little bit more ease. And I basically tell Claude, "Please fire off five threads and task each of them with like proving that this this post is wrong, that I'm thinking about it the wrong way, that I have my facts wrong, that I misunderstood something, and then they go off with a given goal of like break this down, you know, prove that I'm wrong." >> How do you then deal with all the output?
14:33Because I do the same, but I use I just use different models when I'm prepping for the interview. like tell me if I'm digging too deep into this or that. But then I'm giving so much different feedback and I'm like okay now I have to decide cuz this model gave me this this gave me something else and gave this gave me something completely different. How do you then >> decide [laughter] what to use? >> I think you're going to enjoy this because you can see where this is going.
14:55The answer is once again clawed. What we use in coding context is something we call a judge >> where you get the output from the people who are like the the agents whose job it was to attack your output and they're like all right here's our main points of criticisms and then you use yet another cloud >> and their job is your your job is to validate and verify and like opinionate whether or not this is valid feedback.
15:17M >> um and then what you can sometimes do is that ultimately um depending on how you set up your workflow the feedback that is valid should probably go back to the clot that actually made the thing and that claw too that agent too can be discerning and whether or not to act on feedback right because sometimes um and we see this in a coding context fairly often I also see it in like the knowledge work output that I make sometimes the feedback is like maybe reasonable but um has missed a point or it's actually not that actionable or is not that useful and That's usually when I know that my output is pretty good.
15:48That's one way. That's when you are really quite uncertain whether or not the output is good yourself and you can't easily check. But for a lot of a lot of things that I create with AI on the other hand, I actually don't need to fully understand the entire flow because as I mentioned like to me as a person, it is I do have opinions about almost anything I use every single day. I've remade my website with all of these 3D assets. I have no idea how to use Blender.
16:10I've never used it. But I can sort of judge whether or not this looks good, feels good, and that might be enough, right? like you sort of put yourself much more in a client position. For so many other things in our life, we don't really have a good way of verifying whether or not we get a good output other than like giving feedback to the person we're using, right? Like for a contractor or a lawyer or like for many people a doctor, right?
16:31So you sort of just put it in one of these two buckets like either you really have no idea what goods like good looks like there. My answer is more AI just throw more tokens at it. or you do have you do have an understanding what good looks like and then maybe that's enough and you don't really need to like have opinions about the process itself. >> Yeah. I think it it comes down to uh making sure you select only the top projects where you want to understand every single step instead of just want to control everything.
16:57>> Yeah. Yeah. It's like the old problem of like sort of being willing to let go, right? Like how to not micromanage. It's really hard. >> Yeah. Comes Yeah. Now we have this problem with agents. Is there any knowledge process that you have nailed with agents like you completely trust them? They just go I don't know maybe it's >> yeah fact checking like editorial factecking I fully trust now the output I get 100% if I ask claw to verify that all the facts that I'm stating in the slide deck and this presentation um it doesn't even need that many rounds right like it's not like I have 50 agents doing that I really trust one agent I'm like I need you to find primary sources for every number I mention I
17:32want you to verify that it's a good primary source I fully trust that 100%. When you say agent, do you have a special agent for factchecking that you've trained or is just an one agent that you're using for everything? >> It's just just a cloth thread. I feel like this is going to be a spicy take which I think you'll like. I think we're kind of over rotating on this agents thing. And I can back this up with a little bit like I think models are really good generalists and we've seen this now time and time again.
17:58Like when we first started looking at math as an industry, I think the the first company to really look at math like with with a very serious lens was Google Deep Mind and they trained a custom model and a custom harness and they unleashed it on the math Olympiad and they got a silver medal which at the time was state-of-the-art and then four or five months later all the major generalist models come along and win a gold medal.
18:20I understand why people use agents. I have empathy because it's easier for us humans to rationalize what AI can do if we sort of like put on a little hat like you're a data scientist, right? But that same data scientist has all the intelligence as like a fieldleading biologist or mathematician or what have you. >> And I think we're we're sometimes limiting ourselves in our use of AI if we sort of force ourselves to first like make up this cast of characters.
18:45So for a lot of these things that I just meant, I don't really have any agents who are being told who they are. But does the data matter because for every agent you upload certain data set or just because your cloud has access to all your tools it doesn't really matter what you feed to each particular age but then isn't it a lot of data for it to consider when answering each particular question >> that is true but it's quite discerning in what kind of data it wants to use right like I think the right approach here is not to make these slices of data and then give every single agent the slice of data that it wants because many of us if we want to be quite effective at our job we do actually need the full view on the entire business, right?
19:21We're not quite effective if we only get a small slice of data. Like the as an engineering manager, if you only give me code access, I'm probably going to be much less useful to you as opposed to if I also have customer data, access to customer calls. Sometimes I do want to sit in like a like a customer call like when we're trying to sell something, I want to have deep usage information. Ideally, I get business data about what is working, what isn't.
19:42And you want your agents to have the same range of data. Now, how do you deal with the fact that it's probably a lot of data? tools. You give them tools and you document really well what kind of data is behind this tool so that they don't actually have to ingest the whole thing, right? You're just like, if you do want business data, here's the tool to get business data. >> So, when you're setting this up, cuz I'm thinking like when you mentioned this, should I move away from all my projects that have access to all the like specific data?
20:07For example, I don't know, there's a project for ranking a guest and we have the whole uh description of the process and then all the old scripts there so that we know how each guest performed. But now that you're saying this, does that mean that I should move away from that project? Tell my claude, okay, we have this is a folder with all the scripts. If you want to look at them in notion, we have all the intelligence on how every podcast guest performed.
20:30And uh this is how we rank guests and just leave it there. And whenever I want to rank a guest, I just start a new chat without telling it anything and just tell it rank the guest and it's going to >> Yeah, I'm careful to be prescriptive in workflows. >> Like I always say, if it works for you, it works for you and I don't think you should change it. The problem is the bottles are getting better and I want to get better. So this been working for this year but a year is a lot.
20:53>> Yeah. Okay. Well, I mean if if you're looking for the recommendation that I give for like what do I do? >> Again, everyone should do whatever makes them happy and comfortable and what makes them powerful. But it is true that the models have gotten so much better at tool calling and discerning tool calling. Um and tool calling is this process of like the model knows when to use a tool, how to use a tool. Doesn't like get stuck in tool call loops.
21:14doesn't accidentally say, "Oh, I found a data tool call. Let me call this tool until there's no more data left." Right? Like >> um the models have gotten particularly good at efficiently and effectively using tools. >> And for that reason, I don't really make this a priority decision anymore and what kind of data do you need. >> Um it also allows me to like have more threads going and more sessions going without >> easier process.
21:36Much easier process. Do you still use skills? >> We do use skills quite a bit still. Although um our official anthropic guidance as well as my personal guidance is be really careful with skills. We often tell models things that were sort of relevant for like the last generation of models which time is so compressed in AI right that the halflife of a good skill is very short and sometimes what we see is if you completely remove the skill you actually get better output.
22:01>> Yeah I think journey said that like just forget about that just just prompt >> just prompt and see what happens. >> Yeah it is it is very effective. This is obviously most useful to people who have some kind of evil and have an idea of what good and bad looks like. But I'm pretty careful and very discerning and when when I hear skills and when I don't. Okay. >> You know, you know what a what what a great way to think about this is I like to consider the model experience as opposed to like user experience.
22:24>> And model experience is you pretend for a second that you're the model. You're claude, right? What kind of things would you want if you had fulfill whatever task you're being given? And you very quickly discover that you actually do want access to the raw data. You want access to a computer. You probably want to be able to install some software. >> I want to make some decisions myself without somebody telling me what they think already. I just want to see exactly what you said, raw data, and make my own decisions.
22:48>> Like I think the upfront cost is higher. Like if you like even a year and a half ago, I was in this world of like telling Claude, I found another crash, please go and fix the crash. Right? But it very quickly dawned on me, I'm just a conveyor belt for Claude, right? Like I go look at our crashes, pick one up, move it over, drop it off in front of Claude. I'm like, "Please fix this." I'm like, "Okay, this is this is no, I'm just going to claw it needs to have access to all the crash data and then needs to like figure out itself what it needs to fix, what it doesn't need to fix." The upfront cost is really worth it, but it's also like important, I think, to consider that there is upfront cost, but once you set this up, um, it it
23:23opens up so many doors for you, and that's where I find a lot of like real hard value coming out of AI. >> If you've been hearing about AI agents and wondering how you'd actually set up one, I put together a guide that walks you through your first one in about 20 minutes. It helps you choose what to automate and get it working with no coding required. You can get it through my free newsletter. The link is below. For someone who just who's listening to this conversation and who thought they would need like a major setup to start using AI correctly, would you say uh if somebody is starting just today, they're using it for knowledge work.
23:54They don't need all these tone of voice things. Like my business strategy, you just give it access to your email. It learns from it. From your tone of voice, you give access to your newsletters or whatever. So you don't need any of that. >> The best way to get started with claw today, I think, is step one, download the desktop app. It has a built-in browser where you can log into some system you're using and forget about connectors, forget about like downloading all the data, forget about like all kinds of things.
24:20Just let Claude use your computer a little bit. Sort of give away the keyboard. That gets you so far and really shows you what works and what doesn't work. And then at some point, you might discover, oh, Cloud is really helping me with whatever tool you're using, whatever internal or external tool you're using. And once you see that a workflow really works for you, you might consider, okay, can we give Claude something better than like having to manually click in this dashboard around?
24:42Can we give it some kind of connector? >> But just starting with a browser gets you so far. >> Okay. And when it just to wrap up with the agents, when you say agents, that that means you have just basically chats, right? And that's it. There there's no like specific agent who does research, no specific agent who does your tax returns. >> No, no. I I mean I do have projects so I do think about I do think about the different work items but I don't think about different roles.
25:07>> Mh. >> Right. So I don't I I never tell Claude you're an excellent whatever. >> Um I think this was this was very invogue sort of like 2024 we were like Claude you're an excellent engineer please go fix whatever problems are here. Um, >> but you still use that in prompting now. Act like something that we started with. Act like a top whatever.
25:28>> I specifically implore claw to like tell me what tools someone top of the field in this field would use and then I'm like I want you to use the same fields. Um, a problem most models still have today is that if you simply tell it data science is maybe a good example or like accounting. A lot of the models still have the problem that if you tell them to act like a very good accountant that they engage in a little bit of role play >> and role play is not really what we're going for.
25:51We want the techniques, the tools, the knowledge that a professional would use. We don't really want just the vocabulary, right? Like I don't I don't need claw to sound like >> I don't need to put on a costume. I actually wanted to talk to me like I'm five, but I wanted to use all the top tools in the field. >> Is there still anything you would not trust an agent with? >> That's a tricky question. we can just keep layering more validation on top.
26:15So I think the question for me is much more it's less about it's just about what do I not trust Claude with and much more about uh what things do I not trust one single session with. >> And I think the the list of things where I don't trust one single session is actually much larger than the list of like things I wouldn't trust Claude with. I rarely want the current session to judge itself. that I think is still really iffy >> like the current cloud session that is running wherever you have it in cloud code in cloud co-work in chat in even in a project the current session tends to be quite bad at being a being a judge of whether or not the work it did was
26:50quite good. So as much as I can, I try to find that outside signal and the outside signal in a pinch can be another claw session that has proven to be quite effective, but my my preference is to find some hard outside evidence for what works or what doesn't work. And sometimes you need to like trust the process a little bit and give Claude a little bit of time. Like for instance in your field, I'm kind of making this up, but I assume that maybe like the number of views or subscribers is like an important metric that you >> Oh, like the number of views that's that's what we're whenever I give a task to Claude, I tell it like we need to hit this number of views.
27:22>> Yeah. >> Go and try. But you that and that's that's [snorts] something that we're still like struggling with is this feedback loop, right? Because it's me giving feedback to my team and then my team actually going and posting something else, not what Claude gave us. It's like maybe 70% the same, but then the 30% that I've added is actually my taste.
27:43>> Yeah. >> How do you close that loop? >> A thing that has worked for us is there's probably some kind of place where this feedback usually happens. Is it usually in person or does it usually happen? >> It's for us is on Telegram. There isn't a native Claude connection. We have a bot that scrapes every chat, but it's still very manual. I have to trigger Claude to look at the bot, look at the feedback, and it's still not it's it's not there.
28:05even like looking at something we've sent, right? >> Yeah. Because we do we do obviously again in a coding context we have a lot of engineers who you sort of know which engineer you want to review a certain thing that you're working on and the kind of things that they look for. Claude is very good at figuring out from the feedback. It's pretty good like given all of this feedback that I've read >> feedback on Slack, right?
28:28>> Yeah. On GitHub, Slack, wherever else the feedback might happen. And you didn't tell it to look at feedback specifically. Just because you gave it access to Slack, it's looking for feedback cuz >> yeah, that is much easier to do. Like if it has the tools and you don't have to like as a human yourself figure out, okay, let me assemble a list of all the feedback I've ever given. >> Um, but if you do this like three-step process of step one, assemble all the feedback you've ever given.
28:49Step two, tell a claw to like condense that down into a list of rules or principles that you care about. Sometimes it's useful to like then again do a third round of like okay what are the principles that maybe you might believe but you don't actually know you believe and then take that list as like an input for your ongoing reviews that is that has been quite successful in our attempts to I want to say like specialize the feedback that individual sessions give >> um that has really worked I think for for work that is a little different than raw coding or not very textbased I think the problem is still bigger
29:26Um but I think similar things can be done right about like give me feedback on a customer call. How did that go? If you read the transcript or like maybe you're I'm doing a lot of interviews for anthropic which is like a typical knowledge work problem but also like quite interesting >> but then it's teaching me I wanted to teach it. So >> and I think a lot of people who started using I experienced this you want to do more. You see how productive you are but you I don't there is no agent that I trust 100%.
29:51Every output goes through my team and then eventually through me. So I become the biggest bottleneck in the company. >> Honestly, I feel like our products would do that for you pretty well today. I'm actually quite confident like if Claude has the ability to watch you give this feedback for a while. Like the first thing you would probably do is you would sort of run it side by side. This is an experiment I recommend to almost everyone. It's like really fun to do. If you currently have a trust problem, you will probably not be super excited to immediately give over the reins, right?
30:17Like from one day to the next. But sort of run it side by side. like you read something, you write down your feedback and um you do that for a few rounds and just to check what kind of feedback claw would have given and like where do they differ where are they the same that also gives you something where you can actually sharpen the workflow a little bit because I think models will still be imperfect right there's still going to be things that where where the feedback is like a little different and like even for most humans if we take the same human and we ask them to review the same document like on a Monday versus a Friday the feedback will be different but this like sideby-side process is rather powerful because it shows you so much.
30:51>> And how do I build this agent that talks to my team? Is it just something that like a website where they upload stuff and it gives feedback? >> I would recommend claw tag which is cla inside slack. >> Uhhuh. And you build you give it access to my taste and >> Yeah. It's like it's like currently our core multiplayer experience, >> right? Where multiple people can talk to Claude at the same time.
31:12>> Yeah. Yeah. Yeah. >> This is all getting very interesting when it comes to building things, right? because you can build so many things easy. What becomes valuable when you're building software and I'm talking about uh people who are beginner entrepreneurs. I'm thinking about myself to what is it 2015 we came here with a booking platform for study abroad experiences amazing like a modem was building this whole platform because we hired a lot of engineers was th hundreds of thousands of dollars we automated stuff now I can do this in two days by myself >> like what is my entrepreneurial
31:48mode when you think about building software is there any >> spicy opinion I think it actually hasn't changed that much I think the cost of investment like sort of the upfront capital cost was much higher. That is true. And like I think we're now entering this interesting area where like single person teams or like two person teams can build billion dollar companies. But I think the secret to building a good product is actually not changed.
32:10And I believe it's empathy. >> Empathy for the user, empathy for the use case, empathy for like trying to figure out is this actually valuable for the person who's ultimately going to pay for it. I think that has again and I think this is maybe a spicy opinion but I think this has always been the key ingredient in any entrepreneurial team right and I think especially in startups like before we started taping you and I talked a little bit about 500 startups I think this is even like sort of common knowledge within VCs that whatever you show up with doesn't really matter it's probably going to change anyway but you as the founder you're the key ingredient and we can we can go one level deeper what does that mean is it like your talent your skills
32:45I think empathy and like an ability to really build something of value is probably the core thing that matters in a founder and I think that has not changed. It has just gotten much easier to tap into that empathy and like to to execute on it. >> I totally agree with that. But also sometimes I'm thinking like I have this idea but then I see something on X where people say oh this new model just killed this and we see people vip coding their smaller tools etc.
33:09Do you think it's a trend that we're going to v code a lot of tools versus paying for them? like what would us make us pay for a certain tool if we can spend an hour just building something similar. >> I think software that is hopefully going to go away is bad software. >> Like the software that doesn't really serve your use case and you're only using it because you don't have anything better. I think all of us have some tools that come to mind where you're like begrudgingly okay I guess I'm going to use this because I have to and I don't have the time to like >> look for something else or use something else.
33:35I think that will come to an end. Um however at the same time we talked about how hard it is to to build good products. I think there's still going to be tremendous value in like figuring out how to best deliver value and like to have empathy in a way where most of the products I use that I'm like a big fan of I wouldn't have come up with myself. >> Mhm. >> Right. Like I'm on record as like having written blog posts about how the iPhone is stupid because obviously you need a keyboard.
33:59Right. Oh, really? >> Yeah. Of course. I think for most of the products that I really enjoy, someone else came up with an idea, executed on it, and then gave it to me. And I think the main reason I use those products is rarely because it was very expensive to build or because it took them a long time or because they needed a big team. The reason I use the product is because it solves a need I have really really well probably in a way that was better than I thought of. The gap that I think we're going to completely fill is this garbage software and I only have to use it because I have no other choice.
34:25SponsorI think that will hopefully go away and I'm very excited about that future. >> Before we keep going, I'd love to get to know you a little better. I share a lot about my work here and I'm curious about yours. What do you do and what you'd like more help with? We've put together a little survey. Your answers will help me decide what to cover in future conversations and what resources to put together for you.
34:45SponsorI'll leave the link in the description. Please, please, please take a couple minutes and tell me about you, about your business, about your ambition so I can make better content for you. I really, really appreciate you. Thank you so much for doing this. You mentioned something about Windows computers that quietly run work and that they are a big business opportunity.
35:06Can you talk to me about that? >> I build AI products. I work on cloud. When I go out into the world and I look at what are the computers and the machines that are really running the economy. I think it's really easy, especially for like us in Silicon Valley to like sort of consider your typical startup with like bunch of the latest generation MacBooks and that's all that exists and then you go to an airport and you sort of like peek around the corner and you see that they have not actually yet moved to graphical operating systems.
35:30It's still like the black and white like >> oh yeah the transportation authorities the >> Yeah. And I'm doing this for fun now sometimes that I just go to all of these places that I that that run so much of the economy. Like the last place I did this was a hotel. Like I love doing this at a check-in where I'm like, "Hi, I work in software. I build tools. Can I just like can you just show me real quick like what you do on your little machine right there to check me in?" Harrowing.
35:52>> Like all of the all of these devices >> have software that is so clunky, so bad. Truly awful. And I think it is probably an untapped opportunity for many businesses to go in there and like try to clean that up. Well, before AI became a thing in 2010, I I joke that if I ever wanted to retire, I would build really good software for dentist offices.
36:15>> Now you can build it in a day. I guess it's it's the relationship part. >> Yeah, exactly. >> That's becoming Well, this is the hardest, right? Not not just the software. Any other businesses you wish existed? I still think we're barely getting started on software that feels a little bit more personal and has a little bit of a touch of AI that is not just here's a here's a sidebar with a chatbot in it.
36:37I think we've we've sort of permeated the entire industry with low intelligence cheap models that operate sort of at the level of I've read the FAQ page and if you give me a question I'm sort of like spit out the answer. We haven't really like figured out what can we do with topline intelligence across many of our applications. Businesses that I specifically wish existed.
36:58I'm just going to double down on this empathy thing. The industry moves slowly, right? Like even VCs move kind of slowly and it's like really hard to adjust to a new world. And I don't think we've properly adjusted to execution is cheap. We've always said ideas are cheap. That was true. And we said execution kind of expensive, but it's gotten cheaper over time. And now execution has become extremely cheap. I think we need to like really double down on empathy, connections, who's capable of like going into a business and really understanding what the needs are and how to can how can you actually optimize things here.
37:26So this might actually be a nice opportunity like the bigger revenge are the MBAs, right? Like I think MBAs can now make a bunch of software startups that might be quite successful. >> I was just talking to a Nobel Prize laurate in economics and he was telling me that we have such powerful models but we really lack the application layer. Do you agree with him that there is this huge gap between what models are capable of and what average user is using them for?
37:50>> Oh yeah, for sure. Yeah, we definitely have like a capability overhang, right? Like the idea of you have an amazing model in front of you and if you're essentially using it to to ask, hey, what's the capital of France? Again, you're probably not really tapping into what is possible there. Our founder and CEO, Dario, talks about this concept of like a data center of geniuses. My job really is to figure out okay like let's assume we have a data set of geniuses.
38:13How do I empower you the user to actually make use of that? And it requires creativity. It requires like a rethinking of like your needs and like what you think is possible. Um it's a it's a really hard problem that I don't actually think is new, right? It's this notion of people have always been held back what what by what they thought is possible rather than what is actually possible. Yeah, I do agree that we have like we've just scratched the surface of what's possible.
38:38How can one close that on an individual level? >> On an individual level, I thought about this like all weekend. I think on an individual level, the thing that you see on X a lot is this this rallying cry of be more ambitious. We ought to be more ambitious. Like what being more ambitious really means is I think so context dependent. But I would like people to shed all the shackles of like what they believe is possible and rethink why are they doing things, right?
39:05like what like I'm doing a certain task in my business. What I what am I hoping for is going to happen next. And you just keep giving the model that task. Like whatever outcome you're hoping for, give the model that task. You can't play this forever. You will eventually find the limits of the current top intelligence. Like a good counterargument here is okay like the purpose for most businesses is to make money.
39:27And if I just tell clude go make money, it's probably going to like not make it very far. So we we haven't reached like the top level of altitude yet. is somewhere in the middle. But once you bump against that ceiling, you do know that you're at the forefront. You're at the frontier and you're going to really benefit. You're going to benefit from the next model generations. >> Because if if all we're doing is like summarize this email for me or like make this email sound as angry, you're not really like benefiting from the new intelligent.
39:51>> What would be your advice to someone who's just starting out now as a software engineer in this ever evolving >> in this ever evolving like landscape of like how to do software engineering? >> I think it's I think it really depends on what you want as a software engineer, right? like what does it what does it mean to be a software engineer? I think we're like as engineers are we're currently in this like moment of crisis where we're like what does it mean to be a software engineer? Um does it mean that I'm I've completely memorized the entire API service in React?
40:16Is that what I'm valuable for? Is that what I do? Um and I think that's probably less the case now. To me, this is a pretty welcome change. I've always been of the opinion that a good engineer versus a bad engineer. I measure I've always measured my engineers on their impact and the impact is like does the user like this enough to buy the software. This there's always for me been relatively little value in like really elegant beautiful mastery solutions where we can be really proud of the engineering craft here but the user doesn't feel any benefit.
40:46>> How do you develop that? What do you double down on now in order to be relevant in three years if like every position is suddenly a senior position? What would be your advice there? The way I got my start in this career is by like building a strong portfolio and I think that would be the same advice today. I think building a portfolio that is not just like a mock is much easier. So my recommendation for anyone young would be go and build some real things.
41:11Prove that you can do it. Prove that you have the empathy. Prove that you have the chops, the ability to talk to customers. Like figure out what is necessary versus not necessary. Um I do think there's like a slightly new definition of what software engineering is. I personally think that's good. I do see the young people adapt to this new style of thinking much quicker than some of the older engineers who maybe enjoyed the luxury of primarily working on on tickets without having to ask why too many times.
41:36>> I've seen so many people be quite successful in that. Um the things that I would maybe to contrast not recommend anymore. It is true that I think like completely memorizing individual API services I think has become less valuable. What is still valuable is being able to combine different systems in interesting ways. I think the value of that is amplified now because you can so quickly combine things, right?
41:59You can just like point AI like three different systems and be like I want these three like I want these five ingredients mashed together. Go and do it. >> Is that something that you look for when you hire people? Yeah. This ability to >> Is there something else that you're particularly looking for? >> I mean, I don't know how you hire. Do you still do you do you care a lot about the resume? >> I do. When it goes because when you mentioned relationships, they're very important for my industry.
42:20I am looking for someone who knows people. I am looking for someone who knows how other businesses operate because this is not public information like how do I learn how to what what do I charge as a podcaster right what do I charge for I want someone who knows exactly what's going on in the market >> so we do care about relationships but when it comes to the creative part actually with creative part we still look at how many views their videos have produced because yeah >> they know the art I don't look at their degree of course >> yeah yeah exactly exactly and I think this is I don't know if this is still controversial I guess we both work like in a you know like tech adjacent, but I've never cared too much about which degree do you have or which
42:56school you went to. It was always much easier for me to look at what are the things you made. Are those things I can touch, feel, try and I can feel like, oh, you either were the winning player on a team or you at least worked on a winning team. That is sometimes useful for me to know. I think that's really valuable information. And the thing is, if I didn't have that, I might go take the shortcut and look for schools, degrees.
43:17I don't really have to. I will always pick someone who's proven themselves to build something amazing over someone who hasn't built something amazing but has an amazing degree. >> What would you tell your kid when she grows up and says like, "Dad, I wanted to coding." What would be your advice? >> I mean, she's 9 months old right now, so I think things will change a little bit. >> I think you have to be in line for your daycare already. [laughter] Just saying.
43:38>> And it all happens quicker than you think. What would I tell my daughter? Um, I think I would tell my daughter to very early on try to understand how businesses operate, how people collaborate, um, how to really find value. I've not gone to the University of Wateroo, but I'm such a fan of the University of Wateroo's process. >> All the amazing minds, >> all the amazing minds, but I think the University of Wateroo has always had this approach which was quite unique within within computer science engineering, which was this is an outside perspective.
44:03They might disagree with me on this, but my outside perspective was always coming from Germany where computer science is very much computer science. It's informatics. It's a lot of math and very little like only a sliver of how do we actually build a good product. Um, let alone computer engineering. JavaScript was in Germany for a long time universally considered to be like not what real computer scientists use.
44:25And the University of Wateroo's approach has always fascinated me so much because I would get these interns or junior engineers who've worked at five different companies before in their internships. It was pretty common especially in San Francisco that you get an intern and it's their like fourth internship and they previously interned at Tesla, Google and Apple. And they have so many interesting things they learned about how companies operate, what are the upsides and downsides, how do you actually advocate for an idea, how do you get resources, how do you convince legal to get something done.
44:54I think those are the skills that really like for me made University of Wateroo's engineers almost immediately effective. It's their first job maybe with me, but they're in a completely different category than someone who has maybe even gotten a PhD in computer science, but has never worked in a company. They might be very smart what they do. But as we all know, like building a good product and getting it out there is about much more than just knowing what code to write.
45:16You need to be able to talk to people. You need to be able to like work together with other functions. You need to have like a little bit of a of a sense for how does this company run. So I would tell my daughter to probably like really invest in that. >> Love it. >> What do you tell your kids? What do you think your kids should do in the ever evolving economy? >> I tell them to try all the things. And I tell them you can be anything cuz sometimes they're like oh robots.
45:39You can be the queen of like if I tell her like you can build a robot, she's like not very excited. I tell her you can be the queen of robots. >> Yeah. >> And she's like oh I like queen of robots. And then she says and we we bought this self-driving car that you can program yourself. Oh, that is fun. >> She was building it with her dad and like just telling them they can be anything and there is no thing that they cannot touch because it's I don't know out of this or whatever cuz I came here with a notion that everyone here is from another planet.
46:06Yeah, >> if you went to Stanford that was like something that I couldn't even fathom >> but seeing that these are all normal people and like giving this idea to my daughters that anything is possible is just hard work and passion. >> Oh man, I love that. I think the reason I love that so much is because in AI circles we constantly talk about this idea of ambition and the reason people are not ambitious enough is because we're sort of programmed as humans to like only consider certain things possible.
46:34And if we don't consider them possible, we don't even try. So it's like real like what we believe is possible is the real limiter. Like a much bigger limiter than what the models can do or what we can do. Maybe that's another reason like really believe in the new generation is because they will never have experienced all these limitations that we have. >> Okay. I think we covered so much. I just wanted like your final thought on our future.
46:55How do you think our life is going to be different in 5 years? >> That is such a big question. And I'm not sure you have the right person for like I I know that like going on any podcast is like there's a there's an occupational hazard of having to make predictions. Biggest thing that I've noticed so far is that we're all so wrong about the intelligence exponential coming out of AI. It's really hard to like wrap your finger around what does it really mean for for the exponential to exist.
47:20And I keep going back to math, right? Like when when we won the silver medal at the m period when I say we I mean Google um as an industry I was well aware of scaling laws. I was well aware of like how much intelligence progressing and I think if you had asked me at what point will AI solve what we would otherwise consider unsolvable math problems I don't think I would have given you like the middle of 2026 I would have been like that's that's in a year and a half.
47:48What are you talking about? So I think instinctively having a good feeling for how will what will the future look like is really really tricky. That's the first part. And I think the second part that is rather tricky for me is the one thing I can tell you with a decent amount of confidence is that I think the products that we use today will all look completely different. I think I think we're currently we've hit this point where I becoming so mainstream and it's becoming so popular that it's like easy to imagine or we're maybe like this this is sort of the end state.
48:15Um, but I'm quite convinced that all the AI products we use today are roughly in the same phase as all the funny phones that we had right before the smartphone came along. So I think we're still in the pre iPhone era, which makes it very exciting for me to work in. >> Wow, this is really fascinating already like from in our preI era. Uh there are so many things that are changing and you made me rethink my whole process.
48:37I won't even show you what what it looks like now cuz I think it's fine if it works. very outdated [laughter] cuz cuz we give it all the data and tell it exactly what to look at. But I totally understand how giving it access to everything is just so much more powerful. >> It's such [laughter] a tricky thing, right? Like all the advice I've given you today in about 10 weeks, we're probably going to have a new revision on all of it. >> That is also true. So being an always on agent when it comes to new models, [laughter] I think that's the that's the thing.
49:02And trying new things. Thank you so much, Felix. If this conversation with Felix made you want to try Claude in your work, watch my interview with Alli Miller next. Ally works with teams on using AI and in our conversation, she showed me her real workflows and built a Claude morning brief life. It's a practical place to start if you're wondering what to do