The Future of AI-Assisted Coding: Insights from the Cursor Team

the following is a conversation with the founding members of the cursor team Michael truell swall oif Arvid lunark

and Aman Sanger cursor is a code editor based on VSS code that adds a lot of powerful features for AI assisted coding

it has captivated the attention and excitement of the programming and AI communities so I thought this is an

excellent opportunity to dive deep into the role of AI in programming this is a super technical conversation that is

bigger than just about one code editor it's about the future of programming and in general the future of human AI

collaboration in designing and Engineering complicated and Powerful systems this is Le Freedman podcast to

support it please check out our sponsors in the description and now dear friends here's Michael suale Arvid and

Aman all right this is awesome we have Michael Aman suali Arvid here from the cursor team first up big ridiculous

question what's the point of a code editor so the the code editor is largely the place where you build software and

today or for a long time that's meant the place where you text edit uh a formal programming language and for

people who aren't programmers the way to think of a code editor is like a really souped up word processor for programmers

where the reason it's it's souped up is code has a lot of structure and so the the quote unquote word processor the

code editor can actually do a lot for you that word processors you know sort of in the writing space haven't been

able to do for for people editing text there and so you know that's everything from giving you visual differentiation

of like the actual tokens in the code to so you can like scan it quickly to letting you navigate around the code

base sort of like you're navigating around the internet with like hyperlinks you're going to sort of definitions of

things you're using to error checking um to you know to catch rudimentary B um and so traditionally that's what a

code editor has meant and I think that what a code editor is is going to change a lot over the next 10 years um as what

it means to build software maybe starts to look a bit different I I think also code edor should just be fun yes that is

very important that is very important and it's actually sort of an underated aspect of how we decide what to build

like a lot of the things that we build and then we we try them out we do an experiment and then we actually throw

them out because they're not fun and and so a big part of being fun is like being fast a lot of the time fast is fun yeah

fast is uh yeah that should be a t-shirt like like

fundamentally I think one of the things that draws a lot of people to to building stuff on computers is this like

insane integration speed where you know in other disciplines you might be sort of gate capped by resources or or the

ability even the ability you know to get a large group together and coding is just like amazing thing where it's you

and the computer and uh that alone you can you can build really cool stuff really quickly so for people don't know

cursor is this super cool new editor that's a fork of vs code it would be interesting to get your kind of

explanation of your own journey of editors how did you I think all of you are were big fans of vs code with

co-pilot how did you arrive to VSS code and how did that lead to your journey with cursor yeah um

so I think a lot of us well all of us originally Vim users pure pure VI pure Vim yeah no neo just pure Vim in a

terminal and at Le at least for myself it was around the time that C- pilot came out so

2021 that I really wanted to try it so I went into vs code the only platform the only code editor in which it was

available and even though I you know really enjoyed using Vim just the experience of

co-pilot with with vs code was more than good enough to convince me to switch and so that kind of was the default until we

started working on cursor and uh maybe we should explain what copala does it's like a really nice

autocomplete it suggests as you start writing a thing it suggests one or two or three lines how to complete the thing

and there's a fun experience in that you know like when you have a close friendship and your friend completes

your sentences like when it's done well there's an intimate feeling uh there's

probably a better word than intimate but there's a there's a cool feeling of like holy it gets

me now and then there's an unpleasant feeling when it doesn't get you uh and so there's that that kind of friction

but I would say for a lot of people the feeling that it gets me over powers that it doesn't and I think actually one of

the underrated aspects of get up copet is that even when it's wrong is it's like a little bit annoying but it's not

that bad because you just type another character and then maybe then it gets you or you type another character and

then then it gets you so even when it's wrong it's not that bad yeah you you can sort of iterate iterate and fix it I

mean the other underrated part of uh calot for me sort of was just the first real real AI product it's like the first

language model consumer product so copile was kind of like the first killer app for LMS yeah and like the beta was

out in 2021 right okay mhm uh so what's the the origin story of cursor so around 2020 the scaling loss papers came out

from from open Ai and that was a moment where this looked like clear predictable progress for the field where even if we

didn't have any more ideas looked like you could make these models a lot better if you had more computer and more data

uh by the way we'll probably talk uh for three to four hours on on the topic of scaling laws but just just to summarize

it's a paper and a set of papers and set of ideas that say bigger might be better for model size and data size in the in

the realm of machine learning it's bigger and better but predictively better okay this another topic of

conversation but anyway yeah so around that time for some of us there were like a lot of conceptual conversations about

what's this going to look like what's the the story going to be for all these different knowledge worker Fields about

how they're going to be um made better U by this technology getting better and then um I think there were a couple of

moments where like the theoretical gains predicted in that paper uh started to feel really concrete and it started to

feel like a moment where you could actually go and not you know do a PhD if you wanted to work on uh do useful work

in AI actually felt like now there was this this whole set of systems one could built that were really useful and I

think that the first moment we already talked about a little bit which was playing with the early bit of copell

like that was awesome and magical um I think that the next big moment where everything kind of clicked together was

actually getting early access to gbd4 so sort of end of 2022 was when we were um tinkering with that model and the Step

Up in capabilities felt enormous and previous to that we had been working on a couple of different projects we had

been um because of co-pilot because of scaling laws because of our prior interest in the technology we had been

uh tinkering around with tools for programmers but things that are like very specific so you know we were

building tools for uh Financial professionals who have to work with in a juper notebook or like you know playing

around with can you do static analysis with these models and then the Step Up in gbd4 felt like look that really made

concrete the theoretical gains that um we had predicted before felt like you could build a lot more just immediately

at that point in time and also if we were being consistent it really felt like um this wasn't just

going to be a point solution thing this was going to be all of programming was going to flow through these models it

felt like that demanded a different type of programming environment to different type of programming and so we set off to

build that that sort of larger Vision around then there's one that I distinctly remember so my roommate is an

IMO gold winner and uh there's a competition in the US called of putam which is sort of the IMO for college

people and it's it's this math competition is he's exceptionally good so Shang Tong and Aman I remember it

sort of June June of 2022 had this bet on whether the mo like 2024 June or July you were going to win

a gold medal in the Imo with the with like models IMO is international math Olympiad uh yeah IMO is international

math Olympiad and so Arvid and I are both of you know also competed in it so was sort of personal and uh and I I

remember thinking Matt is just this is not going to happen this was like it un like even though I I sort of believed in

progress I thought you know I'm a girl just like Aman is just delusional that was the that was the and and to be

honest I mean I I was to be clear it very wrong but that was maybe the most preent bet in the group so the the new

results from Deep Mind it turned out that you were correct that's what well it technically not technically incorrect

but one point awayan was very enthusiastic about this stuff back then and before Aman had this like scaling

loss t-shirt that he would walk around with where it had like charts and like the formulas on it oh so you like felt

the AI or you felt the scaling yeah I i l remember there was this one conversation uh I had with with Michael

where before I hadn't thought super deeply and critically about scaling laws and he kind of posed the question why

isn't scaling all you need or why isn't scaling going to result in massive gains in progress and I think I went through

like the like the stages of grief there is anger denial and then finally at the end just thinking about it uh acceptance

um and I think I've been quite hopeful and uh optimistic about progress since I think one thing I'll caveat is I think

it also depends on like which domains you're going to see progress like math is a great domain because especially

like formal theor improving because you get this fantastic signal of actually verifying if the thing was correct and

so this means something like RL can work really really well and I think like you could have systems that are perhaps very

superhuman in math and still not technically have ai okay so can we take it off all the way to cursor mhm and

what is cursor it's a fork of vs code and VSS code is one of the most popular editors for a long time like everybody

fell in love with it everybody left Vim I left dmax for it sorry

uh uh so it unified in some fun fundamental way the uh the developer community and then that you look at the

space of things you look at the scaling laws AI is becoming amazing and you decide decided okay it's not enough to

just write an extension Fe vs code because there's a lot of limitations to that we we need if AI is

going to keep getting better and better and better we need to really like rethink how the the AI is going to be

part of the editing process and so you decided to Fork vs code and start to build a lot of the amazing features

we'll be able to to to talk about but what was that decision like because there's a lot of extensions including

co-pilot of vs code that are doing so AI type stuff what was the decision like to just Fork vs code so the decision to do

an editor seemed kind of self-evident to us for at least what we wanted to do and Achieve because when we started working

on the editor the idea was these models are going to get much better their capabilities are going to improve and

it's going to entirely change how you build software both in a you will have big productivity gains but also radical

in how like the active building software is going to change a lot and so you're very limited in the control you have

over a code editor if you're a plugin to an existing coding environment um and we didn't want to get locked in by those

limitations we wanted to be able to um just build the most useful stuff okay well then the natural question

is you know VSS code is kind of with copilot a competitor so how do you win is is it basically just the speed and

the quality of the features yeah I mean I think this is a space that is quite interesting perhaps quite unique where

if you look at previous Tech waves maybe there's kind of one major thing that happened and unlocked a new wave of

companies but every single year every single model capability uh or jump you get model capabilities you now unlock

this new wave of features things that are possible especially in programming and so I think in AI programming being

even just a few months ahead let alone a year ahead makes your product much much much more useful I think the cursor a

year from now will need to make the cursor of today look Obsolete and I think you know Microsoft

has' done a number of like fantastic things but I don't think they're in a great place to really keep innovating

and pushing on this in the way that a startup can just rapidly implementing features and and push yeah like and and

kind of doing the research experimentation necessary um to really push the ceiling

I don't I don't know if I think of it in terms of features as I think of it in terms of like capabilities for for

programmers it's that like you know as you know the new one model came out and I'm sure there are going to be more more

models of different types like longer context and maybe faster like there's all these crazy ideas that you can try

and hopefully 10% of the crazy ideas will make it into something kind of cool and useful and uh we want people to have

that sooner to rephrase it's like an underrated fact is we're making it for oursel when we started cursor you really

felt this frustration that you know models you could see models getting better uh but the coall experience had

not changed it was like man these these guys like the steing is getting higher like why are they not making new things

like they should be making new things they should be like you like like where's where's where's all the alpha

features there there were no Alpha features it was like uh I I'm sure it it was selling well I'm sure it was a great

business but it didn't feel I I'm I'm one of these people that really want to try and use new things and was just

there's no new thing for like a very long while yeah it's interesting uh I don't know how you put that into words

but when you compare a cursor with copilot copilot pretty quickly became started to feel stale for some reason

yeah I think one thing that I think uh helps us is that we're sort of doing it all in one where we're developing the

the ux and the way you interact with the model and at the same time as we're developing like how we actually make the

model give better answers so like how you build up the The Prompt or or like how do you find the context and for a

cursor tab like how do you train the model um so I think that helps us to have all of it like sort of like the

same people working on the entire experience on end yeah it's like the the person making the UI and the person

training the model like sit to like 18 ft away so often the same person even yeah often often even the same person so

you you can you create things that that are sort of not possible if you're not you're not talking you're not

experimenting and you're using like you said cursor to write cursor of course oh yeah yeah well let's talk about some of

these features let's talk about the all- knowing the all powerful praise B to the tab so the you know autocomplete on

steroids basically so what how does tab work what is tab to highlight and summarize it a high level I'd say that

there are two things that curser is pretty good at right now there there are other things that it does um but two

things it it helps programmers with one is this idea of looking over your shoulder and being like a really fast

colleague who can kind of jump ahead of you and type and figure out what you're what you're going to do next and that

was the original idea behind that was kind kind of the kernel the idea behind a good autocomplete was predicting what

you're going to do next you can make that concept even more ambitious by not just predicting the characters after

cursor but actually predicting the next entire change you're going to make the next diff the next place you're going to

jump to um and the second thing cursor is is pretty good at right now too is helping you sometimes jump ahead of the

AI and tell it what to do and go from instructions to code and on both of those we've done a lot of work on making

the editing experience for those things ergonomic um and also making those things smart and fast one of the things

we really wanted was we wanted the model to be able to edit code for us uh that was kind of a wish and we had multiple

attempts at it before before we had a sort of a good model that could edit code for

you U then after after we had a good model I think there there have been a lot of effort to you know make the

inference fast for you know uh having having a good good experience and uh we've been starting to

incorporate I mean Michael sort of mentioned this like ability to jump to different places and that jump to

different places I think came from a feeling off you know once you once you accept an edit um was like man it should

be just really obvious where to go next it's like it's like I I made this change the model should just know that like the

next place to go to is like 18 lines down like uh if you're if you're a whim user you could press 18 JJ or

whatever but like why why even why am I doing this like the model the model should just know it and then so so the

idea was you you just press tab it would go 18 lines down and then make it would show you show you the next edit and you

would press tab so it's just you as long as you could keep pressing Tab and so the internal competition was how many

tabs can we make them pressive once you have like the idea uh more more uh sort of abstractly the the thing to think

about is sort of like once how how how are the edit sort of zero zero entropy so once You' sort of expressed your

intent and the edit is there's no like new bits of information to finish your thought but you still have to type some

characters to like make the computer understand what you're actually thinking then maybe the model should just sort of

read your mind and and all the zero entropy bits should just be like tabbed away yeah that was that was sort of the

abstract there's this interesting thing where if you look at language model loss on on different domains um I believe the

bits per bite which is kind of character normalized loss for code is lower than language which means in general there

are a lot of tokens in code that are super predictable lot of characters that are super predictable um and this is I

think even magnified when you're not just trying to autocomplete code but predicting what the user is going to do

next in their editing of existing code and so you know the gold cursor tab is let's eliminate all the low entropy

actions you take inside of the editor when the intent is effectively determined let's just jump you forward

in time skip you forward well well what's the intuition and what's the technical details of how to do next

cursor prediction that jump that's not that's not so intuitive I think to people yeah I think I can speak to a few

of the details on how how to make these things work they're incredibly low latency so you need to train small

models on this on this task um in particular they're incredibly pre-fill token hungry what that means is they

have these really really long prompts where they see a lot of your code and they're not actually generating that

many tokens and so the perfect fit for that is using a sparse model meaning Ane model um so that was kind of one one

break one breakthrough we made that substantially improved its performance at longer context the other being um a

variant of speculative decoding that we we kind of built out called speculative edits um these are two I think important

pieces of what make it quite high quality um and very fast okay soe mixture of experts the input is huge the

output is small yeah okay so like what what what else can you say about how to make it like caching play a role in this

cashing plays a huge role M um because you're dealing with this many input tokens if every single keystroke that

you're typing in a given line you had to rerun the model on all those tokens passed in you're just going to one

significantly deg grade latency two you're going to kill your gpus with load so you need to you you need to design

the actual prompts use for the model such that they're cach caching aware and then yeah you need you need to re use

the KV cach across request just so that you're spending less work less compute uh again what are the things that tab is

supposed to be able to do kind of in the near term just to like sort of Linger on that generate code like fill empty

space Also edit code across multiple lines yeah and then jump to different locations inside the same file yeah and

then like hopefully jump to different files also so if you make an edit in one file and maybe maybe you have to go

maybe you have to go to another file to finish your thought it should it should go to the second file also yeah and then

the full generalization is like next next action prediction like sometimes you need to run a command in the

terminal and it should be able to suggest the command based on the code that you wrote too um or sometimes you

actually need to like it suggest something but you you it's hard for you to know if it's correct because you

actually need some more information to learn like you need to know the type to be able to verify that it's correct and

so maybe it should actually take you to a place that's like the definition of something and then take you back so that

you have all the requisite knowledge to be able to accept the next completion Al also providing the human the knowledge

yes right yeah can you integrate like I just uh gotten to know a guy named Prime Jen who I believe has an SS you can

order coffee via SSH oh yeah oh we did that we did that uh so can that also the model do that like

feed you and like yeah and provide you with caffeine okay so that's the general framework yeah and the the magic moment

would be if it is programming is this weird discipline where um sometimes the next

five minutes not always but sometimes the next five minutes of what you're going to do is actually predictable from

the stuff you've done recently and so can you get to a world where that next 5 minutes either happens by you

disengaging and it taking you through or maybe a little bit more of just you seeing Next Step what it's going to do

and you're like okay that's good that's good that's good that's good and you can just sort of tap tap tap through these

big changes as we're talking about this I should mention like one of the really cool and noticeable things about cursor

is that there's this whole diff interface situation going on so like the model suggests with uh with the red and

the green of like here's how we're going to modify the code and in the chat window you can apply and it shows you

the diff and you can accept the diff so maybe can you speak to whatever direction of that we'll probably have

like four or five different kinds of diffs uh so we we have optimized the diff for for the autocomplete so that

has a different diff interface than uh then when you're reviewing larger blocks of code and then we're

trying to optimize uh another diff thing for when you're doing multiple different files uh and and sort of at a high level

the difference is for when you're doing autocomplete it should be really really fast to

read uh actually it should be really fast to read in all situations but in autocomplete it sort of you're you're

really like your eyes focused in one area you you can't be in too many you the humans can't look in too many

different places so you're talking about on the interface side like on the interface side so it currently has this

box on the side so we have the current box and if it tries to delete code in some place and tries to add other code

it tries to show you a box on the you can maybe show it if we pull it up on cursor. comom this is what we're talking

about so that it was like three or four different attempts at trying to make this this thing work where first the

attempt was like these blue crossed out line so before it was a box on the side it used to show you the code to delete

by showing you like uh like Google doc style you would see like a line through it then you would see the the new code

that was super distracting and then we tried many different you know there was there was sort of deletions there was

trying to Red highlight then the next iteration of it which is sort of funny Would you would hold the on Mac the

option button so it would it would sort of highlight a region of code to show you that there might be something coming

uh so maybe in this example like the input and the value uh would get would all get blue and the blue would to

highlight that the AI had a suggestion for you uh so instead of directly showing you the thing it would show you

that the AI it would just hint that the AI had a suggestion and if you really wanted to see it you would hold the

option button and then you would see the new suggestion then if you release the option button you would then see your

original code mhm so that's by the way that's pretty nice but you have to know to hold the option button yeah I by the

way I'm not a Mac User but I got it it was it was it's a button I guess you people

it's h you know it's again it's just it's just nonintuitive I think that's the that's the key thing and there's a

chance this this is also not the final version of it I am personally very excited for

um making a lot of improvements in this area like uh we we often talk about it as the verification problem where U

these diffs are great for small edits uh for large edits or like when it's multiple files or something it's um

actually a little bit prohibitive to to review these diffs and uh uh so there are like

a couple of different ideas here like one idea that we have is okay you know like parts of the diffs are important

they have a lot of information and then parts of the diff um are just very low entropy they're like exam like the same

thing over and over again and so maybe you can highlight the important pieces and then gray out the the not so

important pieces or maybe you can have a model that uh looks at the the diff and and sees oh there's a likely bug here I

will like Mark this with a little red squiggly and say like you should probably like review this part of the

diff um and ideas in in that vein I think are exciting yeah that's a really fascinating space of like ux design

engineering so you're basically trying to guide the human programmer through all the things they need to read and

nothing more yeah like optimally yeah and you want an intelligent model to do it like ly diffs Al diff algorithms are

they're like Al like they're just like normal algorithms uh there's no intelligence uh there's like

intelligence that went into designing the algorithm but then there there's no like you don't care if the if it's about

this thing or this thing uh and so you want a model to to do this so I think the the the general question is like M

these models are going to get much smarter as the models get much smarter uh the the changes they will be able to

propose are much bigger so as the changes gets bigger and bigger and bigger the humans have to do more and

more and more verification work it gets more and more more hard like it's just you need you need to help them out it

sort of I I don't want to spend all my time reviewing code uh can you say a little more across

multiple files div yeah I mean so GitHub tries to solve this right with code review when you're doing code review

you're reviewing multiple deaths cross multiple files but like Arvid said earlier I think you can do much better

than code review you know code review kind of sucks like you spend a lot of time trying to grock this code that's

often quite unfamiliar to you and it often like doesn't even actually catch that many bugs and I think you can

signific significantly improve that review experience using language models for example using the kinds of tricks

that AR had described of maybe uh pointing you towards the regions that matter

um I think also if the code is produced by these language models uh and it's not produced by someone else like the code

review experience is designed for both the reviewer and the person that produced the code in the case where the

person that produced the code is a language model you don't have to care that much about their experience and you

can design the entire thing around the reviewer such that the reviewer's job is as fun as easy as productive as possible

um and I think that that feels like the issue with just kind of naively trying to make these things look like code

review I think you can be a lot more creative and and push the boundary and what's possible just one one idea there

is I think ordering matters generally when you review a PR you you have this list of files and you're reviewing them

from top to bottom but actually like you actually want to understand this part first because that came like logically

first and then you want understand the next part and um you don't want to have to figure out that yourself you want a

model to guide you through the thing and is the step of creation going to be more and more natural language is the goal

versus with actual uh I think sometimes I don't think it's going to be the case that all of programming will be natural

language and the reason for that is you know if I'm PR programming with swalla and swall is at the computer and the

keyboard uh and sometimes if I'm like driving I want to say to swallet hey like implement this function and that

that works and then sometimes it's just so annoying to explain to swalla what I want him to do and so I actually take

over the keyboard and I show him I I write like part of the example and then it makes sense and that's the easiest

way to communicate and so I think that's also the case for AI like sometimes the easiest way to communicate with the AI

will be to show an example and then it goes and does the thing everywhere else or sometimes if you're making a website

for example the easiest way to show to the a what you want is not to tell it what to do but you know drag things

around or draw things um and yeah and and like maybe eventually we will get to like brain machine interfaces or

whatever and can of like understand what you're thinking and so I think natural language will have a place I think it

will not definitely not be the way most people program most of the time I'm really feeling the AGI with this editor

uh it feels like there's a lot of machine learning going on underneath tell tell me about some of the ml stuff

that makes it all work recursor really works via this Ensemble of custom models that that that we've trained alongside

you know the frontier models that are fantastic at the reasoning intense things and so cursor tab for example is

is a great example of where you can specialize this model to be even better than even Frontier models if you look at

evls on on the on the task we set it at the other domain which it's kind of surprising that it requires custom

models but but it's kind of necessary and works quite well is in apply um

so I think these models are like the frontier models are quite good at sketching out plans for code and

generating like rough sketches of like the change but actually creating diffs is quite hard um

for Frontier models for your training models um like you try to do this with Sonet with 01 any Frontier Model and it

it really messes up stupid things like counting line numbers um especially in super super large file

um and so what we've done to alleviate this is we let the model kind of sketch out this rough code block that indicates

what the change will be and we train a model to then apply that change to the file and we should say that apply is the

model looks at your code it gives you a really damn good suggestion of what new things to do and the seemingly for

humans trivial step of combining the two you're saying is not so trivial contrary to popular perception it is not a

deterministic algorithm yeah I I I think like you see shallow copies of apply um elsewhere and it just breaks like most

of the time because you think you can kind of try to do some deterministic matching and then it fails you know at

least 40% of the time and that just results in a terrible product experience um I think in general this

this regime of you are going to get smarter models and like so one other thing that apply lets you do is it lets

you use fewer tokens with the most intelligent models uh this is both expensive in terms of latency for

generating all these tokens um and cost so you can give this very very rough sketch and then have your smaller models

go and implement it because it's a much easier task to implement this very very sketched out code and I think that this

this regime will continue where you can use smarter and SM models to do the planning and then maybe the

implementation details uh can be handled by the less intelligent ones perhaps you'll have you know maybe 01 maybe

it'll be even more cap capable models given an even higher level plan that is kind of recursively uh applied by Sonet

and then the apply model maybe we should we should talk about how to how to make it fast yeah I feel like fast is always

an interesting detail fast good yeah how do you make it fast yeah so one big component of making it it fast is

speculative edits so speculative edits are a variant of speculative decoding and maybe be helpful to briefly describe

speculative decoding um with speculative decoding what you do is you you can kind of take advantage of the fact that you

know most of the time and I I'll add the caveat that it would be when you're memory Bound in in language model

Generation Um if you process multiple tokens at once um it is faster than generating one Tok at a time so this is

like the same reason why if you look at tokens per second uh with prompt tokens versus generated tokens it's much much

faster for prompt tokens um so what we do is instead of using what specul decoding normally does which is using a

really small model to predict these draft tokens that your larger model would then go in and and verify um with

code edits we have a very strong prior of what the existing code will look like and that prior is literally the same

exact code so you can do is you can just feed chunks of the original code back into the into the model um and then the

model will just pretty much agree most of the time that okay I'm just going to spit this code back out and so you can

process all of those lines in parallel and you just do this with sufficiently many chunks and then eventually you'll

reach a point of disagreement where the model will now predict text that is different from the ground truth original

code it'll generate those tokens and then we kind of will decide after enough tokens match

uh the original code to re start speculating in chunks of code what this actually ends up looking like is just a

much faster version of normal editing code so it's just like it looks like a much faster version of the model

rewriting all the code so just we we can use the same exact interface that we use for for diffs but it will just stream

down a lot faster and then and then the advantage is that W wireless streaming you can just also be reviewing start

reviewing the code exactly before before it's done so there's no no big loading screen uh so maybe that that is part of

the part of the advantage so the human can start reading before the thing is done I think the interesting riff here

is something like like speculation is a fairly common idea nowadays it's like not only in language models I mean

there's obviously speculation in CPUs and there's there like speculation for databases and like speculation all over

the place let me ask the sort of the ridiculous question of uh which llm is better at coding GPT Claude who wins in

the context of programming and I'm sure the answer is much more Nuance because it sounds like every single part of this

involves a different model yeah I think they there's no model that poo dominates uh others meaning it

is better in all categories that we think matter the categories being speed

um ability to edit code ability to process lots of code long context you know a couple of other things and kind

of coding capabilities the one that I'd say right now is just kind of net best is Sonet I

think this is a consensus opinion our one's really interesting and it's really good at reasoning so if you give it

really hard uh programming interview style problems or lead code problems it can do quite quite well on them um but

it doesn't feel like it kind of understands your rough intent as well as son it

does like if you look at a lot of the other Frontier models um one qual I have is it feels like they're not necessarily

over I'm not saying they they train in benchmarks um but they perform really well in benchmarks relative to kind of

everything that's kind of in the middle so if you tried on all these benchmarks and things that are in the distribution

of the benchmarks they're valuated on you know they'll do really well but when you push them a little bit outside of

that son's I think the one that that kind of does best at at kind of maintaining that same capability like

you kind of have the same capability in The Benchmark as when you try to instruct it to do anything with coding

what another ridiculous question is the difference between the normal programming experience versus what

benchmarks represent like where do benchmarks fall short do you think when we're evaluating these models by the way

that's like a really really hard it's like like critically important detail like how how different like benchmarks

are versus where is like real coding where real coding it's not interview style coding

it's you're you're doing these you know humans are saying like half broken English sometimes and sometimes you're

saying like oh do what I did before sometimes you're saying uh you know go add this thing and then do this

other thing for me and then make this UI element and then you know it's it's just like a lot of things are sort of context

dependent you really want to like understand the human and then do do what the human

wants as opposed to sort of this maybe the the way to put it is sort of abstractly is uh the interview problems

are very wellp specified they lean a lot on

specification while the human stuff is less specified yeah I think that this this SP

for question is both Complicated by what um Sol just mentioned and then also to what Aman was getting into is that even

if you like you know there's this problem of like the skew between what can you actually model in a benchmark

versus uh real programming and that can be sometimes hard to encapsulate because it's like real programming is like very

messy and sometimes things aren't super well specified what's correct or what isn't but then uh it's also doubly hard

because of this public Benchmark problem and that's both because public benchmarks are sometimes kind of Hill

climbed on then it's like really really hard to also get the data from the public benchmarks out of the models and

so for instance like one of the most popular like agent benchmarks sweet bench um is really really contaminated

in the training data of uh these Foundation models and so if you ask these Foundation models to do a sweet

bench problem you actually don't give them the context of a codebase they can like hallucinate the right file pass

they can hallucinate the right function names um and so the the it's it's also just the public aspect of these things

is tricky yeah like in that case it could be trained on the literal issues or pool request themselves and and maybe

the lives will start to do a better job um or they've already done a good job at decontaminating those things but they're

not going to emit the actual training data of the repository itself like these are all like some of the most popular

python repositories like simpai is one example I don't think they're going to handicap their models on Senpai and all

these popular P python repositories in order to get uh true evaluation scores in these benchmarks yeah I think that

given the dirs and benchmarks um there have been like a few interesting crutches that uh places that

build systems with these models or build these models actually use to get a sense of are they going in the right direction

or not and uh in a lot of places uh people will actually just have humans play with the things and give

qualitative feedback on these um like one or two of the foundation model companies they they have people who

that's that's a big part of their role and you know internally we also uh you know qualitatively assess these models

and actually lean on that a lot in addition to like private evals that we have it's like the live

the vibe yeah the vi the vibe Benchmark human Benchmark the hum you pull in the humans to do a Vibe check yeah okay I

mean that's that's kind of what I do like just like reading online forums and Reddit and X just like well I don't know

how to properly load in people's opinions because they'll say things like I feel

like Claude or gpt's gotten Dumber or something they'll say I feel like and then I sometimes feel like that too

but I wonder if it's the model's problem or mine yeah with Claude there's an interesting take I heard where I think

AWS has different chips um and I I suspect they've slightly different numerics than uh Nvidia gpus and someone

speculated that claud's deg degraded performance had to do with maybe using the quantise version that existed on AWS

Bedrock versus uh whatever was running on on anthropics gpus I interview a bunch of people that have conspiracy

theories so I'm glad spoke spoke to this conspiracy well it's it's not not like conspiracy theory as much as they're

just they're like they're you know humans humans are humans and there's there's these details and you know

you're doing like these quzy amount of flops and you know chips are messy and man you

can just have bugs like bugs are it's it's hard to overstate how how hard bugs are to avoid what's uh the role of a

good prompt in all this see you mention that benchmarks have really uh structured well formulated

prompts what what should a human be doing to maximize success and what's the importance of what the humans you wrote

a blog post on you called it prompt design yeah uh I think it depends on which model you're using and all of them

are likly different and they respond differently to different prompts but um I think the original gp4 uh and the

original sort of bre of models last last year they were quite sensitive to the prompts and they also had a very small

context window and so we have all of these pieces of information around the codebase that would maybe be relevant in

the prompt like you have the docs you have the files that you add you have the conversation history and then there's a

problem like how do you decide what you actually put in the prompt and when you have a a limited space and even for

today's models even when you have long context filling out the entire context window means that it's slower it means

that sometimes a model actually gets confused and some models get more confused than others and we have this

one system internally that we call preum which helps us with that a little bit um and I think it was built for the era

before where we had 8,000 uh token context Windows uh and it's a little bit similar to when you're

making a website you you sort of you you want it to work on mobile you want it to work on a desktop screen and you have

this uh Dynamic information which you don't have for example if you're making like designing a print magazine you have

like you know exactly where you can put stuff but when you have a website or when you have a prompt you have these

inputs and then you need to format them will always work even if the input is really big then you might have to cut

something down uh and and and so the idea was okay like let's take some inspiration what's the best way to

design websites well um the thing that we really like is is react and the declarative approach where you um you

use jsx in in in JavaScript uh and then you declare this is what I want and I think this has higher priority or like

this has higher Z index than something else um and then you have this rendering engine in

web design it's it's like Chrome and uh in our case it's a pre renderer uh which then fits everything onto the page and

and so you declaratively decide what you want and then it figures out what you want um and and so we have found that to

be uh quite helpful and I think the role of it has has sort of shifted over time um where initially was to fit to these

small context Windows now it's really useful because you know it helps us with splitting up the data that goes into the

prompt and the actual rendering of it and so um it's easier to debug because you can change the rendering of the

prompt and then try it on Old prompts because you have the raw data that went into the prompt and then you can see did

my change actually improve it for for like this entire evil set so do you literally prompt with jsx yes yes so it

kind of looks like react there are components like we have one component that's a file component and it takes in

like the cursor like usually there's like one line where the cursor is in your file and that's

like probably the most important line because that's the one you're looking at and so then you can give priorities so

like that line has the highest priority and then you subtract one for every line that uh is farther away and then

eventually when it's render it to figure out how many lines can I actually fit and it centers around that thing that's

amazing yeah and you can do like other fancy things where if you have lots of code blocks from the entire code base

you could use uh retrieval um and things like embedding and reranking scores to add priorities for each of these

components so should humans when they ask questions also use try to use something like that like would it be

beneficial to write jsx in the in the problem where the whole idea is should be loose and messy I I think our goal is

kind of that you should just uh do whatever is the most natural thing for you and then we are job is to figure out

how do we actually like retrieve the relative EV things so that your thing actually makes sense well this is sort

of the discussion I had with uh Arvin of perplexity is like his whole idea is like you should let the person be as

lazy as he want but like yeah that's a beautiful thing but I feel like you're allowed to ask more of programmers right

so like if you say just do what you want I mean humans are lazy there's a kind of tension between just being lazy versus

like provide more is uh be prompted almost like the system pressuring you or inspiring you to be

articulate not in terms of the grammar of the sentences but in terms of the depth of thoughts that you convey inside

the uh the problems I think even as a system gets closer to some level of perfection often when you ask the model

for something you just are not not enough intent is conveyed to know what to do and there are like a few ways to

resolve that intent one is the simple thing of having model just ask you I'm not sure how to do these parts based in

your query could you clarify that um I think the other could be maybe if you there are five or six

possible Generations given the uncertainty present in your query so far why don't we just actually show you all

of those and let you pick them how hard is it to for the model to choose to speak talk back sort of versus

gener that's a that's hard sort of like how to deal with the uncertainty do I do I choose to ask for

more information to reduce the ambiguity so I mean one of the things we we do is um it's like a recent addition is try to

suggest files that you can add so and while you're typing uh one can guess what the uncertainty is and maybe

suggest that like you know maybe maybe you're writing your API and uh we can guess using the

commits uh that you've made previously in the same file that the client and the server is super useful and uh there's

like a hard technical problem of how do you resolve it across all commits which files are the most important given your

current prompt and we still sort of uh initial version is ruled out and I'm sure we can make it much more

accurate uh it's it's it's very experimental but then the ideaas we show you like do you just want to add this

file this file this file also to tell you know the model to edit those files for you uh because if if you're maybe

you're making the API like you should also edit the client and the server that is using the API and the other one

resolving the API and so that would be kind of cool as both there's the phase where you're writing the prompt and

there's before you even click enter maybe we can help resolve some of the uncertainty to what degree do you use uh

agentic approaches how useful are agents we think agents are really really cool like I I I think agents is like uh it's

like resembles sort of like a human it's sort of like the like you can kind of feel that it like you're getting closer

to AGI because you see a demo where um it acts as as a human would and and it's really really cool I think um agents are

not yet super useful for many things they I think we're we're getting close to where they will actually be useful

and so I think uh there are certain types of tasks where having an agent would be really nice like I would love

to have an agent for example if like we have a bug where you sometimes can't command C and command V uh inside our

chat input box and that's a task that's super well specified I just want to say like in two sentences this does not work

please fix it and then I would love to have an agent that just goes off does it and then uh a day later I I come back

and I review the the thing you mean it goes finds the right file yeah it finds the right files it like tries to

reproduce the bug it like fixes the bug and then it verifies that it's correct and this is could be a process that

takes a long time um and so I think I would love to have that uh and then I think a lot of programming like there is

often this belief that agents will take over all of programming um I don't think we think that that's the case because a

lot of programming a lot of the value is in iterating or you don't actually want to specify something upfront because you

don't really know what you want until youve seen an initial version and then you want to iterate on that and then you

provide more information and so for a lot of programming I think you actually want a system that's instant that gives

you an initial version instantly back and then you can iterate super super quickly uh what about something like

that recently came out rep agent that does also like setting up the development environment installing

software packages configuring everything configuring the databases and actually deploying the app yeah is that also in

the set of things you dream about I think so I think that would be really cool for for certain types of

programming uh it it would be really cool is that within scope of cursor yeah we're aren't actively working on it

right now um but it's definitely like we want to make the programmer's life easier and more fun and some things are

just really tedious and you need to go through a bunch of steps and you want to delegate that to an agent um and then

some things you can actually have an agent in the background while you're working like let's say you have a PR

that's both backend and front end and you're working in the front end and then you can have a background agent that

doesn't work and figure out kind of what you're doing and then when you get to the backend part of your PR then you

have some like initial piece of code that you can iterate on um and and so that that would also be really cool one

of the things we already talked about is speed but I wonder if we can just uh Linger on that some more in the the

various places that uh the technical details involved in making this thing really fast so every single aspect of

cursor most aspects of cursor feel really fast like I mentioned the apply is probably the slowest thing and for me

from sorry the pain I know it's it's a pain it's a pain that we're feeling and we're working on

fixing it uh yeah I mean it says something that something that feels I don't know what

it is like 1 second or two seconds that feels slow that means that's actually shows that everything else is just

really really fast um so is there some technical details about how to make some of these models so how to make the chat

fast how to make the diffs fast is there something that just jumps to mind yeah I mean so we can go over a lot of the

strategies that we use one interesting thing is Cash Waring um and so what you can is if as the user is typing you can

have yeah you're you're probably going to use uh some piece of context and you can know that before the user's done

typing so you know as we discussed before reusing the KV cache results and lower latency lower cost uh cross

requests so as a user starts type in you can immediately warm the cache with like let's say the current file contents and

then when theyve pressed enter uh there's very few tokens it actually has to to prefill and compute before

starting the generation this will significantly lower ttf can you explain how KV cach works yeah so the way

Transformers work um I like it I mean like one one of the mechanisms that allow Transformers to not just

independently like the mechanism that allows Transformers to not just independently look at each token but see

previous tokens are the keys and values to tension and generally the way tension works is you have at your current token

some query and then you've all the keys and values of all your previous tokens which are some kind of representation

that the model stores internally of all the previous tokens in the prompt and like by default when you're doing a

chat the model has to for every single token do this forward pass through the entire uh model that's a lot of Matrix

multiplies that happen and that is really really slow instead if you have already done that and you stored the

keys and values and you keep that in the GPU then when I'm let's say I have stored it for the last end tokens if I

now want to compute the the output token for the N plus one token I don't need to pass those first end tokens through the

entire model because I already have all those keys and values and so you just need to do the forward pass through that

last token and then when you're doing attention uh you're reusing those keys and values that have been computed which

is the only kind of sequential part um or sequentially dependent part of the Transformer is there like higher level

caching of like caching of the prompts or that kind of stuff could help yeah that that there's other types of caching

you can kind of do um one interesting thing that you can do for cursor tab is you can basically predict ahead as if

the user would have accepted the suggestion and then trigger another uh request

and so then you've cashed you've done the speculative it's it's a mix of speculation and caching right because

you're speculating what would happen if they accepted it and then you have this value that is cach this this uh

suggestion and then when they press tab the next one would be waiting for them immediately it's a it's a kind of clever

heuristic slash trick uh that uses a higher level caching and and can give uh the it feels fast despite there not

actually being any changes in the in the model and if you can make the KV cach smaller one of the advantages you get is

like maybe maybe you can speculate even more maybe you can get seriously 10 things that you know could be useful I

like uh like predict the next 10 and and then like it's possible the user hits the the one of the 10 it's like much

higher chance than the user hits like the exact one that you show them uh maybe they typeing another character and

and he sort of hits hits something else in the cache yeah so there's there's all these tricks where um the the general

phenomena here is uh I think it's it's also super useful for RL is you know may maybe a single sample from the model

isn't very good but if you predict like 10 different things uh turns out that one of the 10 uh that's

right is the probability is much higher there's these passid key curves and you know part of RL like what what RL does

is you know you can you can exploit this passid K phenomena to to make many different predictions and and uh one one

way to think about this the model sort of knows internally has like has some uncertainty over like which of the key

things is correct or like which of the key things does the human want when we ARL our uh you know cursor Tab model one

of the things we're doing is we're predicting which like which of the hundred different suggestions the model

produces is more amendable for humans like which of them do humans more like than other things uh maybe maybe like

there's something with the model can predict very far ahead versus like a little bit and maybe somewhere in the

middle and and you just and then you can give a reward to the things that humans would like more and and sort of punish

the things that it would like and sort of then train the model to Output the suggestions that humans would like more

you you have these like RL Loops that are very useful that exploit these passive K curves um Oman maybe can can

go into even more detail yeah it's a little it is a little different than speed um but I mean like technically you

tie it back in because you can get away with the smaller model if you are all your smaller model and it gets the same

performance as the bigger one um that's like and SW I was mentioning stuff about KV about reducing the size of your KV

cach there there are other techniques there as well that are really helpful for Speed um so kind of back in the day

like all the way two years ago uh people mainly use multi-ad attention um and I think there's been a migration towards

more uh efficient attention schemes like group query um or multiquery attention and this is really helpful for then uh

with larger batch sizes being able to generate the tokens much faster the interesting thing here is um this now

has no effect on that uh time to First token pre-fill speed uh the thing this matters for is uh now generating tokens

and and why is that because when you're generating tokens instead of uh being bottlenecked by doing the super

realizable Matrix multiplies across all your tokens you're bottleneck by how quickly it's for long context um with

large batch sizes by how quickly you can read those cache keys and values um and so then how that that's memory bandwidth

and how can we make this faster we can try to compress the size of these keys and values so multiquery attention is

the most aggressive of these um where normally with multi-head attention you have some number of quote unquote

attention heads um and some number of kind of query query heads U multiquery just preserves the query heads gets rid

of all the key value heads um so there's only one kind of key value head and there's all the remaining uh query heads

with group query um you instead you know preserve all the query heads and then your keys and values are kind of in

there are fewer heads for the keys and values but you're not reducing it to just one um but anyways like the whole

point here is you're just reducing the size of your KV cache and then there is MLA yeah multi- latent um that's a

little more complicated and the way that this works is it kind of turns the entirety of your keys and values across

all your heads into this kind of one latent Vector that is then kind of expanded in frence time but MLA is from

this company uh called Deep seek um it's it's quite an interesting algorithm uh maybe the key idea is sort of uh in both

mqa uh and in other places what you're doing is sort of reducing the uh num like the number of KV heads the

advantage you get from that is is you know there's less of them but uh maybe the theory is that you actually want a

lot of different uh like you want each of the the keys and values to actually be different so one way to reduce the

size is you keep uh one big shared Vector for all the keys and values and then you have

smaller vectors for every single token so that when you m you can you can store the only the smaller thing as some sort

of like low rank reduction and the low rank reduction with that and at the end of the time when you when you eventually

want to compute the final thing uh remember that like your memory bound which means that like you still have

some some compute left that you can use for these things and so if you can expand the um the latent vector

back out and and somehow like this is far more efficient because just like you're reducing like for example maybe

like you're reducing like 32 or something like the size of the vector that you're keeping yeah there's perhaps

some richness in having a separate uh set of keys and values and query that kind of pawise match up versus

compressing that all into one and that interaction at least okay and all of that is dealing with um being

memory bound yeah and what I mean ultimately how does that map to the user experience trying to get

the yeah the the two things that it maps to is you can now make your cash a lot larger because you've less space

allocated for the KB cash you can maybe cash a lot more aggressively and a lot more things do you get more cash hits

which are helpful for reducing the time to First token for the reasons that were kind of described earlier and then the

second being when you start doing inference with more and more requests and larger and larger batch sizes you

don't see much of a Slowdown in as it's generating the tokens the speed of that what it also allows you to make your

prompt bigger for certain yeah yeah so like the basic the size of your KV cache is uh both the size of all your prompts

multiply by the number of prompts being processed in parallel so you could increase either those Dimensions right