AI Safety Research, LLM Auditing, and Governance Challenges for AGI

[Music] so yeah I was just going to kind of talk really briefly about a number of the

things I've been doing recently yeah um sure so just for some background context uh my group has seven PhD students in it

now and they all work on somewhat different things mostly on AI safety um but not any one particular topic um so

it's it's a lot of pretty Broad and diverse set of things that I'm working on uh recently I've been trying to do

more work related to governance so I've sorted my scholar by year so you can see some of the most recent ones if you do

that as well um and maybe roughly half of the papers from 2004 are relating to governance so this conference I was just

at in Rio um fact we had a paper called blackbox access is insufficient for rigorous AI audits so AI auditing is

something that has been uh increasingly popular and governments are starting to look pretty serious about doing some

serious auditing um but unfortunately uh so far all of the auditing that is being done or proposed is is almost all just

what we call blackbox Access so this is like here's a model that we're going to deploy and you get to interact with it

some before it actually gets deployed uh the auditor uh gets to interact with it some and in this paper we argue that

that's really not sufficient and we would like to enable more access like white box access would mean that you get

the auditor gets full access to the model weights um but we also talk about things in between uh that might be

useful and we also talk about ways that you could sort of simulate white box access using

um uh like cryptographic techniques and stuff like this to give to allow an auditor to make particular queries which

could be arbitrary queries the same way that you could if you actually had you know the model there like as a you know

a pytorch model or something let's say um but without giving the auditor direct access to the model so that queries can

still be vetted to see if they are you know legitimate or they can be contested if the company is like no that's going

to give you too much information about our system for instance um so yeah uh one of my the next one here visibility

into AI agents is sort of part of a series of papers so we had another paper at fact last year also led by alen Chan

on harms from increasingly agentic algorithmic systems so a lot of the concerns about AI systems are to do with

particularly agentic systems and here were um Allen is is really focused on agent governance these days so right now

uh we don't really see like AI agents or especially llm agents being deployed much in practice but there's a huge

amount of work trying to make those systems and deploy them and uh I think there's a a many more safety concerns

become more serious and severe when you have more agentic systems um and so the paper from last year was sort of

articulating and explaining why that would be the case um I guess uh yeah moving down a bit so

we we also have some work recently um that's more technical but is still quite related to questions of governance and

and auditing so um that would include the stress tting capability elicitation What mechanisms does knowledge

distillation distill um an earlier paper from 2023 uh which I think you'd have to

click the show more to for it to show up mechanistic mode connectivity um so we're kind of also doing some work on

what but uh like how does fine-tuning work and how do current safety methods work and why are they insufficient and

why they might be best viewed as hiding safety problems rather than fixing those problems and that's one of the main

reasons why blackbox audits might be insufficient is because the auditor might fail to um identify and surface

the problematic behavior and it can be much easier to figure out what's wrong with the model if you had more access

like access to the weights or maybe access to the the training process or the training data as well

um I guess uh if I now were to sort it by the citation count um there's a lot of my sort of old work from during my

PhD and Masters on just sort of generic deep learning stuff um but some of the more recent ones here include defining

and characterizing reward gaming and goal miss generalization and deep reinforcement learning so those are both

from two years ago um and these are a couple papers where we tried to really uh introduce to the

machine Learning Community two of these classic failure modes that again are sort of particularly acute or concerning

when thinking about agentic AI systems so these are um ways that you can end up with a misaligned system and a

misaligned agent in particular I should say um and yeah so just really briefly reward hacking is basically when you

have the the wrong reward function and then you optim it and um something bad happens you actually make things worse

by optimizing it and uh in that paper we wanted to see what are the conditions that a reward function needs to satisfy

in order to be unhackable so that it's safe to optimize you know that optimizing that reward function won't

actually make things worse um so in practice you know we train models with uh reward models which are imperfect

approximations of what we actually care about and so we want to know how how good the approximation needs to be in

order for this to be you know guaranteed safe and it turns out uh the the the requirements are like extremely strict

basically you need to have the correct reward function um or else there's always some chance that optimizing

against a proxy will actually make things worse according to a ground Truth uh reward function and then the other

one go Miss generalization is about if you have the right reward function but your system Mis generalizes and in

instead um ends up learning yes that's the one right at the bottom of the screen here now um ends up

optimizing a different reward function because it was maybe really correlated with the true reward function on the

training distribution and so this is another this is really similar to some sort of well-known problems in computer

vision like the cow on the beach problem um but this is a a version of it that could be particularly severe because you

can end up with a system that is competently pursuing the wrong goal and so sort of actively working against uh

the the sort of goal that you had intended for it to pursue um yes that that gives like a

little bit of a flavor um oh maybe one more thing I want to mention is from the very recent papers again from this year

there's two papers um affirmative safety and safety cases that are really just uh saying instead of just doing audit so I

already said like blackbox audits are maybe not enough um I think actually even any form of auditing is really not

in my mind sufficient in terms of governance I think we want developers to actually be making a positive deploying

AI systems to show that they are safe and that they are going to have a positive impact um and so this these

papers were both kind of describing ways that you can imagine doing that uh yeah so I'll leave it at that uh

super super thanks David uh for for giving that uh background uh meaning as I said in the email just to reiterate we

we will record this uh interactions and it will go as part of a lecture online course that we teaching um yeah so the

first one you you uh nicely said uh you actually said about facts some one of the student actually possibly looked at

your Tweet or something uh said that look you attended fact recently is there anything interesting uh that uh that you

picked up from attending fact technically in terms of problems yeah um I guess um some of some

of the most interesting things for me there were uh not really on the technical side but again more on the

governance side so understanding what people are doing right now um in terms of like legal challenges around AI

systems so I think there's a lot of concerns that you know systems can be biased or U trained on copyright data or

just harming people being deployed irresponsibly and so we heard from some of the people who are like really

involved you know um on the front lines so to speak of of these battles um so that was really cool um yeah technical

stuff so I I mostly when I go to conferences these days I mostly just talk to people a lot so I I didn't even

attend too many sessions but another sort of and and I should mention this is only the second time I've attended this

conference so I really enjoyed it and I highly recommend it I last year I was only there for like a day and a half um

as well um but I I was kind of trying to get you know a Vibe check and figure on the

pulse of like how that Community is is thinking about things and so it's quite interesting because um yeah there's a

lot of people who are starting to think more about AI safety partially because that is sort of a a more powerful brand

the these days if you will and partially because um historically people at fact and people from the ethics Community

have been more likely to sort of have a skeptical or dismissive attitude towards advances in AI saying like these things

don't really work it's it's like snake oil you know it's it's a scam um and I think

more and more people are starting to say oh that's not really the case anymore with these systems they actually do have

important impressive capabilities and so we really need to start um you know catching up and figuring out like how

they're being used and and what we can do about that um so yeah it was it was pretty interesting you know just sort of

from from that point of view for me to um meet a lot of people from this other community that I haven't interacted with

that much and um yeah one of the one of the things I liked about it was well two of the things I liked about it was it's

a much more diverse communities so not just technical researchers but researchers from many other disciplines

um and also what was the other thing I was going to mention um well it skips my mind now so I'll let

you know if I think of it later yeah yeah I I know uh fact for a couple of years now and uh yeah the the the

audience is quite uh diverse uh in terms of policy researchers actually one of the speakers that we had in the summer

school earlier uh did attend last year actually okay so now let's uh get into more questions uh so in one of your

recent tweet you actually mentioned that llm built by current leaders I'm just reading it from the Tweet itself so uh

uh current leaders will lead a transformative AI in a few years they will likely be sufficiently smart and

aligned to solve any remaining AI safety problems within next one to two years can you can you throw a little bit light

on it what are you thinking yeah so import context here is that I was trying to characterize a view that I think is

um like quite common or some some variation of this view is quite common within the AI safety Community these

days um and there's more to it as well in the in the Tweet thread this is not my view I mean I think this is a

possibility but I think it's not super likely um but uh I guess and in fact the the point of this tweet and this tweet

thread was sort of to for me to um to rail against this and say I I I disagree with all of this um I think that it's a

a view that is too like too prominent and getting too much um attention and is misleading people in terms of how they

approach the question of AI safety strategically um but I think there is um you know there it certainly starts to

look possible in my mind that like we're in shooting distance of like transformative systems and it could be

you know just a few more generations of scaling will get us to systems that are like really really powerful um and the

hope and I think it is just a hope and I think it's a long shot and not you know at the end of this tweet thread I say

the this these strategies are incredibly desperate and fragile um but the hope is that uh the leading developers or some

of the leading developers will be able to you know leverage these transformative AI capabilities

um either themselves or maybe in cooperation with uh governments um or uh International organizations to sort of

solve what other whatever problems are remaining to prevent AI from uh causing an

existential catastrophe and yeah it's kind of it's a little bit unclear how that's supposed to happen um but a lot

of people would say there are some technical problems that remain to be solved but at a certain point we'll get

AI research assistants and if those research assistants are not actively working against us um so I mentioned

earlier how like reward hacking or goldness generalization could cause AI systems to be actively working against

us um but if that is not the case uh then maybe we can get a lot of work out of these research assistants and it will

be as if we can you know do a thousand or a million times more AI safety or AI alignment research in a year than we

normally would be able to because of because of that um so that's kind of the one of the the line of thought that I

was commenting on and really critiquing um and I do think it's it's it's not like totally impossible I think it's

just not something I want us to be counting on or betting on yeah um I think it's you know when you're talking

about trying to solve these massive problems in this in the span of a year or two I think there's just so much so

much potential for things to go off the rails um for you know a variety of reasons like you know political reasons

for instance if if uh there's you know competition and disagreement about like how to proceed and how to use these

systems and stuff like that and then I think there's also the question of like are we actually going to get there in a

few years and you know etc etc there's there's a lot of so I didn't really lay out all the problems I have with this

view in in the Tweet but um a few people have ask me to and maybe I will at some point in the future yeah okay good good

to know the context of the Tweet uh this is connected to the other question that we had right what is Agi according to

you uh when do you think it'll come come if it comes what will it look like and what will you do with

it yeah um I mean so in my mind AGI refers to like AI systems that have no qualitative or quantitative cognitive

deficits compared to humans and um let's say expert humans just to to be clear um so I think you know the the basic idea

of AGI in my mind was always like um AI systems that really surpass human intelligence in in every sense of the

word um and I think uh if when this happens it's going to like pretty quickly uh unleash a lot

of chaotic power struggles and probably lead to um yeah the extinction of humanity or a something really bad

because I think it's um like going to be even if it's technically possible to keep these

systems in control um I think we will still see that uh people will use them to pursue selfish and short-term

objectives uh because they'll feel that it is necessary for to maintain their own competitiveness and really to

survive as you know a nation or an organization or a company or an individual um so I think when I look

around the world today I see a lot of this happening right now already we're not very good at solving Collective

action problems and coordinating you know to to work towards the common good you can see climate change as a great

example of this um and you know even things like the Persistence of Wars uh these are just you know destructive

conflicts that nobody really thinks are you know should should exist necessarily but we don't have a way of coordinating

to avoid them um so I think AGI is likely to intensify and accelerate all of these competitive Dynamics and lead

to sort of a race to the bottom where um yeah people are are just forced to uh yeah sort of act uh in their ve their

very short-term and and um selfish interests in order to not sort of fall behind and be outc competed and I think

that's going to be really bad for for everybody and for the world yeah no I think uh this this power equation is

definitely a uh important uh thing to to uh keep in mind we'll come back to this power later at some point in time um but

let's get to so Usman actually interacted with us last week uh he he presented the report and gave some of

the open challenges um so so from your side can you can you sort of say uh list down

high level takeaways from the report I'm also curious to hear about how did the report idea uh get conceived uh what

process procedure did you follow in actually putting this report together there why was it exciting to write this

report yeah um yeah thanks for asking so uh basically this report was usman's idea and really like so much of the

credit goes to him so um I have to make that uh clear if it certainly wouldn't have happened without him and he did the

more much more work than anybody else myself included um and originally uh he started talking to me about this about a

about a year ago and saying we should make like a concrete problem s in AI safety for the llm era so um I'm not

sure if people are familiar with this classic paper from 2016 concrete problems in AI safety um but to some

extent it's kind of outdated because uh so much has changed with the Advent of Ls I consider it a new paradigm along

the lines of how deep learning was a new paradigm when I got into the field about 11 years ago um and so I think you know

everything needs to be Revisited in in light of um llms and Foundation models and it just kind of got bigger and

bigger in scope and um you know one thing led to another so um we ended up with a 175 page document instead of like

a 25 page document as we had originally envisioned um and yeah what else to say about that I mean I think I'm pretty

happy with how it turned out despite it drifting so far from the vision and it did take a long time to finish just

because there was a lot of work to do you know corralling uh and organizing this large project with so many people

and then trying to maintain like a consist consistent editorial style and standard of quality and just getting

people sort of on the same page about what we were trying to do like really making it a research agenda instead of

something more like a survey or review paper um in terms of the big takeaways I mean

I think to a large extent it was more in my mind about sort of expressing things that were already you know in in our

heads me and me and Usman about like you know what we think are the interesting problems um I guess I think uson told me

that he updated some towards uh some of the directions that are really popular in the AI safety Community being much

less promising so interpretability being one of the main ones that's incredibly popular in the AI safety Community but

um really doesn't look very promising um to to either of us um so I think it's sort of a good thing to work on but it's

really overinvestment sections that uh or I guess two of the sections that I proposed and contributed

the most to were about safety performance tradeoffs so that gets back to the previous question um and the

basic idea there is like uh if you if there is a trade-off between safety and performance then

there will always be pressure for people to sacrifice safety in order to have a more performant system and that's one

way that you can get these really problematic sort of Game Theory Dynamics where where

people deliberately build and deploy and use systems that they know are not safe and they know might even you know go

Rogue and get completely out of control um because not doing so is just not competitive so if we think about AI

agents again um you know one of the Key properties of an agent is that it you give it a ton of autonomy and so you

could say like just run this company for me so one of the things I'm very worried about is I think once we get to AGI it

will be very to keep humans in the loop in any meaningful sense and there'll be a huge pressure for organizations to

just automate all decision- Mak um and you know compare that to where AI can make you know uh tons of decisions like

really quickly with like actually having meetings and having conversations and you know making decisions on the like

you know on the time span of days instead of like seconds and it's easy to see why it would be really hard to

compete if you don't handoff decision- making to an AI system um so that was one really interesting one another one

was uh section 2.2 which I forget what we call it but it's basically about the notion of capabilities in AI systems so

all of this stuff about auditing or evaluations is sort of predicated on the notion of um systems capabilities and in

particular the idea of dangerous capabilities so uh the current you know sort of international governmental

consensus seems to be we certainly don't want there to be easily available uh AI systems that can help you build weapons

of mass destruction so let's can let's call that a dangerous capability and then let's try and make sure that we

have a proper evaluations regime such that any system that's going to be made available to people won't help them

build weapons of mass destruction um I think this is a very like kind of weird and limited way of viewing the problem

of AI safety but let's just go with it for now so that's where people are at in terms of policy um but this whole notion

of capabilities is a little bit confused and I really it's a word that I think people really started using a lot more

like two years ago along with Eva and these both kind of come out of the safety community and neither of them has

much rigorous grounding and so there's a question of like what does it even mean that a system has a capability so you

can talk about the performance on a benchmark but we know that benchmarks don't always really reflect underlying

capabilities because they can be very narrow and if you look at performance on even the slightly different distribution

you can find that the system performs much differently and much worse so this idea of just measuring capabilities in

terms of performance on a benchmark isn't really satisfying and so we sort of talk about how could we go beyond

this and have a more rigorous way of talking about capabilities um because they are such a big part of the

conversation right now and it's really not technically grounded in a in a very rigorous way yeah sure so so one uh

followup questions is how I meaning what was the so is the idea in putting the authors together I meaning they're

they're from all across the world uh including dance uh group on one side right so how did that

happen yeah um so I think we we basically just you know started thinking about who we knew who worked on these

different areas that we wanted to cover and then reached out to people um and in some cases I think we like would contact

a somebody more senior and and ask them to help recruit Junior people so we sort of tried to have a senior author who

would um sort of be the edit have have like editorial control and then more Junior authors like actually writing the

content um and then besides that we also then had like rewriting stuff A lot of times and me sort of exercising the

final editorial control or me and Usman collaboratively doing that um and one thing that we were aiming for is to

really like have primarily or exclusively academic authors so you know there's sort of a meta issue with the

whole field of AI and AI safety in particular where um a lot of it is happening in companies and these are the

companies that are building the dangerous AI systems um and so there's kind of a conflict of interest there and

uh a lot of the times concerns about AI safety and existential risk in particular get characterized and

mischaracterized as just some sort of corporate ploy to distract from other issues or to like hype up their products

and we really wanted to make sure that we weren't you know falling victim to that um misperception interesting you

referred as dangerous AI systems I'm curious to know what are the what are the companies and products that you

think are dangerous AI systems as of now yeah um so I think it's it's kind of a question of like yeah what you mean by

dangerous I think uh I I have another tweet about this I mean I think the release of gp4 was dangerous and

irresponsible and I think that um you know there wasn't any reason a priority to to be confident that the system

wasn't going to lead to an existential catastrophe or cause otherwise like massive societal harms um so I think

that was like a pretty unlikely um risk but you know the the magnitude of these risks are large enough that I think uh

ideally everyone would be moving slower and being more careful with um deploying these systems so that comes back to like

the safety performance trade-offs where uh if you you know if you do more testing more safety testing or if you

try and ensure that you really understand how the system works then that's going to really limit your

ability to build and deploy systems quickly um and I think you know so far we haven't really seen people

successfully build powerful AI agents on the back of systems like gp4 um and so it's really unclear you

know whether and to what extent these kinds of safety concerns will manifest when that happens um

so um yeah so I I think uh basically any any systems that are like at that

capability level and above I would consider potentially dangerous and I think you know again depending on what

you mean by dangerous potentially dangerous is just a version of dangerous because it means you don't know if it's

dangerous or not and so you should you know if you're being conservative you should treat it as dangerous um so like

broadly speaking I think uh yeah like my my view on the whole state of AI right now is that it we need like an

indefinite pause on advancing AI capabilities in order to like actually you know get our together and try

and coordinate to have a more effective um means of you know uh controlling uh the rate of progress and the way in

which systems are deployed and used so I think these are incredibly difficult governance problems that we don't really

have any promising ideas for right now and I worry that you know within a few years time or like a decade's time it

will be pretty clear that AI is completely reshaping the world um and poses a serious risk to human existence

or at least existence as we know it um and we'll be in a similar situation to where we are with climate change or we

have been for several decades where there's scientific consensus but we still aren't able to coordinate

effective uh Global action on the problem um and I would like us to you know buy as much time as possible to

sort of start working on those global coordination issues before we get to like really powerful and dangerous

systems um yeah this also relates to the thing about capabilities where uh and and one of the reasons why I think

audits are insufficient is that you don't really know what what a system can do when you combine it with other

systems so like when you um you know give it access to more tools and more kinds of data and when you have like

interactions of many systems and so I think like some of the capabilities that gb4 exhibits are like really incredible

and could be um really really powerful and dangerous if people figured out how to make it into an effective agent and

gave it access to more real world sensors and actuators and I say could be because I it's it's just we don't know

um so I'm not saying like certainly would be a problem but I think there's like you know there there is some risk

there already so yeah sure uh so you mentioned this couple of times so let's uh let's go deeper there global

coordination couple of times you mentioned so what are your thoughts on global coordination what is going on now

uh meaning it I mean there's a connected question also which is like this EU AI act that has come is is that uh s to say

going to look at some of these Global coordination um particularly with EU us India any any

thoughts on uh what's going on in this space of coordinating between countries yeah so like the eui act as as to my

knowledge is just about the EU um so yeah um I think like there's been uh these safety Summits and there's been

some you know some signs of progress here so it does look like there's an effort at least to coales around some

sort of International auditing or evaluation scheme um and that's you know that that does represent some meaningful

progress on the other hand um I think it's it's woefully inadequate and I'm skeptical that it will be enacted

effectively in a way that like has real teeth um so I think a lot more is needed um what I would say is basically I would

like uh all of the major World governments right now to be like sitting down and talking to each other about um

what are we going to do if we arrive at a scenario where we know that there are certain AI systems that we don't want to

exist um because they are too dangerous and so we just all agree that like this thing just shouldn't exist we don't want

anybody to have it what can we do like is there any solution that we can imagine um where we would actually be

able to say you know that kind of system is not going to exist because we all agree that it shouldn't and the the

problem again is like maybe individual Nations will have reasons to cheat and build such a system anyways you know for

instance for autonomous Warfare or for just um economic competitiveness or other geopolitical reasons and the way I

usually think about this is like um very simplified picture of future AI systems um or maybe even like well yeah of the

kind of a systems I'm worried about is like it's like a button that you can push and every time you push it you get

a million dollars but there's a one in a million chance it destroys the world we don't want anything like that to exist

because everyone's going to just like push that button a bunch of times really fast and then the world is going to end

and I think when you're talking about systems that might actually like go Rogue escape human control and be able

to you know take over the world essentially um you you are running that risk um and so I think because we don't

really have any any way of assuring that a system won't do that now um the only reason that we

aren't that worried is because the system don't seem capable of doing it so none of the current systems seem like

quite smart enough like they're pretty smart but they're not they're not there yet they're not at the point where like

if we had them plugged into to all sorts of infrastructure and if they were you know running companies or running

militaries or these sorts of things um they would actually be able to you know uh successfully dis Empower

Humanity but in the future uh that does seem like something that's going to happen at some point because we will

have AGI and then superhuman AI systems and it also seems like in the future um there will be lots of competitive

reasons to plug these systems into exactly those things into the military into infrastructure into you know

leadership positions in companies um into you know Supply chains and just just everywhere because it'll be so

effective so efficient and be able to make really um High quality decisions really fast and at that point uh I I

think we don't want to do any of that unless we really have a really good reason to believe that these systems are

going to stay under human control or you know do things that are in Humanity's interest and I just don't see any

pathway towards having that level of confidence right now um so I think I think in my mind it's like quite likely

that we will reach this situation um before too long where we we see you know the writing on

the wall we see that this is where things are headed and everyone you know wants to slam on the brakes and wants to

stop they don't want you know us to hand over all of the running of society AI systems but at that point it'll be

really difficult to stop it so I think we need to sort of think ahead a little bit and think about what could we do to

coordinate to prevent that from happening sure let's get to privacy for for uh uh few questions uh so as all of

us understand these models have been trained using uh social media data uh uh which may have some of our personal data

including your tweets including my tweets uh how can these AI systems be designed to respect and protect users

privacy yeah it's a really big problem in my mind I think um you know there there are some technical tools like

differential privacy here that might be useful but um I think this is like mostly another sort

of structural problem where um there just isn't an a proper um incentive to protect users privacy and so all of the

business models are not really set up to do that in the way that I'd like to see so um I think it would take like a

pretty significant transformation of how business is being conducted right now because um you'd really want to redesign

the whole process of from the ground up with privacy in mind um and you know so there's things like uh the um gdpr sort

of gives you these rights over your personal data but it's really hard to see how rights like that might be

enforced um sorry so gdpr that's another EU regulation from a few years back yeah yeah we we have seen that as part of the

CL summer school um how things like that could be enforced in the context of AI systems so like if this system was

trained on data we don't actually have any technical means of undoing that right now like there's work on it but

none of the methods are are like actually very good um and so you know it's not like we're going to throw away

gp4 and and train it all over again just because it used a little bit of somebody's data and they decided they

they didn't want that um that's just not something that I I see happening although maybe it would be good if if we

did do that sort of thing and take these things really seriously so um yeah I think it's it's more of like a

structural societal problem at the end of the day where we need to um yeah find a way to like get give people more power

over their data and like really enforce that um but again that that would maybe cause a lot of issues for the way that

AI is currently being practiced um and you know there are like these big powerful companies who are the ones who

are building these systems and benefiting from them and they are certainly not going to to just suddenly

make this kind of a change um that would really be giving away a lot of their power and putting more power into the

the hands of the people whose data are used to train the systems um yeah so this this connects to the legal

challenges that I mentioned so um one of the interesting legal issues right now in

the states is the notion of fair use of copyright data so uh I think so far llm developers have success F argued that

they are allowed and and I guess Foundation models more generally um so or large models like I think um

stability and image generation models as well but they're allowed to train on copyrighted data because um their use of

the data is transformative so this is they argue that it's similar to um like if you're an artist and you sort of make

a collage where you like take you know pieces of somebody else's art or like pieces of Brands and you like put them

onto some some new art piece in a way that is you know expressing something different but using bits and pieces of

stuff other people have produced um and uh it to my mind this is not a valid argument because one of the important

criteria for establishing that uh use of copyright data falls under fair use is that you aren't directly competing with

the creators of the copyright data and so if you think about something like you know translation or like generating art

it's pretty clear that these systems are directly competing with translators and artists right now and so I think this

argument is is bogus and I hope that it will be um you know successfully challenged in court and I think that

sort of thing um could give a lot more you know privacy and just rights over data more generally to people um

producing the data because you know data is so Central to building these AI systems and Society really was not set

up um with this in mind right so even besides that point there's sort of a higher level point of like data is so

valuable now and we need to actually account for that properly so like when when somebody uses your data and like

uses it to build a really powerly eye system they should not you know just get away with all of the benefits and profit

from from doing that you as the person who created the data that they trained the system on regardless of whether it

was copyright you should still you know be compensated in some way and even more than compensated I think you should have

some real power um so not just financial compensation but power over the use of your data um but yeah this is all like

stuff that I don't know that much about but this this is just kind of my opinion on the high level take of like how

things need to work if we want to really take this issue and do better yeah yeah as part of the summer school we did see

uh content for uh machine unlearning catastrophic forgetting and and those ideas so yeah very good very good uh so

you talked about these large companies my next question was around that uh let's see if you have anything more to

add here given Apple's collaboration with open AI to integrate Char GPT into their devices and uh

Elon musk's concern about data usage uh is is the privacy and data of users at risk if AI assistance take over um and

then is added to all these devices also you mentioned about AI assistance agents before so yeah um yeah yeah yeah

so I actually I I think this is sort of the first that I heard this news specifically but

um I think so so far all of the like personal and private data that exists um on the web is like it's not actually you

know personalization is pretty weak so far in general and I think Ai and LMS could change this in a big way in the

next few years and that has um huge potential to cause more problems so I think um and and again it comes back to

the safety performance trade-off thing so again imagine in the next few years that you have um big companies coming

out with these AI assistants that will learn to adapt to your work habits and to customize you know everything they do

to be like a really good personal assistant that you would maybe pay like $100,000 a year for right now uh and you

can get it for like you know a hundred bucks a month or something so like way way cheaper so now there's a huge

financial reason for you to adopt this system and use it on the other hand by learning all of this about you learning

how you work learning your routine learning your you know how you how you um how you do your job like how you

think about things how you approach problems you know all of these sort of personal and psychological factors and

even you know things about your relationships just all of these things right are really relevant to doing this

kind of job and if all of that knowledge is now um in the hands of these big companies who are selling you these

assistance that gives them such an incredible and dangerous amount of power because they don't have to use that in

your own interests right there's like some economic incentive to do that on the basis of competitiveness maybe but

there's also a huge potential for um for misuse of that data to do things like manipulate you um you know sell you

things maybe manipulate your political opinions um and that's you know that's sort of arguably like the tip of the

iceberg right so I think the idea of having you know one of these really incredible AI assistant is just it

sounds great right and I I I really want one I would I would love that um this is one of the things that I imagined when I

was first getting into the field of AI and thought would be so cool but the way that you know the way that AI is

developed and sold and everything right now makes this just really really sketchy it's like really um not a

position I want to be in where a a a big company or a government for that matter like I don't want anybody to know that

level of detail about me unless you know maybe an individual person who I know and have a relationship with and I and I

trust like a like a you know human personal assistant maybe but even that you know that's that's that's uh

depending on your on your views on privacy um not something you might want but yeah as this becomes very cheap and

very accessible um it's going to be really hard to resist the the Temptation if you will or really the economic

incentives to adopt this kind of system but it also will you know just put you in a very vulnerable position I think um

and I I don't really see any way around that again it's just like uh sort of um coming out of the current way that the

technology is being developed sorry I shouldn't say I don't see any way around it I mean I think fundamental changes to

how the technology is developed and deployed could address this but that is a very big project and one that we like

urgently need to start undertaking now even if these things aren't here present and live um yeah now I was going to go I

was going to ask that question if you were to be uh restarting the whole uh uh sort of say area domain of gen and if

you were to uh design all of these well and not have these privacy issues that you're discussing what would you do

where would you start how should one do go it uh yeah it it starts to get outside of my area of expertise so like

I can give you these high level takes about like the way that it works right now is bad and scary but how should it

be different I mean I think um I think there are ways that you could set this up where um sort

of we sort of you know it's like the the company um has like half of a a key and I have the other half of a key and if

the two halves don't come together nothing can get done and also um they are sort of only put together on a

case-by Case basis so like when there's a particular thing that we both agree should happen we you know combine forces

and we make that thing happen um but I never give away like my half of the key to the company and so you know every

time that the company would want to do anything or like you know run any computation about about me it would sort

of need that half of my key um and I think there's still there's maybe still some issues and some trade-offs you'd

have to make in terms of you know big part of what makes these systems powerful is generalization so like

there's the personalization using my data but um the system also benefits so much from using everybody else's data

and that can help teach it you know how to help me uh and so we we want those benefits of like generalization and sort

of aggregating the data in some sense into a powerful system that has you know somehow informed about human preferences

in general or people like me and can use that to help serve me um but I think I think you know we could find the right

way to balance these trade-offs and manage them if sort of the structurally things were set up in

the right way and I think that would mean something like this where essentially I'm like every

time any of my data is is used it's used in a very like limited way and there's sort of like accounting and tracking of

it to make sure that it's not misused and stuff like that but it's yeah I I I couldn't say how this all could be imp I

just believe that such a thing is possible based on my conversations with people who know more about these things

than I do yeah yeah sure [Music]