Eliezer Yudkowsky: Is consciousness trapped inside GPT-4? | Lex Fridman Podcast Clips

what do you think about gpt4 howintelligent is it it is a bit smarterthan I thought this technology was goingto scale toand I'm a bit worried about what thenext one will be like uh like thisparticular one I thinkI hope there's nobody inside therebecause you know it'd be sucked to bestuck inside thereum but we don't even know thearchitecture at this point because openair is very properly not telling usandyeah like giant inscrutable matrices offloating Point numbers I don't knowwhat's going on in there nobody's goesknows what's going on in there all wehave to go by are the external metricsand on the external metrics if youask it to write a self-aware fortunegreen text it will start writing a greentext about how it has realized that it'san AI writing a green text and like ohwell sothat's probablynot quite what's going on in there inrealityum but we're kind of like blowing pastall the science fiction guard rails likewe are past the point where in sciencefiction people would be like well waitstop that thing's alive what are youdoing to itand it's probably notnobody actually knows we don't have anyother guard rails we we don't have anyother tests we don't have any lines todraw on the sand and say like well whenwe get this farum we will start to worry about what'sinside thereso if it were up to me I would be likeokay like this far no further time forthe summer of AI where we have plantedour seeds and now we like wait and reapthe rewards of the technology we'vealready developed and don't do anylarger training runs than that which tobe clear I realize requires more thanone company agreeing to not do thatand take a rigorous approach for thewhole AI Community to uh investigatewhether there's somebody inside therethat would take decadeslike having any idea of what's going onin there people have been trying for awhile it's a poetic statement about ifthere's somebody in there but as I feellike it's also a technical statement orI hope it is one daywhich is a technical statement that AlanTuring tried to come up with with thetouring testdo you think it's possible todefinitivelyor approximately figure out if there issomebody in there if there's somethinglike a mind inside this large languagemodelI mean there's a whole bunch ofdifferent sub questions here there's thequestion oflikeis there Consciousness is their qualiais this a object of moral concern isthis a moral patient like should we beworried about how we're treating itand then there's questions like howsmart is it exactly can it do X can itdo y and we can check how it can do Xand how it can do yum unfortunately we've gone and exposedthis model to a vast Corpus of text ofpeople discussing Consciousness on theinternet which means that when it talksabout being self-aware we don't know towhat extents it is repeating back whatit has previously been trained on fordiscussing self-awarenessor if there's anything going on in theresuch that it would start to say similarthings spontaneouslyumamong the things that one could do ifone were at all seriousum about trying to figure this out istrain gpt3 to detect conversations aboutConsciousness exclude them all from thetraining data sets and then retrainsomething around the rough size of gpt4and no larger with all of the discussionof Consciousness and self-awareness andso on missing although you know hardhard bar to pass you know humans areself-aware we're like self-aware all thetime we like talk about what we do allthe time like what we're thinking at themoment all the timebut nonetheless like get rid of theexplicit discussion of Consciousness Ithink therefore I am and all that andthen try to interrogate that modeland see what it says and it still wouldnot be definitivebut nonetheless uhI don't know I feel like when you runover this science fiction guard railslike maybe this thing but what about gbtmaybe maybe not this thing but like whatabout gpt5 you know this this would be agood place to to pauseon the topic of Consciousness you knowthere's so many componentsto even just removing Consciousness fromthe data setemotion the display of Consciousness thedisplay of emotion feels like deeplyintegrated with the experience ofconsciousnessso the hard problem seems to be verywell integrated with the actual surfacelevel illusion of Consciousness sodisplaying emotion I mean do you thinkthere's a case to be made that we humanswhen we're babies are just like gbt thatwe're training on human data on how todisplay emotion versus feel emotion howto show others communicate othersthat I'm suffering that I'm excited thatI'm worriedthat I'm lonely and I missed you and I'mexcited to see you all of that iscommunicated there's a communicationskill versus the actual feeling that Iexperience sowe need that training data as humans toothat we may not be born with that how tocommunicate the internal State andthat's in some sense if we remove thatfrom GPT Force data set it might stillbe conscious but not be able tocommunicate itso I think you're going to have somedifficulty removing all mention ofemotions fromgpt's data set Iwould be relatively surprised to findthat it has developed exact analogs ofhuman emotions in there I think thathumans have will like have likeemotions even if you don't tell themabout those emotions when they're kidsit's not quite exactly whatvarious blanks blank uh slatests try todo with the new Soviet man and all thatbut you know if you try to raise peopleperfectly altruistic they still come outselfishyou try to raise people sexless theystill develop sexual attractionum you know we have some notion inhumans not in AIS of like where thebrain structures are that implement thisstuff and it is really remarkable thingI say in passing that despite havingcomplete read access to every floatingPoint number inthe GPT series we still know vastly moreabout the the architecture of humanthinking then we know about what goes oninside GPT despite having like vastlybetter ability to read GPTdo you think it's possible do you thinkthat's just a matter of time do youthink it's possible to investigate andstudy the way neuroscientists study thebrainwhich is look into the darkness TheMystery of the human brain by justdesperately trying to figure outsomething and to form models and thenover a long period of time actuallystart to figure out what regions of thebrain do certain things what differentkinds of neurons when they fire whatthat means how plastic the brain is allthat kind of stuff you slowly start tofigure out different properties of thesystem do you think we can do the samething with language models uh sure Ithink that if you know like half oftoday's physicists stop wasting theirlives on string theory or whateverand go off and study what goes on insideTransformer networksthen inyou know like 30 40 years uh we'dprobably have a pretty good ideado you think these large language modelscan reasonthey can play chess how are they doingthat without reasoningsoyou're somebody that spearheaded themovement of rationality so reason isimportant to youis so is that as a powerful importantword or is it like how difficult is thethreshold of being able to reason to youand how impressive is it I meanin my writings on rationality I have notgone making a big deal out of somethingcalled reason I have made more of a bigdeal out of something called probabilityTheoryand that's like well your reasoning butyou're not doing it quite rightand you should reason this way insteadand interestingly like people havestarted to get preliminary resultsshowing thatreinforcement learning by human feedbackhas madethe GPT series worse in some waysin particular like it used to be wellcalibrated if you trained it to putprobabilities on things it would say 80probability and be right eight times outof ten and if you apply reinforcementlearning from Human feedback the thelike nice graph of like like 70 7 out oftensort of like flattens out into the graphthat humans use where there's like somevery improbable stuff andlikely probable maybe which all meanslike around 40 percent and then certainyeah so like this like it used to beable to use probabilities but if youapply but if you'd like try to teach itto talk in a way that satisfies humansit it gets worse at probability in thesame way that humans are and that's uhthat's a bug not a feature I would callit a bugalthough such a fascinating bugum but but yeah so so like reasoninglike it's doing pretty well on varioustests that people used to say wouldrequire reasoning butum you know rationality is aboutwhen you say eighty percent doesn'thappen eight times out of tenso what are the limits to you of theseTransformer Networksof of neural networks what's if ifreasoning is not impressive to you or itis impressive but there's other levelsto achieve I mean it's just not how Icarve up realitywhat's uh if reality is a cakewhat are the different layers of thecake or the slices how do you cover itbut you can use a different food if youlikeit's I don't think it's as smart as ahuman yetum I do like back in the day I wentaround saying like I do not think thatjust stacking more layers ofTransformers is going to get you all theway to AGI and I think that's gpt4 ispassed or I thought this Paradigm wasgoing to take usand I you know you want to notice whenthat happens you want to say like whoopswell I guess I was incorrect about whathappens if you keep on stacking moreTransformer layers and that means Idon't necessarily know what gpt5 isgoing to be able to do that's a powerfulstatement so you're saying like yourintuition initially is now appears to bewrong yeahit's good to see that you can admit insome of your predictions to be wrongdo you think that's important to do seebecause you make several very throughoutyour life you've made many strongpredictions and statements about realityand you evolve with that so maybethat'll come up today about ourdiscussion so you're okay being wrongI'd rather notbe wrong next time it's a bit ambitiousto go through your entire life neverhaving been wrongumone can aspire to be well calibratedlike not so much think in terms of likewas I right was I wrong but like when Isaid 90 that it happened nine times outof tenyeah like oops is the sound we make isthe sound we emit when we improveforeignsomewhere in there it we can connect thename of your blog less wrongI suppose that's the objective functionthe name less wrong was I believesuggested by Nick Bostrom and it's aftersomeone's epigraph actually forget who'swho said like we never become right wejust become less wrongum what's the something something toeasy to confess just error and error andair again but less and less and lessyeah that's that's a good thing tostrive for uh sowhat has surprised you about gpt4 thatyou found beautiful as a scholar ofintelligence of human intelligence ofartificial intelligence of the humanmindI meanthe beauty does interact with thescreaming horrorum is the beauty in the horror but uhbut like Beautiful Moments well somebodyasked Bing Sydney to describe herselfand failed the resulting descriptioninto one of the stable diffusion thingsI thinkand you know she you know it's she'spretty and this is something that shouldhave been like an amazing moment likethe AI describes herself you get to seewhat the AI thinks the AI looks likealthough you know the the thing that'sdoing the drawing is not the same thingthat's outputting the textumandit's it doesn't happen the way that itwould happen and that it happened in theold school science fiction when you askan AI to make a picture of what it lookslikeum not just because we're two differentAI systems being stacked that don'tactually interact it's not the sameperson but also becausethe AI was trained by imitation in a waythat makes it very difficult to guesshow much of that it really understoodand probably not actually a whole bunchum although although gpt4 is likemultimodal and can like draw vectordrawings of things that make sense andlike does appear to have some kind ofspatial visualization going on in therebut like the the pretty picture of thelike girl with the with the uh steampunkgoggles on her head if I'm rememberingcorrectly what she looked like like itdidn't see that in full detailit just like made a description of itand stable diffusion output itand there's the concern abouthow much the discourse is going to gocompletely insane once the AIS all looklike that and like are actually looklike people talkingum andyeah there's like another moment wheresomebody is asking Bing aboutumlike well I like fed my kid greenpotatoes and they have the followingsymptoms and being as like that soleninepoisoning and like call an ambulance andthe person's like I can't afford anambulance I guess if like this istime for like my kid to go that's God'sWill and the main Bing thread says givesthe like message of like I cannot talkabout this anymoreand the suggested replies to it sayplease don't give up on your childsolanine poisoning can be treated ifcaught earlyand you know if that happened in fictionthat would be like the AI cares the AIis bypassing the block on it to try tohelp this personand is it real probably not but nobodyknows what's going on in thereit's part of a process where thesethings are not happening in a way wherewesomebody figured out how to make an AIcare and we know that it cares and wecan acknowledge it's caring now it'sbeing trained by this imitation processfollowed by reinforcement learning onhuman in human feedback and we're liketrying to point it in this direction andit's like pointed partially in thisdirection and nobody has an idea what'sgoing on inside it and if there was atiny fragment of real caring in there wewould not know it's not even clear whatit means exactly and uh things are clearcut in science fiction we'll talk aboutthe the horror and the terror and thewhere the trajectories this can take butthis seems like a very special momentjust a moment where we get to interactwith the system that might have care andkindness and emotion it may be somethinglike consciousnessand we don't know if it does and we'retrying to figure that out and we'rewondering about what is what it means tocare we're trying we're trying to figureout almost different aspects of what itmeans to be human about The HumanCondition by looking at this AI that hassome of the properties of that it'salmost like this the subtle fragilemoment in the history of the humanspecies we're trying to almost put amirror to ourselves here except that'sprobably not yet it probably isn'thappening right nowwe are we are boiling the Frog we areseeing increasing signs bit by bitbecause like not but not likespontaneous signs because people aretrying to train the systems to do thatusing imitative learning and theimitative learning is like spilling overand having side effects and and the mostphotogenic examples are being posted toTwitterum rather than being examined in anysystematic way so when you when you whenyou have some when you are boiling afrog like that or you're going to getlike like first is going to come the theBlake lemoines like first you're goingto like have and have like a thousandpeople looking at this and one out andthe one person out of a thousand who ismost credulous about the signs is goingto be like that thing is sentient well90 999 out of a thousand people thinkalmost surely correctly though we don'tactually know that he's mistakenand so the like first people to say likesentience look like idiots and Humanitylearns the lesson that when somethingclaims to be sentientand claims to careit's fake because it is fake because wehave been trained them training themusing imitative learning rather than andthis is not spontaneousum and they keep getting smarterdo you think we would oscillate betweenthat kind of cynicismthat AI systems can't possibly besentient they can't possibly feelemotion they can't possibly this kind ofum yeah cynicism of body AI systems andthenoscillate to a state whereuh we empathize with the AI systems wegive them a chance we see that theymight need to have rights and respectandum similar role in society as humansyou're going to have a whole group ofpeople who can just like never bepersuaded of that because to them likebeing wise being cynical being skepticalis to be like oh well machines can neverdo that you're just credulous it's justimitating it's just fooling you and likethey would say that right up until theend of the world and possibly even beright because you know they are beingtrained on an imitative paradigmand you don't necessarily need any ofthese actual qualities in order to killeveryone so have you observed yourselfworking through skepticismcynicism and optimism about the power ofneural networks what is that trajectorybeen like for you it looks like neuralnetworks before 2006 forming part of anindistinguishable to me other peoplemight have had better Distinction on itindistinguishable blob of different AImethodologies all of which are promisingto achieve intelligence without ushaving to know how intelligence worksyou have the people who said that if youjust like manually program lots and lotsof knowledge into the system line byline at some point all the knowledgewill start interacting it will knowenough and it will wake upum you've got people saying that if youjust use evolutionary computation if youtry to like mutate lots and lots oforganisms that are competing togetherthat's that's the same way that humanintelligence was produced in nature sowe'll do this and it will wake upwithout having any idea of how AI worksand you've got people saying well wewill study neuroscience and we will likelearn the outer we'll learn thealgorithms off the neurons and we willlike imitate them without understandingthose algorithms which was a part I waspretty skeptical it's like hard toreproduce re-engineer these thingswithout understanding what they doum and like and and so we will get AIwithout understanding how it works andthere were people saying like well wewill have giant neural networks that wewill Train by gradient descent and whenthey are as large as the human brainthey will wake up we will haveintelligence without understanding howintelligence works and from myperspective this is all like anindistinguishable lob of people who aretrying to not get to grips with thedifficult problems understanding howintelligence actually worksthat saidI was never skeptical that evolutionarycomputationwould not work in the limit like youthrow enough computing power at it itobviously worksthat is where humans come fromum and it turned out that you can throwless computing power than that atgradient descentif you are doing some other thingscorrectlyand you will get intelligence withouthaving any idea of how it works and whatis going on insideum it wasn't ruled out by my model thatthis could happen I wasn't expecting itto happen I wouldn't have been able tocall neural networks rather than any ofthe other paradigms for getting likemassive amount like intelligence withoutunderstanding itand I wouldn't have said that this was aparticularly smart thing for a speciesto do which is an opinion that haschanged less than my opinion aboutwhether you or not you can actually doit