Neilson Koerner-Safrata (introduced as Nelson Bershad, working at World Labs) argues that photorealistic image and video generation functions as 'a protocol for knowing'—that creating a photoreal image or 3D world from a single photo demonstrates a form of knowledge about reality, inverting the historical arrow from reality-to-realism into realism acting back on reality. He traces this through his own tools (image-to-3D-world generation with physics and sound, camera-roll 'telepathy' search) and historical precedents like Muybridge's chronophotography, Duchamp's response to it, Ed Catmull's hand animation, and AlexNet, concluding that the same technology deepening our computational grasp of reality is simultaneously destroying the shared trust in photorealism as a stable representation of the real.
Shared by the speakers for the symposium. Email addresses have been removed.
Transcript
YouTube’s automatic captions, unedited, so names and technical terms are often misheard. Each timestamp opens the video at that moment.
0:05 Okay, thank you so much for having me IKEA and um happy to be the final speaker in memory track. Thanks everyone for sticking around. Uh so yeah, my name is Nelson Bershad. I have worked uh a long time in in 3D rendering and uh video games and and 3D art and and kind of photorealistic rendering. So, today uh I want to talk about uh and what what what photorealism has meant from a technological perspective uh but also how it’s quickly becoming uh what I’m sort of referring to now is kind of a a protocol for knowing and and in a sense that uh to generate an image is to uh somewhat know something about the world and that uh we’re seeing that uh a lot of the ways that that people are thinking about uh uh uh like robotics and things like that uh in some fields is that um to to generate a photoreal image is also thinking about the world. Um So, some of this uh talk will will will refer directly to work I’ve done uh both uh both in my career but also uh side projects as well.
1:13 Um so World Labs uh uh is is where I’m at right now. So, I just want to quickly get that out of the way and just show kind of my my my my my day job just because uh it actually does materially affect kind of how I think and talk about these things. Um so uh and it’s a good example of kind of what I’m talking about. Um recently I’ve been working on a project that uh lets you place images in 3D space. So, you’ll see that these images on the left side uh are actually mapped to kind of virtual cameras and um you’re looking at a model that can interpolate between images placed in 3D space and actually uh create a consistent spatial environment just by trying to understand uh what these relationships could have been uh by literally taking these images seriously as actual things you would have taken in real life and then it’s able to interpolate between them. Um And so this is this is kind of what some people are calling a world model. And actually here’s for example the the ground truth version of this where where I’m talking from right now, my studio just by taking you know, nine images of it. Um this model is virtually placing cameras inside of that space and creating a video completely in its own model space of this studio based off nine reference images. And you can kind of think it think of it like how an LLM does next word completion.
2:38 This is kind of doing like almost like a next video completion but using actual 3D coordinates in space. So this is this is like a kind of like a a a LLM architecture but for for for 3D space and kind of 3D knowledge. Um so I think what actually prompted this talk is not so much these technical ideas or or the work I do during my day but kind of I think the confusion we have right now about um the the the problem of like photoreal images being very cheap to make and easy to make and kind of more an issue for things like culture and art around like hoaxes or slop or fakes and and kind of how that’s like an accident that’s kind of happened on on the road to generating um on the on the road to generating like intelligence about what the world is. Like I think of this novel Roadside Picnic which talks about like a a fantasy situation where an alien alien visits Earth but they kind of like leave their trash on the side of the road as if like you used to do at a picnic before people actually cleaned up after themselves. And and the idea is like they leave all this alien technology on Earth and the humans are like don’t even know what it is.
3:52 They’re like flabbergasted, but it’s just kind of garbage that aliens actually just dumped dumped on the planet. But, I kind of think of like the the photoreal image generation kind of like that. Like, on the way to create kind of synthetic or artificial intelligence, we kind of then accidentally created the conditions under which we could create photoreal images as well. And that’s kind of how I feel about this this like initial trauma about the ability to create photoreal images using models is is like kind of this accident of like just on the path of trying to learn about the world through computation, we’ve created this this technology for doing that. Obviously, an accident is a bit of a misnomer. Like, it’s very much been a through painstaking evaluations, benchmarks, and billions of dollars in compute, we’ve done that. So, it’s not truly an accident, but um So, just to kind of clear some terms, I think like what is through researching this talk, I thought a lot about how how to frame these ideas and and and originally I started with kind of the the image as as like the main thing I was talking about, but I think I I kind of honed in on realism and and kind of reality and realism and and photorealism as as as as something like an an object that’s kind of created from reality and we can think of reality as sometimes it’s referred to as as the actual what is actually kind of happening moment to moment and then realism as as kind of when you take reality and you approximate it into other so photos, you know, the photoreal being kind of one of the most ideal versions of capturing reality, freezing it into a frame, stopping its time, and then it kind of acts now all of a sudden as its own object.
5:35 And so, what what it feels like to me is we’ve we’ve we’ve had, you know, over 100 years now of kind of this realism, technological realism we’ve accomplished through cameras and things like that. And and what it what it seems like is we’ve had this space on the left here, this realism bucket, and we’re all of a sudden we’re finally starting to reason on it computationally. And and it feels like this arrow that’s kind of moved from the real reality bucket into the realism bucket is kind of starting to move in the other direction where now realism is kind of now acting back on reality because of our ability to reason on on images and videos that have been digitized. Um And I think that’s the key here is that a lot of these models I’m talking about are the photorealism is is all kind of the all these things are trained on image and video. So at the end of the day they’ve really been they’ve been taught about reality through the lens of the photoreal image or the photoreal video. Um and I think Kia brought this up succinctly in one of our in one of our uh group group chats which was um that that kind of the awkwardness right now is that we’ve treated images and and photoreal images and as as kind of faithful representations of reality. So it’s it’s very awkward all of a sudden now that we kind of have objects and models and things like that which can now kind of chuck chuck these things back at us, but um that that kind of don’t have so much grounding in reality as as we kind of come to expect, which I’ll talk a bit more about in this uh presentation.
7:02 So I think just to maybe we say one final thing on the image, you know, I think they they they are these photoreal images are these kind of static representations of the world that um that that by by by in and of themselves are are doing more than just capturing reality. They’re they’re they’re clearly like generating new realities, and we can like sit and look at images and kind of the way that our minds work where we can hold thoughts and freeze time and kind of uh create slow thinking or slow moments or or have memory or imagination, images kind of have the same property of of being able to kind of freeze reality and and create something like a surface which we can we can look at and operate on or or just uh do whatever you want with. Um So, uh some of the ways I’ve been framing it just quickly is I think that you know, we have this image as this compressed world state. It’s like a snapshot of uh time and space in this frame. And um now we have the ability to create new real images or photo photo-realistic images that, you know, could produce or kind of infer or uh uh seem to indicate possible world states which they be like fake or real.
8:12 Um you know, before world model like modeling or like AI modeling, we used to do this with like CGI um or or other like computational techniques or we just got really good at painting or draftsmanship. Um and then I think finally the the kind of one main takeaway of this this talk what is kind of that how how both the ability to create like a photo-real image before machine learning and after is kind of like infers that that actually deeply under these technologies is not like this um escape from reality of like oh no, all of a sudden we have all these fake images, but more actually indicates like we’re actually getting closer to deeper kind of knowledge about some some some fundamental features of reality. But actually kind of what the awkward paradox is here is we’re creating we’re so good at creating fake images now because we’re actually creating some some kind of deep uh understanding of the baseline reality that we’re in uh rather than kind of like escaping it or or leaving it. Um So, that’s maybe the one kind of I think takeaway from this talk is that that that paradox of uh of deeply understanding reality maybe through through some of these generative models or or getting to some higher dimensional understanding of it, but at the same time kind of now destroying uh kind of this stable-ish surface we had under which we could trust maybe sometimes trust photo-real images to to depict some reality for us.
9:28 Um just looking at time, I think I’m going to skip a lot of this talk. So, don’t be too worried if I skip by slides. Um So, rather than just rehash some of the San Francisco conversations around, you know, like the fake image. I just think that to to quickly go over these, um we just remember when we’re dealing with images, we’re dealing with like truly what like a virtual a virtual object. And that um like people have pointed out in the past, we have we kind of think of like Plato’s cave as this fundamental example of the virtual, which is something um which isn’t which quite approximates reality but isn’t actually reality. And that she wrote extensively about this with regards to photographs and how it makes us feel like reality is kind of more available than it really is.
10:10 And I really loved uh some of uh Don DeLillo’s writing, especially about he he makes a point about how uh kind of like he kind of gets ahead of social media in one of his books where some the the main cast of the book goes to a barn and they they they go to this barn, it’s the most photographed barn in America, and they realize it no one’s taking pictures of the barn. Uh they’re kind of like taking pictures of taking pictures. Uh that that the fact that this barn is the most photographed barn is kind of this now like duplicating quality. Um And then I think the final point on on on you know, what this this reality that’s getting disrupted by the image model that can create photoreal images is like this this like something that always like raises a red flag in me or not a red flag but like makes my hair stand up is when someone around me says this feels like a movie or something like that. Like that within the um generation of images from the last like 50 to a 100 years through cinema, we’ve kind of developed this this kind of uh emotional or sense sensitive quality to uh narrative and film and kind of um uh the space of images that that kind of guides our everyday reality. And I I highly suggest if anyone is interested in some of those ideas that you read. I think Norman Klein uh is one of my favorite thinkers and has really developed my um uh my ideas around this and talks about, for example, scripted space. Um that that that everything kind of has a cinematic quality about it. Um, and and that this is maybe now probably one of the most confused scripted space, I’d say, uh, as as more of these technologies decentralize.
11:47 Uh, so just maybe I think the rest of this talk I’ll just talk about a couple more pieces of work to ground some of these ideas and and things I’ve done. So, one of the one of the first, uh, kind of versions of of this talk came out of this project that I done earlier this year, which was a a image to what I was calling like an image to world, um, toolkit, which let you take any photoreal image and generate an entirely 3D scene from that single image. So, what you’re seeing are outputs from this, uh, AI tool, which orchestrates many different AI models to diagnose kind of like understand an image. So, the image is in the bottom right corner, and the generated 3D world is you’re seeing in the background, and it uses many many different models in concert to understand and generate, uh, this 3D environment. And it also does things like create sound effects for the objects, so that when their physics and things like that are hitting other objects, it creates, you know, some some kind of, uh, canned audio for each one. And so, this is kind of what led me to to this this feeling of like, well, yes, we’re clearly moving towards some like understanding of images if we can kind of do complex operations like this, um, computationally. Uh, I guess the we here is the computer, not necessarily the human. We already have an intuitive understanding when we look at images of how heavy an object is or what it might sound like or how warm the room might be, for example, or even what it might smell like.
13:11 Um, so, maybe just for the technical minded of of us who kind of interested in how this works, um, you generate a caption of the image. That’s kind of the toolkit’s like orchestrating many things and it starts by generating a caption. So, it says, here’s a room and here are, for example, the this is actually the literal instruction that I wrote in the file for for the AI agent is separate any object that is that a human could carry or lift into a separate 3D object and give it physics. And the way it does that is it uses a image generation model to actually remove those objects from the scene. It sends the static environment without those objects in it to one 3D generator. And then it it it actually um uh it actually then creates then new photoreal images of every object we wanted physics for. And and kind of now creates these these these kind of pristine uh versions where they can be masked out. And then it sends each one of these objects to its own 3D generator to generate a a unique uh kind of one of one of one of that object.
14:19 It then puts these all back in the scene, and then you kind of get the physical simulation just using like a classic game engine with physics and things like that. And um and that’s that’s kind of the the gist of this toolkit. Uh but I think the point is is like you can do this uh this is just from one image for example, so you can you can do this with any image you want. Uh it you know, as long as it’s semi uh it doesn’t have to be be photoreal, but it kind of works best on photoreal images. And I think someone I was showing this to had a bit of vertigo from this feeling of like um of this feeling of the this this way you could kind of extract things from images and create uh virtual or synthetic environments from from images. Um These are by no means perfect environments. Just I’m not saying that these worlds are um uh uh they’re still very flawed, I’d say. They’re still based on a lot of like 3D rendering and graphics assumptions just to make that clear. Um And what was interesting is I think uh we we we go now from this this kind of paradigm where rendering and creating images computationally kind of moved from like a build the universe from scratch or model into kind of inverse rendering, which is now kind of a paradigm I think we’re moving more towards, which is from any image or any photo you can actually kind of extract things from it. So, rather than a film maybe being built or or maybe like a computer graphics image or ad or whatever being built from like modeling individually each object, lighting it, and then putting it through a rasterizer or like lighting engine or things like that, you just kind of create like the final frame of what you wanted and it can kind of like decompose that into its parts by reasoning on it.
16:01 Uh so, for the last like 5 minutes and then I’ll leave some time for questions. I just I I was kind of curious about like well, where did photorealism really come from? Um you know, photorealism actually is like a term that kind of emerged in the ’70s around painting. So, it’s actually some somewhat of a of a newish term, but I I found it interesting that actually some of the first uh photos uh kind of like the panoramas and motion motion graphics before film was invented actually came out of San Francisco by a kind of photographer/inventor Eadweard Muybridge. And this is one of his famous panoramas of San Francisco. And um I think what’s interesting is is is in this moment of um of worrying about generative images kind of displacing the arts, I found it really just kind of been relaxing and informative that how how these early inventors of photography actually how the arts responded to them rather than kind of how the arts maybe are responding right now, which was one of the main um uh one of the main inventions before we had motion pictures was called chronophotography, where they just re-exposed the same person on a plate many times like this.
17:06 And um uh one of the main responses to that came from famously Marcel Duchamp, who painted a painting in the style of chronophotography, uh abstracted as well, which made it more controversial as well, but um that this collapsing of time and space into into photos and this capturing of reality was definitely a shock in the in the end of the 19th century as much as we’re having now, but and and the the arts kind of responded in this experimental and very bracing way. And then even even after that, you know, directly affecting my life was, you know, the invention of 3D graphics and I I found these images very compelling about how they they first went about kind of trying to understand reality. You know, we couldn’t do it in this way that we can now. We can’t just generate image from scratch and it almost was this approach of, well, how do we make photo real image with computers? Well, let’s let’s literally go out into the world. Let’s map our car with individual points and then try and recreate it in 3D space or let’s take this hand. Um this is Ed Catmull who who was the co-founder of Pixar and then this was I believe his doctorate project where he he created animation of a human hand and the way they did that was create 3D sculpture sculpted versions of it then map it to points that they could then animate.
18:31 Um and then I won’t go too much into a retelling of of the machine learning current paradigm we’re in, but I mean I feel like it is interesting that that actually what started the entire kind of modern AI race was actually it was called AlexNet and it was a model that successfully was able to recognize objects in images. So, it was kind of the first model to map language and images inside of computation. And and that was back in 2012. So, we kind of see that this this whole moment kind of was actually uh somewhat predicated on connecting photo real images with language and kind of this this understanding of the world and that this is kind of why we arrived at the ability to generate images was this deep question of how do we how do we understand shapes and objects and images and map them to language. This is Art Breeder, another fun project, and you can before before we got good at making fake images, we had kind of these fun not so good ones.
19:34 Well, and then maybe two more things. One was a project I worked on that that kind of used this technology from from AlexNet, but but kind of like modernized versions where you have now like this high high dimensional mapping of of language and images, and so that means you can just like type words into your phone and the images come up with it. But then I also realized that you could actually send an image that you have to your friend’s camera roll, and it actually uses because all these images are lifted into like a higher dimensional space, that you can actually kind of search their camera roll with one of your own images and like kind of prompt each other’s camera rolls in high dimensional space, both with words and images. And I thought like this is kind of reminded me this book where it was a it was a game played with concepts. So it was like a book written about called the Glass Bead Game where where the moves of the game were kind of chess-like, but you played concepts instead of actual kind of moves on a board. So you you could I feel I feel like we’re in this moment with with kind of this high dimensional understanding of of language and images.
20:41 And I think this continues to blow my mind to this day.
20:46 So yeah, thanks for listening. I’m going to wrap it up. I think just with trying to really get get get around back to this point of like generation as a protocol for knowing. So like what are these abilities to generate and understand photoreal images? Like what does this matter and what you know, for example, these are photoreal paintings from the 70s and and photorealism was a response to instead of using our eyes as kind of the main thing that eyes and specifically a live subject for painting, we’re going to use photographs as the subjects of our paintings. So, we as painters are going to go into the world with our cameras. We’re going to take a moment just capture a moment with our cameras and then we’re going to paint exactly what was in that photograph. And we’re we’re going to directly use this technology when creating paintings. And usually the subjects were quite austere. They were usually like American diners or you can see here cars or buses or or or snapshots of kind of the American American countryside. Um And I think we’re kind of in this kind of like in the in a in a in a moment like that where uh I started seeing these these these synchronicities between, you know, for example, on the right is this if when you first train a machine learning model, you train it on handwriting. So, you look at all these handwritten digits and you try and create a neural network that can guess the number of any handwritten character from zero to nine.
22:11 And I and I it feels like we’re in this moment with photorealism and world models where Eadweard Muybridge feels like maybe the most relevant artist of our times where when you look at his his famous books that he published on the left, The Human Figure in Motion, you see this person just walking over a set of stairs and it would it was like hundreds of pages of humans doing actions like this, walking, sitting, jumping, throwing things, um dancing, running. And and that these things are basically the data sets for what what we now train our models on. Like this these were the world models of of the 19th century in a way. And and they start to kind of come back now. Uh And that that when painters were doing these, you know, when when we when we paint or we draw or we do these things and we generate um usually the first thing you learn when you’re when you’re young and you’re learning how to draw a paint is that you know do not do not try and look. You have to see. Or or if you’re a musician, don’t hear, you have to listen. And there’s this is motion between going from just pure perception to to noticing.
23:17 And that maybe what you’re starting to see is is that underneath all of our perception there’s kind of then a leap you have to make. After which you then you actually start to really catalog or or or kind of start to know something about the world. So I think it’s not clear you know what kind of knowledge we’re we’re we now have from from the ability to create photo real images. Like how much is this knowledge? And I think Kia pushed me on this which is you know how much how much is actually known when when the when a model creates a photo real image. I guess the the question is is is um how the technology is kind of unfolding is is the idea is if you can create a a a video of something completing a task and that that constitutes knowledge of it doing it and that you could then follow along with that video and then that creates kind of like a knowledge about the world. But I guess that you know how deep that knowledge is kind of unclear and it feels like when we have a lot of models solving very difficult math problems and things like that no one really knows you know is it is is there understanding or knowledge underneath that? Um Something is happening, something’s being generated, something’s being known about the world and that’s kind of the promise of of a lot of the industry I work in is that world models will know something about the world and be able to act back on it.
24:33 But we don’t quite know what that knowledge actually means at the end of the day. It’s a kind of prediction. We know that that much we know. You’re you’re making a kind of prediction about the world.
24:44 Um so I think I think that’s that’s most of the talk and and I’d be happy if anyone has questions at this time. But I think two two main things to leave you with which is a theme that came up in the memory reading group which is kind of the between remembering and imagination as being somehow connected to this. And um and that’s um this this play between real realism and reality kind of feels like this play between imagine and remembering in some way. And that by generating we are usually then saying we know something about the world that by being able to either draw or create or render something, we’re kind of demonstrating a kind of knowledge. And I think the paradox is that we’re generating extreme amounts of knowledge about the world by generating photoreal images, but we’re also helping to possibly um create very confusing moments where we kind of confuse uh being able to look and understand and think about the world and look at images with kind of the ability to to understand the world. And that that these wires will continually probably get crossed over the course of the next 5 to 10 years. Thank you.
25:54 Uh thank you so much for that wonderful talk. I wanted to ask a little bit uh I’ll say it briefly, but um you know, this this world from an image and the sense of also um like creating virtual worlds in order to produce an image or to you know, there’s the concept of render spaces, but also setting them up in order to produce an image or to create something in the world and how it’s kind of bidirectional now is also something that your talk really brought up for me and um unfortunately, I don’t know if there’s time to ask a question or for you to respond, but I wanted to say that that was a really interesting thread and I’m thank you.
26:32 Yeah, thank you very much. It’s not enough time, but please reach out to me anyone if you’re interested about talking about this more Yeah, and we’ll see you at the next memory reading group.
26:39 Yeah, and thank you all for participating. I’ll be in touch with all the participants and like
Daniel Schmidt argues that mazes are a minimal formal instantiation of protocols, not just a metaphor, using his own maze-drawing history and games to illustrate concepts like hard vs. soft protocols (physical walls vs. crossable paper lines), scale-entangled vs. scale-lined navigation, and stigmergic memory. He demonstrates an ant colony pheromone simulation, a multiplayer infinite maze with shareable 'pebble' markers, a maze-design platform separating protocol-author from protocol-navigator with heatmaps of friction points, and an art-gallery maze as medium. Venkatesh Rao joins to explain his 'clockless clock' maze, a topological map of thought-space (rat/goat/crow vertices) designed by Schmidt to represent arrested development and spiraling toward nirvana.
Simone Cicero of Boundaryless presents O2A (Open Organisation Alliance), a proposed open standard for making organizations 'composable' by replacing implicit coordination mechanisms like hierarchy and meetings with an explicit, machine-readable contract graph of units, offerings, and relationships. He argues the protocol's compact grammar—three economic primitives (purchase, revenue split, investment) plus composition mechanisms—can represent bespoke institutional arrangements like Haier's Rendanheyi model or joint ventures as 'sentences' in a common organizational language, becoming essential as AI agents enter firms and transaction costs collapse.
Sachin Benny argues that AI modernity is reorganizing our relationship to the past the way industrial capitalism reorganized our relationship to the future, coining 'Archival Time' as a condition where traces of prior events become retrievable without recreating the event. He contrasts this with Bakhtin's 'Carnival Time,' bounded periods where accumulated traces lose authority and things get unmade and remade, drawing on William Sewell's eventfulness framework (path dependency, temporal heterogeneity, contingency) to argue eventful history requires a mix of both archival inheritance and carnivalesque rupture.