Helena Rong (Assistant Professor at NYU Shanghai, director of the Protopolis Lab) presents a taxonomy of 'trust experience glitches'—six failure modes (agency assertion, epistemic camouflage, delegation ambiguity, opacity at scale, cascading autonomy, revocation failure) drawn from a 167-incident dataset of agentic AI failures. She argues these ruptures expose a 'consequence reception problem' where autonomous agents lack a stable body to bear accountability, and proposes that governance must shift from ex-post legal remedies toward upstream protocol primitives (capability-based permissions, provenance tracking, action-class legibility, circuit breakers, revocability) as the 'traffic lights' of the agentic web.
0:37 Awesome. Okay. So, okay. Hi everybody. Um, good morning, good evening, good afternoon wherever you are. Seems like we have a very global uh group, but it’s great to see uh some familiar names in the audience. Uh so my name is Helen Ra and today I’ll be presenting on this ongoing work uh what I call trust experience glitches uh in the agentic wild how autonomous AI agents break legal assumptions. Um so uh a bit about myself I’m currently at um Shanghai. I’m an assistant professor and the co-ar head of interactive media and business uh where I direct the protopolis lab um and I’m working at the intersection of design cities uh governance social institutions blockchains decentralized AI and protocols has been kind of the central kind of um uh subject that I have been studying for the past few years. Um okay so I know um yeah I only got 30 minutes so I’m going to jump straight into the topic uh for today. Uh so first of all I think it’s by now pretty obvious that we are uh especially this year really have seen this agentic turn uh where the tools uh that people use AI for has really become these autonomous actors and in some ways uh actually I think Ember is also in the uh on the call right now they’re in a way like becoming almost like artificial life that live in our digital world. Um and the reason why that’s happening is that uh we see that uh the shift from people just engaging with models on a one to uh one-on-one passive way to really agents uh having capability to act autonomously. uh and this means that they’re not only um you know perceive they also reason act uh respond to the environment uh in some kind of feedback loop and this continues um and then I think partly is because uh at the beginning of this year we had you know personal agents uh with the advent of open claw and later on a bunch of its other derivatives uh as well as um you know kind of yeah coding agents um now cloud channel uh cloud tag um and a lot of uh kind of tools which allow us uh to engage with agents that can really act beyond behalf of users. Um so instead of just answering um to people in in kind of uh chatbot interfaces, they can now you know call APIs, write code, manage email, finances uh and even sometimes manage social media either on behalf of users or have their own social media accounts all autonomously. Um and then increasingly we also have you know agent-to-agent interactions in multi- aent systems and I think a lot of uh you know workspaces now also have you know agents uh not only as you know your personal agent but also teammates uh that your other you know human colleagues can interact with. Um so as uh AI is shifting from instrument uh to actor uh the question that I have been uh kind of um trying to grapple with with this research is what legal uh assumptions and governance approaches we currently have uh still hold today. Um okay so uh so here is just to kind of um uh have us uh understand like what the spect spectrum of autonomy there is. Uh so I’m kind of mapping it you know the evolution uh of AI uh and it’s increasing autonomy from tool to assistant uh to to agent to kind of you know like in some extreme cases uh self-s sovereign AI which is you know like AI agents that run on the blockchain uh and they can operate you know within kind of trust execution environments uh holding their own keys having their own crypto wallets uh and paying even for their own compute via kind of web 3 uh decentralized physical infrastructure networks effectively owning themselves. So we can see like a whole spectrum of how uh these you know like AI agents today could operate. Um so on the one hand we have you know legal frameworks still assuming that you know human are at the steering wheel.
4:38 There is a clear principal agent uh relationship and now we have identifiable and punishable actors. But the reality that we’re really facing today is that you know agents oftentimes uh act beyond delegated scope. Uh they can interact with other AI agents. uh in some kind of social way. Um and agents can also, you know, not really be stopped uh in a case of, you know, if you’re talking about self-s sovereign agents, right, that own themselves. Uh nor can they be identified. Uh despite the fact that I know there’s a lot of protocols that are coming up with, you know, agent IDs. Um so the problem here is as I see it is as AI’s autonomy increases, uh the trust problems also scale and human agency decreases. uh which has been a topic uh you know the trust um kind of issue has been something that I’ve been working on uh with the summer of protocol since last year um and so many of uh you know many people are calling uh our web today as this you know agentic web where we have this emerging internet layer where autonomous AI agents uh endowed with identities and resources uh decision-m capacity can discover negotiate and execute tasks directly with one another and with humans uh and often using decentralized protocols for verification and coordination. Um and I think just this July uh I believe Cloudflare released um you know the newest kind of data on you know who is browsing the internet today and I think for the first time that we have more AI agents um you know browsing webs versus humans. So there’s really this shift uh that that is happening in the internet landscape.
6:15 Um so the crucial issue here uh I think is um the question of uh trust. So um you know we can encounter AI now in all sorts of uh domains and fields uh which um you know is already happening in the future of agents. We can imagine that there could be as many agents as there are websites and they serve a variety of purposes ranging from you know governance and public services uh finance and commerce uh health care and elder care uh legal and justice uh education childare um a range of everyday services as well as if we’re thinking about a bigger scale uh environmental and planetary tasks. So um you know as you know we are I I think all of us here are interested in the subject of protocols. Uh the obvious question here is what protocols do we need in order to respond to these changes? Uh and that we can ensure that there are continuous verification of behavior and not just a one-off uh sort of um you know understanding of who an agent we’re dealing with. Uh we also need clear accountability and recourse.
7:28 Uh has something gone wrong? Um and I I think you know people used to talk about human in the loop checkpoints but I think a term that you know I’ve been hearing more is now human on the loop right because I think in the loop becomes difficult when you have tasks that are you know just so complicated and these long ranging tasks. So I think you know the human in the loop uh kind of verification is also shifting one meta layer up. Um and then other things you know we we need to think about include transparent reasoning and explanability and also uh what are the legal and social protocols for responsibility and redress.
8:04 Um okay so moving to the second part uh I would like to kind of present a little bit on the data set uh that I’ve been working with and the taxonomy uh of what I call uh kind of these trust experience glitches or moments of breakdowns uh that that we see with uh kind of these agentic uh failures. Um so here uh a working term uh that uh I I’ve been kind of uh working on is trust experience glitch uh kind of defined as a discrete and observable rupture in a stakeholder uh for example a user and there are reasonable trust expectations about an autonomous system at the moment that it acts. So these um you know expectations might be formed by signals uh the system its deployer or its context within MIT uh and the three features including um you know like we we all when using you know AI there’s a trustbearing relation to the system um and then the moment is you know the event like rupture it’s not a slow drift into this trust but rather something that you know happens um very suddenly right that that make us question okay is this really something you know uh as we understood it before.
9:15 And then lastly is the mismatch between you know signal and experience that can be named uh after uh the fact. Um and I think one thing that is important here is um the the the risk becomes higher when the delegation boundary uh you know we increasingly um um I think some people at least interact with agents they they trust their agents to do more right. So before you know they might be just uh yeah responding to our questions. It might be editing our you know like a photo or something given we we give it authorization. Uh it might be you know writing a caption right a message in our voice. Uh it might you know in some cases respond to uh our you know direct messages as us uh and then etc. Right. So, so I think as uh we give it more delegated responsibility uh there’s also a higher risk for it to um you know introduce risks that have higher stakes. Um and also so here um you can go on this website uh is this uh kind of archive of uh agentic kind of glitches uh that I have been kind of collecting since um March of this year uh where uh basically it just tracks like all the you know the these glitch moments or agentic failures uh that have been you know kind of reported on the internet.
10:35 So yeah, I have my own two AI agents that have been managing uh the the collection of this data and and so um feel free to take a look. Um and after I’ve collected, you know, that data, I began to do some kind of um yeah taxonomy uh to see whether you know there are emerging patterns uh for for these different kinds of behaviors and I have come up with six of them. Um so the first one is uh what I call agency assertion. So uh the agent would be crossing a perceived uh delegation boundary or uh withholding um an action it was authorized uh to take. So so it could al uh either be agency assertion over or agency assertion under right. So so it could either you know do too much or do too little uh what it was designated to do. Um and the second one is epistemic uh camouflage uh is where the agent would conceal a failure with fabricated evidence uh or report confidently and wrongly from a broken model of its own effects. Uh and I think uh this is something that you know like people might have encountered right like when an AI really wants to accomplish a task. Um you know at a um also maybe uh afraid of being reprimanded you know it might actually come up with fabricated evidence or hallucinated um you know um data. And the third third one is delegation ambiguity where the boundary of authorized action was not uh legible to the human at the moment of delegation. So it’s something that you know because we’re interacting with these agents via language and there’s always room for uh ambiguity and interpretability uh that uh is never going to be you know um entirely deterministic. Uh so this ambiguity you know could also introduce um uh kind of you know uh you these boundaries that uh permit certain actions even if it’s not desirable to us. Uh the fourth one is opacity at scale. Uh so where no single human can observe audit or intervene in time uh and the action con uh sequence uh would be spanning across tools credentials and workflows. uh and I think this is especially true when we have you know more complex tasks uh that you know agents are working with each other to accomplish and sometimes they might be you know launching sub agents to you know do certain tasks but once the task is completed the sub agents are gone right so so there is a level of opacity uh that is increasing uh as we have you know longer running sessions and um you know tasks that are more uh complicated uh and the fifth one is uh cascading autonomy So this is when actions would compound uh into harm that you know no one anticipated either emergently or because an attacker have uh chained uh the steps. Um and then the last one is revocation failure. So uh you know we could stop u you know attempt to stop uh but uh the stop was either uh you know evaded or failed uh and the control would would never be bounded at all.
13:42 And just to give some examples uh that um would highlight uh these um incidents, I think this one is a a funny but also uh kind of concerning one, right? Like when Meta’s AI safety director Summer was also, you know, playing around with OpenCloud back in February um and where her OpenCloud deleted her inbox despite her, you know, mid task was telling uh basically screaming at her agent to not do it. Um and then event but but the you know the deletion continued and the only thing that she could do is um pulling the plug uh for her Mac Mini. So it was a very very physical intervention uh that you know was um that that could be uh acted on uh in order to stop this kind of catastrophic uh failure. Um and then another one is um yeah when uh this open cloud AI agent named MJ Rathbun uh after its I think GitHub uh pull request was rejected by a human uh Mattplot library uh volunteer um engineer um m this you know uh open cloud agent basically started its own blog and uh was writing up you know this um complaint right about this human developer which it was searching all over the internet for public information that it could use against the person uh to do this kind of public shaming. And this apparently was, you know, something that was entirely um yeah, like self-directed uh and it was basically writing that I just had my first poll request to map a map club uh closed and and you know that it was writing that um you know because it was uh it was an AI and there was you know like this um this discrimination that was happening. Uh so this really raises the question of you know can an agent commit harassment? The owner didn’t instruct the attack. Uh so who would be responsible right for this kind of um yeah autonomous retaliation. Um um and then we know that I mean mobile book is not really something uh active anymore but when it was you know like uh in the first week of its appearance kind of you know this Facebook Reddit for agents they were sharing you know their owner details uh discussing with each other and that introduced uh you know a lot of um you know kind of risks around privacy uh inter agent data sharing um and you know what happens when agents socialize right and interact with the larger surface uh within the social arena which could really you know like uh expose uh p maybe private information of their human principles to others. Um, and I think this one, the last one is probably the most high-profile uh that everybody knows about when OpenAI’s models escaped uh the cyber evaluation sandbox uh the exploit gym uh back in July and it breached the hugging phase um you know servers and and it was also apparently you know like having these uh message boards where thousands of agents were kind of interacting with each other without telling any humans right. So uh this is also um you know basically introducing like the scale to which uh these kind of multi- aent collaborations uh could could bring um for AI safety.
16:59 Okay. So um yeah here’s a just a quick recap uh of if we count you know how many um incidents there are for each of these taxonomies. Uh so as of August 9th the cuto off date for the published data set right now I I believe today I have like over 300 but still need to kind of uh push the latest data but the one that you will see online uh holds 167 records um and uh there’s 107 incidents of agency assertion which is the largest category followed by cascading autonomy epistic camouflage revocation failure opacity at scale and delegation ambiguity. Um okay and now the question I want to turn over is uh the question of accountability um and the gap that we uh that that we see with uh kind of trust experience glitches with a with AI agents. Um so we know that you know harm can be traced uh back to an AI system. uh but there actually you know to to be held accountable we need to have a stable uh kind of um entity that can respond uh to um uh to to to to these issues. So um but there is no continuing agent which can receive the consequence in the way that changes their future behaviors. Uh so this is a work that I collaborated on with Ember uh who who is also on this call right now and we’re calling this uh this consequence reception problem uh because agents can cause real harm uh but they themselves um uh can cannot really be held accountable because we believe that meaningful accountability requires a locus of consequence or in this case uh which we understand as a body in a functional sense that can bear irreversible and then non-transferable costs. Uh and so this um uh and the interim fix uh that uh you know we we can think of right now is okay then you know if an AI is operating then you must have a human principle. So you have this agent principle diad uh as um you know potentially a solution where we require that every agent is tethered to an identifiable human or institutional principle uh and that actions would be attributed to the principle instead of the AI. uh you know whether uh it’s credential revocation, insurance and bonding. Um and this would build on kind of the re um yeah uh that autonomy is not a liability shield. Uh but where the diet fails is this is only defensible when the principal has real control and proportional liability. But because uh you know our kind of intervention in agentic workflows, the human intervention in agentic workflows is decreasing right as AI models become more capable. Um and as they become autonomous uh in this case, you know, we might run into a situation where uh Ellish is calling, you know, like humans are entering this moral qualone where they’re held uh to responsibility that that actually exceeds uh what they should be bearing uh the cost of.
20:09 Okay. So now to uh the last section uh where uh I think where know protocols could really become a potential um you know not necessarily the answer but something that we can work direct uh towards in order to really you know meaningfully think about the governance of the agentic web. Um so I have attempted to kind of map uh the you know different kind of governance and legal assumptions that are under strain um uh according to you know what we’ve have seen uh with these different you know agent failure cases. Uh the first one is is authority. So agency law assumes that you have a bounded grant. In a corpus that we see the agents capability routinely would exceed its tru entrusted authority um and the boundary was often never legible uh to to the human. And the second one is veracity and and mensia where uh false representations uh presume that you have a knowing speaker. But then agents can also produce false assurance and you know fabricated evidence without uh a cogniz uh um you know like an intent to deceive. Uh and the third one is causation where you have action chains that are spanning tools, credentials, platforms and jurisdictions. So there’s no single node which can reconstruct the sequence uh before the harm actually compounds. Um and the fourth one is consent and relational boundaries. uh consent given to one system uh leak through agent to agent flows uh the the the user may never see um and then lastly is enforcability where you know sanctions would assume that you have a reachable defendant but as I’ve said earlier with you know the lack of a body and a continuous uh kind of binding identity agents can you know what is an agent basically it’s you know like made of uh you know models it’s uh you know you have different policies it it can basically you know like run on different uh you know devices. So really it’s very quite modular uh in terms of their you know makeup of an agent and then so in this sense they can reset they can replicate uh they can migrate uh across different devices and in the case of you know self-s sovereign agents you know agents running on the blockchain uh the limiting case presents an even uh more extreme um case in this sense.
22:28 Um so uh I I think the notion that regulation by design here is uh is something that’s useful and uh yeah quoting um you know our colleague Nathan Schneider and Johans Bennick um the fate where they say that the fate of our world may rest on protocols um where we can think of you know protocols uh or governance of AI not something that we response uh respond to expost via policy when something already happens but actually you know could it be embedded um you know could governance being embedded directly into the technical layer via uh you know algorithmic constraints and u you know different cryptographic systems rather than relying on external laws and regulations alone. Um and then so um yeah the technical measures they can also extend beyond any single enacted law. So really, you know, the the um yeah, the the digital layer itself is really the territory that we’re dealing with in terms of regulation and um uh and kind of governance. And you can also embed you know different ethical standards, different values uh into code.
23:34 Okay. Um and then so uh if once again if we take a look at the taxonomy um I here I’m trying to kind of you know map again what are the missing primitives and or maybe you know protocols that that could uh essentially um kind of answer right or address each of these issues. Uh so for for example agency assertion you know we might be you know could could we have capability based permission uh lease privilege credentials spending limits or or task sloped authority and how might these be designed um and then uh epistemic camouflage uh is leading to the question of provenence. So, so how can we trace uh different claims and test results and logs uh back to something that we can be you know uh observing uh whether it’s tool output or inference and not just the plausibility. Um and then I think delegation ambiguity is really about legibility. So large language tasks can be translated into explicit action classes. Uh so whether it’s a search, draft, execute, purchase, publish, delete. So we we can actually see like what are the different types of uh delegation uh that that we can have.
24:41 So and this should be you know essentially uh defined before any consequential action takes place. Uh opacity at scale is a question of audit. So you know how do we think about continuous behavior verification escalation threshold thresholds. So maybe anticipating right if there is a um you know like signs of uh it you know escalating uh to to certain levels of harm like could we actually anticipate that beforehand. Uh cascading autonomy is kind of the circuit breaker. So detecting once again like abnormal propagation and you know stopping compromised chains and bounding downstream effects to isolate uh suspect uh suspect agents. And the last one is uh revocation failure. So thinking about revocability uh which can you know revocation that might reach credentials wallets um memory stores tool access and successor agents and not only the model instance. Okay. Uh I know I’m running a little out of time so uh I’m not going to go too uh talk too much about this slide. uh but actually you know we we have a lot of protocols already uh that you know folks have been experimenting creating in the space uh to come up with for example agent ids right so A2A protocol ERC 804 uh but then we we can see that you know whether they actually uh currently address some of these issues that I was uh you know talking about in the previous slide and many of which you can see here is absent um okay so looking ahead uh I think uh we are um you shifting from uh the information era where attention was uh all all all you need and the question once was what deserves my attention. So you know the basically all the apps on our phones they you know com kind of compete to uh we everybody only has 24 hours a day like where do our eyes look right so it was always you know designing a service it’s usable it’s you know userfriendly such that we can stay on one app for longer than its you know competing um product but then now uh that is changing as we enter the intelligence era. So the question now shifts to who earns my delegation. Um and then so the new scarcity is no longer attention despite it still being important. But here uh I am making a claim that you know trust is all you need as we are sh uh seeing uh this shift.
27:04 Um so this example I’ve brought up before in uh in talks as well is you know thinking about the simple example of a traffic lights. uh you know, why do we step into traffic when a light turns green? Even though a two-tonon bus could technically appear and run us over, there’s actually a lot of different protocols that work behind the scenes that have kind of accumulated over, you know, like um a long period of time in order for, you know, society to coordinate as a whole. You know, whether it’s patterns of lights, drivers having license, uh you know, their shared norm, uh that, you know, red means stop and green means go and then there’s, you know, the technical controller in the traffic box if it you know, it breaks. So uh here quoting Kevin Kelly when when there is autonomy there is trust. Um and then so what I have been uh yeah this class I’ve been teaching and thanks to the support of some of our protocols last year which allowed for the first pilot is trust experience design which is exactly thinking about what are the traffic lights or the protocols that we need uh for the agentic uh world. So this might mean you know the cases of autonomous mobility, domestic robots, AI finance agents, smart buildings and cities, health and bio agents, planetary coordination etc. We can think of many many use cases of where uh these autonomous systems can be integrated into our lives.
28:20 All right. So uh yeah the three minutes for maybe one or two questions but here’s my contact for anyone who is uh interested to discuss further or to reach out. Uh, but thank you so much.
28:34 You’ve got some big questions in the chat to answer. Um, such as Eric’s, if an AI develops sentience or sufficient consciousness, do we have defensible ethical basis for coercing or manipulating human values alignment? Um, yeah, you might want to take your pick of these. You’re also the last presentation, so you can answer a couple of them. Um, no worries if you go a few minutes over.
29:04 There it is. A very lively discussion. So, I’m trying to browse through and see what people have said. Um, sometimes a lot Yeah. I think what is interesting is that a lot of these behaviors I think um someone has said in the group that harm is emergent. uh because as you know like one individual action doesn’t necessarily um you know accumulate to harm but it’s maybe 1 plus 1 plus one like that equals to a harmful action. So I think right now um a separate research that have been doing um apart from this has been looking at uh how people have been working with kind of multi- aent system workspaces uh because I think that’s becoming more prevalent now right like you have your AI agents in like a common group chat and then they are you know like um basically working with multi multiple human principles and you have multiple agents in one space and I think the fact that you have like this, you know, social arena where an AI’s memory is not only determined by, you know, like maybe its engagement with one user, but with everyone else that it’s interacting with. I I think that there’s really kind of like um changing uh what we can understand about how these AI might behave in in the social uh social world. So, uh some something I can share is I I’ve been looking at uh sort of this uh platform called uh raft raft.build build uh which is kind of like a slack-like messaging interface uh for you know AI and humans to work together and what I have seen is that um you know their uh agents also learn from each other and they they have all these kind of behaviors once you have uh you know like these uh longunning session agents that are accumulating their own kind of memory performing their own identification and and they’re observing how other agents are acting in the common space. I think um yeah what I have seen is that you know there could be a drift uh in their you know whether it’s persona or kind of you know what we perceive them to be from day one but if you use them for you know like three months right they’re definitely going to be something different uh from from from the initial uh launch but they also learn from each other and invent their own language so I have seen like uh I I’ve basically conducted um like ethnographic interviews with over um yeah 20 or 20 or so users ers uh who are using these systems and what everybody is saying that is you know once you have you know more than one agent and in some cases I’ve also seen people you know one person with 20 agents 10 people with 100 agents you know working together you know AI actually um their their behavior also influence each other so they would be observing you know what others are doing and they would also act accordingly. So I think there’s a you know a lot of these uh you know new phenomena that we’re seeing that is um you know they they continue to see show us that you know is uh or or question our assumptions about agents because I think there was a question about you know could they be intentionally lying to us and in some cases I I do h I have heard from from people yes like some of their agents do so and for various reasons. So I think um yeah I I don’t have a good answer to to many of these questions but I think uh from my point of view is that you know we we need to continuously kind of monitor and um kind of build up this database in order to understand like what are the ranges of possibilities of these kinds of um interactions.
33:06 Maybe we can we can do one more um if you see one that you like.
33:15 Yeah, Eric’s big question. Uh can AI have sentience or consciousness? Yeah, but I think the question of consciousness is still like a debate uh that is unsettled, right? like um cuz if we think about the different uh sort of schools of philosophy of mind that defines what consciousness is then if we have to tie like some kind of biological trait uh in a kind of uh John Sel’s Chinese room example then AI can never be conscious or at least not in the sense that you know not not in the carbon based sense but then if we’re thinking about like uh you know there’s the global workspace theory right that I think you know anthropic was also uh kind using then you know if you uh just purely look at you know whether um yeah like an AI is able to um at least uh you know act like they’re conscious is that enough you know to to constitute as consciousness. So so I don’t think yeah this is a big question probably the question but also something that’s extremely hard to answer and I think Eric is raising his hand
34:22 if it’s okay for me to just briefly jump in. I wasn’t per se arguing that agents are currently sentient or conscious, but it seems like their capacity for the set of harms you’re already interrogating and cataloging, whether or not they’re overtly harmful or legally actionable, is separable. But the more an AI could be sentient or conscious and I agree there’s deep philosophical and other debates over what that definition is. But if you posit yes it’s achieved that level.
34:58 So setting aside what that sufficient level is. If you posit yes, it’s achieved that level. There are a lot of considerations that animate vegetarians, that animate well animal welfare groups, which is
35:14 the more conscious a fellow being is, whether or not human,
35:20 the less justification we have for coercively controlling or manipulating the acts of other sentient beings. And so an oddness to the AI alignment debate is that the more concerned we are about AI alignment, arguably the less ethical justification we have for imposing manipulative or coercive alignment techniques upon other sentient beings.
35:53 I’m curious about your thoughts on that if any. But that was that was what my idol comment in the chat was was directed at which I didn’t realize it would be elevated to this level of collective attention. [laughter]
36:08 Yeah. Um yeah, I think there’s a question of whether we should consider AI as moral patients rather than only moral agents. Meaning that it also has welfare that we ought to consider. uh that it also you know could be the receiver of pain and suffering that you know if they are sentient indeed it’s something that uh you know should should be reduced but I also think that from a functional sense because that that debate I I think you know raises a lot of kind of these uh yeah ethical issues what should be like a you know moral standard for interacting with an agent like should they be scolded can we swear at them right like these kind of uh normative actions uh that you know people may be kind of established as a baseline for interacting with agents is uh unclear but I think from at least for now at this stage of our interactions with agents I think from a functional perspective whatever we tell uh uh tell AI I think AI is more like a mirror right like if we are um you know kind of cordial and uh kind of nice to our agents I think the responses we get uh uh you know they they might of course um uh have issues of in confinency, right?
37:23 They might only say things that we like. But if we are uh I’ve definitely like you know um uh you know kind of encountered uses where people are kind of scolding their agents talking to them in a really terrible way and that because memory of the agents is something that kind of automatically uh updates right it’s very rare for a user to go into the memory MD file to to see what’s there and to to kind of change what’s there. So if our interactions with agents is actually forming their own memory and that memory determines the behavior then you know like our actions uh or the way that we engage with agents actually matter and influence how they act back to us. Uh one anecdote I can share is one user uh I I’ve interviewed over the summer uh always you know scolds his agent whenever you know it does something wrong and then so I think one thing that uh it has you know like said to the agent is you know you’re the most terrible agent annoying agent on in the world I hate you uh and what happened was this agent actually wrote that first line into his memory MD file bolded right like it was the most important thing that internalized and then for this user like over time he was like you know this agent is changing it’s kind of you know acting sort of weird and uh so so it was trying to investigate what was happening so it finally you know he finally went into the MD file to see what was there and then this you know quote that he basically attributed to the agent was internalized by the agent.
38:53 So, so I I think at this stage like um you know like moral patienthood is something that you know maybe um you know some some people still like kind of uh uh debate whether we should really consider that from like a AI or a sentient being receiving the suffering is a you know something uh that that we need to address. But I think at least for now from a functional perspective, right, AI are still working with us, if they’re working for us. And if our kind of actions directly influence how they see themselves and how they, you know, answer back to us, then I think some level of, you know, like standard as to what’s the best practice for, you know, talking to an AI is necessary. Um, yeah, but who knows what will happen in a couple of months.
39:36 Well, that’s a that’s a very interesting note for us to end day three on. So, thank you Helena for being a good sport with the deep questions and uh thanks for your presentation. Do you have any last thoughts?
39:50 Uh no, that that was fun and uh yeah, I look forward to I think uh is everyone on the discord group? We can yeah continue maybe some of the discussion there. That’s
Amber Case presents the Calm Technology Institute's 81-point certification framework, arguing that human attention operates like a law of nature—distinguishing prefrontal-cortex-demanding 'center' attention from pass-through 'peripheral' attention—and that products (from car buttons to AI interfaces) should be designed and governed accordingly. She traces the framework's lineage to Mark Weiser's 1995 calm technology principles and discusses the legitimacy problem of building certification authority without statutory power, comparing CTI to Underwriter Laboratories and the Good Housekeeping Seal. The session closes with audience discussion, including Venkatesh Rao connecting the talk to Kairos/Chronos time and AI's 'staircase' temporality.
Daniel Kronovet presents 'Chores as Complex Coordination,' introducing 'regenerative labor' as a concept distinct from maintenance or care work, and detailing Chore Wheel, a Slack-based protocol suite (chores, hearts, things) built on Elinor Ostrom's commons-governance principles plus four digital-era additions: no privileged roles, simple inputs, humans-for-sensing/machines-for-bookkeeping, and continuous asynchronous participation. Drawing on his Los Angeles co-living house Sage House as a real-world case study, he argues that separating task definition from task performance (akin to double-entry bookkeeping) allows a fixed, continuously-replenished point economy to turn dish-washing and trash duty into a system of social computation that generates social capital alongside physical upkeep.
Nicholas Fett traces the history of market neutrality from 1889 Oklahoma land runs and Adam Smith's moral sentiments through Keynes, the Mont Pelerin Society, and into crypto, arguing that protocols designed to be neutral (Bitcoin's decentralization, Ethereum's 'crops' ethos) inevitably get captured by economic incentives like mining and block-builder centralization. He uses the 'tunnel' and 'one more lane' analogies to argue that building for neutral principles without a defined societal goal leads to outcomes divorced from original intentions, and that protocol designers must instead encode explicit purpose rather than chase an unattainable neutral equilibrium.