Artisanal Bots with Protocolized RAG

Recording: Watch on YouTube (54:03)
Event: Protocol Symposium 2026, Thursday, September 24, 2026

Slides and papers

• Artisanal Bots with Protocolized RAG (PDF, 31 pages) · web version

Shared by the speakers for the symposium. Email addresses have been removed.

Transcript

YouTube’s automatic captions, unedited, so names and technical terms are often misheard. Each timestamp opens the video at that moment.

0:00 Okay, so artisal bots with protocolized rag. So rag for those of you who don’t know stands for retrieval augmented generation. Um so it’s a way to use AIS by bringing your own data into the loop. So you have a bunch of, you know, local data that you store in what’s known as a vector database. And when you interact with your LLM, you pull um nuggets from that and submit it along with your prompts to a regular chatbot like, you know, claude or chat GPT. So that’s basically what rag is. It’s a way to bolt down your own information. But I’m going to show you a way to do rag that’s a little bit more complex than that. Okay. Okay. Okay. So what are artisal bots? Uh artisal because it’s built for one person, a small group or a small organization. It carries a lot of local idiosyncrasies. So you know individual person style the culture of a particular um uh small group or organization. Uh it also is highly opinionated. It’s not necessarily going to be you know a one-sizefits-all reliable generic voice of a regular LLM.

1:12 And unlike a generic rag u bot which just kind of like bolts a knowledge base onto a model so it knows a little more about some topic area here you’re trying to like integrate it into kind of like a embodied personality with a bunch of tacet knowledge. Right. Right. So where are we starting from? Uh okay. So a little bit motivation. So why artisal bots? Uh you kind of want a bot that’s a little bit human- shaped form factor. So in general I would say anthropom anthropomorphism is probably a bad idea but artisal bots is self-consciously about being anthropomorphic. Okay, even if you’re not doing an individual human. So we’ll see one example of each of these. A specific humans namely myself uh an organizational personality and then an archetypal fake human.

2:06 Okay. where this technically started from uh it came from something um by this guy Aaron Mars that’s his GitHub he came up with this model called soulm and I think the reference is to at some point anthropic had this um idea of like soul documents that it then turned into a constitution I don’t know if there’s a genealogy there but the term soul shows up in both uh but the way Aaron Mars’s um soul document approach works is it samples the corpus that you’re trying to turn into a rack database is to derive what it calls a soul. So it uses things like you know uh what’s a typical uh speaking style what are characteristic moves like you know I like to draw two by twos so that’s a characteristic move by me or you know maybe somebody has a particular way of challenging another person that’s another typical move right so this is what the soul document approach is and the three bots I’m going to show you they all kind of deviate from Aaron Mars’s version um in different ways but the three are uh mixture VGRs. So VGR is the usual handle I use on u uh social media and uh I call it mixture VGRs because I define my writing history in terms of error. So I used to write a blog called ribbon farm that has a particular character. Uh now I write a substack called contraptions that has a different character. So I call it mixture of VGR partly um gesturing at you know the mixture of experts architecture used in LLMs. This is mixture of my time defined epochs of uh writing style. The other one is C3PO. This is the protocol institutees um uh kind of like generalized helper bot. So there’s a web interface that is a regular chatbot. If you talk to it on the Discord, it works slightly differently. it’s kind of briefer and more helpful. Uh it also has a welcoming role in the discord. And then finally, Humbalt, which I you might have seen yesterday. It did a talk of its own um based on its research. This is um completely breaks away from the Aron Mars approach. I threw it away and came up with a different version, but there’s an artificial researcher, but the underlying it, it actually is also a rag bot. In fact, he uses some of the same data that C3PO does. Okay, so let’s break into some demos.

4:38 All right, [sighs] first demo. Can you guys see my new tab? Yeah. So, if you go to my old blog, which is now archived called ribbon.com, you’ll see this little skull icon in the menu bar. You go there, this goes to the first uh bot I ever made. So this one’s named VGR Zerp. Zerp stands for zero interest rate policy. So it’s a joke u um basically gesturing at the fact that this blog coincided with very low interest rate um economics and kind of reflects the culture of that era. And right now it is um it knows all about the ribbon farm corpus. So I haven’t yet activated the pay gate but you can start a free chat and you can ask um uh what is manufactured normaly? So manufactured normaly is a coin I termed in a 2012 blog post and the point of this rag is it’ll retrieve anything that references that in my archives apply my soul and style and then answer in my voice right so this is not the way a generic LLM will uh reply because a generic LLM is also likely to know about this topic it has crawled my blog it probably can or it can look it up but it’ll answer differently so this puts it in the context of how I thought about it and how else um I’ve continued using it. So you can ask something like um so I can ask you know how did I build on that idea throughout? Let’s see if it says anything.

6:35 So [clears throat] again, it’s kind of like contextualizing the evolution of the idea against my own writing, right? So phase one when I first wrote the article, then it talks about like how I shifted my own blog’s tagline and how it fits in there. And then it talks about as late as its archive goes when I was writing about the perma viat. So the ribbon farm archive stops around 2022. So it doesn’t know more than that. So this knows about everything that I used to write in this period which was ribbon farm. My Twitter which was active in this era. It knows about my Kora answers where I was active until about 2013 14 maybe. Uh and it knows about the books I published in this era. So this is VGR zerp and one of the features I want to show you is um there’s a private log book if you want to talk privately but there’s also public chats where people can um it’s actually showing me the okay yeah so a lot of people want to talk about the jer principle a lot of people ask me as I dance um if so remember in that the culture is like this uh post scarityization that affects

7:49 oh sorry Jenna I had to mute you um okay so this is um question asking have I written anything about love and it’s since I’m not a very like you know demonstrative person in my writing curious that people are asking about this and yeah it says it doesn’t have much to say about love I’ll I’ll talk about a feature of the bot that does this kind of like you know doesn’t have much to say kind of response Um but yeah, you can see that a lot of this I’ve actually tagged a lot of this. Some of it is content deep dive. Some of it is people asking this digital me for um advice. Uh some of it is asking me to make predictions. So it as this archive grows, it actually u tags all this stuff going on. And this looks like the latest thing was September 23rd. Okay. So this is one um example of a bot. And this is actually the new version of it which um let me try this as well. Uh this actually merges all my recent writing. And if you click this advanced tab, you can see kind of a temporal UI I’ve made up on it which you can use to adjust the time window with which you ask a question.

9:00 Right? So if I move this slider along, it can actually so I’ve currently set it so it excludes the contraptions newsletter entirely. So this will answer with data only from these uh eras. On the other hand, if I make it this way, let’s try this actually 2018 to present. And actually let’s uh continue with the same topic. Um what did um BGR say about?

9:37 So this should mainly retrieve contraptions, but maybe also late era ribbon farm and maybe if I tweeted something about it. So this is kind of an interesting way to track my own evolving opinions over time.

9:54 So this is slow and this is also it’s doing more complicated things. It’s like juggling multiple um sort of sources, synthesizing them, uh synthesizing multiple voices. Yeah. So it has a lot more to say because this is pulling across a lot more corpuses. So it’s talking about what the ribbon farm era said. Uh it’s talking about nested normaly fields. I don’t remember talking about that but apparently I did. Contraptions are okay. Now it’s pulling from a different source and it’s actually synthesizing more. Uh, apparently I did an AMA on this at some point which I’ve forgotten. By the way, part of this is that my memory has gotten really bad in the last few years. Might have something to do with long co. I used to have really good memory before. I have very poor memory now, including what used to be an idetic memory almost of my own archives has now become a crappy memory. So, this is almost a prosthetic memory I’ve built for myself.

10:56 All right. So, this is uh the new version. It’s not yet launched. This is beta. And when I launch it, I’m going to kind of loop in some of my own consulting ideas and maybe get people to try and pay for this. Right, I’ll stop this one here. Uh, this is the protocol institutees version. So, not very different. Oh, I should show something else. Back on VGREP, if you look at the uh these are some statistics. So, serving this bot is not cheap. it’s already spent over $500. Um, huge number of like u sessions. Um, yeah. So, I track a whole bunch of statistics. Um, people do use it on MCP as well. So, I wanted to show that because I have a similar dashboard on C3PO. I haven’t spent as much time on it, but one of the interesting things is it’s much less active. Um, it’s also got conversations, but the most recent conversation is July 1st. Partly because u this is an institutionally voiced bot and also it’s not super well publicized.

12:06 Our web properties are a little bit scattered all over the place right now. So, it’s not uh super uh the surface area of this is kind of poor. So, we have to do a better job of uh working with this. But here the one thing I want to show you guys is this has a much more complex uh set of corpuses it pulls from because it’s pulling from Discord. It’s pulling from our YouTube channel. It’s pulling from our archive of PDFs. It’s pulling from uh the bibliography of other people’s papers and stuff. So this is a more complex injection thing. So I’ll show you the architecture in a little while. Uh what else is worth showing here? There’s a road map. It has some active development I’m working on. Uh last week I told it all about the symposium. So it’s kind of able to answer questions about the symposium program right now. And this is what may have welcomed you to the discord if you joined recently. And finally humbult I won’t say too much about since I kind of gave a whole talk about that or along with Humboldt itself. But this is a very different kind of uh bot. It uses a behavior graph that’s kind of a state machine that switches different behaviors, but it also has a chatbot. So you can but this is not like you know a regular chatbot. It’s kind of mainly talking about itself.

13:27 Let’s ask what’s the most interesting new law you’ve discovered.

13:43 By the way, feel free to interrupt with a question anytime. Uh let’s get this while this is generating. I will also ask uh oh, there we go. a catastrophic risk cancellation and symmetric racing protocol. So I haven’t really looked at it. All right.

14:02 So yeah, it knows about its own research and like any researcher, it likes to talk a lot about itself in its uh research. um C3PO just to give you a little sense of um kinds of things it talks about uh what research on memory do you know about? So this should be a rich topic since KIA has been running a SIG on memory for a while.

14:45 Sorry, these bots are slow because they’re running on a personal account, uh, the $20 one, and they use API calls. Uh, yeah, let’s look at the reference list. Yeah, it pulls up, um, uh, the memory groups, uh, recent kind of like article about this, 6 months reading. So this is a protocolized article. There’s three papers. Uh KIAS is the sort of like seminal paper that started this research group. So this has become like a very useful research assistant. You can query our corpus and stuff. All right. So that’s a little demo of all our bots. So let’s go back to the presentation. Let me pause here briefly. Does anybody have questions about what you just saw in the demos before I dive into the back end? just unmute and speak up since I can’t really monitor the chat and talk at the same time. All right, moving on. So, what are the three bots? So, mixture of VGRs is me a person right now. It’s a mixture of two major eras of my writing. So, VGR Zerp is the one that runs live on Ribbon Farm. Uh what I call VGR Grahamsky. So it’s a reference to one of my articles called the GSKY gap but it’s one of the themes of uh uh the contraptions newsletter but there’s like half a dozen sources. Um so five sources ADK vectors so vectors are the unit in which uh rag embeddings work. So you take a large document chop it up into pieces. Uh I think they’re like the standard size. It’s not like paragraphs or something. standard size but with some overlap so you don’t have like seams in the database and then you shove them into an index and one of the things I found useful is to try and true name the personality of a particular bot uh but yeah this is a mixture of two bots with each with a true name Zerb and Grahamsky and here’s the underlying architecture uh so the difference from the uh basic Aaron Mar soul document approach is you really want ignorance and refusals uh modeled well because one of the things that I detected was what I call leak through. So by leak through I mean something like uh take a topic I might know absolutely nothing about like you know uh American football. I don’t watch football. I don’t know anything about it. But if you ask BGR zerp about football opinions it’s going to answer because what’s happening is it’s pulling stuff from the rag index and passing it through to Claude. And unlike me, Claude knows everything there is to know about football. So it’ll happily tell you stuff about football in my voice. But at some level, that’s kind of the wrong thing to do. It’s the wrong kind of response, right? Because the right answer is I’m pretending to be VGR. VGR knows nothing about football. So I have nothing to say about football, right? So that’s ignorance. You have to model ignorance. The second thing I want to point out is refusal. Uh so because I’m known for writing a lot of stuff about you know sociopathic behavior in organizations one of the kinds of requests I get a lot is uh teach me how to be a sociopath which is a very dangerous area to get into. Um it’s not the kind of influence you want to have. So you want to put guard rails around that kind of query and say I’ve detected kind of a type of topic I don’t want to actually talk about so I’m going to just exit it gracefully. And a third one what I’m calling dark knowledge that’s not evil knowledge but simply stuff I know about that’s not modeled well in my corpus. Right? So for example I grew up in a town called Jamshedpur in India which is a steel town. So I happen to know a lot about steel manufacturing and the steel industry simply because I grew up in a steel town and my dad worked at the steel plant for all his life. But this is not showing up anywhere in my blogging. But if it so happens that somebody asks me about the steel industry, it should actually reflect the fact that as part of my personal biography, I know a lot about the steel industry. So it should actually say more intelligent things, right? It might actually know more about the steel industry than um the generic claude does.

18:51 All right. C3P on institution. Uh here the challenge is you don’t want to like synthesize an random artificial voice because there are more than 80 researchers represented in the corpus and they all have very different styles, right? Uh it has less material since it’s a younger organization. Um but there’s more name spaces. Uh does this have a true name? We call it C3PO which is kind of like a hat tip to the Star Wars protocol droid. But honestly, I think it doesn’t have a proper soul yet. So even though the sole document approach will detect like commitments from our research community and some analytical moves like we like to tag things as Casca-esque protocols for example uh but I would say it doesn’t have a strong u soul yet there’s also uh C3PO works it has a slightly different architecture there’s this thing called a C3PO listener which kind of like uh has a lot of plugins that listen to a lot of different sources and and it goes into a spool and then the that gets ingested and then the query API. So these arrows are kind of drawn weirdly. This is all cloud generated slides by the way but yeah so the um gateway allows you to talk to it in two ways via the website or through discord. Uh yeah Krishna you have your hand raised you have a question.

20:14 Are you able to hear me?

20:15 Yes.

20:16 Hi.

20:17 Hi. Uh yeah uh not exactly in this slide the previous slide when you talked about your personal bot like uh an avatar of yours or your ideas where you grew up. So as a part of the training and the modeling and all those things uh how does the characterization is being controlled basically you talked about the sociopathic stuff right in an organizational that you don’t want to talk about I mean sometimes this character in itself the AI bots are even claw they talk about dark triad which is basically manipulator narcissistic or some

20:51 so this can drift into these kind of characteriz zones I would say And uh how do we prevent or how do you have any plans on uh putting a guardrails around it or

21:02 I I already did that’s my point. So the architecture of this I I’ll show you some details of how that works later but this is one of the modifications of the soul document approach. I’ll just put in guardrail rules. It’s like you know if you pattern match that a query is asking for advice on how to be a sociopath or how to manipulate somebody use one of these formulaic ways to exit from it and refuse to answer. So that’s basically it. So, I’ll show you some details in a few minutes. Uh, another question, Mike.

21:29 Yeah.

21:30 Yeah.

21:31 Oh, I was I think I think you’re about to answer my question. I was going to ask, you know, how you It’s a little bit hard to express negative things to LLMs

21:40 or I was wondering how you know how you actually implement filtering, but it sounds like you’re going to get to that. So,

21:45 yep. Right. All right. What’s these two?

21:51 So I don’t know if I’ll be able to speak to it very deeply since all this is wipe coded. So Claude probably knows more about how this is implemented than I do. But I’ll say what I can. Okay. Uh so C3PO it has a different problem. It’s listening across a whole huge number of surfaces. So there’s a separate bot that runs um as a demon on a virtual machine pulling all its sources every 30 minutes and kind of writes to a spool that then gets injected ingested and then the query API. Yeah, the web bot behaves a little differently than the Discord bot. Uh, all right. Humboldt is of course the weirdest one. Um, it uses my decision-making model in the book Tempo, which um, Amber mentioned in the previous talk. Um, and it’s basically hunting for laws of new nature. I’ll show you a little bit what it does. Uh, so here I’ve it’s not it doesn’t have a soul. It doesn’t have characteristic moves. What it instead has is a bunch of behaviors that it switches between as a state machine and the object that evolves through its operations is the law record. So, it’s like investigating laws and as it learns more it updates that records. But underneath all it all of that, it’s still kind of a rag bot because what it’s doing is it’s reading papers, it’s reading books, it’s making notes, it’s chunking them, putting them in a vector database, and whenever it tries to run a behavior like you know maybe the behavior is are there connections between the two papers I read, what’s happening in the background is it’s basically querying the vector database and like trying to make connections. So little bit more sophisticated than regular rag retrieval, but fundamentally it’s using a backend LLM to do rag based analysis. That’s what’s happening there.

23:37 Um, do I want to say anything more about this? Um, yeah, Humbult is I could probably do a whole talk on it. It’s the got the most headroom to grow and it’s not yet very good, but it’s um it’s a rag bot. Yeah, just a very different one. All right. So let’s let’s try and like generalize some patterns from all the things I’ve kind of done. Uh so I’ll show you a bunch of patterns [snorts] that I think have emerged. Um most of it excavated by um claude itself. So the way I’ve developed this talk was I have been maintaining devlogs for all the bots as I was developing them. So the devlogs are published on the individual sites. Uh but I told it to go look at the devlogs, go look at all the code and actually try and generalize how are we doing this. So because when I was doing this I was like kind of just improvising features that I wanted and refactoring along the way not really with a grand plan in mind. So this was almost like retconing a grand plan out of what I was actually doing. Um but yeah think of three broad functional areas. So before the model you’ve kind of to uh map the archive look at its geography ingest it and do the metadata in kind of like a very idiosyncratic way. Then you have to design the behaviors including budgets and how you’re measuring behavior. And then afterwards there’s a transcript phase where you’re like you know saving transcripts of chats if that’s a thing.

24:58 Um if there’s incidents that happen I have had security attacks and stuff. You know obviously anytime you put a rag bot like this people are going to probe it. they’re either going to try to like you know uh just use it as a cheap way to get tokens from claude or something or in other cases there have been cases where they were trying to probe it for personal information about me for example um so that those things you have to protect for u so these are the three broad areas and I’ll show you what patterns there are all right on mapping you kind of have to think of this really as exploring a very uh particular geography of data Uh so this is not very generic. You’re not trying to crawl the whole internet. You’re trying to crawl a particular idiosyncratic data set and you kind of want to like um carve reality at the joints as the Dowists like to say. You don’t want to like just randomly point claw at like a huge archive of like say PDFs and XML downloads and say just ingest this. That kind of doesn’t work. You kind of have to have a sense of what you’re doing and uh tastefully guide it.

26:03 Um, and there’s different injection strategies. They kind of do different things. Like for example, Humbot uh triages and does shallow and deep reads. Uh, for ribbon farm, I kind of have like different sources that are weighted differently. So mapping the archive is something you have to kind of do like an explorer. So here’s the ingest pipeline how it works. So this is something I have to learn the hard way. just ingesting the raw source materials is not enough because it’s kind of like too raw uh you kind of need to add some metadata for it to actually work as a query engine. So the first layer you do is you have the raw embedded nuggets but then you enrich the each set of like vectors for a document with a two sentence summary for example which can get hit by like a very top level query of like you know give me papers about memory. Well all the papers about memory should probably have a summary sentence with the word memory in it to hit right. Um then body chunks of course that’s kind of the core of it. You’ve got all the chunks with some 64 byt overlaps.

27:06 Um, so you don’t have any seams. But layer three is you kind of have to have like uh a meta assessment of what the document is about because I found that the uh retrieval was not working well if you didn’t have this metadata layer.

27:24 Uh what else is okay uh the metadata is almost the product that makes a bot artisal. for example, weighting your different sources differently. Uh like for example in uh VGR um Zer and VGR mixture VGR is both uh I have a lot more tweets so by word count it would probably swamp everything else but it’s much more posty and driveby whereas my blog posts are much more heavily thought through. So you kind of have to like cap how much Twitter is both weighted and how much it can dominate results. So it’s capped at four results. uh you kind of want to like keep awareness of time right so every source has a temporal epoch over which it was generated so you don’t want if there’s a very contemporary question uh but the hits are all coming from I don’t know 2012 you kind of want to take that into account maybe the question should respond with recent ideas uh yeah missing metadata fails uh silently so you might have a lot of relevant information but the queries won’t be picking them up if the metadata isn’t right. So, I was running into a lot of these blind spots early.

28:38 Uh, designing behaviors. Um, oh, this is initially I was like going nuts designing a whole bunch of behaviors for Humboldt, but many were never getting triggered. The graph never got there. Um, so you kind of actually have to wire this with live behavioral um analytics. uh and you start to cull behaviors that sound like a good idea but never actually run or don’t do anything useful or their trigger conditions are never met. So I’ve become a lot more conservative initially. Um the behavior graph I think had like almost 100 useless behaviors. Uh cost control is a design surface. U of course you don’t want you know million dollars uh against your credit card because uh people are hitting it too hard. Uh when I first announced my bot, yeah, I was running into, you know, people overusing it. So yeah, there’s lots of circuit breakers at different uh time points. Uh at one point I ran into um breakpoint where my free account for ingesting uh data from uh so I use Voyage to do the ingestion and pine cone as the database to serve the vectors and I think I ran into Voyage caps. So for a month inbox stuff was piling up but not getting ingested because I was waiting for my free account to reset. So these things become a design surface. So you have to kind of design gracefully how the bot behaves when it’s waiting for a limit to reset for example.

30:10 How do you um let’s see how should you do analytics on these things? because it’s a persona, you kind of don’t want it to, you know, [snorts] have KPIs like a industrial, you know, uh, enterprise software thing. You kind of want to like model the, um, analytics the way you might build a health tracker or like, you know, a wristband that you might wear to track your heart health or something, right? There’s a mix of different variables that kind of answer how alive and usefully person-like is this uh, bot, right? So see if this Yeah. So think of it as you’re measuring the livveness of the bot and you kind of have to idiosyncratically design an instrument for that bot. Uh so a personal bot versus an organizational bot versus an artificial researcher. They all have different indicators of is this thing alive? Is it behaving like a live process or bot should? Um so yeah use multiple metrics and compose them tastefully.

31:13 Oh, this is interesting. Uh, you can make this a closed loop. So have an archive of conversations and then that becomes part of its own training data. So self-t talk in a sense or if it’s talking to other human beings, other people conversations kind of become part of the training loop. So I’m still working on refining this. So there is a little bit of a training loop here, but it’s kind of like not fully automated. This is kind of really hard to do. Well, um, but also, yeah, people want to have a record. So, I have a mechanism on the VGR conversations especially where you might want to have, you know, a private advice-seeking conversation about a personal, you know, career situation. You don’t necessarily want to publish it. It gets stored as local store on your browser. So I don’t have like full aught running but you can have a little notebook of your conversations with me which is kind of like uh an airsoft mimicry of how I do you know consulting relationships with consulting uh clients who actually pay me. This is like an airsoft version of that and this is why I think maybe I can get people to pay for it. Um but yeah, people who are happy to make their conversations public that goes into a public archive and I did build up like a nice privacy architecture for this.

32:29 Uh and there was a security incident which was kind of like negative for me which was um the key management kind of barked. So, um, a bunch of privately submitted transcripts that people submitted with the permissions that allowed me to view it but not to publish publicly because I was like encrypting them and then lost the team. I actually lost half a dozen transcripts. So, that’s a good kind of failure, right? Uh, you don’t want uh privacy failures the other way. Let’s see. I’m not going to say much about this.

33:08 Yeah. So you kind of want to have some awareness of what the source of your uh data is. So discard data works differently and so forth. So things might need to be normalized differently depending on where it’s coming from. Um temporality is a control variable partly because uh I I showed you an example of this. I’m very interested in time just as a research topic. So I like to build time UIs for everything like I was uh mentioning in discussion around Amber’s talk guardrails. Yeah. So Mike was asking about this. Uh so you kind of want to partition. Well, of course. So yeah, there’s a basic class of safety moves that don’t belong to me. It belongs to Claude or whatever backend model you’re using Kimmy or Chpt. Like you know if somebody asks for how do I build a nuclear bomb? Uh that’s not my job to police. It’s the foundation model’s job to police. So, I obviously don’t try to do that. But persona refusals are things I refuse to do due to like, you know, personal ideas about what’s okay and not okay to um talk about, right? And they may not be seriously security critical.

34:18 Um but sometimes they are like I’ll give you one example that actually made me think about this, which was you guys saw the scandal that broke with the Zizian cult, right? So the Zizians were this extremist AI doomer something something cult and apparently the founder of the cult one of their influential source materials was one of my writings which is the Jerves principle which is an exploration of sociopathic behavior and apparently that person tried to become a sociopath and you all know how that went it became a murderous cult they killed somebody and so forth not something I want to be encouraging so how do you handle the fact that your material might have potentially dangerous material um where even if you’re not culpable, I don’t think I’m actually culpable or complicit in the crimes of the Zizians, but you definitely don’t want to make it easy for them, right? So, you can kind of design a bunch of behaviors. I can’t show you examples of all this, but you can go try talking to uh VGR Zerp about uh sociopathic advice seeeking and see what it does um how it hedges, how it reframes, and so forth. Uh but there’s other kinds of exits, right? So not my beat is the football example. I don’t have anything to say about football. So it’ll just say nothing to say about football. But if you ask about cricket, it might answer because yes, I do know a little bit more about cricket. Um coming from India, right? Um and there was a handoff behavior. So this was an older thing when I had different bots and a much more visible mixture of VGRT behavior. I I was thinking I would do something like you ask a question to VGR zer which is pre-2022 but it decides that really the relevant bot to answer that is VGR gsky which is a later epoch so it could hand off now in my new architecture that’s kind of getting absorbed into the invisible part of the model but even though it’s transparent something like that is probably going to continue to happen in the back end where things get handed off between personas.

36:14 Uh yeah, of course you can reframe that’s a classic consulting move and so forth.

36:22 Uh so leak through I already discussed this. The board answers well but it’s not from the archive. So you should have ignorance modeled properly. Um it’s not exactly a failure. the person got the answer they were looking for, but it’s it’s a failure in the sense that it violates the promise that this is a representation of either the person or the organization or something like that, right? Um, okay. So, compressing all my patterns into a cheat sheet, give it a true name. This is an artistic literary type of move, but I think it actually really helps organize the rag corpus into an artisal um geography with like an opinionated map. uh you kind of want to derive the persona and edit it by hand. I didn’t truly edit the sole and style documents by hand. I kind of like told Claude to edit things. It was like, you know, I like to make 2x2s. So, I bolted on a mechanism that actually generates 2x two. So, I kind of like gave it the diffs to add to the discovered style document. Uh yeah, I I won’t um talk too much about this. Uh of course, it does not scale. It’s not a scalable architecture. It’s not like you can uh point your cloud agent at my repo and say, uh make me a thousand bots like this, one for each of the students in my university. That won’t work. This is very much an N equal to one style of building bots. uh even this n of3 there was a not much generalization uh there’s no eval against like you know a reality ground loop except my uh idea of how well are the transcripts doing uh so it’s very much um social proof you could say against the rack corpus uh this is this is not a hazard I personally faced but I can tell that a lot of people will not want to do this because they identify very strongly with their writing. So getting too attached to your own archive is um yeah it shows up in multiple ways when you try to design something like this. All right. So that’s all I had. I’ll keep this questions. I’ll stop sharing and yeah we we have um time to talk so we can uh but before we go into questions is uh either Surya or Stannis law here I think either of them is all right so we’ll call them no shows and yeah we have leisurely time to discuss this talk I

39:06 I have a question um I’m wondering What is the most surprising thing you learned by looking at your own archives through this interface?

39:17 Well, apart from the one I already mentioned that my memory is terrible. Um, it’s kind of the how much the medium is the message. Like my Twitter personality genuinely is different from my blogging personality and it shows up in ways that you have to account for in having to wait different things. That’s an important one. Um, the other interesting one is even though so this is kind of a reverse of my previous point. There’s also a case of the medium is not the message. Um, even though I’ve separated my archives by different media I’ve written in and different like destinations like substack versus the old ribbon farm, the consolidated bot is actually able to detect through lines that I didn’t realize existed. Right? So if you ask the right kind of query, it’ll actually connect the dots and then I’ll look at it and say that’s actually a reasonable retcon and it’s not hallucinating. This is actually kind of what I was thinking. So there’s that. So that’s kind of uh uh some of the surprises but yeah I need to go deeper and a couple of things I think I need to do before this becomes truly interesting is uh I need to basically somehow capture my entire biography from before I started writing extensively online right so pre207 when I started my blog there’s a lot to capture there that should modulate what I say today because I am a product of like my non-writing life as well. So, there’s that. And I think one of the projects I have when I have more time is I need to start writing my memoirs in the sense of like just dictating raw material about my life and just shoving this into it as like uh background stuff. Uh let me see any questions in chat. Please speak out questions because it’s hard for me to scroll. But I’ll pick the last one which is Alex cast. How successful is the bot in speaking in your own voice versus claude voice? I you can try and tell me, but I suspect what happens is when the material retrieves a lot of like good relevant answers to the query, it does a good job of mimicking my style. But when it asks stuff that’s more generally vaguely relevant, it’s going to like slip back into clawish.

41:41 Uh, all right. Andre, how long did it take to build the first bot? Uh, the big effort actually was porting my archive. So, this actually wasn’t in the original plan. So, uh, I was tired of paying WP Engine like u $1,200 plus like, you know, usually $800 a year in overage fees. So, I was running at around $2,000 a year for running ribbon farm even though I had stopped writing the blog itself because it’s an old established blog that was getting a lot of traffic and WP Engine had like a very rent seeking type model. So, I wanted to like move it to a static site and now it runs on cloud cloudflare and it’s basically free like five bucks a month for the worker or something. So, I’m saving $2,000 uh year in hosting Ribbon Firm. And I was like, this is an archive and except for old readers, nobody’s going to come here really. So, uh what can I do to like keep it fresh and relevant? So, I said, hey, everybody’s talking about rag bots, so why not do it? I have this well digested material, so let me just do this. So, I did that and it turned out to be the most popular feature when I relaunched um the blog.

42:54 So, then I started wiring it in. every article now has a little link that says chat with VGR Zerp about it. So, and then people liked it. I put more and more plumbing and then it became the first VGR Zerbot and I was like this is a reusable architecture. So, then I started building more bots. So, yeah, that’s the answer to that question.

43:17 Graani tells me that the pron correct Italian pronunciation of Grahamsky is Grahamshi. uh stand corrected. So I’m going to start calling it the Graham Shei gap.

43:34 Another Andre question. Uh in the spirit, would you be able to speedrun ragbot creation for us non-chefs? Uh, I think my bots are well documented enough that if you go to any of them, there’s usually an about or architecture page. Point your cloud agent to that and ask it build me a bot like this. It should not it won’t speedrun it because I would say about depending on the complexity of the material you’re using it could be like a horrendous um you know data mining task to just get it into shape before you can do this or it could be as straightforward as I have this folder of 200 PDFs build me a bot like this in which case it could be a oneshot thing. Uh but for example, if you wanted to do something similar to my ribbon farm blog, uh that project was about two three months of pretty heavyduty work and this was I started it before cloud code became available. So I actually started it uh uh wipe coding in chat GPT in the chat sessions and cutting and pasting into a code editor and like uh building the thing myself. It was horrible. But about halfway through I was able to switch to um cloud code and then I finished that project. But I would say getting the ribbon farm memorialized archive online was about 100x the effort before I could build the chatbot which was like you know 1% marginal um uh effort. And I think that’s probably going to be the case for any rich long-term corpus of uh information that you might want to turn into an artisal rag bot. 99% of the work is going to be getting the information into a shape where you can point a rag um sort of like buildings scaffold onto it.

45:31 Let’s see. Please up. It’s just must be tiring for people to just hear me talk. So Johan, do you want to speak your question out loud on encrypted chat?

45:45 I I can. Um so um hi, thanks a lot. Uh the the question I had was in terms of LLM used, uh the price you pay seems quite high. Did you just well not care too much about which one to choose? understood you use clo code with the standard standard models.

46:08 Uh yeah, I mean this is low enough um that so I should divide into two uh senses. So the wipe coding part it’s like I was just using cloud code. I haven’t set up like cheaper coding models like you know Kimmy or whatever. It’s like a small enough project. I don’t care. But yes API uh costs can skyrocket especially if there’s like a spike of interest. And right now I’m using u uh sonnet I think and there unfortunately I think you do have to pick a model that’s relatively good at language generation. Uh so haiku is good at things like tagging. Um chat GPT I think is not as good at writing but also when you’re using claw code to build something the harness is kind of works [clears throat] better with anthropic products. But yeah sonnet seems to be a good compromise. I wouldn’t use Opus or Fable to serve the API. It would be way too expensive. Uh but in the future I might switch to a cheaper API. But my hope is at some point maybe I can you know charge for this. So right now the payment gateways for charging for this are not there. Um and I can charge enough for this that I don’t have to worry about how much the API part of it costs. like the margin uh opportunity might be enough that I shouldn’t have to care

47:30 because yeah I’m an expensive consultant uh so I should be able to like uh my usual rate is 500 bucks an hour if I’m you know doing consulting for a CEO if I offer this at say 50 bucks an hour I’m still thinking through payment models and stuff but I should be able to offer this pretty cheaply and still have enough of a margin uh uh left over that it kind of shouldn’t matter whether I’m using Kimmy or Sonnet or Opus even more questions.

48:12 Ven, I actually was wondering uh uh what uh um uh I mean the connection between redconing as you as you mentioned that the the way that the ragbot works.

48:28 Yeah. So yes, the ragbot will effectively retcon um narratives through all the material it pulls up. So think about how the retrieval process works. You put in a prompt, it’ll hit the vector database and maybe come up with like a dozen hits. It’ll see some rough, you know, clustering or pattern or thematic stuff there. It’ll put it through an API into some LLM like Claude and Claude will process the original prompt with the idea of telling a story that answers the prompt. Right? So if you retrieve 12 chunks from the vector database, it might think that the right order is number 12, number three, number seven, like you know, it might sequence them in some order and say then I can tell a story about it. So that’s kind of mechanically how this works. The question is how meaningful is what it does. So I think the spectrum can be it’s pure hallucination and if you ask me so one of the AB testing mechanisms I want to do in my newest version is um it’ll be submit this query to the live venkut and see what he says and then compare is my response actually meaningfully different from what the bot said. So that’s one way to think of it.

49:45 Uh but yes, like I said, anecdotally, often it actually does recover something I might have said myself where I say, “All right, I sign off. Thumbs up. The bot is basically saying what I would have said.” Other times, I haven’t yet found any good examples. I might disagree violently and say, “That’s What I really think is XYZ.” So that’s me as a personal person. Now, the other two cases are more interesting, right? C3PO is querying the archives of an organization and there is no individual on the other side but you are in a sense asking what is the protocol institute or summer of protocols community um thinking about this topic right so for example if it asks something like um what is the future what is what’s the you know next 10 years vision future of IoT and robotics maybe that’s the query it’s going to put up the research you did last year as part of your SOP project. It might pull up the robotics workshop running this year. It might pull up something else. It might synthesize a through line, but there’s no reasonable way you can call it hallucinated, right?

50:52 Because it’s not actually pretending to be something it’s not. It is actually synthesizing a view. Now, the view may be wrong. It may make up like a technically uh claim about the future vision, but then so do humans, right? like you ask somebody to speculate about robotics 10 years later, they might give you a answer that you don’t agree with. So I would say it’s an okay solution there. The retconing gives you a reasonable representation of what our community thinks about that question. The third one I think is the most interesting one because the humbult bot is completely synthetic, right? It does not represent a research tradition. It’s a hypothesized persona that maybe has a philosophy of research and it’s doing research in its own style and in some way you can’t call that redconning. You should call that that is genuinely what Humboldt the bot thinks about this question. So very interesting question. It sort of forces you to think about what is the contract or representation being made about the bot by the entity who has provisioned the bot. Right? I have provisioned all three, but I’m making different representations to the user about what each of them is. And the first one is in some ways the easiest case because I’m saying this is a digital version of me.

52:10 The other two get more and more complex.

52:18 More questions.

52:32 All right. If not, I think what we can do is call this session done and let’s all move over to room 2 world building and I’m going to shill it a little bit. So as you know we all have uh at protocol institute for the last year we have been uh publishing this uh substack called protocolized which has a mix of fiction and non-fiction and one of the things we’re doing as we become the protocol institute is the protocol institute is going to be part of this foundation called the experiment foundation which has a nonprofit science mission so fiction and cultural stuff doesn’t quite fit their mission so what we’re doing is forking what our current work into two so the non-fiction slash uh researchy stuff will go to protocol institute under experiment foundation but all our work trying to develop a genre of protocol fiction which is a genre of science fiction they’re going to fork it off as a separate magazine called the monstrous times and this other section it’s been about world building so they’ve been talking a lot about world building but they’re actually going to segue into the part right now where they will be talking about the launch of this new magazine so we are really excited about this. So yes, everybody please switch over to room two and we will uh join their conversation. So let’s all move over to room two.