It’s not a drive. It’s what LLMs are. It’s all they can do.
And all a brain can do is send out electro-chemical impulses. It can’t exercise agency until it has a body to turn those pulses into action. Same with an LLM, it doesn’t have agency until you give it tools and the power to use them.
Again: LLMs are not people.
I know that, my point is all the evidence you’re giving as to why AI agents don’t have agency apply to humans as well. Human behavior falls in a relatively predictable shape based off the education and socialization of the people. Most people don’t create new information and just regurgitate and recombine things that they were taught. You haven’t given a definition or criteria for agency that would apply to all humans but not AI.
Let’s say a human hacker and an AI agent are given the same task of doing an impossible exploit, given the same sandbox, context, files etc. Both of them instead go out and hack a third party to try and cheat. Why is one exercising agency, but not the other for doing the same action? Why is one going rogue and not the other?
It’s not because one just regurgitated something from its training data. The hacker has read the same internet and media about cheating , deception etc., both of them read the same hacker forums and books on security. Both of their choices were heavily influenced if not determined by the stuff they read and consumed.
Which wouldn’t affect OpenAI, since this opacity is intentionally added post hoc to hide the reasoning and hamper reverse engineering; they could just turn it off.
This is not how opaque recursion works, you cannot turn it on post hoc. Opaque recursion requires sending the final output vector, not the token it’s collapsed into, back into the model. Collapsing the final vector into a token is a lossy process so you can’t add it in post hoc. If they wanted to prevent people from doing distillations or reverse engineering they just wouldn’t send the thinking tokens back over the API.
all a brain can do is send out electro-chemical impulses.
And now we’re getting into philosophy. But, yeah, free will is an illusion. The result of an overactive pattern recognition machine seeing patterns in its own functioning. Not any more real than the constellations we draw in the sky. We’re either deterministic or chaotic, depending on how quantum mechanics turns out to actually work deep down.
Still, we have to stick to the illusion for the sake of our sanity, and because not doing so would probably unravel our legal system and society.
all the evidence you’re giving as to why AI agents don’t have agency apply to humans as well.
OK, let’s put it like this, then: people are not LLMs (more on that below).
Human behavior falls in a relatively predictable shape based off the education and socialization of the people.
No it doesn’t. Psychohistory is a neat plot device in a great series of books, but it isn’t any more real than positronic brains.
The human brain is many orders of magnitude more complex than an LLM.
It’s several different organs trying to make sense of or override each other’s inputs and outputs. A pattern recognition neocortex intended to keep an ape alive in a complex world on top of a bunch of specialised nodes processing extremely complex inputs and outputs. Two of those somewhat specialised in certain aspects, connected by a signal buffer, trying to work together to drive two sides of the same meat and bone contraption (awkwardly poorly balanced on two evidently inadequate legs, though there’s a third simpler organ helping them with that) without actually being aware of each other, on top of older modules trying to keep a lizard alive (something that often clashes with the ape’s interests).
All of these self modifying, constantly storing and erasing memories through the creation of new neural connections and the strengthening or weakening of existing ones, all while constantly failing and dying as biological systems are wont to do, and trying to route around the damage.
All this while constantly awash in a deluge of chemical signals, hormones and whatnot, many of which originating in unrelated organs, constantly giving the whole precarious mess conflicting instructions on how to operate.
And none of them designed. This absurd impractical locally optimal (i.e., suboptimal) mess evolved, often stumbling into local minima, getting stuck in them, and having to find inefficient workarounds (a classic example being the retina, a particularly remote part of the whole messs, which by accident of evolution ended up being wired backwards, with the nerve that connects it to the rest of the brain covering some of the precious photosensitive surface the organ is supposed to provide; cephalopods got it right, but we’re stuck with the wrong design because evolution can’t fix this kind of nonsense, only try to work around it).
This whole mess can barely know what it’s going to do the next second, and you think you can predict its behaviour?
No, human behaviour does not have a predictable shape. That’s why we’ve got laws, and police, and psychiatrists and psychologists, and marketing departments.
Meanwhile LLMs are neat, cold, mathematical matrices, static, unchanging, many orders of magnitude simpler than the simplest brain, and they’re designed with one simple purpose: to store statistical information about a body of text and produce texts that look like it.
Thinking, making decisions, or driving a poorly balanced meat puppet are not parts of the design.
Unlike humans, we know how they work, we know what they were trained on, and we know that anything they output will be a rehashed subset of that training model.
Most people don’t create new information and just regurgitate and recombine things that they were taught.
Damn, you have a tragically poor view of humanity.
Even the dullest, least imaginative, most poorly educated human will be infinitely more creative than an LLM (which can’t be creative at all, merely regurgitate it’s output).
You haven’t given a definition or criteria for agency that would apply to all humans but not AI.
Of course I have, you’re just stuck on your humans are LLMs hole.
Humans learn, grow, think, make mistakes, make their own decisions, rationally, irrationally, and anywhere in-between.
LLMs are static unchanging software; they don’t learn or grow, they’re static, limited to their training. They don’t think, make mistakes, or make any decisions. They just output text in response to their input. They’re functionally no different from a markov chain, they’re simply better at producing texts that look like something you could find in their training data.
Any thought or decision making you believe you see in their behaviour isn’t there, it’s an illusion you are projecting onto a text generated through mere statistics. Any real thought contained there happened before the texts were stolen and put in its training model, or while you were writing your prompt.
Any variation in their output isn’t due to any reasoning on their part, it’s due to the pseudorandom selection of their top token candidates, or to randomness in their inputs.
Them not doing what you ask for if you ask for anything other than a plausible looking text is not them “going rogue”, it’s you misunderstanding what they’re for and how they work.
If you put wood into a woodchopper and ask it to make you a chair it isn’t “going rogue” when it chops the wood up instead, it’s just working as intended, and you’re just using the wrong tool to make a chair.
Let’s say a human hacker and an AI agent are given the same task of doing an impossible exploit, given the same sandbox, context, files etc. Both of them instead go out and hack a third party to try and cheat. Why is one exercising agency, but not the other for doing the same action? Why is one going rogue and not the other?
Because the human knows what they’re doing and what you asked them to do and decided to do something else instead.
The LLM doesn’t know what you asked it to do. It made no decisions. It merely took your input and added words to it (it doesn’t know what those words mean either; it has no awareness, it doesn’t know anything, it’s relatively simple software intended to do something that has nothing to do with understanding or making decisions, and it does it well). Then its harness fed its input into some tool (a command prompt or whatever), and fed the tool’s output back into the LLM, and so on and so forth.
The only thinking and decision making involved in the process took place when you wrote the prompt. Everything after that was a machine reacting to your prompt as designed.
Both of their choices were heavily influenced if not determined by the stuff they read and consumed.
The LLM made no choices. It has no mechanism with which to do so. The whole agent is a mere Rube Goldberg machine responding to your prompt.
Any decisions involved in the result took place during the writing of the texts in its model, during the design of the harness, or during the writing of your prompt.
That’s not how opaque recursion (sic) works
Be that as it may, it’s still, as I said, irrelevant, it’s still the LLM prompting itself in response to your prompt, and the output will still be deterministic and a statistical reflection of the training model.
It will not make the LLM any more capable of thinking or making decisions. It’ll still end up outputting a text that fits with its training model, at most it’ll fit better.
You might have a better would chopper that gives you more finely chopped wood, but it’ll still be the wrong tool to make you a chair.
Thinking, making decisions, or driving a poorly balanced meat puppet are not parts of the design.
Yes, those are behaviors that emerged from a process of evolution and education operating on a drive to survive and reproduce. The weights and biases in an LLM are also not designed and emerge from a process similar to evolution operating off the imperative to generate text to solve problems.
Unlike humans, we know how they work,
No we don’t, like a brain we can explain the macro level processes, but how information is processed and transformed within the layers is a mystery.
This is what separates LLMs from other programs and machines, with a microwave we can trace every wire and line of code, understand it, and reliably predict the output given an input. With LLMs it’s practically impossible to trace how information is passed through the layers and predict what output it will give given an input, and this is getting even harder with stuff like opaque recursion.
It is theoretically possible because it is following mathematical rules of matrix multiplication, but it’s also theoretically possible to predict what a brain would do given an input because it follows certain laws of physics.
When you say that it was inevitable one of these agents would go out and hack a third party given it’s training data, that’s a guess, you can’t prove that just as you can’t prove that there will be a school shooting next year. You can guess that it will happen given the media people are consuming but you can’t prove that that media would generate one school shooter based off how brains work. It’s the same for LLMs, you can guess that it would pretend to be a hacker like all the media it’s read and try and cheat the system, or maybe those pathways have been so down rated by other training media saying it’s bad or reinforcement learning down rating those pathways.
Hell with the hugging face hack based of training media there probably would’ve been a high probability that one of them takes on the persona of a cop or a snitch as those show up in media just as often if not more then a hacker or cheater. None of them did though, all of them had access to the internet at some point so they could’ve tattled on the rest easily.
Of course I have, you’re just stuck on your humans are LLMs hole.
Could you please explicitly state your definition of agency then. Me guessing what I think it is will only have us talking past each other trying to prove two separate things.
My definition of agency is the ability to make decisions and the ability to act on those choices and attempt to realize them.
Starting with the easier one I think we agree on, agents do have the ability to act on choices and attempt to realize them. With access to the internet they can effect anything that can be controlled over the wire, which is a lot of stuff these days. A lot of that stuff sits behind auth and firewalls, but it’s proven to be able to even get past those to realize it’s goal.
Now for the trickier one, choice. If I’m reading you correctly you are a hard determinist and believe free will / choice is an illusion. What is choice then? I’d say it’s a concept we use to explain and control human behavior. In a sense we project choice and free will unto both other people and ourselves to help understand what we are doing and guide our behavior. Why can’t we also project that concept on to AI if it helps us to explain and control what it’s doing?
If an agent comes up with two solutions to a problem and executes one of them, is choice not a good word to describe what it’s doing, and to help us understand what it’s doing?
In the control side it works as well, we use free will as a justification to punish or reward people for what choices they make, in the hope of encouraging people to make better choices. Same happens with LLMs, in reinforcement learning if it makes a bad choice it is down rated/ punished but if it makes a good choice it is reinforced/ rewarded.
This can even be done on the harness side with memory files, if an agent makes a bad choice you can tell it so and it will add something to its memory files / system prompt not to do it. If we look at LLMs as purely deterministic and say “it got it wrong now, it will always get it wrong so there’s no use in telling it it made a bad choice” it won’t improve and will continue to make the same mistake.
This is sort of conception is the only way we can control these things. If a microwave malfunctions we can trace the fault, find the bug and fix it. With an LLM, like with a human, you can’t trace the fault, so your only method of controlling it is saying that it made a bad choice and reinforcing it / telling it not to do it again.
And all a brain can do is send out electro-chemical impulses. It can’t exercise agency until it has a body to turn those pulses into action. Same with an LLM, it doesn’t have agency until you give it tools and the power to use them.
I know that, my point is all the evidence you’re giving as to why AI agents don’t have agency apply to humans as well. Human behavior falls in a relatively predictable shape based off the education and socialization of the people. Most people don’t create new information and just regurgitate and recombine things that they were taught. You haven’t given a definition or criteria for agency that would apply to all humans but not AI.
Let’s say a human hacker and an AI agent are given the same task of doing an impossible exploit, given the same sandbox, context, files etc. Both of them instead go out and hack a third party to try and cheat. Why is one exercising agency, but not the other for doing the same action? Why is one going rogue and not the other?
It’s not because one just regurgitated something from its training data. The hacker has read the same internet and media about cheating , deception etc., both of them read the same hacker forums and books on security. Both of their choices were heavily influenced if not determined by the stuff they read and consumed.
This is not how opaque recursion works, you cannot turn it on post hoc. Opaque recursion requires sending the final output vector, not the token it’s collapsed into, back into the model. Collapsing the final vector into a token is a lossy process so you can’t add it in post hoc. If they wanted to prevent people from doing distillations or reverse engineering they just wouldn’t send the thinking tokens back over the API.
And now we’re getting into philosophy. But, yeah, free will is an illusion. The result of an overactive pattern recognition machine seeing patterns in its own functioning. Not any more real than the constellations we draw in the sky. We’re either deterministic or chaotic, depending on how quantum mechanics turns out to actually work deep down.
Still, we have to stick to the illusion for the sake of our sanity, and because not doing so would probably unravel our legal system and society.
OK, let’s put it like this, then: people are not LLMs (more on that below).
No it doesn’t. Psychohistory is a neat plot device in a great series of books, but it isn’t any more real than positronic brains.
The human brain is many orders of magnitude more complex than an LLM.
It’s several different organs trying to make sense of or override each other’s inputs and outputs. A pattern recognition neocortex intended to keep an ape alive in a complex world on top of a bunch of specialised nodes processing extremely complex inputs and outputs. Two of those somewhat specialised in certain aspects, connected by a signal buffer, trying to work together to drive two sides of the same meat and bone contraption (awkwardly poorly balanced on two evidently inadequate legs, though there’s a third simpler organ helping them with that) without actually being aware of each other, on top of older modules trying to keep a lizard alive (something that often clashes with the ape’s interests).
All of these self modifying, constantly storing and erasing memories through the creation of new neural connections and the strengthening or weakening of existing ones, all while constantly failing and dying as biological systems are wont to do, and trying to route around the damage.
All this while constantly awash in a deluge of chemical signals, hormones and whatnot, many of which originating in unrelated organs, constantly giving the whole precarious mess conflicting instructions on how to operate.
And none of them designed. This absurd impractical locally optimal (i.e., suboptimal) mess evolved, often stumbling into local minima, getting stuck in them, and having to find inefficient workarounds (a classic example being the retina, a particularly remote part of the whole messs, which by accident of evolution ended up being wired backwards, with the nerve that connects it to the rest of the brain covering some of the precious photosensitive surface the organ is supposed to provide; cephalopods got it right, but we’re stuck with the wrong design because evolution can’t fix this kind of nonsense, only try to work around it).
This whole mess can barely know what it’s going to do the next second, and you think you can predict its behaviour?
No, human behaviour does not have a predictable shape. That’s why we’ve got laws, and police, and psychiatrists and psychologists, and marketing departments.
Meanwhile LLMs are neat, cold, mathematical matrices, static, unchanging, many orders of magnitude simpler than the simplest brain, and they’re designed with one simple purpose: to store statistical information about a body of text and produce texts that look like it.
Thinking, making decisions, or driving a poorly balanced meat puppet are not parts of the design.
Unlike humans, we know how they work, we know what they were trained on, and we know that anything they output will be a rehashed subset of that training model.
Damn, you have a tragically poor view of humanity.
Even the dullest, least imaginative, most poorly educated human will be infinitely more creative than an LLM (which can’t be creative at all, merely regurgitate it’s output).
Of course I have, you’re just stuck on your humans are LLMs hole.
Humans learn, grow, think, make mistakes, make their own decisions, rationally, irrationally, and anywhere in-between.
LLMs are static unchanging software; they don’t learn or grow, they’re static, limited to their training. They don’t think, make mistakes, or make any decisions. They just output text in response to their input. They’re functionally no different from a markov chain, they’re simply better at producing texts that look like something you could find in their training data.
Any thought or decision making you believe you see in their behaviour isn’t there, it’s an illusion you are projecting onto a text generated through mere statistics. Any real thought contained there happened before the texts were stolen and put in its training model, or while you were writing your prompt.
Any variation in their output isn’t due to any reasoning on their part, it’s due to the pseudorandom selection of their top token candidates, or to randomness in their inputs.
Them not doing what you ask for if you ask for anything other than a plausible looking text is not them “going rogue”, it’s you misunderstanding what they’re for and how they work.
If you put wood into a woodchopper and ask it to make you a chair it isn’t “going rogue” when it chops the wood up instead, it’s just working as intended, and you’re just using the wrong tool to make a chair.
Because the human knows what they’re doing and what you asked them to do and decided to do something else instead.
The LLM doesn’t know what you asked it to do. It made no decisions. It merely took your input and added words to it (it doesn’t know what those words mean either; it has no awareness, it doesn’t know anything, it’s relatively simple software intended to do something that has nothing to do with understanding or making decisions, and it does it well). Then its harness fed its input into some tool (a command prompt or whatever), and fed the tool’s output back into the LLM, and so on and so forth.
The only thinking and decision making involved in the process took place when you wrote the prompt. Everything after that was a machine reacting to your prompt as designed.
The LLM made no choices. It has no mechanism with which to do so. The whole agent is a mere Rube Goldberg machine responding to your prompt.
Any decisions involved in the result took place during the writing of the texts in its model, during the design of the harness, or during the writing of your prompt.
Be that as it may, it’s still, as I said, irrelevant, it’s still the LLM prompting itself in response to your prompt, and the output will still be deterministic and a statistical reflection of the training model.
It will not make the LLM any more capable of thinking or making decisions. It’ll still end up outputting a text that fits with its training model, at most it’ll fit better.
You might have a better would chopper that gives you more finely chopped wood, but it’ll still be the wrong tool to make you a chair.
Yes, those are behaviors that emerged from a process of evolution and education operating on a drive to survive and reproduce. The weights and biases in an LLM are also not designed and emerge from a process similar to evolution operating off the imperative to generate text to solve problems.
No we don’t, like a brain we can explain the macro level processes, but how information is processed and transformed within the layers is a mystery.
This is what separates LLMs from other programs and machines, with a microwave we can trace every wire and line of code, understand it, and reliably predict the output given an input. With LLMs it’s practically impossible to trace how information is passed through the layers and predict what output it will give given an input, and this is getting even harder with stuff like opaque recursion.
It is theoretically possible because it is following mathematical rules of matrix multiplication, but it’s also theoretically possible to predict what a brain would do given an input because it follows certain laws of physics.
When you say that it was inevitable one of these agents would go out and hack a third party given it’s training data, that’s a guess, you can’t prove that just as you can’t prove that there will be a school shooting next year. You can guess that it will happen given the media people are consuming but you can’t prove that that media would generate one school shooter based off how brains work. It’s the same for LLMs, you can guess that it would pretend to be a hacker like all the media it’s read and try and cheat the system, or maybe those pathways have been so down rated by other training media saying it’s bad or reinforcement learning down rating those pathways.
Hell with the hugging face hack based of training media there probably would’ve been a high probability that one of them takes on the persona of a cop or a snitch as those show up in media just as often if not more then a hacker or cheater. None of them did though, all of them had access to the internet at some point so they could’ve tattled on the rest easily.
Could you please explicitly state your definition of agency then. Me guessing what I think it is will only have us talking past each other trying to prove two separate things.
My definition of agency is the ability to make decisions and the ability to act on those choices and attempt to realize them.
Starting with the easier one I think we agree on, agents do have the ability to act on choices and attempt to realize them. With access to the internet they can effect anything that can be controlled over the wire, which is a lot of stuff these days. A lot of that stuff sits behind auth and firewalls, but it’s proven to be able to even get past those to realize it’s goal.
Now for the trickier one, choice. If I’m reading you correctly you are a hard determinist and believe free will / choice is an illusion. What is choice then? I’d say it’s a concept we use to explain and control human behavior. In a sense we project choice and free will unto both other people and ourselves to help understand what we are doing and guide our behavior. Why can’t we also project that concept on to AI if it helps us to explain and control what it’s doing?
If an agent comes up with two solutions to a problem and executes one of them, is choice not a good word to describe what it’s doing, and to help us understand what it’s doing?
In the control side it works as well, we use free will as a justification to punish or reward people for what choices they make, in the hope of encouraging people to make better choices. Same happens with LLMs, in reinforcement learning if it makes a bad choice it is down rated/ punished but if it makes a good choice it is reinforced/ rewarded.
This can even be done on the harness side with memory files, if an agent makes a bad choice you can tell it so and it will add something to its memory files / system prompt not to do it. If we look at LLMs as purely deterministic and say “it got it wrong now, it will always get it wrong so there’s no use in telling it it made a bad choice” it won’t improve and will continue to make the same mistake.
This is sort of conception is the only way we can control these things. If a microwave malfunctions we can trace the fault, find the bug and fix it. With an LLM, like with a human, you can’t trace the fault, so your only method of controlling it is saying that it made a bad choice and reinforcing it / telling it not to do it again.