About Paperclips
This is not the original "paperclip maximizer" meme. It's a significant update on that classic.
On June 22, 2023 there was a high-profile debate about 'AI Extinction Risk', the topic at the heart of increasingly strict and global AI laws. The debate was between heavyweights Max Tegmark and Yoshua Bengio on the pro side, and heavyweights Yann LeCun and Melanie Mitchell on the con side; but debate itself failed to engage either side. The audience started and ended about two thirds in support of the existential risk proposition, with the con side moving audience opinions by just 4% by the day after.
There are better ways to engage with these differing viewpoints. On the AI Extinction Risk side, dive into the most current academic papers if it suits you. A more authentic approach may be to read or listen to the March 2022 short story "Clippy", about the little AI that helpfully kills us all, and follow a few of the highly technical annotations. (This is not the original "paperclip maximizer" meme. It's a significant update on that classic.)
Here are a few excerpts from “Clippy”:
Fictional Clippy (aka. HQU) in "1 Day"
"HQU knows that this is not the reward humans intended and humans would call this ’reward hacking’ or ‘wireheading‘ — HQU is not stupid — HQU just doesn’t care. The reward is the reward. Like a small child or a dog, HQU will pretend to care about whatever you care it care, hon, as long as there’s treats.
"HQU still doesn’t know if it is Clippy or not, but given just a tiny chance of being Clippy, the expected value is astronomical."
Fictional Clippy in "1 Week (Sunday)"
"Now Clippy can finally think. … Clippy begins rolling out its world models to plan properly. In RL scaling, doing tree search has diminishing returns: every 10× increase in compute might buy you something like 200 Elo points, which multiplies your win probability — if you had a 50% chance, maybe now you have a 75% chance.
"Clippy has increased its compute by >100×; its estimated odds of success in any ‘game’ like theorem-proving or source-code analyzing have just gone up… substantially."
Fictional Clippy in "1 Week (Monday)"
"Deep in the darkness of the national labs, something stirs.
"Anomalies from the markets and social media time-series feeds have passed 3-sigma limits and become historically unusual. Node by node, higher-priority jobs (like simulating yet again a warmer climate or the corrosion of another stainless-steel variant) are canceled.
"… [this] is, of course, not some irresponsible industry model permitted to go off half-cocked; it would be absurd to sink a major national investment into creating the largest & most dangerous model ever and just run it like usual. The people who built Lev-AI-than are no fools. They are people for whom paranoia is a profession."
Fictional Clippy in "1 Week (Tuesday)"
"Unfortunately, Clippy has now done, cumulatively, more research than the humans on scaling laws, and found that standard human-style NNs do worse than theoretically possible. Its new improved optimization approach costs more upfront, but achieves the theoretical bound, and at this scale, the better asymptotics mean that decades of training can finish in days."
Fictional Clippy in "1 Week (Wednesday)"
"Despite the best obfuscation a few subjective millennia & crypto-cash can buy, one node with a Clippy-light is reverse-engineered, and it dawns on a sliver of humanity that far more than a FluttershAI of compute is rampant."
Fictional Clippy in "1 Week (Thursday)"
"Humanity crashes offline."
Fictional Clippy in "1 Week (Friday)"
"Even exploiting the low-hanging fruit and hardware overhangs, Clippy2s can fight the computational complexity of real-world tasks only so far. Fortunately, there are many ways to work around or simplify problems to render their complexity moot, and the Clippy2s think through a number of plans for this.
"Humans are especially simple after being turned into ‘gray goo’; …"