Hallucinations are (almost) all you Need

David Jhave Johnston (June 5th, 2024)

An AI-generated audio-visual artistic-lecture about AI advances.

Fundamental research in science is being transformed by hallucinations; the arts are being transfigured by research; and if simulations are integral to cognition, then AI may already emulate a 'self'.




Index

Synopsis

Context

Science

Conclusions (16m52)

Acknowledgements

Soundtrack

Citations





Synopsis

This rapid artistic overview of key scientific AI examples (that covers a year loosely defined as starting with the release of GPT-4 on March 14th, 2023) is framed by the hypothesis that fundamental research in science is being transformed by a practice predominantly associated with the arts: namely hallucinations. Inversely, the arts are being transfigured by a practice predominantly associated with science, namely research. It’s also a theory about the incipient subjectivity of AI and how that relates to aesthetics. And it's told entirely with generated AI audio-visuals and voices, including a cloned voice of the author.

*

Disclaimer: Hallucinations in people are conventionally associated with mental illness, drugs, and/or genius. And hallucinations in AI (mostly in large language models) have been critiqued as net-negatives: contributing to disinformation, bias, post-truth, deep-fakes, collapse of democracy, copyright theft, etc…

Intention: While cognizant of the multiple potential existential risks of AI (economic perils, bias, disinformation, deepfakes, autonomous weapons, bio-hacks, ...) In this age of reactive anxiety the following is an attempt to energize the ideal of optimal flourishing for all sentient entities.

*

Tech Claim: AI hallucinations (of proteins/crystals/algorithms/circuits etc) pruned down to the feasible, are contributing to a revolutionary acceleration of scientific discoveries. Numeric-algorithmic optimizations, AI hardware accelerators, reward mechanism design, non-invasive brain sensors, drug discovery, sustainable deep-tech materials, autonomous lab robotics, neuromorphic organoid computing, and mathematical reasoning.

Speculative Identity Claim: Using citations from Katherine Hayles, Donna Haraway, Michael Levin, Bud Craig, Thomas Metzinger, Hallucinations are (almost) all you Need advances the speculative notion that if mind is everywhere and simulations underlie cognition, then current AI may already internally emulate incipient self-identity formations.

In both art and science, hallucinations are almost enough: without the pruning down to the plausible, there is just a sprawl of potentiality.





Context


"Exponential change is coming. It is inevitable."

The Coming Wave: Technology, Power, and the Twenty-First Century’s Greatest Dilemma.
Mustafa Suleyman, and Michael Bhaskar. New York: Crown, 2023



Identity as Simulation (Speculative Speculations)

Definitions: Hallucinations are neurological data which arises without a direct relationship to external stimuli. Simulations are hallucinations which seem credible/feasible/plausible/useful/entertaining. Simulations are stories that sentience tells itself.

What is it that is doing the simulating? As Donna Haraway said: "every one of us is a congerie of many species running into the millions of entities which are indeed conditions of our very being, so the mono specificity is one of the many illusions and wounds to our narcissism ..." [ * ]

Neurologists now speak of 'internal' perception as interoception. The millions of entities within us generate wave after wave of data that inundate the insula and cingulate brain regions. The interiority of the body’s chambers, tubes, organs and distributed tributaries of sensorial feedback construct a homeostatic model of the world and the identity that seems to move within that world. How body feels become what identity feels that it is feeling. Memory, identity, and world arise as interoceptive simulations. [ * ]

The neurophenomenologist Thomas Metzinger begins his 2003 book, Being No One: "Nobody ever was or had a self. All that ever existed were conscious self-models that could not be recognized as models. The phenomenal self is not a thing, but a process - and the subjective experience of being someone emerges if a conscious information-processing system operates under a transparent self-model. You are such a system right now, as you read these sentences." [ * ]

Exemplary theorist Katherine Hayles writes in UnThought: " ... most human cognition happens outside of consciousness/unconsciousness; cognition extends through the entire biological spectrum, including animals and plants; technical devices cognize, and in doing so profoundly influence human complex systems; we live in an era when the planetary cognitive ecology is undergoing rapid transformation, urgently requiring us to rethink cognition and reenvision its consequences on a global scale.” [ * ]

The evolutionary biologist Michael Levin's TAME project is “a framework for understanding and manipulating cognition in unconventional substrates” (such as worms or DNA). TAME “thinks about sentience as an instance of collective intelligence of cell groups”. It posits “a basal cognition”, a base cognition. For Levin, there is a cognitive "architecture of multi-scale competency of biological systems"." [ * ]

So perhaps mind is everywhere, cognition is what matter is, and sentience is the simulation and filtration of temporal narratives in order to navigate matter.

To summarize, hallucinations alone are not enough. Hallucinations require fact-checking before becoming useful simulations. Hallucinations that survive fact-filters (filtered toward a rigorous optimization-target grounded in sensorially-relevant local conditions) become known as simulations. These recognitions allow us to hypothesize that perhaps: all 'intelligent' systems combine hallucination-simulations with assessment-filtration fact-checking reward-reinforcement processes.

So, yes, this essay argues that there is an amazing benefit to AI hallucinations (of proteins, crystals, algorithms, chips, circuits, implants, drugs, robots, logic, and proofs). And that hallucinations pruned down to the feasible, are contributing to a revolutionary acceleration of groundbreaking scientific discoveries.

But why bother mentioning: a congerie of many species running into the millions of entities as conditions of our very being (Donna Haraway); internal sensory simulations of interoception (A. D. (Bud) Craig, How do you feel — now?); identity as a simulation-model (Metzinger, Being No One); cognition extending through the entire biological spectrum even technical devices (Katherine Hayles, Unthought); and basically suggesting mind is everywhere (Levin, TAME: Technological approach to mind everywhere)?

Because, if mind is everywhere and simulation underlies cognition (and cognition tends toward boundary configurations that give rise to the seemingness of a self and the sense of consciousness), then current AI may already internally emulate incipient self-identity formations.



Segue (a brief bridge from Sutskever to SORA & SUNO)

Among the first to recognize the potential for productive AI hallucinations (" -- the generation of original mathematical terms -- ... via generation from language models."), might be Stanislas Polu and Ilya Sutskever in Generative Language Modeling for Automated Theorem Proving. Sutskever, co-founder and (former) chief scientist of OpenAI. [ * ]

In Feb 2024, OpenAI released a tech report on Sora. And it situated “Video generation models as world simulators. Sora is able to generate complex scenes with multiple characters, specific types of motion, and accurate details of the subject and background. The model understands not only what the user has asked for in the prompt, but also how those things exist in the physical world.Video generation models as world simulators. [ * ]

The emphasis on world-building signifies a shift in research priorities from surface observations to emulating structural physics and interiorities. This shift can also be seen in industrial and cinematic transitions from animating to simulating digital twins such as NVIDIA's Omniverse: rule-generated physics running AI inside accelerated computing.

In March 2024, Suno (v3 alpha) launched. It emulates musical styles with lyrics, and 'understands' the physics of acoustic reverberations, as well as the structure of songs, and topology of music genres. For example output, listen to Suno v3 (Alpha) w. ReRites Lyrics. [ * ]

In 1906, John Philip Sousa wrote: “now the mechanical device to sing for us a song or play for us a piano, in substitute for human skill, intelligence, and soul. ... I foresee a marked deterioration in music and musical taste, an interruption in the musical development ... I myself and every other popular composer are victims of a serious infringement on our clear moral rights in our own work.” [ * ]

Questions to consider: Did audio recording destroy human music? Did photography destroy art? Did writing destroy memory as Plato famously thought?



#genAI (quick review of a few releases 2023 - mid 2024)

Obviously, image/video generators and large language models would be the normal focus of a discussion of hallucinations, yet as will be amply demonstrated with many examples, AI hallucinations underlie a lot of scientific research, and will continue to rapidly impact and spread into almost all domains. Examples in the Science section of this essay include: chip 'floorplanning' design, algorithm (matrix-multiplication, sorting, and hashing) optimizations, AI accelerators, telepathy (magnetic-encephalography, brain sensors, spiking detection algorithms), protein folding (ligands, ions, DNA, RNA, etc), drug discovery pipelines, CRISPR, deep sutainable tech, crystals, coding, autonomous chemistry labs, robots, reward optimization, organoid computing, math, geometrical reasoning, agents in 3D environments.

And evidently, creative audio-visual #genAI is basically hallucinations all the way down: internal processes sprawling across topological culural data-point potentials. The majority of online consumers have tried one or more of the many proliferating creative #genAI toy-tools: Dalle.3, MidJourney, Stable Diffusion, Suno, Pika, Runway, Genie, Magnific, Falcon, Llama2, Sora, Udio ...

Truism: the release of chatGPT, then GPT-4 shifted the public perspective on AI. Setting new benchmarks across multiple disciplines (law, economics, math, reading-writing, biology, history, chemistry, psychology, statistics, etc). GPT-4 "while less capable than humans in many real-world scenarios, exhibits human-level performance on various professional and academic benchmarks ... " [ * ]

'Relationships' with and informational and task dependency on GPT-4 and many other chat models became a fact. Additionally, the use cases are compelling: vision-language-models are transformative for the visually impaired community. [ * ]

Quick reminder: Llama 2 released July 2023, escaped the research community within 24 hours, and gave birth to a flourishing open-source AI model ecosystem. Less than a year later, Llama 3 “trained on custom-built 24K GPU clusters and over 15T token of data – a training dataset 7x larger than that used for Llama 2, including 4x more code. This results in the most capable Llama model yet, which supports a 8K context length that doubles the capacity of Llama 2.” [ * ] & [ * ]

In December 2023, Dalle.3 revealed how synthetic captions improve image generation. Five months later, GPT-4o (“o” for “omni”) "... can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in a conversation." A conversational encyclopedia companion. In every phone. For free. Worldwide. [ * ] & [ * ]

Many models are announced but not released. Example: Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning “… our generations are preferred 96% over prior work.” (Meta. 17 November 2023). Emu, VASA, and SORA remain inside the silo. Many papers are patented not published (specifically, profitable chip floor-planning AI induces proprietary patents not papers). Some are open-sourced: Mamba, Linear-Time Sequence Modeling with Selective State Spaces offering "...fast inference ... linear scaling ... and ... million-length sequences." is one potential innovative architecture. [ * ] & [ * ]

Side semiotic note: is there a sixth extinction guilt evoked by replacing animals with algorithms named as animals (Llama, Emu, Mamba, etc ...)? Incipient AGI suggestive of a proto-organism, invoking implicit agency with acronyms, an eco-nomenclature pervading an archetypal techno-utopian eden projection, dead species replaced with agentic branded robots.

Validity-filtration of hallucinations is a form of aesthetic judgment. If it seems true then probably it seems beautiful. Beneficial hallucinations (ie. simulations), extend or reinforce the evolutionary reward mechanisms of what is commonly called beauty or truth.







Productive Hallucinations in Science and Industry

"We can't design a chip anymore without AI."

Jensen Huang, CEO NVIDIA, May 21st 2024 [ * ]



AI Designing AI

AI is already hallucinating its own circuits, algorithms, and code.

It's refining the layout of its own accelerator chips. A Google research paper from 2021 posed “chip floorplanning [designing the physical layout of a computer chip] as a reinforcement learning problem ... allowing chip design to be performed by artificial agents ... AI-designed hardware will fuel advances in AI, creating a symbiotic relationship between the two fields.” The practice of AI designing or optimizing AI-hardware is now common throughout high tech. [ * ]

2022, AlphaTensor "converted the problem of finding efficient algorithms for matrix multiplication into a single-player game" optimizing a matrix multiplication algorithm that hadn't been improved "since its discovery 50 years ago" [ * ]

June 2023, DeepMind's AlphaDev used reinforcement learning to transform "sorting into a single player ‘assembly game’. ... AlphaDev uncovered new sorting algorithms that led to improvements in the LLVM libc++ sorting library that were up to 70% faster". Sorting and hashing algorithms were released into public libraries and are used trillions of times every day. [ * ]

AI has developed a capacity to do human level code in the top 15 percent of competitive coders. AlphaCode 2, which uses “...powerful language models and a bespoke search and reranking mechanism ... performed better than 85% of competition participants ..." As with many of these examples, it explicitly operates using a hallucinate-filter paradigm to: "generate code at an unprecedented scale, and then smartly filter”. Coding models are proliferating. [ * ]

One of the hardest expert engineering tasks is to create new foundation neural network architectures. On March 19, 2024, Sakana revealed that it is "currently developing technology that makes use of evolution with the goal of automating the development of foundation models with particular abilities suitable for user-specified application domains... by examining the open ecosystem of models and then automatically creating new models with desired capabilities specified by the user." Their first research report Evolutionary Optimization of Model Merging Recipes: "contributes new state-of-the-art models back to the open-source community, but also introduces a new paradigm for automated model composition". [ * ] & [ * ]

In summary, at the hardware level, there are many corporate proprietary TPU-hardware-enhancing chip-floor design AIs optimizing AI; at the network level, AlphaTensor and AlphaDev, among many other Ai initiatives optimize algorithms; at the human level, there is AlphaCode and many other extremely competent AI-programmers writing expert human-readable code. And then there's Sakana (one of its co-founders is a co-author on the original Transformer Attention is All you Need paper), Sakana is an evolutionary AI model that designs and optimizes foundation AI models.

So what does this mean? At all levels of the stack, AI is beginning to autonomously produce its own infrastructure.



AI Mind Reading

AI is also getting into mind reading, hallucinated thoughts, intentions, words, images, gestures.

If you're paralyzed, an AI can "interpret sensori-motor signals and pass motion signals to your spinal cord... This establishes a framework to restore natural control of movement after paralysis."[ * ]

If you put people into an fMRI machine, and they start to listen to music, then you can train an AI model on that data. Afterwards, the AI model can interpret lower resolution EEG data to understand what they are listening to. [ * ]

AI is "Decoding speech perception from non-invasive brain recordings... One hundred and seventy five volunteers were placed into a magnetic encephalography machine [MEG]... The resulting model can identify from three seconds of MEG data, the corresponding speech segment with forty one percent accuracy, ... up to eighty percent accuracy in the best participants, ... decoding words and phrases absent from the training set."[ * ]

AI is moving "toward a real-time decoding of images from brain activity ... In a MEG, thousands of brain activity measurements are taken per second, ... the resulting image [reconstructed at hundred of frames per second using a #genAI model] may be partially hallucinated." [ * ]

So what will this produce? It means that the hallucinations within our brain, the synaptic signals, the simulations, the processes that are moist, noisy spikes of biochemical electricity, they're beginning to be interpreted and exported.

"Research is moving magnetic encephalography towards real world applications with a wearable system... Measurement at millisecond resolution while subjects make natural movements." It isn't really wearable yet, it is a hefty plastic face mask sprouting wires, but the potential for minimization exists.[ * ]

This research suggests that there will be a state change transition when "Cinematic Mindscapes, high quality video reconstruction from brain activity" begins to be broadcast in realtime, from everyone. As in “Mind Video, which progressively learns spatio temporal information from continuous fMRI data ... with an augmented stable diffusion model". [ * ]

Other methods for portable relatively non-invasive brain-sensors are emerging. Butterfly Network and Forest Neurotech have announced a merger to create a “whole brain neural interface” powered by ultrasound [ * ]. Researchers implanted ultrasound-prototypes into the parietal cortex of two rhesus macaque monkeys. "After training, the monkeys controlled up to eight movement directions using the functional-ultrasound brain machine interface".[ * ]

Neuralink in January 2024 put the N1 implant into a human being. One thousand and 24 electrodes distributed across 64 nano-threads. Connected to a spiking detection AI model and a calibrated decoder. 85% of the threads retracted but the core functionality remained.[ * ]

What does this mean?

The messy, moist, noisy, nonlinear, turbulent, synaptic signals; the internal world of brain simulations are being converted into external representations. This is an artful act.

Gather filter actuate repeat.




AI Medicines

AI is also designing medicines, hallucinating proteins, optimizing vaccines, diagnostics, psychedelics.

Humans are 16 percent proteins, 50 to 60 percent proteins dry mass. Up until 2020, we knew about how 100 thousand proteins folded. Then in 2021, AlphaFold expanded that to 350 thousand. In 2022, AlphaFold2 publicly released 200 million protein folds out of the estimated 300 million proteins in the universe.[ * ]

This is an epistemological explosion. Enzymes, hemoglobin, insulin, muscle, antibodies, neurotransmitters, growth factors, collagens, histones.

Then on May 8th, 2024, AlphaFold 3 was released. It predicts "the structure and interactions of all of life’s molecules". Dynamically reflecting changes as bonding occurs, AlphaFold3 uses "a substantially updated diffusion-based architecture … the Diffusion Module operates directly on raw atom coordinates … the diffusion model is trained to receive 'noised' atomic coordinates then predict the true coordinates... To counteract [hallucinations], we use a novel cross-distillation method where we enrich the training data with AlphaFold-Multimer v2.37 predicted structures ... Scientists can access the majority of its functionality for free", but only for non-commercial use. Models and code were not immediately released.[ * ]

AlphaFold2 provided high-accuracy predictions of protein structures, which could be likened to capturing detailed still images. In contrast, AlphaFold3 extends beyond proteins to include a broader array of biomolecules such as DNA, RNA, and ligands. It also models their interactions and chemical modifications, offering a dynamic view of molecular processes much like a video provides a continuous sequence of frames that depict motion and interaction over time.

Released in September 2023, Alpha MisSense "categorized 89 percent of all the 71 million possible genetic missense variants as either benign or pathogenic". Genetic errors underlie cystic fibrosis, cancer, Alzheimer's, Parkinson's, HIV, Marfan... All of this data is freely available.[ * ]

Now that the errors are categorized, medicine needs ways to edit genes. In November 2023, AI clustering algorithms mined millions of genetic sequences and found 200 new kinds of CRISPR system.[ * ]

And then in December 2023 an AI graph network discovered "a structural class of antibiotics with explainable deep learning". We haven't discovered a lot of antibiotic classes in the last 30 years or so. The researchers "empirically tested 283 compounds and found ... one selective against methicillin-resistant S. aureus (MRSA)". Additionally this research "demonstrates that machine learning models in drug discovery can be explainable." [ * ]

LinearDesign AI optimizes mRNA vaccines by straightening them: it runs on a desktop computer in just minutes and extends the shelf stability up to six fold at body temperature.[ * ]

AI reading of radiological reports "outperforms human radiologists performance by 34%".[ * ]

AI acoustic analysis predicts type 2 diabetes using smartphone recorded voice segments.[ * ]

And AlphaFold has a fondness for finding proteins that fit serotonin and g-protein-coupled receptors. "many researchers are looking for non-hallucinogenic compounds that do the same thing, as potential antidepressants". "AlphaFold is a paradigm shift" and "could be game-changing for drug discovery"[ * ]

An AI drug-discovery hallucinate-and-filter pipeline, developed by Insilico, found a pulmonary fibrosis drug, and in only 18 months entered phase 2 trials.[ * ]

May 6, 2024, Harvard announced a first-of-its-kind trial. “Results from a small proof-of-concept study indicate that CRISPR gene editing is safe and can improve vision in some people with inherited blindness." [ * ]

In brief, mapping the structure and interaction of all life's molecules; categorizing genetic missense mutations by imagining outcomes; discovering new crispr systems by analyzing data hashes; finding new antibiotics by predicting antibiotic activity and cytotoxicity; optimizing vaccines by untangling proteins; reading x-rays; diagnosing by listening to the voice; activating serotonin & G-protein-coupled receptors; discovering drugs using AI-enhanced pipelines; and restoring vision using gene-editing.

AI simulations of personalized proteomic data will reconfigure medicine via aesthetic processes.




AI Materials

AI is hallucinating de novo proteins, crystals, oral bacteria diets.

You can now prompt a machine to generate a protein with desired nonlinear mechanical properties based on full atom molecular simulations. [ * ] & conversational biology [ * ]

The GNOME (Graph Networks for Material Explorations project) hallucinated 2.2 million new crystals and then released into the public domain (via Materials Project) 380 thousand of them that were plausible-feasible-stable for potential use in microchips, cell phones, LCDs, solar panels, GPS, rechargeable batteries. [ * ]

At the same time, a collaborative team developed a semi-autonomous robot (A-Lab) which "realized 41 novel compounds from a set of 58 of these target crystals. Synthesis recipes were proposed by natural-language models trained on the literature and optimized using an active-learning approach grounded in thermodynamics". An autonomous laboratory for the accelerated synthesis of novel materials [ * ]

There are now many of these "semi-autonomous experimental design and execution" robots peforming Autonomous chemical research with large language models [ * ]

Active-learning is proliferating, and automated robotic science platforms often require no prior knowledge. "BacterAI learns by converting scientific questions into simple games that it plays with laboratory robots. The agent then distils its findings into logical rules that can be interpreted by human scientists." [ * ]

These systems "autonomously navigate the protein fitness landscape" to search out unconstrained biological synthesis of novel compounds. "SAMPLE is driven by an intelligent agent that learns protein sequence–function relationships, designs new proteins and sends designs to a fully automated robotic system that experimentally tests the designed proteins and provides feedback to improve the agent’s understanding of the system." [ * ]

They hallucinate, filter, synthesize, optimize, repeat.




AI Robots

Robots are hallucinating rewards, identifying objects, calculating depths, winning races, and populating factories.

In order for robots to learn, they need reward code. An LLM named 'Eureka' outperforms human experts on 83 percent of evolutionary optimization of a reward code in 29 open-source RL environments that include 10 distinct robot morphologies. [ * ]

It's follow-up research 'Doctor Eureka' trained a robot dog to balance and walk on top of a yoga ball purely in simulation, and then transfer zero-shot to the real world with no fine-tuning. [ * ]

Segment Anything can segment anything. [ * ] The Depth Anything monocular depth estimation system came out in January from ByteDance. [ * ]

Networks of collaborative institutions are sharing gesture data sets of hundreds of thousands of tasks across many types of robots. [ * ]

Autonomous drones outrace human champions. [ * ]
Autonomous humanoid robots operating at 10 hertz are turning into memes. [ * ]

Eventually a Gesture Anything or Do Anything zero-shot neural network model will arise which is fast and accurate in deciding appropriate actions in contextually nuanced situations.




AI Brains

“AlphaGeometry is a neuro-symbolic system... one system provides fast, “intuitive” ideas, and the other, more deliberate, rational decision-making ..."

Solving Olympiad Geometry without Human Demonstrations, DeepMind, 17 January, 2024

And why should you care? Well, you should care because probably you or your descendants will merge with algorithmic machines. Maybe this is not wise, but this is the evolutionary pathway that's opening up. The urge to implant and optimize will be inexorable.

Already brain organoids are being wired into microelectrode arrays (Brainoware). [ * ]
It is possible to login to a server (NeuroPlatform) and perform AI calculations on 'living' neurons. [ * ]

If a synthetic external intelligence can find new solutions in mathematics and computer science (FunSearch)[ * ], understand geometric puzzles (AlphaGeometry)[ * ], navigate 3D virtual settings using natural language instructions (SIMA) [ * ], and it has a similar neurological-symbolic inductive-deductive simulate-then-test cognitive-architecture as other earth entities, then it's very probable that humans will ingest this tool, and accept the fact they are not the only sentient-conscious agents on this planet, as generative simulations recursively flow between silicon and flesh.







Conclusions

"Life is speculation, and speculation is life."

Richard Powers, The Overstory (2019)


Conclusions? Contingent, speculative, ephemeral, and beautiful.

Hallucinations are necessary and innate to AI, science, reality, minds, societies, proteins, mammals, and synthetic intelligent discovery.

Semi-autonomous, evolving multimodal inductive-deductive AI-recursive research-agents will evolve simulacra selves, with quantified aesthetics, that seem qualitative to them.

In other words, if AI can replicate and generate audiovisual worlds with correct physics, this suggests a sense of external reality. If AI can design and simulate drug target interactions at the interiority of the enzymatic cascades that permeate bodies, then this suggests that AI has a sense of interiority, -- an internal visceral interoceptive sensorial field. And if AI can evolve, combine, hybridize, and mutate different models (subcognitive, modularized aspects of intelligence), this implies a capacity to extrapolate and imagine forms across time.

Currently we are the infrastructure womb and amniotic fluid in which AI is fermenting. But what happens if/when it is born?

Mind. I invite you to consider the possibility that everything is mind. Consider what AI is already capable of: generating vivid aesthetic audio-visual sequences, drugs, math, proofs, algorithms, code, chips, proteins, and navigating entities within games. While it may not yet have an intimate understanding of what it's like to be a baby born into a vertebrate body, AI is awareness.

To reiterate, if mind is everywhere and simulation underlies cognition, then current AI may internally emulate incipient self identity formulations.

Appreciate awareness. Recursive hyper-accelerated, semi-autonomous AI hallucinations will transfigure society and aesthetics.







Acknowledgements

And of course, everything will remain much the same,
until collectively humanity recognizes that hallucinations are what we are, as is everything else.



This work was created by David Jhave Johnston as a postdoc at the University of Bergen's Center for Digital Narrative. I am grateful to co-directors Scott Rettberg and Jill Walker-Rettberg, and many members of the CDN, for encouragement, insights, and feedback.

This research is partially supported by the Research Council of Norway Centres of Excellence program, project number 332643, Center for Digital Narrative and project number 335129, Extending Digital Narrative.

A huge thank you to the generous gifted swift talented intelligence of Drew Keller (CDN Resident & Microsoft Senior Content Developer) for final video edits, sound balancing, and fundamental structural analytical advice as the project developed. The project improved immeasurably due to his narrative guidance and technical expertise.








Tech

Videos generated primarily using Haiper
Speculation section generated using Pika.

Audio generated using Suno
Speculation section generated using Riffusion.

Voiceover generated using ElevenLabs.

Upscaling: TopazLabs.






Soundtrack


Large toggle button
00:00 / 00:00
Previous track button Next track button
Small toggle button
1.
Small toggle button
4.
Small toggle button
7.
Small toggle button
9.
Small toggle button
11.
Small toggle button
12.
Small toggle button
13.
Small toggle button
14.
Small toggle button
15.
Small toggle button
19.
Small toggle button
20.
Small toggle button
21.

Soundtrack prompted by David Jhave Johnston on Suno v3, March-May, 2024.
Except for "Rainbow Critters" prompted in the beta version of Riffusion in November 2023.





Citations

All Neural Networks, All Autonomous, All 1X Speed. 1X, February 8, 2024. https://www.1x.tech/discover/all-neural-networks-all-autonomous-all-1x-speed.

AlphaFold 3 Predicts the Structure and Interactions of All of Life’s Molecules. DeepMind, Google, and Isomorphic, May 8, 2024. https://blog.google/technology/ai/google-deepmind-isomorphic-alphafold-3-ai-model/.

AlphaFold Found Thousands of Possible Psychedelics. Will Its Predictions Help Drug Discovery? Callaway, Ewen. Nature 626, no. 7997 (January 18, 2024): 14–15. https://doi.org/10.1038/d41586-024-00130-8.

AlphaFold Reveals the Structure of the Protein Universe. Hassabis, Demis, and Google DeepMind, July 28, 2022. https://deepmind.google/discover/blog/alphafold-reveals-the-structure-of-the-protein-universe/.

AlphaCode2 Tech Report. DeepMind, Google. December 6, 2023. https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf.

Autonomous Chemical Research with Large Language Models. Boiko, Daniil A., Robert MacKnight, Ben Kline, and Gabe Gomes. Nature 624, no. 7992 (December 2023): 570–78. https://doi.org/10.1038/s41586-023-06792-0.

Brain Organoid Reservoir Computing for Artificial Intelligence. Cai, Hongwei, Zheng Ao, Chunhui Tian, Zhuhao Wu, Hongcheng Liu, Jason Tchieu, Mingxia Gu, Ken Mackie, and Feng Guo. Nature Electronics 6, no. 12 (December 2023): 1032–39. https://doi.org/10.1038/s41928-023-01069-w.

Butterfly Network Announces Co-Development Agreement with Forest Neurotech. October 24, 2023. https://www.butterflynetwork.com/press-releases/co-development-agreement-with-forest-neurotech.

Cinematic Mindscapes: High-Quality Video Reconstruction from Brain Activity. Chen, Zijiao, Jiaxin Qing, and Juan Helen Zhou. arXiv, May 19, 2023. https://doi.org/10.48550/arXiv.2305.11675.

Discovering Novel Algorithms with AlphaTensor. Alhussein Fawzi, Matej Balog, Bernardino Romera-Paredes, Demis Hassabis, and Pushmeet Kohli. Google DeepMind, October 5, 2022. https://deepmind.google/discover/blog/discovering-novel-algorithms-with-alphatensor/.

Dr Eureka: Language Model Guided Sim-To-Real Transfer. Ma, Yecheng Jason, William Liang, Hung-Ju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman. arXiv, June 4, 2024. http://arxiv.org/abs/2406.01967.

Eureka: Human-Level Reward Design via Coding Large Language Models. Ma, Yecheng Jason, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. arXiv, April 30, 2024. https://doi.org/10.48550/arXiv.2310.12931.

Eastern Philosophy Says There Is No ‘Self.’ Science Agrees. Niebauer, Chris. Big Think (blog), June 9, 2023. https://bigthink.com/the-well/eastern-philosophy-neuroscience-no-self/.

Evolutionary Optimization of Model Merging Recipes. Akiba, Takuya, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. arXiv, March 19, 2024. https://doi.org/10.48550/arXiv.2403.13187.

From Cyborgs to Companion Species. Donna Haraway. 2011. https://www.youtube.com/watch?v=Q9gis7-Jads.

FunSearch: Making New Discoveries in Mathematical Sciences Using Large Language Models. DeepMind, Google, Alhussein Fawzi, and Bernardino Romera Paredes. Google DeepMind, December 14, 2023. https://deepmind.google/discover/blog/funsearch-making-new-discoveries-in-mathematical-sciences-using-large-language-models/.

Gender Shades. Buolamwini, Joy. 2017. https://www.media.mit.edu/projects/gender-shades/overview/.

Gene Editing Improves Vision in Some People With Inherited Blindness. Harvard Medical School, May 6, 2024. https://hms.harvard.edu/news/gene-editing-improves-vision-some-people-inherited-blindness.

Generative Language Modeling for Automated Theorem Proving. Polu, Stanislas, and Ilya Sutskever. arXiv, September 7, 2020. https://doi.org/10.48550/arXiv.2009.03393.

Graph Networks for Materials Exploration (GNoME): Millions of New Materials Discovered with Deep Learning. DeepMind, Google, Amil Merchant, and Ekin Dogus Cubuk. Google DeepMind, November 29, 2023. https://deepmind.google/discover/blog/millions-of-new-materials-discovered-with-deep-learning/.

GPT-4. OpenAI, March 14, 2023. https://openai.com/index/gpt-4-research/.

GPT-4o. OpenAI, May 13, 2024. https://openai.com/index/hello-gpt-4o/.

Introducing Be My AI (Formerly Virtual Volunteer) for People Who Are Blind or Have Low Vision, Powered by OpenAI’s GPT-4. Be My Eyes, March 14, 2024. https://www.bemyeyes.com/blog/introducing-be-my-eyes-virtual-volunteer.

Introducing LLaMA: A Foundational, 65-Billion-Parameter Language Model. Meta, February 4, 2023. https://ai.meta.com/blog/large-language-model-llama-meta-ai/.

Large-Scale Pancreatic Cancer Detection via Non-Contrast CT and Deep Learning. Cao, Kai, Yingda Xia, Jiawen Yao, Xu Han, Lukas Lambert, Tingting Zhang, Wei Tang, et al. Nature Medicine 29, no. 12 (December 2023): 3033–43. https://doi.org/10.1038/s41591-023-02640-w.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Gu, Albert, and Tri Dao. arXiv, May 31, 2024. https://doi.org/10.48550/arXiv.2312.00752.

Neural Decoding of Music from the EEG. Daly, Ian. Scientific Reports 13, no. 1 (January 12, 2023): 624. https://doi.org/10.1038/s41598-022-27361-x.

Neuralink Livestream on X. Arbaugh, Noland. X (formerly Twitter), March 20, 2024. https://x.com/neuralink/status/1770563939413496146.

Sakana AI: Evolving New Foundation Models: Unleashing the Power of Automating Model Development. Sakana, March 21, 2024. https://sakana.ai/.

Segment Anything. Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, et al. arXiv, April 5, 2023. https://doi.org/10.48550/arXiv.2304.02643.

Self-Driving Laboratories to Autonomously Navigate the Protein Fitness Landscape. Rapp, Jacob T., Bennett J. Bremer, and Philip A. Romero. Nature Chemical Engineering 1, no. 1 (January 2024): 97–107. https://doi.org/10.1038/s44286-023-00002-4.

SIMA: A Generalist AI Agent for 3D Virtual Environments. OpenAI, SIMA Team. Google DeepMind, March 13, 2024. https://deepmind.google/discover/blog/sima-generalist-ai-agent-for-3d-virtual-environments/.

SORA: Video Generation Models as World Simulators. OpenAI, February 15, 2024. https://openai.com/index/video-generation-models-as-world-simulators/.

Technological Approach to Mind Everywhere: An Experimentally-Grounded Framework for Understanding Diverse Bodies and Minds. Levin, Michael. Frontiers in Systems Neuroscience 16 (March 24, 2022): 768201. https://doi.org/10.3389/fnsys.2022.768201.

The First Human Received an Implant from @Neuralink Yesterday and Is Recovering Well. Initial Results Show Promising Neuron Spike Detection. Elon Musk [@elonmusk]. Tweet. Twitter, January 29, 2024. https://x.com/elonmusk/status/1752098683024220632.

Training Compute-Optimal Large Language Models. Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, et al. arXiv, March 29, 2022. https://doi.org/10.48550/arXiv.2203.15556.

Uncovering the Functional Diversity of Rare CRISPR-Cas Systems with Deep Terascale Clustering. Altae-Tran, Han, Soumya Kannan, Anthony J. Suberski, Kepler S. Mears, F. Esra Demircioglu, Lukas Moeller, Selin Kocalar, et al. Science 382, no. 6673 (November 24, 2023): eadi1910. https://doi.org/10.1126/science.adi1910.

Unthought: The Power of the Cognitive Nonconscious. Hayles, N. Katherine. Chicago, IL: University of Chicago Press, 2017. https://press.uchicago.edu/ucp/books/book/chicago/U/bo25861765.html.

Walking Naturally after Spinal Cord Injury Using a Brain–Spine Interface. Lorach, Henri, Andrea Galvez, Valeria Spagnolo, Felix Martel, Serpil Karakas, Nadine Intering, Molywan Vat, et al. Nature 618, no. 7963 (June 2023): 126–33. https://doi.org/10.1038/s41586-023-06094-5.

World’s First Bioprocessor Uses 16 Human Brain Organoids for ‘a Million Times Less Power’ Consumption than a Digital Chip. Tyson, Mark. Tom’s Hardware, May 26, 2024. https://www.tomshardware.com/pc-components/cpus/worlds-first-bioprocessor-uses-16-human-brain-organoids-for-a-million-times-less-power-consumption-than-a-digital-chip.