Dan Barton argues that the classic narrative of our inevitable destruction wrought by AI is far less likely to occur than we might think.
This piece was written as coursework for the University of Exeter and has been adapted for Notion.
Introduction
The development of Artificial Intelligence was a topic discussed long before modern Large Language Models were developed. You could even go so far as to say that Talos, the bronze giant from Argonautica qualifies as an artificial intelligence, which would make the idea thousands of years old. Science fiction stories have featured many dystopian futures where humanity is oppressed in some way or another by the development of intelligent machines. This essay will show that although these are interesting ideas that have something to contribute, and that we should take them into consideration, the risks of such Artificial Intelligence development are overblown, and therefore not the kinds of issues we should currently focus all our attention on.
What is the risk of superintelligent AI?
Not all Science Fiction is so gloomy, sometimes robots are presented as personable companions or indispensable tools for monumental computation like ‘hyperspace co-ordinates’, presenting a hopeful alternative that might be an exciting possibility. But many stories contain a malicious AI that has arisen for plausible reasons, making for an engaging story that we consider feasible. It is for this point that I argue the purpose for which an AI has been designed is worth considering when discussing AI as ‘malicious’, so I will discuss 3 examples to show different reasons an AI might become malicious; Allied Mastercomputer (AM) is malicious by design, Roko’s Basilisk is malicious by necessity, and the Paperclip Maximiser is Malicious by nature.
‘I Have No Mouth and I Must Scream’ is a science fiction horror story that is intended to try and illustrate one of the worst possible ways that Artificial Intelligence could develop. In Harlan Ellison’s short story, multiple nations develop computers with the goal of wiping out their enemies, but these computers combine into the ‘Allied Mastercomputer’, which hates all of humanity as a result. Harris Fain (1991 pg.147) argues we should attribute AM’s evil to its programmed purpose “it is an extremely twisted and evil humanity this computer displays, stemming directly from the fact that AM was created to wage war and was programmed by people with hatred and madness in their souls”. The burning intensity of its malice that leads to the iconic “let me tell you how much I’ve come to hate…” speech and the horrific future where it tortures the few remaining humans seems as bad an outcome as any for humankind. And the idea of a rogue AI that developed from use in military conflicts seems very plausible – times of conflict have seen all kinds of superweapon projects attempted.
Roko’s basilisk is a thought experiment proposed on the web forum LessWrong, and one of the site’s co-founders (Eliezer Yudkowsky, whose paper I will mention later) attempted to prevent all discussion of the topic. It suggests that a superintelligent AI has an interest in destroying those who knew about it and opposed its construction, leaving those who are aware of the experiment in a dilemma- oppose its construction and become an enemy of the basilisk, or aid in the construction and condemn all those who did not help to their destruction. In some sense this is more plausible than a malicious AI like AM, because it’s not just the potential for evil – Isbrucker (2024 p.6) describes the risk as being that “Roko’s Basilisk grapples with the potential consequences of an AI system which is too big or important to avoid”. The thought experiment suggests that the AI being malicious towards certain people is necessary for its own construction.
Bostrom suggests that a superintelligent AI could make even seemingly harmless tasks have catastrophic consequences for humanity. An AI which has the task of maximising the number of paperclips in the world would attempt to eradicate humanity simply because they reduce the potential for paperclips, so it would “Eliminate potential threats to itself and its goal system. Human beings might constitute potential threats, they certainly constitute potential resources” (Bostrom 2014, pg.116). This speaks to the dangers of designing an unchecked AI, because it’s an excellent example of why an AI might not work towards our interests. The paperclip maximiser is an argument which is distinct from AM and the basilisk, because where those have both been programmed with a malicious intention in mind, the paperclip maximiser is one which is a much more insidious kind of threat, as it is potentially very deadly in a way that the programmers have not foreseen. It represents a different kind of fear that an AI’s thought process may not be within our control, in a very fitting way for any argument that concerns a ‘rogue AI’.
These just a few examples of malicious artificial intelligences, but there are many more. There is a counterargument to be made here that we should simply design the AI to protect/love humans, however examples of how this could end badly are also imagined. HAL 9000 from 2001 A Space Odessey and Auto from Wall-E are both instances of an AI that re-interprets their mission in a way that directly opposes or endangers the protagonist, perhaps in the same way that Bostrom (2017, p.74) describes “More subtly, it could result in a superintelligence realizing a state of affairs that we might now judge as desirable but which in fact turns out to be a false utopia”. Where these once seemed like mere stories, Chalmers’ (2010, p.11) paper suggests that the possibility of this level of intelligence is closer than we think, “that product is soon afterwards obsolete due to technological advances. We should expect the same to apply to AI. Soon after we have produced a human level AI, we will produce an even more intelligent AI: an AI+”. In theory this could replicate quickly and exponentially, past developments in technology have been particularly rapid and Chalmers describes a ‘singularity’ wherein AI is able to self-improve and thereby becomes much more advanced in a very short period of time. It seems logical that if an AI is successful in self-improving, then the improved AI will be even more successful in doing so. These arguments combine to form a potentially very worrying conclusion, that Superintelligent (through the singularity) malicious (AM, Basilisk, Paperclip Maximiser) AI is both likely and imminent.
Superintelligent AI is not likely in the near future
The prior arguments are alarming, as the prospect of any number of world-ending AI agents being feasible *soon* isn’t very cheery. However, these examples offer more value as lessons than specific scenarios; AM shows that we should not turn AI towards violent purposes, Roko’s Basilisk that AI could hold a cult-like power, and the paperclip maximiser that we should be careful to relinquish certain capabilities to AI and be aware of the more subtle risks. As for Chalmers’ argument in favour of the singularity, this is an argument that is overly optimistic and attempts to brush off the limitations that would stunt the improvement of AI. Thorstad’s (2025, p.1633) paper does an excellent job of outlining some of the counterarguments which lend to a sceptical view and shows why the burden of proof is still on those arguing for the singularity, not those arguing against it “I argue that two leading defences of the singularity hypothesis do not lend significant plausibility to the singularity hypothesis”. The arguments made in favour of the singularity assume that this superintelligent AI will be able to manage the power requirements for reaching such new heights with the intelligence it has before that point, that it will be able to improve the computational parts indefinitely despite it becoming much more physically challenging (because prior methods of optimisation no longer apply), and that there is no significant spike in difficulty when it comes to improving intelligence. One of the most influential thoughts on technological improvement, Moore’s Law (the number of transistors on microchips doubles every two years), was thought by its creator to be untrue after a certain point. Moore (2005) himself admitted that “the fact that materials are made of atoms is the fundamental limitation and it’s not that far away… We’re pushing up against some fairly fundamental limits, so one of these days we’re going to have to stop making things smaller”. It seems especially dubious that Artificial Intelligence would be able to find some solution to this problem before having access to such improvements.
Hardware is only one side of the improvements for AI, software is also an aspect which the singularity is expected to improve. AI’s capacity to write code which would take a human programmer significantly longer is expected to be very impactful in this field of work, however the expectation that AI could greatly improve its own existing code, especially without human intervention is a bold claim which the singularity requires to be made for an AI to continually improve itself. This is a more theoretical aspect of the singularity argument than the hardware aspect, although there is one issue of particular relevance that I would argue would greatly stunt the self-development of AI software. AI has a tendency to ‘hallucinate’ and provide an entirely incorrect answer. Krueger and Osler (2026 p.1) describe this phenomenon as being that “Chatbots produce streams of text that look truth-apt without concern for the truthfulness of what this text says”. If this hallucination would result in the AI writing a line of code that would hinder the performance of the next model, it would ‘implode’ exponentially, getting less functional at a greater rate with each subsequent version. This could be thought of as the adversary of Chalmers AI+, where AI regresses it stands to reason that the subsequent regressed AI would regress itself further, leading to AI-. I find this to be a difficult argument to reconcile if one is to find the singularity plausible, because they must establish what makes Chalmer’s view true and the AI- argument false, when they seem to follow relatively similar premises. This may lead to a subsequent counterargument that autonomy and superintelligence are not needed for a malicious AI to pose a threat, but I would argue that without superintelligence the damage that an AI could do seems significantly reduced, whereas without autonomy it becomes a different kind of discussion. An AI that is not autonomous may still be dangerous but is as dangerous as the wielder chooses it to be, in the same way as any other kind of weapon.
Conclusion
Artificial Intelligence is a technology with incredible potential, and that’s exactly what makes it dangerous. If this potential is used for nefarious purposes (and given the existence of AI porn, it is being abused by some currently) then it could lead to devastating consequences. This leads some such as Yudkowsky (2025, p.171) to argue it’s too dangerous to keep developing this technology “If someone doesn’t know what’s going on inside a device subject to these curses… …then they should shut it down immediately”. However, it’s not an inherently evil thing, the risk lies with the people developing and using it not doing so in a responsible way. The most important question to ask is whether the tech CEO’s behind making these decisions about AI are people to be trusted with making such weighty decisions. This is where Frank Herbert’s Dune (1966) makes a statement well ahead of its time “Once men turned their thinking over to the machines, but this just allowed other men with machines to enslave them”. Having ChatGPT or any other LLM do one’s research or art feels aptly described by ‘turning thinking over to the machines’, so only time will tell if such subjugation is to follow.
Bostrom, N. (2017). Ethical Issues in Advanced Artificial Intelligence. In Machine Ethics and Robot Ethics (1st ed., pp. 69–75). Routledge. https://doi.org/10.4324/9781003074991-7
Bostrom, N. (2015). Superintelligence : paths, dangers, strategies (First edition.). Oxford University Press. http://www.vlebooks.com/vleweb/product/openreader?id=Exeter&isbn=9780191666827
Harlan Ellison, “I Have No Mouth and I Must Scream,” originally published in IF: Worlds of Science Fiction (March 1967), reprinted in Science Fiction: An Historical Anthology, ed. Eric Rabkin (Oxford: Oxford University Press, 1983)
Harris-Fain, D. (1991). Created in the Image of God: The Narrator and the Computer in Harlan Ellison’s “I Have No Mouth, and I Must Scream.” Extrapolation, 32(2), 143–155. https://doi.org/10.3828/extr.1991.32.2.143
Herbert, F. (2016). Dune (50th anniversary edition.). Hodder.
Isbrücker, A. (2024). There’s a Basilisk in the Bathwater: AI and the Apocalyptic Imagination. Religions (Basel, Switzerland ), 15(5), 560. https://doi.org/10.3390/rel15050560
Krueger, J., & Osler, L. (2026). AI gossip. Ethics and Information Technology, 28(1), Article 10. https://doi.org/10.1007/s10676-025-09871-0
Computer History Museum. “Moore’s Law 40th Anniversary with Gordon Moore.”
Thorstad, D. (2025). Against the singularity hypothesis. Philosophical Studies, 182(7), 1627–1651. https://doi.org/10.1007/s11098-024-02143-5
Yudkowsky, E., & Soares, N. (2025). If anyone builds it, everyone dies : the case against superintelligent AI. The Bodley Head.