Why do we obsess about the end of the world as we know it? The answer may seem obvious: it’s happening. Covid-19 has killed millions of us, and is still spreading. Month by month, the climate emergency accelerates at terrifying speed. Wars and political instability on every continent threaten our fragile civilisation. Even the greatest technological advances of our time come with built-in existential dread. The unleashing of ever more powerful artificial intelligence, we’re told, runs a non-negligible risk of producing some kind of extinction-level catastrophe – the machines might run amok. No wonder young people are depressed about the state of the planet. Most of the time it can seem like we’re all just helpless bystanders, doomscrolling our way to oblivion.

Yet, as Dorian Lynskey argues in his clever and voluminous new book, there’s more to it than that. Yes, we live in perilous times. But the world has always been in a terrible state. What’s different is that we keep increasing our exposure to far-flung horrors. Both the sheer amount of (mainly bad) news that now washes daily over us and the incredible frequency with which we consume it are unprecedented. Every time you refresh your phone, it’s there.

More deeply, though, this outlook is not new at all. There may in fact be a basic psychological explanation for why humans have always been fascinated by the end of days. Everyone knows that they are going to die. But we also all like to think that our particular span on Earth is somehow special. Perhaps speculating about the end of the world is a way of connecting those two things, of creating a grand narrative that aligns what are otherwise two random, unconnected plots. After all, by placing ourselves at the end of the biggest story of all, we simultaneously place ourselves at its centre: Après moi, le déluge.

For centuries, religions had a monopoly on this subject: the end of the world would come about through divine intervention, rather than by human or natural actions. For Christians this became a central dogma. Over more than 20 chapters of bizarre, gore-filled plot twists, the biblical Book of Revelation laid out what was going to happen, and who to look out for in the season finale – the antichrist, the whore of Babylon, the four horsemen of the apocalypse, the seven seals, the number 666, the battle of Armageddon. As George Bernard Shaw once quipped, it reads like “a curious record of the visions of a drug addict” – yet even today the legacy of this imagery remains profound.

Everything Must Go is about how, over the past 200 years, writers and artists have built on this inheritance to create new kinds of non-Christian eschatology. Ever since Lord Byron’s poem Darkness (1816), which dispensed with God, people have been creating secular fictions about the three main non-divine ways in which things might end – the annihilation of the planet, the extinction of humankind or the collapse of civilisation. Movies, radio broadcasts, comic books, pop songs, plays, novels, paintings, television shows, video games – it turns out that these scenarios have inspired a huge amount of detailed invention, mainly for entertainment. We love to wallow in our worst nightmares.

The form that such stories take is always influenced by the scientific and environmental events and concerns of their time. In May 1941, months before the start of the top-secret Manhattan Project, author Robert Heinlein published a story about a clandestine scheme to build a weapon from uranium-235. Many of the project’s own scientists were avid readers of the magazine Astounding Science Fiction, to which he also contributed. “Have you heard of U-235?” a character in one of its stories asked in March 1944; “Who hasn’t?” another replies.

In turn, fictions can affect real life. The novels of Arthur C Clarke inspired new efforts to detect asteroids. Similarly, our current worries about AI cannot but echo imaginary rogue examples like Skynet in the Terminator films, or HAL 9000 in 2001: A Space Odyssey. And, as the scientific outlook changes, so do our fictional priorities. To keep things manageable, Lynskey distinguishes three main epochs (the period from 1816 until 1945, the atomic age and the 21st century), and seven different kinds of scenario – ranging from interplanetary impact to nuclear winter, zombies and climate breakdown.

It’s a long book – not just because the subject is so large, but because Lynskey crams in every interesting fact and obscure example he has come across, and has seemingly never met a narrative byway he doesn’t feel like marching his readers down. His approach also necessitates endless plot summaries of texts and films that sound as if they have been deservedly forgotten. The capaciousness can be wearying; at times, even the author seems overwhelmed by the mass of material he has uncovered.

Perhaps we don’t need to be told the name and exact size of the largest meteorite thus far discovered, or survey the entire oeuvre of the late 19th-century Minnesota politician and crackpot fantasy novelist Ignatius L Donnelly. Then again, probably no serious connoisseur of the end of the world could resist mentioning Rudolph Valentino’s 1921 war movie The Four Horsemen of the Apocalypse, or that the phrase “the second world war” was coined by HG Wells in 1930, nine years before it actually broke out. Despite the occasional longueur, Everything Must Go is so engagingly plotted and written that it’s a pleasure to bask in its constant stream of remarkable titbits and illuminating insights.

• Everything Must Go: The Stories We Tell About the End of the World by Dorian Lynskey is published by Picador (£25). To support the Guardian and Observer, order your copy at guardianbookshop.com. Delivery charges may apply