The first long solitary walk was 10 years ago. On occasional visits to Edinburgh, I’d always been taken by that beautiful stretch north of Newcastle, where the North Sea sidles up and the train seems to skirt the cliff edge, travellers looking up from books and phones, and crossing the carriage aisle to take in the view. The estuary, the town and the beach – it always seemed as if the passengers wanted the train to slow, then stop, so that we might clamber down and walk to the shore. Perhaps not all the passengers, but I certainly did and there seemed to be no reason why this couldn’t be achieved. I looked up the spot on the map, bought a guidebook to the Northumberland Coast Path, measured out distances and put aside four days. I bought new boots and socks, Ordnance Survey maps and an unnecessary compass, because how do you get lost on a coast walk? I bought roll-up waterproofs and dense, futuristic protein bars and too many novels, and on a bright, sharp day in the early summer of 2014, set out, catching the first departure from London to Newcastle, changing for Alnmouth, walking to the shore, then turning left and heading towards Berwick.

It sounds, I realise, like a textbook reaction to middle age, with that obsessive and eccentric quality, a need to achieve something measurable and definable. As midlife missions go, it wasn’t even that impressive or ambitious, hardly the Camino de Santiago or Everest; walking, not running, an almost marathon. Neither was this some new-discovered passion. I’ve been going on long walks for most of my adult life and my children have happy and less-than-happy memories of hikes across fells and moors in rain and snow. The only features that marked out this journey were the distance and the solitude. I had never walked so far nor spent so much time alone. This was new.

Four days of walking, four days of thinking. My father had died the previous August after a period of illness that had stretched into years, the complications and conditions accumulating with each month. It was a relationship that still felt unresolved, though it was hard to imagine what that resolution might have looked like, or how walking and thinking might help. If anything, thinking was the problem: overthinking, circular thinking, the pressure of work and the demands of parenthood, of getting older and stiffer. Stuck, distracted, perpetually exhausted yet unable to sleep, prone to gloom, introspection and withdrawal, to catastrophising and protracted silences. “Crisis” can seem like a melodramatic word for the familiar, predictable preoccupations of midlife, especially when, to an outside eye, things seem to be going well. Work to do and happiness at home; to someone whose life is in turmoil, there must be a kind of luxury in feeling merely “stuck”, and there was no reason to imagine that walking farther than usual might help, or that solitude rather than conversation was the answer. And yet once planned, the journey felt like a compulsion. At the very least, if I walked for long enough I might actually sleep.

And that first walk was wonderful, those vast, unpopulated beaches with castles on the skyline at Dunstanburgh and Bamburgh. I walked all day and stayed the night in pubs and small hotels, eating fried food alone and reading conspicuously (“Who is that mysterious traveller?”), slept well and ate kippers for breakfast, the taste my companion for the day. It rained, of course, late spring squalls, and the latter stages were dull – fine golf courses make for terrible hiking – but I loved the ancient paths, St Cuthbert’s and St Oswald’s Way, taking a diversion off the official route to pad bare-footed across low-tide sands to Holy Island.

I sound like a tourist, I know, with this talk of barefoot pilgrimages and castles in the mist; but I grew up in a southern suburb with no car and no trip north of Milton Keynes until I was 18. There must have been ancient history in my home town, too, but it had long since disappeared beneath housing estates and shopping arcades and light industrial zones, so that the English countryside, the real thing, seemed like another country. I can’t imagine that I was ever more than 200 metres from another human being and so this lonely landscape still felt relatively new, particularly on that eerie, three-mile walk across the wet sands on the Pilgrim’s Way, following the poles and passing the refuge huts high – high enough? – on their stilts, reassuring myself that I’d checked the tide tables, had plenty of time before the sea crept over my ankles, but hurrying on nonetheless. I dried my feet in the grass, walked a circuit of the island, took the causeway back, and on the afternoon of the fourth day, caught the train home from Berwick.

Traditionally, this is where I report some transformation, an insight or sense of resolution. A pilgrimage changes the pilgrim and has a purpose and destination other than a distant railway station, and nothing much had altered, not my feelings about my father’s death, not my regrets or anxieties. No inspiration or great unblocking of emotion, none of the traditional epiphanies, just a pleasant feeling of exhaustion from exertion rather than insomnia, and a suntan solely on the right side of my face.

I’d never been alone for that long in my life and apart from brief calls home and ordering in the pub, I’d spent the days either in silence or muttering to myself about the rain. I am not an antisocial person, have colleagues, friends and family whose company I treasure, and would go so far as saying I have a dread of loneliness that has found its way into everything I write. A long, solo journey should have been a trial and yet I was determined to do it again as soon as possible, and have done it every year since, sometimes twice a year. Distance and solitude, a long walk alone.

* * *

The conditions are always the same, a hike of 60 to 90 miles in early spring or autumn. The paths are well-trodden and unadventurous, taken from a list of routes that end in “way”: the Dales Way, the Cumbria Way, the Dales High Way, the Cleveland Way, the South Downs and Peddars Way, the Ridgeway. I always carry my own rucksack and have never for even one moment considered a tent. Pubs are favoured over B&Bs and I’m now quite comfortable with the smell of beer at breakfast. My diet is a nutritionist’s fever dream of cheap bacon, sweets and crisps, fish and chips, salt and vinegar, so that I’m in a permanent state of dehydration, justifying the calories by the astronomical step-count. “Walking quite some way” is unlikely to be recognised as a sport but it’s the nearest I’ve got to competence in a physical activity and while I like to think there are people who might come with me, being alone allows me to be quite insufferable, boring only myself with maps and routes and schedules, free of the need to make jokes or conversation, or apologise for the weather or the clouds obscuring a view as if they were my fault. I’ve developed preferences, too, the kind of wild, provocative opinions that alienate other hikers, like slagging off a favourite band. Northumbria and Cornwall aside, I don’t like coastal walks. The Dales High Way is far superior to the Dales Way. I have no love for the South Downs. There, I’ve said it.

And I try not to look at my phone but there are other ways in which I wonder if I’m quite “in nature” enough. Rather than the song of the pipit or the wind in the heather, I listen to music, often for hours, strongly flavoured playlists of Kate Bush and Joni, Can and Sondheim and the 20-minute prog rock tracks I loved at 15; music that can clear a room in minutes but which I find comforting, like some terrible foodstuff you associate with childhood but wouldn’t dream of offering to anyone else. Picture me, thirsty and wet on some desolate moor, high on glucose and listening to The Kick Inside with 12 miles still to go. The dream.

At other times, I “read” while I walk. I’ve been lucky enough to adapt some of my favourite novels into screenplays, the first stage of which is to absorb the book, learning its structure and dialogue, something best achieved on the move, listening while I walk, then returning to the beginning and listening again like an actor learning lines. Five hundred pages is roughly 50 miles and now Dentdale will always sound like Patrick Melrose; the Thames Path like Great Expectations.

But I think there has always been a connection in my mind between literature and landscape, beginning with a childhood preference for books with a map at the front, tracking the characters’ progress as if they were really there, like some kind of pre-digital satnav. For a kid who rarely travelled farther than school or the library, I was thrilled by the urgent flight across Europe in The Silver Sword and later loved the improbably long journeys in Victorian novels as characters fled their desperate circumstances. To open the door, walk out and keep going; it was such an appealing idea, that combination of freedom and peril, and I particularly loved the young David Copperfield’s trek from London to Dover, his courage and vulnerability along the way. Dickens himself was an obsessive walker, claiming speeds of 4mph, walking briskly in search of inspiration or, once, as an escape from domestic turmoil, setting off at night to walk from Tavistock Square in Bloomsbury to Gad’s Hill in Rochester, the first leg of Copperfield’s walk, too. At just under 30 miles, I’ve considered it myself, though the thought of the A207 at five in the morning means I’m unlikely to set out any time soon.

But if any author fuelled this desire to walk and walk, it was Hardy. The Brontës’ landscape was wild and dramatic but distant and unknown to a southern suburban boy. “Hardy’s Wessex” was not so far away, my home town squeezing unmarked on to the eastern edge of the map in Penguin editions, that strange parallel realm where Dorchester is Casterbridge but Southampton remains stubbornly Southampton. I’d been on day trips to Bournemouth/Sandbourne, where Tess stabs Alec, and on book-buying trips to the WH Smith in Winchester/Wintoncester, where she meets her terrible fate, though not in Smith’s. Casterbridge was too far away to recreate those journeys. Where could I go if I walked out my door?

I was experimenting with ornithology at the time, lugging immense cast-iron binoculars along the local riverbanks, through business parks and new-build estates, alongside the railway yards, shadowing the commuter line to Basingstoke and Woking, walks that were only fitfully scenic, the hum of the M3 never far away. I was an inept and complacent birdwatcher, easily satisfied with moorhens and wagtails, but these nature walks felt bookish and learned, the kind of thing a writer might do. I read the Romantics, too, disappointed at the lack of romance, stuff about girls. Instead, they wrote about the sublime, nature as a living thing, the power of a landscape to transform and inspire. Perhaps this scrappy, un-sublime, municipal landscape would transform and inspire me. I liked David Attenborough, O-level Biology and early Genesis, was practically a nature poet already, and while I don’t think I actually wrote any poetry, I wouldn’t have put it past me, Keats by way of Adrian Mole.

Distance and solitude. It was, looking back, eccentric to be wandering lonely at an age when sociability and popularity are at a premium, and I did have good friends, more than might be imagined from the above, but no one who wanted to spend the weekends counting chaffinches on brownfield sites. Clearly, solitude suited me and here another reading memory seems relevant, dating from some years earlier when, as a zealous member of the Puffin Club, I tore through Tove Jansson’s Moomin books. The first few volumes felt familiar enough – illustrations, a cast of characters, journeys and adventures – but as the series went on the tone changed and a literal darkness set in, the books now less about family and community, more about isolation, loneliness, the absence of light. Books had made me sad before but never in this rarefied way, not the pathos of a character’s death, Aslan or Hazel, but something more delicate and indefinable. “The faint whisper of rain and running water was still there and it had the same tender solitude and perfection but what did the rain mean to him as long as he couldn’t write a song about it?” This is from Moominvalley in November which I must have read when I was 10. It’s unusual, almost inappropriate, to find such distilled melancholy in the pages of an illustrated book and I don’t think that I was a particularly morose child, yet I still remember the sensation of reading it, remember loving it, too. Company is an intrusion and if being alone is sometimes frightening, then the remedy – the presence of other people – brings its own concerns. It’s an ambivalence I feel on those long walks, too. I relish the solitude but inevitably there are moments when I think, my God, I really am alone, a twist of panic that will inevitably have me reaching for my phone, specifically for social media, its rough approximation of friendship and connection.

Best to resist and just keep walking. Melancholy is a solitary pleasure and something I’ve been drawn to for as long as I remember. Certainly, it’s in my novels which are all, to some degree, romantic comedies about sadness and loss, a feature of the long walks, too, the reason why I don’t mind trudging through drizzle along the verge of a B-road or sitting in empty cafes on grey February afternoons. Extraordinary sunlit vistas are spirit-lifting but I like windblown, out-of-season seafronts, too, infinitely preferable to a Riviera beach in July.

I realise I’m not selling this very well, and perhaps this is the reason why I’ve never had to dissuade my family from accompanying me. I confess I don’t always understand the pull of it myself, except that perhaps melancholy bears the same relationship to depression as solitude does to loneliness, in that it’s a deliberate, self-inflicted state that can be shaken off or will at least pass in time. There’s a route out of melancholy and solitude, whereas the great fear with depression and loneliness is it will never end. We all sometimes crave peace and quiet, but who would ever wish for loneliness?

* * *

Solitude or company, contemplation or conversation. My latest novel, You Are Here, is an attempt to explore the merits of each and its working title, Walking and Talking, is as good a synopsis as any. A comedy in a landscape, it’s a love story about two isolated people, Michael and Marnie, who meet on a weekend hike and decide to keep going, talking, arguing, telling jokes and stories and listening to music along the way. The setting is the Coast to Coast, the famous 192-mile route devised by the great Alfred Wainwright, forming a high belt from St Bees Head in the Western Lakes to Robin Hood’s Bay on the edge of the North York Moors, so that my most recent expeditions have been the longest and must purposeful. Traditionally it takes 12 or so days to cross from west to east, but work and family commitments meant it was undertaken in three parts, also the three acts in the novel, approximating to the Lakes, the Dales and the Moors. If this felt like cheating it was still demanding, beautiful, sometimes boring and desolate. Walking is famously a spur to creativity and I always carry a notebook in the hope that something, the rhythm of boots, a glimpse of the sublime, will unleash some great idea. Until now, I’ve never taken the notebook from my pocket, but I’ve loved writing onto a landscape, matching the sections of the walk to the stages of their relationship, indifference on day one, hostility on day two, curiosity on day three and so on, a novel that comes with maps like the books I loved as a kid, that could even be read on location. Like the best journeys, there are diversions and the final destination is not what’s expected, but it’s a chance to explore these preoccupations with lonesomeness and company, the south and the north, nature and what we hope to take from it.

* * *

I’m aware of the great claims made for the benefits of walking, to mental health in particular, the notion of a walking cure, and I’m sure that’s what I was hoping for on that first long journey, turning north on the beach at Alnmouth, and on the later walks, too, a lifting of the spirits at a dark and disorienting time, the difficult years that bracketed my 50th birthday. I’m also sure that they helped, though I’m wary of overstating their power. I’ve felt profoundly unhappy in places of extraordinary beauty and in the novel, as in my own experience, hope and change seem far more likely to spring from conversation and connection, rather than solitude and landscape. It’s too much to expect happiness solely from a view.

And yet I know that I will get restless and down if I’m in the city too long, that once or twice a year I’ll feel the need to go for a long walk, and that this is best done alone. This is new. Certainly in my 20s, just one evening spent by myself felt like defeat and even though I don’t “love my own company”, I no longer mind it quite so much. The part of me that wants to be alone is the same part that leaves the party too early and delights in the cancelled plan, the part that sometimes forgets the pleasurable challenge of conversation – what Bill Callahan, in one song, calls “the discipline of company”.

Still, it feels important to hold on to this time alone, to walk and think and sit in the solitariness, knowing that after a while the novelty will fade, counting myself lucky that it’s a temporary, voluntary state and that I’ll soon want to return as quickly as I possibly can to the complicated joys of other people.

• You Are Here by David Nicholls is published on 23 April by Sceptre at £20. To support the Guardian and the Observer, buy a copy at guardianbookshop.com. Delivery charges may apply.