Most directors build their genre movie by coming up with a story, or at least a concept. But when the Americana-flavoured auteur Jeff Nichols first came up with the idea for his new film, Midnight Special, he paid protocol no mind. The movie is a sci-fi chase piece that evokes the pop entertainments a 1980s-era Steven Spielberg used to make. Yet plot and hook were distant notions.

"The first time Jeff mentioned Midnight Special to me, he described it as a chase film where I'd be driving a cool old car late at night – and that was about it," says Michael Shannon, Nichols' frequent collaborator and a star of Midnight Special. "He saw the film very clearly even without knowing what it was about."

Nichols smiles when asked if he tossed the story out of an airlock. "I didn't create this to have a kind of plot concision. That's just never been a priority," he says.

Nichols, 37, has long been about filling familiar vessels with unfamiliar materials, becoming the kind of accomplished and distinct American voice people like to complain doesn't exist much in film anymore.

In 2011's Take Shelter, Nichols crafted an apocalypse drama that was less about the disaster and more about the waiting. His follow-up, Mud, was a child-adventure movie in the spirit of The Goonies – but designed to feel like Mark Twain. Over the course of this work, he has become a kind of unique hybrid, drawing from a long tradition of American storytelling but combining it with a modern independent sensibility that eschews back story and embraces minimalism.

Yet the writer-director has perhaps never worked in a genre so recognisable, nor ended up with a piece so thoroughly subverting of that genre, as he has in his new work.

Midnight Special begins at a cult, as a possibly prophetic child named Alton (Jaeden Lieberher) is kidnapped. It turns out he has been not so much taken as taken back by his father, Roy (Shannon), and father's friend Lucas (Joel Edgerton). Soon the chase is on. The trio is joined by Roy's ex-wife, Sarah (Kirsten Dunst), holing up in a motel room as the FBI bears down. Also pursuing them is a government agent (Adam Driver) with his own theory and agenda.

A road picture about a sci-fi exceptionalist, the movie continues to bob and weave, building to a big-canvas, otherworldly confrontation. Echoes of Starman, E.T. and Close Encounters of the Third Kind are evoked – until they aren't. The climactic rendezvous not only leaves questions unanswered but also seems to ask the audience a meta one of its own: Why is it so important for you to know?

At a moment when talented young directors either toil in rarefied indie space or quickly give themselves over to the studio-blockbuster overlords, Nichols is an anomaly: a populist auteur. Or, perhaps, an anti-populist auteur. His movies form recognisable story patterns, feature known stars and, at least in this case, come from a major studio (Warner Bros).

But he is agnostic about plot, draws decidedly outside the screenwriting lines of exposition and payoff and generally crafts movies that defy expectation."Jeff paints half the canvas and then hands the audience the brush," Edgerton says.

Nichols is not trying to challenge viewers, mind you. Not consciously, anyway. He simply has different goals in mind.

"All I want to do is convey one palpable emotion and have it sustain itself through the process of filmmaking. The details are window dressing," he says. "My thought about the third act, whatever the ... that is, is emotional transference. Do you feel something at the end of having spent two hours with these people? That's all I care about.

He adds: "I think that's why some people are put off by my endings. Because I don't really give a ... about endings. My stated goal is to transfer an emotion to an audience."

As the director had lunch, fittingly, at an upmarket deep-fried restaurant in Los Angeles, he was in a state of what might be called activist Zen. He tends to keep an even keel, a mood that suits his Arkansas upbringing and current home in Austin, Texas. But he also carries some resentments, especially after the early industry struggles of his last film, Mud, and the occasional naysaying on this one. "[Mud] kind of sent me in to an existential crisis. It was like 'what if all the fundamental things I believe about storytelling were not true? Do I just not know what makes a good story?' I felt like I would know if something was total crap. And the industry was treating this like total crap."

He returned to Texas that summer and began to write Midnight Special. "With Mud I was checking all the boxes of a classic American story, and no one wanted it. So it was like ... I'm going to write the leanest thing I possibly could. I'm not here to make anyone happy. There definitely was a chip on my shoulder writing this thing."

Mud ended up finding an audience, earned Jeff Nichols a Palme d'Or nomination at the Cannes Film Festival and confirmed his desire not to kowtow.

Nichols says he continues to want to shape movies out of his personal concerns.

He made some of his earlier work to honour his father, who when Nichols was growing up carried a love of classic westerns – the books of Louis L'Amour, the movies of John Ford. Nichols has in turn also sought to grapple with feelings about becoming a father himself. Midnight Special, with its underlying dread of a son who might be taken away at any moment, came when Nichols' own child was ill several years ago and he and his wife had to face the possibility of losing him. (His son recovered and all turned out OK.)

"I think Jeff combines a deep knowledge of cinema with very unique personal experiences," says Shannon, who has had a role in every Nichols film, including his next, a period interracial romantic drama titled Loving that might well upend conventions of that subgenre. "And he kinds of rides on these two tracks, combining genre with the stories of his personal truth."

After Loving, Nichols hopes to make a bikie movie. He has little idea for the plot, of course, but was simply inspired by social truths uncovered in a book of vintage photographs.

"The scary thing is I'm starting to see a personal style developing that I may not have full control over," Nichols says, when asked if – four films in and far from the unknown upstart who caught the industry's eye with Take Shelter – he feels like he wants to change his style or has been cowed by a filmgoing culture that still seems polarised between effects-laden blockbusters and austere indies.

"It's all got me a little intimidated, to be honest," he says. "But I'm trying to see it differently. I'm trying to see it as every time you make a film that's not a total failure, it's a victory."

Los Angeles Times

Midnight Special opens in Australian cinemas on April 21.