As campaign mis-steps go, it was a small but telling one.

Grilled at length in a cafe by a young reporter from digital news-site BuzzFeed on why Labor had supposedly lagged the Greens on support for same-sex marriage, Anthony Albanese's patience ran out when a woman at the next table leaned over to intervene in the exchange.

"Bullshit" he exclaimed – twice – as the eavesdropper hammered him with accusations that his party had come to the issue well after the Greens. "I moved the first private member's bill about same-sex relationships before anyone in the Greens" he insisted. A dig at the reporter's age ("You are young, aren't you") didn't help, as he dished out a quick social history lesson from the late 1990s and early 2000s on the issues that had then most galvanised the LGBTI community.

Several days later, the Labor luminary admits to having had "a difficult conversation with someone". He tells Fairfax Media: "I regret it, full stop. But it is a fact that before 2004, no-one raised with me the issue of marriage equality. The issues being raised by the gay and lesbian rights lobby were different, migration, superannuation equality, visiting rights [for partners of HIV patients] in hospital, a whole range of issues – they were of more practical pressing concern at the time."

Albanese, Labor's shadow minister for infrastructure and cities, is a fighter and his instincts usually serve him well. He breezed through a day on the hustings when Fairfax Media accompanied him recently. Canvassing for votes at Summer Hill railway station in Sydney's inner west, there were a lot of cries of "Hey Albo!" from people he'd never met. He has huge personal recognition. Blame Kevin Rudd, Albanese says. "He once introduced me as Deputy Prime Minister Albo and bang, now its a brand."

Despite the morning rush, several commuters take him on over Labor's asylum seeker policy. He is firm, but civil, taking the time to argue it through with them. But the exchange in the Marrickville cafe hints at how the pressure of the first five weeks of the campaign has been getting to him.

During those weeks he's been staring down a potentially lethal Greens challenge in his inner city seat of Grayndler, after a redistribution earlier this year radically reshaped the electorate, losing him Labor heartland and gaining the more prosperous suburbs of the Balmain peninsula which sit on prime harbourside land next to the CBD. This is territory which the Greens picked up at the last state election, and are now gunning for federally.

Used to being feted as a lion of the Left and champion of progressive causes inside the ALP, Albanese has become infuriated at his portrayal by the Greens as a relative dinosaur on asylum seekers, same sex marriage, and climate change. It's a different battle to those he's been fighting for most of the past 20 years as an MP, a change caught neatly when two teenage boys want to debate him on feminism. "Class is often forgotten as well", he reminds them. "Working class people are discriminated against all the time."

He's still getting the measure of his new electoral turf. "Gentrification is the big thing that's happened," he tells me driving around the older parts of his electorate one morning. "It's not so much Labor people changing their minds, it's Labor people selling their houses and professionals who are more likely to vote Green buying those houses. And one of the things I've found about people who are more likely to be Greens voters, one of the things that makes a campaign difficult, is they don't require the state in their lives. There is no hip-pocket issues, they tend to be just not so concerned about the direct impact the government can make."

Later, at a private meet-and-greet at the Harris Farm family's home in Balmain, Albanese tells his newest constituents "its a demanding community, a politicised community, that's a good thing.... people in this area are passionate about the world, not just themselves."

Deputy Labor leader Tanya Plibersek has been facing a similar challenge from the Greens in her inner-city seat of Sydney, though less so than Albanese or Labor's David Feeney in the Melbourne seat of Batman.

This week Plibersek took her turn at lashing the minor party, declaring them dangerous ideologues who were imperilling, not advancing, the progressive cause in politics.

"Far from making more progressive change possible, the Greens are more likely to prevent real change from happening – because their electoral strategy now relies on preventing the election of a Labor government," she told the McKell Institute in Sydney.

Feeney has had a troubled campaign, dogged by failure to declare a negatively geared property on the parliamentary register, and still faces an uphill battle to retain his seat.

But the Greens' threat to Albanese and Plibersek has largely evaporated thanks to Malcolm Turnbull's pronouncement that the Liberals - after much speculation to the contrary - will preference Labor ahead of the Greens in lower house seats.

Greens candidate in Grayndler Jim Casey, who heads the NSW Fire Brigades Employees Union, admits Turnbull's announcement almost certainly kiboshes his chances in Grayndler. His hope of victory largely depended on picking up Liberal preferences if he'd polled second behind Albanese on July 2.

But Casey insists that the optics of horse-trading between the two major parties will benefit the Greens in the medium term. It will "confirm the position that we've been trying to develop that the major parties are increasingly in lockstep on a whole range of issues, that there really is this Coles and Woolworths model between them," he says.

He cites what he calls a "cosy duopoly deal" between the Liberals and the ALP. This sees Labor giving the Liberals a leg-up over the Nationals in several Victorian and WA seats while reaching a tacit non-aggression pact with the government in lower house seats in South Australia (including Christopher Pyne's), mitigating the threat to the Liberals there from a rampant Nick Xenophon Team.

The sole lower house Greens MP, Melbourne's Adam Bandt, is even more scathing, singling out Albanese for running a "deliberate and coordinated campaign of misinformation" about a Liberal-Greens preference deal which never eventuated (though the idea was floated by Victorian Liberal powerbroker Michael Kroger early in the campaign.)

He warns Albanese the Greens are still gunning for him, and will succeed down the road.

"We'll still be putting the same level of effort and resources into the seat because even if we don't win it this time, a higher vote will send Labor a message and put us in a good position for next time," Bandt tells Fairfax Media.

For all the finger-pointing between Bandt and Albanese, pragmatism rules at the Greens' national office. It quietly announced on Monday it had "come to an arrangement" with Labor whereby it would preference the ALP over the Coalition in the Senate and in "more seats than ever before".

Meanwhile Turnbull and his frontbench are keeping up their attacks on the "Greening of Labor" (the schlock horror video ad released by Treasurer Scott Morrison this week being a prime example) even while they've greatly reduced the Greens' threat to Labor in seats like Grayndler this time around.

The convoluted logic of it all smacks of Through the Looking Glass. But Albanese can probably heave a sigh of relief and get back to what he once said he liked doing best: "fighting Tories".

Follow us on Twitter