It was a heart-warming beginning to an extraordinary judicial hearing, one where a lion of the Australian law was excoriated by legal counsel for Australia's trade unions as unfit to preside over a royal commission.

As some of Australia's best - and best paid - silks took their seats in the hearing room of the royal commission into trade unions, Commissioner Dyson Heydon spoke in his soft cadence to John Agius, acting for the Construction, Forestry, Mining and Energy Union.

"Incidentally Mr Agius, I congratulate you on your speedy return to health. All is fine?" he inquired.

"I'm indebted to the care of medical professionals at Royal North Shore hospital," Agius replied.

"Well, they must be very able people."

Some two hours later, Agius would launch an attack on his credibility that may have made Heydon wish he was still convalescing.

At issue was whether Heydon's plan to deliver the Sir Garfield Barwick Address at a NSW Liberal Party event, for which attendees were invited to make donations to the state election campaign, left him unable to continue as commissioner because it would create an appearance of bias.

Perhaps more pointedly, legal counsel for the unions forensically attempted to discredit Heydon's explanation that he was ignorant the prestigious oration was a fundraiser; that he forgot it was even a Liberal Party occasion and "overlooked" his earlier undertaking that he wouldn't do it at any rate while he was presiding over the royal commission.

$61 million operating budget over two years

$28 million spent to June 30, 2015 (except AFP work)

26 former and current union officials referred to 11 agencies

4 people arrested by TURC police taskforce

441 witnesses given evidence

149 public and private hearings conducted

The political stakes are immense, as are the consequences for the reputation of Heydon. He's a man adored by many as a cultured and erudite individual of the highest integrity, loathed by others as pompous and overbearing, but universally acknowledged as a brilliant lawyer and academic of uncommon ability.

Heydon's exit, if it happened, could also mark the end of the royal commission itself, viewed in some quarters as at the vanguard of the Abbott government's attempts to cruel the Labor Party and its leader Bill Shorten, paving the way for its re-election and delivering an unrecoverable blow to the union movement, the main financial backer of Labor.

Dramatic and compelling, Friday's historic hearing was conducted with the veneer of courtroom decorum but was - nonetheless - a clear attempt to ravage his reputation.

It was also somewhat surreal. As the attacks came, the challenges to Heydon's truthfulness and credibility, they were done through the vessel known as the hypothetical layman, aka the fair-minded observer.

"It just, in our respectful submission, may beggar the belief of the fair-minded observer that … such a matter would have been overlooked," said Agius as he attempted to dismantle the contention that Heydon was unsure he was attending a Liberal Party function, despite discussions going back years and emails headed with the subject matter "Liberal Party Lawyers and Legal Policy branch".

The fair-minded observer is a construct that - as Heydon rules on his own ability to continue as commissioner - he must inhabit.

That neutral person is not a lawyer and he or she is not a random individual pulled off the street for their opinion, either.

"He or she is assumed to have a detailed knowledge of the situation - usually more detailed than appears from media reports - and is slow to jump to conclusions," says Anna Olijynk, a law lecturer at the University of Adelaide.

Or as Robert Newlinds, representing the ACTU, put it to Heydon: "Your own subjective view is irrelevant. The reason I say that is not because you are an irrelevant person, it's because if ever there is someone who ever doesn't fit the description of the reasonable hypothetical lay observer, it is you, Commissioner."

It was a reference to Heydon's glittering career, also referred to by Agius as his "exquisite experience". Stellar student at Sydney's Shore school, brilliant history student, Rhodes scholar and dean of the University of Sydney law school at the age of 30 before rising to become a judge of the NSW Court of Appeal and then the High Court.

The test for Heydon to recuse himself is not that this layperson would be convinced he was biased. Rather it is the "double might" test.

"I only have to persuade you that the hypothetical observer, firstly, might think, and secondly, what he might think is that you might harbour a political prejudice [for wanting to give a speech at a Liberal Party function]," Newlinds told Heydon.

At the heart of the unions' case is Heydon's now infamous aborted plan to deliver the Sir Garfield Barwick Address, exposed last week by Fairfax Media journalist Latika Bourke.

Organised by NSW Liberal Party lawyers, a formal branch of the party, his appearance was first canvassed informally "several years ago".

Heydon agreed in April via an email exchange with the convener, Gregory Burton, SC, that he would make the address in August.

By June, Heydon had been emailed a copy of the invitation, emblazoned with the Liberal Party logo and complete with a form offering recipients to make a donation to the Liberal Party, noting that such donations would have to be declared if they exceeded $12,100.

"All proceeds from this event will be applied to state election campaigning," the invitation said.

Heydon's explanation is that he "overlooked the connection between the person or persons organising the event and the Liberal Party", something he had also been previously aware of.

While Heydon is undoubtedly a very busy man, the oversight is surprising, not least because Sir Garfield Barwick is a Liberal Party legend, a former attorney-general under prime minister Sir Robert Menzies who later became chief justice of the High Court, when he advised governor-general Sir John Kerr that the dismissal of the Whitlam Labor government was constitutional.

More surprising still is that a stream of emails presented to the commission repeatedly mention the Liberal Party connection, typically in the subject heading but also in the body of the email.

Lawyers for the unions made much of the point that when Heydon's assistant emailed the convener of the function, Burton, on Thursday last week, just 12 minutes before he received a media inquiry from Fairfax, it was purporting that he was still unsure if it was a Liberal Party function.

"If there was any possibility the event could be described as a Liberal Party event, he will be unable to give the address, at least while he was commissioner," it read.

This despite the fact that the email was responding to one from a day earlier that had clearly highlighted the nature of the event.

As well as a faulty memory, Heydon cited the fact that he had not examined the invitation sent to him, which was in an attachment to an email from Burton that began "I thought you would like a copy of the formal invitation…".

"My understanding at all times has been that the dinner was not to be a fundraiser," Heydon told the commission this week.

On Friday, Newlinds and Agius countered that, even if he hadn't read the invitation, he surely would have noted that the attachment was headed "state political donations". This included the one sent on Wednesday last week by Burton.

The attachment was not included in the original documents presented to the court - prompting Agius to make the sensational allegation the email had been "doctored".

The allegation was later withdrawn after counsel assisting the commission, Jeremy Stoljar, SC said it was an oversight, rather than deliberate.

Even so, Agius maintained an attachment to an email should have been produced by the commission and there is "no explanation" for why it was not, although he was not blaming Heydon personally.

A black letter lawyer

Ian Callinan, who sat on the High Court with Heydon, describes Heydon as "absolutely punctilious", awe-inspiringly intelligent, and a man of "conservative social views but apolitical".

"He consistently demonstrated complete independence on the High Court," he told Fairfax Media.

In his own words, Heydon favours "beautifully clear, precise, black letter propositions" when it comes to interpreting the law. He is contemptuous of activist judges who apply modern values and concepts such as human rights to override the letter of the law.

Friends and colleagues believe the political atmospherics won't sway a black letter lawyer like Heydon.

But lawyers for the union say that the intense politics surrounding the royal commission mean that negotiating to deliver the Sir Garfield Barwick Lecture - indeed, still vowing to do so once the commission has ended - leaves him fatally compromised

University of Sydney law professor Ben Saul agrees it's a relevant factor.

"It's a super politically charged inquiry," he says. "From the outset, it's been perceived by a large segment of the community as one side of politics attacking the other. This context is important.

"The commissioner has to be super squeaky clean. They have to be absolutely beyond reproach."

If Heydon does recuse himself, the royal commission enters truly unchartered waters.

"It's tricky, tricky territory," said one observer.

Agius made it apparent that, if Heydon does go, the CFMEU will likely push for the royal commission to be wound up. Newlinds indicated the ACTU was still considering the matter but may take the same route.

Shorten, however, has said he would be happy for the commission to continue its work if Heydon is removed.

There are also questions about what would happen to 26 individuals referred to police or prosecutors by the commission, or as a result of its work.

Even so, no evidence before the commission, or finding made by it, can be used in a court of law, so any prosecutions may not be considered to be prejudiced by the disqualification of Heydon.

If Heydon refuses to recuse himself as commissioner, a whole other set of problems - and contingencies - would arise.

First of all, Heydon's admissions about his faulty memory and his failure to read the crucial attachments sent to him in emails leaves it open to future witnesses to come up with similar excuses.

The unions could also appeal to either the Federal Court or High Court, or both - potentially forcing proceedings to grind to a halt for months.

But such a strategy contains high political risks for Shorten and the union movement.

When the CFMEU went to the Federal Court to have Terence Cole dismissed from the Building Industry royal commission in 2002, it failed.

In a bizarre irony, the CFMEU union official who led that challenge against Cole - Andrew Ferguson - was in the witness box on Monday when Heydon adjourned proceedings to first explain the circumstance surrounding his non-appearance at the Barwick oration.

"It's much better for Labor if Heydon stays. If he won't recuse himself, they have a much better case to say the commission is tainted in the long run. If other courts back Heydon, that argument isn't as strong," one legal observer said.

The Abbott government, too, insists Heydon - a "national treasure" - should not bow to what it deems outrageous bullying by Labor and its union supporters. It would be deeply embarrassed if Heydon ruled he should be disqualified.

Despite the fulminations and protestations of Labor, there may be an odd - and rare - confluence of interests among the political parties that Heydon remains as commissioner.

Heydon will deliver his judgment on Tuesday.

"What I'm going to do is to commence work on a consideration of this interesting and in some respects complex matter," he says.