Kevin was jittery. Over the previous couple of weeks, media reports had suggested his leadership was beginning to wobble. Conversely, I had received some praise because, on the whole, the budget had been well received for the impressive fiscal consolidation it set out. That morning, journalist Laura Tingle wrote that I had "emerged as a reliable anchor" and "a power in [my] own right within the government", and that there was "a lot of commentary" about my "growing confidence". Similarly, Laurie Oakes had also reflected that if Kevin were to hit the fence as prime minister, I had emerged as a possible successor. These comments clearly hadn't been missed by Kevin and his staff.

The reality is that there was bugger-all chance I would have challenged him. Absolutely zero. I held no burning ambition to be PM and, beyond that, unseating a prime minister would be a horribly messy business - too bloody and too brutal so close to an election. And if I knew one thing about Kevin, having seen him up close for more than 20 years, it was this: he wouldn't go quietly. I also believe that at this point Julia held no immediate ambition for the top job, either.

Then Kevin asked me the question: "Are you with me?" His direct manner was mildly surprising. I replied "I am" and let my answer hang in the air as I gave him a reassuring nod. I deliberately resisted being any more effusive than that. I wanted Kevin to know that he had to make real changes to the way he was operating. Personally, I believed the way he worked - in particular, the way he managed the cabinet and his relations with the caucus - meant his prime ministership was not built to last. Its shelf life, being built on celebrity, was likely to be quite short unless he changed.

I wanted Kevin to be under no illusions that the ship of government was listing and needed to be righted. He had to lead that process. And that's what I tried to convey on that flight to Brisbane. I wanted him to know that while I wasn't against him at the moment, there were problems with the way he was operating.

Over the previous six months, both Julia and I had regularly raised our concerns directly with Kevin, and his senior staff, about the policy gridlock and his abrasive style. The fact that he had asked me about a possible leadership challenge meant that it was to the forefront of his own thoughts.

On the leadership, and specifically on the idea that we should replace Kevin with Julia, I was a very reluctant starter. I could see Kevin was leading us into the wilderness, but I was torn between the dread of that and the undoubted ugliness that would accompany his ousting. It would be the first time a first-term Labor prime minister had ever been removed from office by caucus. This was the proverbial rock and a hard place.

It was a diabolical situation for a Labor prime minister, who had been lauded as a hero just 2 1/2 years earlier for ending John Howard's 11-year Liberal government and, as a bonus, tipping Howard out of his parliamentary seat. And here he now was, wondering whether his leadership was under threat.

Over the previous six months, senior members of the government had routinely put Kevin on notice about the state of drift. The problems resulting from our continued failure to sequence policy priorities and to make definitive decisions, and his continued refusal to engage with pivotal policy issues, were building up.

Above and beyond these problems, Kevin continued to treat people poorly. I don't suffer fools easily myself, I am direct at times, and when I'm out of my comfort zone I have a tendency to sweat the small stuff. For this reason, my staff used to tell me a lot, I'm not always the easiest person to work with. But Kevin's treatment of people was extraordinarily vindictive and juvenile, and it was frequently on display.

It is one thing to dismiss advice or get testy. It is quite another to seemingly take delight in humiliating staff members for doing their job. It got to the point where Kevin would give certain individuals the cold shoulder, making it almost impossible for them to do their jobs. This is unprofessional and unproductive.

The year had not started well: policy issues such as climate change, health and asylum seekers hung in the air unresolved. And things would only get more difficult as 2010 wore on. Since late 2009, the media had been reporting regularly on problems with the home-insulation program.

There were numerous reasons why some in caucus, strongly backed by sections of the labour movement, went in for the kill so swiftly against Kevin. So many of them relate to his conduct as prime minister - his treatment of others and his decision-making style - that I do want to give a picture from the inside that helps explain what subsequently happened.

I freely give Kevin great credit for the role he played in the way we dealt with the GFC and that imminent threat to the nation's economic security and long-term prosperity. I believe that history will regard the Rudd government collectively as having responded magnificently to this external risk. And, of course, Kevin is also one of a handful of people who have led Labor to government from opposition, a very substantial achievement. But he and the history books also deserve an unvarnished appraisal of his overall success at governing.

The role of prime minister comes with enormous responsibilities, and has huge demands attached to it, in terms of work ethic, management style, decision-making and even time management. Kevin had acute weaknesses in these areas apart from his work ethic - he worked bloody hard. Above and beyond this, the question of his true character also hung in the air, as well as over Kevin himself, whenever colleagues discussed his leadership. He once requested the ALP national secretariat to conduct a comprehensive poll and advise him what his "one core belief" was.

Kevin was prone to vengeful behaviour. Early in the life of the government, in late 2007 when we ministers were recruiting our personal staff, he sought to block us from recruiting anyone he believed had been instrumental in supporting Kim Beazley in the previous year's leadership showdown. I experienced this first-hand with my own staff, but did not yield. Others were forced to do so. Talented senior staff are effective political operators in their own right, and my staff would have been viewed as a political threat in the same way that I was a potential threat in his eyes.

There were also unreported incidents that can only be described as bizarre, the result of an unstable personality. At one point Kevin snapped a pen in a fit of anger in a regional hotel room and dark ink splattered all over the light-coloured décor. The damage bill was in the thousands and had to be settled privately.

Kevin had a pathological fear of leaks, so he centralised a great deal of decision-making in his office. This was at the heart of the malaise that fell upon the government through this period. Too often, Kevin would insist on making decisions himself, but then fail to do so. This was partly indecisiveness and partly simple overload. Whatever the cause, the result was decisions not being made, or being made by default. When this occurred, sometimes better policy options had simply evaporated over time, occasioning least-best outcomes rather than first-class ones.

The cabinet itself, including the Strategic Priorities and Budget Committee (SPBC), was a case study in inefficient process. Kevin was an extremely poor chair of cabinet meetings. They would drag on for hours as he sought to use them as his own personal briefing session because he had not carried out the necessary preparation by reading his papers in advance. Staff and officials would dutifully prepare complex briefs, but they would go unread. Ministers became frustrated that their valuable time was being reduced to a briefing session for the cabinet chair.

Worse than this, cabinet meetings rarely functioned with punctuality or efficiency. It is regarded as a great sin to waste cabinet time unnecessarily, but this is what customarily happened at meetings chaired by Kevin. Ministers, understandably, either left meetings before they concluded or did not attend at all.

The SPBC is worthy of mention here. Much has been written about this supreme decision-making entity. Indeed, the moniker "Gang of Four" has come to form shorthand for its members - Rudd, Gillard, Lindsay Tanner and myself. It was envisaged that the big, strategic decisions of the budget would be taken here. Initially, this worked quite well because the committee was a very effective way of tackling the decisions that needed to be taken to handle the GFC. Kevin particularly liked the SPBC because it reduced the potential for leaks but it suffered from the same problems of cabinet with Kevin as its chair. Membership of this group meant hour upon hour spent in meetings, flying to all points of the compass and failing to reach decisions.

The longer we governed, the more Kevin's idiosyncratic style infected the way officials and staff interacted with him, his office and the cabinet (including its committees). He was intolerant of detailed advice, especially of a deep and highly technical nature - the kind that comes from public servants with decades of experience.

This meant that there was often a high degree of pressure to reach agreement quickly in Cabinet on any decision taken by the SPBC. (To many ministers, SPBC decisions became almost a rubber stamp on a proposal.) Kevin was very intolerant of debate or dispute, possibly because it undermined his sense of control of the minutiae. As a result, issues and ideas suffered from not being thrashed about and stress tested and their pitfalls highlighted.

The number of issues coming before Cabinet and the SPBC also resulted in overload. Instead of trusting his ministers to manage their portfolios and only bring the big, important matters to Cabinet, Kevin's micro- management meant agenda items piled up. This wasted the valuable time of ministers, their staff and our highly skilled public servants. More than that, during late 2009 and early 2010 the agenda of unresolved matters was getting bigger, not smaller, alarming ministers and political advisers when we should have already begun clearing the decks for an election year.

Because of his incessant focus on either the media or the political dimensions of a problem, advice from departments and ministerial staff was often unheeded by Kevin. Too often his focus was on having something to announce and the political upside of any decision, which became the reason for either doing or not doing something. This was untenable. Rather, our priorities should have been about the merits of the policy. For all Kevin's reputation as a policy wonk, it's something that often took a backseat in his decision-making process. Too frequently it came a distant third, behind media and political considerations.

And finally, there was a culture of fear and blame that had its origins in Kevin's temperament. He was often quick to anger and his outbursts were regularly disproportionate to the matter at hand. This was invariably followed by retributive action of some sort. In most instances it would not be the senior minister or departmental official who bore the brunt of these outbursts (although this did sometimes happen); rather, a more junior staff member would be the recipient. The turnover in his office was extraordinary. He burned through staff like a child flicking matches from a box.

It is telling that a long list of very senior staff either left the government entirely or departed Kevin's office for positions with other cabinet ministers. One senior staffer departed before we even got into government. It was not simply that the hours were brutal and the pace intense - in my experience, political staffers rarely expect short working days that meander along at a gentle pace. Instead, it was the fact that they were routinely treated very poorly and, more than that, their advice was not listened to.

For all of these reasons, Kevin's position as leader of the party and the government had become imperilled. Julia, Penny Wong, myself and a host of other senior cabinet ministers were unable to get him to change his ways, despite our best efforts.

On the way up to Brisbane that May morning, I did my best to be forthright with Kevin about the government's rapidly eroding performance, and its subsequent declining fortunes. I remember thinking that he surely had to be aware that his behaviour could not continue. It was clear to all that Julia and I were stepping in to deal with more and more issues. It is inconceivable to me that he could have been completely in the dark about the seriousness of the situation. Indeed, the fact he bluntly put that question to me was evidence he knew he was in an increasingly dicey position.

Kevin's behaviour had left him wide open to attack. He had few friends, so when the critical moment arrived, a tsunami of frustrated and angry MPs and senators vented their spleens. The constant condescension, the intemperance of speech and the acts of abuse were still fresh in their memories and could no longer be countenanced. And, when they looked into their hearts, many could not resolve the nagging question of his character. Was he really one of us? Did he believe in a greater sense of purpose?

Through the first couple of weeks of June 2010, stories began to emerge that suggested Kevin's leadership was at risk. None said in plain language that a challenge was imminent, but it was becoming clear the caucus was increasingly restless. I thought it would be sensible to try to dampen some of this speculation, because it was distracting us from the important business of governing. For example, on Monday June 14, 2UE's Tim Webster pressed me numerous times about the leadership. "Now, a simple question, requiring yes or no," he said. "Will Kevin Rudd lead the Labor Party to the next election?" To which I resolutely replied: "Too right he will. Absolutely."

As that parliamentary sitting week rolled on, it was mostly all quiet on the leadership front. The mining tax debate was in full swing and the week was punctuated by the annual Midwinter Ball on the Wednesday night. Then, as the week drew to a close, the leadership reared its head again. On Friday night, I received a phone call from a senior party official, who suggested that a move was possible. I rejected it in emphatic terms. I then spoke to a number of senior colleagues, who agreed with me that such action would be highly divisive and counter-productive. These colleagues were well aware of my concerns about Kevin's performance, but I indicated in no uncertain terms that I was opposed to a change.

On the Saturday night I flew back to Canberra ahead of an interview with Hugh Riminton on Channel 10's Meet the Press the next morning. Ironically, the leadership was not discussed in that interview, which Riminton opened by asking me about the thumping Labor had copped in a NSW state by-election for the seat of Penrith in western Sydney the day before. The 25 per cent swing was the largest ever recorded in a by-election in NSW. I subsequently formed the view that this drubbing was one of the trigger points for a group of frustrated NSW right-wing MPs to begin agitating in earnest for Kevin's removal.

On Wednesday morning, The Sydney Morning Herald published a story by Peter Hartcher that would trigger a political earthquake. At its core, Hartcher's story made the untrue assertion that Julia had been disloyal to Kevin and had been canvassing support. This led to Alister Jordan sounding out MPs about the leadership. Sometimes in politics, a brush fire can burn quickly out of control and can't be extinguished. The Hartcher story was such a brush fire. Kevin's own behaviour meant that, once the match had been lit, we now had a raging bushfire. It was on.

In Question Time, parliament paused for a motion of condolence for military personnel who had lost their lives in Afghanistan. Shortly afterwards, a trickle of caucus members came through my door, and by early evening it had become a flood. All expressed support for a change. At this stage, it was unclear to me whether Julia would challenge.

During the course of the afternoon, I called my chief of staff, Chris Barrett, and deputy, Jim Chalmers, into my office to tell them that if a ballot occurred, then the result needed to be clear and decisive. I did not play an active role in delivering that outcome. Caucus support was already overwhelmingly behind Julia by the time it became obvious a ballot would take place.

It was then clear Kevin's leadership could not survive. Indeed, there's a possibility Kevin could have been moved on earlier had I not held the line as strongly as I had for the previous few months. It was to no avail, though. By the time the moment came, there was no stemming the tide.

After Julia concluded her discussions with Kevin, she came to my office to tell me that she was challenging him for the leadership. She wanted me to stand for the deputy's position. I knew that removing a first-term prime minister was fraught with danger. Sadly, Kevin had brought these events upon himself and we now had no choice but to make the best of the situation. I was also seized with the historic importance of the moment - Australia was to have its first female prime minister. I had no doubt Julia was the best person for the job and that under her the government would be much more effectively run.

By the Wednesday night, when the ABC's Mark Simkin broke the story publicly on the 7pm TV news, the mild rumble had become a thunderclap.

In the middle of all this I phoned my wife Kim at home - she was glued to the TV. She told me she felt sick in the stomach with worry. I told her it was moving at breakneck speed. The floodgate had been opened and torrents of frustration and disappointment were pouring through. Knowing Kevin as well as I did, she also had a sense of foreboding about what he would do in the future. We both hoped the party would survive what was about to descend upon it.

At about 10pm Kevin and I spoke on the phone. I told him his prime ministership was over. He could not win the ballot. I also said it was important that the party and the government have a decisive result. With that, the call ended.

As the numbers hardened in Julia's favour, I knew Kevin would be an unlikely starter in the ballot. Had he stood, he might have secured up to 20 votes at most out of a possible 114. Kevin's decision not to contest the leadership meant there was no real opportunity to have a full and frank debate in the caucus about the reasons for his removal. It also meant, as time passed, that he and his supporters could exaggerate his levels of support. It denied Julia an important pillar of legitimacy that a thumping ballot win would have conferred.

Over 21 years, Kevin and I had been the best and worst of friends. Our families had been close for over a decade. He was godfather to my only son, Matthew, and had thoughtfully given Matt a St James Bible and, later, a small package of Fosters shares on the anniversary of his baptism. But over the years our relationship had grown distant, and then purely professional as events in the parliamentary Labor Party unfolded through the leadership battles of late 2003 and early 2005.

We first met when Kevin came to Brisbane from the Department of Foreign Affairs in the second half of 1988 to become chief of staff to Wayne Goss, who at that time was leader of the Queensland opposition. I was campaign director and state secretary of the Queensland ALP, and Goss, Kevin and I became known (not always affectionately) as "The Troika" as we worked to defeat the 32-year-old National Party government, effectively ending the Joh era. We regularly dined at each other's homes, the kids usually off somewhere busily working up a dance or acting performance for us all before we headed home. We all shared the joy of Wayne's election win in 1989 and the disappointment of his premature and drawn-out defeat in 1996.

When I entered federal parliament in March 1993, Kevin was also keen to go further into politics. I worked assiduously within the Queensland ALP to give him a leg up to the seat being vacated by my good mate, and then Canberra housemate, Ben Humphreys, the Labor member for Griffith. By then, Kevin was Director-General of the Premier's Department and it was during this period that he became a much more dominant force in the Goss government.

This was the first time I started to hear complaints about his behaviour towards staff and about difficulties in his relationships with ministers. I had certainly seen him under pressure when we worked together, and had seen him in some short-tempered episodes, but this was an exhausting period and I wasn't always in perfect control of my temper, either. So I easily dismissed these stories, and the unkind "Doctor Death" references that accompanied them, as the product of a feverish political and administrative environment in which a strong hand was required.

In the 1996 federal election wipeout, I lost the seat of Lilley and Kevin missed out in Griffith. We worked together to win back these seats for Labor in 1998. That year, I joined the frontbench as Labor's spokesman for Families and Community Services. Kevin was a very active backbencher. It was the worst-kept secret in Canberra that he wanted the job as spokesman for Foreign Affairs, then held by Laurie Brereton.

I was fully supportive of Kevin's ambitions for a frontbench position. He was a rising star and deserved recognition. Kevin was duly elevated to the frontbench by Kim Beazley as Foreign Affairs spokesman in 2001; but, in retrospect, there was something about Kevin's stalking of Laurie Brereton that should have jagged with me. From that point on our friendship grew more distant, although my wife retains gratitude to him to this day for handling the media for her when I had a brief run-in with prostate cancer that year.

I acknowledge that it's a legitimate view that the conflict between Kevin and me was a product of our ambitions and abilities. This is correct, but in one sense only: I was ferociously ambitious for the Labor Party and all it stood for, but began to see that Kevin's ambition was ferociously centred on himself. So, it was inevitable that we would eventually fall out, as indeed we did, in the bitter leadership battles of 2003 and early 2005. I believed that Kevin behaved in a way that jeopardised Kim Beazley's campaign in both these contests. There is little doubt that when I was shadow treasurer he briefed journalists with criticisms of my political performance.

Our falling-out was not about who should be leader or whether I was ahead of him in the queue. As I watched Kevin operate in Canberra, I worried more each day about the character traits and values he brought to the table. And I worried specifically about those issues in regard to the leadership, which by then he was quite openly seeking.

The "Sunrise Kevin" was a seductive image for many in the parliamentary party, and for viewers of Channel 7's early-morning program (where he was a regular guest alongside Joe Hockey). But the man I saw performing there was not the Kevin I knew. When the case for Kevin was put to me at that time - in terms of his energy, his ability and his public image - in frustration I would say privately: "The problem with Kevin is that he's always at his best when he's not himself." The private Kevin can, of course, be utterly charming. But he can all too frequently be dismissive and vengeful. And it was this Kevin I was seeing more and more frequently.

Kevin and I worked very closely together throughout the election campaign of 2007. I have nothing but admiration for his performance and the quality of his presentation to the Australian people. At that time, and then through our first three years of government, I believed I was able to work closely and more effectively with him than any of his other colleagues precisely because I was aware of his peculiar style of working and his approach to relationships. Where others would take great offence at his crudities, his lateness and his high-handedness, for me it was simply water off a duck's back.

In fact, I took on the role of pastoral carer, as did Julia, telling colleagues and staff they needed to turn the other cheek with Kevin. He might not be a delight to work with, but we were in government, and the job before us was too important to let hurt feelings derail us. I do believe he displayed far greater courtesy to me and Julia than he did to anybody else. So we were an effective team during the GFC from 2007 to early 2010, and Australia was in dedicated and capable hands. But, as the GFC abated and new challenges arose, Kevin's method of operation and approach with colleagues started to have a substantial impact on policy formation and relationships within the government.

As I watched Kevin give his resignation speech that morning in Canberra and leave the podium, I knew in my heart that the contest for the leadership of the ALP, and the prime ministership, wasn't over by a long shot. But I also knew, given the circumstances of his removal and the need to try to heal the divisions within the party and get on with governing, that it was not going to be possible to publicly air my doubts and reservations. In retrospect, I obviously wish I had ripped that bandaid off straight away.

This is an edited extract from The Good Fight: Six Years, Two Prime Ministers and Staring Down the Great Recession by Wayne Swan, published by Allen & Unwin next week.