While the brutalist structure has provoked a fury of debate about social inclusiveness, public assets and architecture, it is eerily quiet on the inside.

Built more than 30 years ago to allow working class residents to remain in the area during a period of major construction, the Sirius building is now all but deserted. Unadorned corridors lead to dozens of empty units.

Living alone on the top floor, Myra Demetriou, 89, said she shares the entire building with just three other residents.

A ground floor space - with lines of vinyl chairs, a framed list of rules that include not making too much noise and raised wooden animal artwork - is nearly always empty.

The garden has been abandoned, and nobody knows who has the key to the rooftop anymore. The NSW government plans to sell the site, and it will almost definitely be knocked down.

"There's nobody else up here," Ms Demetriou said. Each night she switches on an SOS or Save Our Sirius light sign in her window.

On Saturday, Ms Demetriou was among at least 1000 people who rallied outside the harbourside building demanding it be heritage listed and saved from demolition.

The Construction, Forestry, MIning and Engineering Union officially launched a "green ban" on any redevelopment of the building, which means no unionised workforce will be allowed on the site and it will be picketed when construction starts.

Jack Mundey, famous for coining the term in a campaign to protect Sydney from overdevelopment in 1970s, told a cheering crowd that the Sirius needed to be saved.

"It's a great pleasure for an old 87-year-old bloke to be here, " Mr Mundey said. "Let us resolve to keep the fight going."

NSW Environment and Heritage Minister Mark Speakman announced in July that the controversial building would not be state heritage-listed despite a unanimous recommendation by the Heritage Council.

Mr Speakman said the listing of the building would reduce the value of the site by as much as $70 million, money which could otherwise be used to fund the construction of 240 new social housing units elsewhere.

Labor, The Greens, City of Sydney Lord Mayor Clover Moore and the Australian Institute of NSW Parliaments have thrown their support behind the "Save Our Sirius" campaign. Artist Reg Mombassa was at the rally and comedian Tim Ross was MC.

Federal Labor MP Anthony Albanese, the shadow minister for infrastructure, transport and tourism minister, also weighed into the debate.

"I've been coming to Millers Point for years. When I was a child growing up in public housing in Camperdown I had mates who lived there," Mr Albanese wrote in a message read out at the rally.

"The Sirius building should be retained for public housing. It is not just a place that people call home it is a symbol of inclusion."

One protester, Diane Marr, from Rozelle, said she had never attended a rally before but would be upset to see the Sirius demolished and replaced by a generic apartment building.

"I'm always driving over the Harbour Bridge seeing this incredible building which always seemed so ugly but so interesting," Ms Marr said.

"I think it should stay. It's an eyesore for some and it's a great building for others."

Millers Point: a community under the hammer