Census 2016: IT experts say Bureau of Statistics should have expected website crash

The websites for the Census and the Australian Bureau of Statistics went down about 8pm on Tuesday, with the agency saying on Twitter it expected them to remain down until at least Wednesday morning. An estimated 16 million people were expected to log on to complete the census on Tuesday.

Paul Brebner, a Canberra-based software performance expert who is familiar with large-scale public sector websites, said the Bureau of Statistics was prepared for 1 million people to be online simultaneously. However, he performed his own calculations and said they should have been ready for 3 million simultaneous users between the hours of 6pm and midnight.

"You can't assume a uniform load across the 24 hours," he said. "If they were expecting 16 million people over 24 hours, that would be okay, but that's not how loads on websites work at all. There's often one very big spike, and it's hard to see how long it will stay in that level."

Mr Brebner said such projects were often done on "limited budgets and very short time constraints", which would have taken a toll.

"I noticed a lot of comments on Twitter were saying 'why didn't they use Amazon Web Services [cloud computing] or something'. I guess with all the media discussion in the past week, with the privacy and the name retention and stuff, is probably the answer to that point.

"They certainly couldn't be using a multinational public cloud to be hosting it, so that limits your options. Then you've got to have the resources yourself to scale up, and that takes time and money unfortunately."

Census was delivered by technology company IBM using its Australian SoftLayer cloud. Figures from the Australian Government's procurement agency AusTender show IBM was paid $9,606,725 in 2014 to design, develop and implement the "eCensus".

Melbourne-based company Revolution IT has also been paid $378,332 by the Australian Bureau of Statistics since January 11 for IT consulting and 'load testing' on the census and agricultural census.

'Load testing' is a process intended to ensure a website can handle a high volume of simultaneous users without crashing. Fairfax Media has contacted Revolution IT for comment.

Glenn Drew, the CEO of IT firm Squizz – which deals with eCommerce sites with high volumes of customers – said it would have taken "very serious and sophisticated" software to successfully deliver the census, and $10 million may not have been enough to pull it off.

"The IBM web sphere technology upon which the 2016 Census is built has not been battle tested and most likely contains inefficient and bloated code," he said.

"DNS errors, HTTP errors [and] unknown error codes highlight the poor quality of software engineering within.

"$9.6 million sounds like a lot but in reality it is 90 people working on the system for one year. Break that up into admin, operations, software design, architecture, development, testing and go live, [and] this is a fairly limited budget for such a large-scale project."

Mr Drew said the Government needed to learn a lesson from what has been branded the "#CensusFail".

"The ABS will blame IBM, IBM will blame the ABS and both parties will blame the Government," he said.

"In reality, it's very difficult to find quality talent to pull off a project like this. I can only hope our industry and government take note of this disaster and scrutinise the talent and technology they are employing."

On social media, some people began pointing fingers at IBM, which was also behind the implementation of an unsuccessful statewide government payroll system in Queensland. ​

However, some said the Bureau of Statistics might also be to blame. In 2015, then-Treasurer Joe Hockey said the Bureau's IT infrastructure was "highly vulnerable to failure and error".

"Critical IT infrastructure has components that are over 30 years old," he said in a statement. "One in three applications are classed as unreliable, with issues occurring daily or weekly."