22 June 2004

Power outage

Lessons to be learned from a recent power outage.

On Friday 11 June 2004 at around 12.40am power failed throughout large areas of the University and the Belconnen Town Centre. Power was restored just after 3am, and there is apparently no explanation for the outage.

All servers in the Division managed by the Technical Services Unit shut down: most are connected to an Uninterruptible Power Supply (UPS) and did so gracefully.

When power was restored most of the servers came back up without any problems. Others were restarted manually on Friday morning and all but the Network Attached Storage came back on line.

With some help from BLIS, the computer controlling the NAS was restored, but all the network shares on the NAS itself were lost. Backup of these shares is scheduled nightly, but the last recoverable backup was the previous Monday night, 7 June 2004. A TSU Technical Officer spent many hours over the weekend restoring the shares (he had to do it twice, and each restore takes about twelve hours). By Tuesday morning (Monday was a holiday), most of the services were again available and with fine-tuning all the services were restored by Wednesday.

The TSU is to be congratulated on their efforts to restore the services, and credit to BLIS IT for their support.

We are investigating why the NAS did not shut down properly and restart, and checking the backup regime to see if a better experience can be planned if such an outage happens again.

Desktop computers
A number of power supplies, particularly in AUVA computers, were lost when power was restored. Replacement computers have been provided and a number of extra new computers purchased to cover the losses.