Importance of IT Resilience

• Disclaimer: The posting written here is solely of my own comment and feedback based on my experience in IT for more than 15 years and is not meant to link to any companies, organisations, etc that I am involved with.

Many Malaysians would have been aware that Malaysia Airports Holdings Berhad (MAHB) had a major IT disruption for both KLIA and KLIA2 as well as areas around it that connects through them (>https://www.thestar.com.my/news/nation/2019/08/23/m039sia-airports-disruption-at-klia-due-to-network-failure-equipment-replaced).

Such major downtime that happened on the night of 21st August and until the time I am writing this article, the Total Airport Management System (TAMS) network is still not fully restored.

This major downtime shows that how important an IT infrastructure is build to support critical business services such as an airport. It is important that every organisation or IT infrastructure must have redundancies or least a backup or contingency plan in place in event such major IT failures.

Most IT teams faces challenges in getting funding for IT because most company management question the Return on Investment (ROI) versus the money invested in various different systems. While some business applications would provide ROI and improve productivity, most IT infrastructure does not immediately show value. In many of the organisations I worked with, the most common question asked is if there is no IT problems, why should I be spending this money?

It is tough for any IT head or teams to get approval on funding to ensure that proper IT systems and infrastructure including providing redundancies and backup because in most cases there are usually no major issue experienced for a period of time. However, when major IT issue does happen, most often the cost of recovering from the downtime is more than the initial investment put into the infrastructure, maintenance and support. By then, it would have been too late and it will cost the organisation more money to rebuild the systems. Most IT don't spend for the sake of spending but not to keep everyone awake at night.

The other factor is that old legacy systems must be replaced when its due. Hardware normally have a lifespan for 6 to 8 years depending what kind of hardware it is. The replacement lifecycle must be adhered to in order to ensure systems run smoothly for the organisation.

In the case of MAHB recently, it looks like the teams are quite prepared for any downtime and ready to swing in action of running the entire airport manually. Now this is important as in current days we are so heavily dependent on systems that most often we cannot run manually. Which brings me to a point where I always emphasize to my colleagues in past and present work place, the importance of testing and running manually regularly so that they are familiar with all the manual process in event such similar major downtime occurs. Having regular test run on running manual without system helps prepare users to operate manually without system and get them prepared for any possible downtime. This will ensure business can run as usual while IT teams work on restoring services.

Do note there is no guarantee that event with proper IT infrastructure setup with redundancy as there are times where redundant systems also failed simultaneously and this is the case for MAHB core switch where both failed at the same time, which is unusual. Many analyst suspect that this could also be due to cyber attack or security breach where someone has attempted to bring the system down, though this is unconfirmed until MAHB releases update on the root cause. It is also important that all organisations put high priority in ensuring cyber security is taken care of and secured. Appoint a proven vendor to help perform IT security risk assessment of the entire organisation and close the gap of any findings quickly to ensure you have your systems secured.

One final factor I would point out is that there must be proper maintenance and support in place with vendors and with proper Service Level Agreement (SLA). This will safeguard the organisation and have the ability to get help immediately. Vendor management and relationship is also very important as you will never know when you need their help urgently.

I hope we can all learn from MAHB's major IT system disruption and start reviewing our organisation's IT infrastructure and ensure that proper contingencies and recovery plan is in place.