Business Continuity in Practice
“I love it when a plan comes together”
Flood Water Line in Hawley Manor Basement
A very wise and very nice man (divorce lawyer) recently used an expression I had not heard before. When talking about some personal challenges he used the phrase “we are the cobbler’s children, worst shod”. A brilliant expression in many an industry that describes “should know better”. The fact that a plumbers house usual has a leak, a builders house is usually in need of some DIY and that many a best informed doctor is working a stressful 90 hour week and surviving on coffee and take away food, while advising others on healthy living all substantiate this great phrase! The same is often true in IT and while we have all the knowledge and resource to hand, how often is it that we put the needs of our customers to the fore, and leave our internal systems to the bottom of the pile. Well luckily for us, that wasn’t the case with business continuity this weekend at Astro, but wow! What a weekend!
As you can see from our previous post and video below, we moved into our new office all but a month ago. While we had the usual issues to deal with, we were pretty pleased with ourselves to have a fully functioning office by 7am on ‘moving in’ day with the usual low level snagging and nothing more. Most importantly, aside from having a new invoice address and a better location for lunch, our customers were none the wiser. So, all was well with the world. However, have you ever had an experience where something you said or thought comes back to bite you on the rear-side, and you think… if only I had kept my mouth shut!
“…we are fine, it is a tiny little brook called the River Darent”
In the process of moving, our insurance company said “you are within 100 metres of a river, and there is a risk of flooding”. We took the relevant insurances as is sensible but along with my colleague Rob, I looked at a map and thought “we are fine, it is a tiny little brook called the River Darent (a Kentish tributary to the Thames) and the next significant waterway is the river Thames and if that goes, we have more than our little office to worry about!”
Well at 02:24am on Saturday morning, the basement (plant room) in the main manor house of which we lease the stable block (no actual horses) filled with two and a half feet of water (see photos). This, in turn took out our mains electricity supply and our office equipment failed over to UPS mode and the office alarm became sponsored by the Duracell bunny.
Not a pleasant experience at all, but a real world example of proper preparation preventing… well… ‘unnecessary challenges’ is the best way I can describe it. Saturday morning our Business Continuity plan kicked into place for the first time in our 30 years history. Critical services such as email and file storage are, dare I say it “Cloud Based” so we were good on that front. The phone system (ShoreTel distributed architecture running over SIP) has a facility to redirect calls from the exchange across a selection of out of hour’s mobiles which just left the monitoring platform and ticketing systems. All of which can be failed over to a location in the midlands in under 2 hours. However, in the circumstances it was quicker to kick the generator in and get essential services back up and running in a jiffy.
“…a real world example of proper preparation preventing… well… ‘unnecessary challenges’”
Then came the coordination of the landlord, our alarm company, key holders, electrician, fire and rescue service and UK Power Networks – all of whom were simply fantastic along with our guys on site who were also brilliant. By 6:00pm that same day normal service had resumed and we could cut over our failover systems back to business as usual. Anyone in the office not on the incident management communication plan will be none the wiser, Amazing! Had it not worked out so well, our Disaster Recovery plan would have taken over and our service desk home worker kits had been readied so that customers would know no different.
We will have a Major Incident Review on Monday, at which I expect we will decide to move a copy of our VM hosts into the cloud or our midlands DC to and use a snapshot technology to keep a copy of it as, should this event have lasted a week, could have caused us an issue, but that is what BC planning is all about isn’t it? Keeping essential services running so customers feel no pain and then learning from an experience like this one, to ensure that next time, if there is a next time, we are even better prepared?
Normal Service Resumed
I am really proud of the team for their response and dedication this weekend. Although we were doing something that comes naturally to us for our customers (we have responded to many incidents like this over the past 20 years for our remotely located customers such as Butlins and Haven Holidays) this was the first time for Astro. Our team made a molehill out of what in some businesses would have been a mountain but most importantly, we kept our customer services in the forefront of our thinking at all times. While like the proverbial swan, under the surface at times we were paddling hard but on the surface we were as elegant as ever.
I hope other businesses were as prepared and as fortunate as us this weekend. Our thoughts go out to everyone affected by this weather. But it goes to show that thinking about the “what if’s”, one day just might pay off and at Astro, we don’t just preach, we practice too!