IT Challenges of the Mergers, Movers and Shakers – Part 1

IT Challenges abound as a new broom sweeps around.

You may be wondering who the mergers, movers and shakers are. Mergers and movers are businesses relocating or merging with another business. The shakers are incoming Heads of IT acquiring an unknown IT estate from their predecessors. These situations generally result in similar challenges for IT teams and any of these scenarios can without doubt present some serious challenges to the business.

This blog is inspired by recent events when some of our friends have moved on to pastures new and have engaged us to make sense of their inherited IT estate. We have a great deal of experience in this area going back decades. Including many projects where we were engaged after a move or acquisition (business acquisition or Head of IT taking acquiring an IT estate) to restore order. Some of our discoveries were shocking to say the least.

In part one of this three-part blog I will share some of these as they may serve as a warning to those about to undertake relocation projects or acquire an infrastructure. It may even be a wakeup call for others with no compelling event on the horizon. In the second and third parts, I will take a look at some methods I use to get on top of unknown IT estates.

“Why is it when you are pushed to your absolute limit things go wrong that require you to keep a clear head to think things through?”

To set the scene this was a merger between two of the largest organisations of their kind in the City. The relocation of staff took place over an Easter weekend. I had several teams on site responsible for delivery of all of the dealer services at the new head office. There were around 1000 dealers (from memory). The actual move element of the project was intense as soon as it commenced on Friday morning.

There was a delay from the start due to lift congestion in the two buildings being vacated and the new combine head office. Kit was stacking up in reception, on the pavement and in the delivery lorries. To recover some time my shift (the first on site at 7am Friday) decided to work through to Saturday evening, the end of their second shift. This gave us a fighting chance to catch up. We were all staying in a nearby hotel so when we did eventually finish our shift we were a short walk to our beds.

Why is it when you are pushed to your absolute limit things go wrong that require you to keep a clear head to think things through? After so little sleep and such as intense 32 hours, even the simplest of problems can seem insurmountable.

“The Bridge was staffed by representatives of each team to enable complex issues to be dealt with quickly and efficiently.”

It was 3pm on a sunny Bank Holiday Saturday afternoon. The WAN serving all of the external services and remote sites that had been stable and reliable for a month went down in its entirety. The new WAN link had been commissioned several weeks prior to the move and the new head office had been hosting the main business servers reliably for a month. So why should it fail now? Whatever the cause, it was all hands to the pumps and I was asked to assist with the troubleshooting.

It was quickly established that the WAN circuits were all up. The power was stable in the server room. Having ruled out any circuit or hardware failure we dug a little deeper. I soon realised we had a duplicate IP address problem and on further investigation identified the duplicate device as an HP printer.

I reported this back to the Bridge (the Project Manager had set up a ‘Bridge’ in the new office as a ‘go to’ place for any project issues. The Bridge was staffed by representatives of each team to enable complex issues to be dealt with quickly and efficiently.

“Who’s IP address is that anyway?”

We soon discovered that due to a problem with the DHCP service, a team of IT technicians were moving around the new head office configuring static IP addresses on some of the devices, including printers. The hunt was now on for the culprit. But there was another problem. The WAN and LAN was managed by a third party and no one on site had access to the consoles to trace the rogue MAC address. Nor could we disconnect sections of the network to localise the problem as this would have hampered the commissioning of all of the local services being tested on each dealing desk.

All that was left to do was to hunt down the static IP addressing team and ask them what printers they were working on from around 2.30pm onwards. Eventually, we found the rogue device and the technician and discovered that she was issued wrong IP address and although she had noticed a difference in the sequence but didn’t think it was worth mentioning. Before continuing with her tasks she asked. “Who’s IP address is that anyway?” There was a separate investigation as to why the WAN router was sat on the office LAN.

“Whoever installed the network at the new office had connected the entire LAN to the outside of the firewall.”

Several years later we were asked to investigate some network issues following a simultaneous office move and network upgrade. There were all sorts of strange happenings on the network, it was very sick indeed.

After a few hours’ investigation we realised there were some very serious problems with the network as the public internet was appearing on the internal LAN. Whoever installed the network at the new office had connected the entire LAN to the outside of the firewall. With a mix of local IP addresses and public IP addresses on the office machines it was a wonder anything was working. This organisation had a very lucky escape as far as they appeared to have got away without having their systems compromised, as far as they were aware.

This is an extreme example of poor network security and one that was very easy to identify. The majority of holes in security are not so easy to spot. My team were involved in the investigation of an ongoing online fraud some time ago. The perpetrators had hacked into the supposedly very secure system and were initiating small regular credit card refunds. The fraudsters were very knowledgeable about the business as they understood the nature of the business and knew exactly what they could get away with and remain under the radar. Although they were small transactions they amounted to an equivalent today of around £100,000 per annum and it had been going on for several years. If it wasn’t for the diligence of the Head of IT spotting something strange in the data, it may have continued for many more years.

“Industrial espionage may sound like the stuff of Hollywood or Pinewood but it is a reality.”

The real problem is that this type of fraud often does go unnoticed and the villains are not always twilight individuals hacking from their bedrooms seizing opportunities as they arise. Industrial espionage may sound like the stuff of Hollywood or Pinewood but it is a reality, and attacks are likely to be very well organised and highly targeted. I have known of several customers over the years who have been victims of industrial espionage. One of these victims was a commodity broker in the City. The Head of IT discovered their systems had been compromised by a competitor. It took several months of undercover work on his part to expose the illegal activity. Although the access was blocked the case was deemed far too complex to take any further. They never found out the true extent of the damage caused but my customer’s business folded a few years later.

Staying on the information security theme for a few minutes, we were engaged with a customer to help with their PCI DSS compliance. As part of the exercise we were asked to examine and challenge the firewall rules. They had several firewalls at a number of locations and there were so many rules opening up paths from the outside to internal systems it was questionable whether these firewalls were actually doing anything to secure the network. We identified around six rules that were current but the remaining rules – hundreds across the firewalls -were a mystery. When I asked the IT team what these rules were for no one knew. If no one knew what they were for, there was even less chance of anyone knowing if they had been used for illegal access.

“Take you through it? How long have you got? It takes around hour and a half to login.”

The last of these examples of acquisition focusses on perception, in particular how an IT team’s perception of a problem can be a world apart from the end user’s experience. Some time ago one of my customers acquired a fairly substantial business. The acquired business already had a network serving all of their sites that was quoted as being ‘state of the art’ and more than capable of supporting the business into the future. As a result, we were stood down as we were surplus to requirements now everything was in hand.

As the acquiring business deployed their systems it was becoming fairly clear that something was not right. My customer’s business systems that worked very well over their existing WAN kept failing over the acquired network. Systems had become so unreliable they were business threatening.

I was asked to investigate the issues on the acquired WAN and chose the least reliable site to start. When I arrived I asked one of the staff to take me through the login process. He laughed and said. “Take you through it? How long have you got? It takes around hour and a half to login.” And, he wasn’t joking. I went off to make a cup of coffee, made a few phone calls, went for a walk and when I returned the hour glass was still there. At approximately the 94th minute we were in.

“The IT team didn’t believe the users, they put their claims down to exaggeration and had never sent anyone to site to investigate.”

What amazed me was that 80 miles away the IT team were totally unaware of the problems faced by their users. When I investigated further, this was the norm rather than the exception. If two people tried to login at the same time it would take well over two hours. Unbelievable! And that was the problem. The IT team didn’t believe the users, they put their claims down to exaggeration and had never sent anyone to site to investigate.

Lessons to be learned in all of these incidents? Absolutely! Some IT people dread these major events such as relocations and mergers. But I believe they are an amazing opportunity to go through the IT estate with a fine toothed comb. When you know your IT estate intimately it is easier to manage, easier to secure, easier to maintain business continuity and, you get to sleep at night. Of course it is possible to gain this level of detail as a standalone project. However, unless you can devise a comprehensive test regime to verify the gathered data, the lack of the ultimate ‘real test’ that a relocation provides means there is a higher risk of something slipping through the net. Anything that slips through the net may come back to bite where it hurts at a later date.

“…as the old saying goes, ‘the devil is in the detail’ so that is where we have to start.”

So, what are the lessons? The lessons are many, but as the old saying goes, ‘the devil is in the detail’ so that is where we have to start. Gathering information.

I will go into that in the next part of this two-part blog, but before I close the first part I just want to reiterate the ideas shared here apply to relocations, mergers, acquisitions (business acquisition or a new head of IT acquiring an existing IT estate) as well as development or redevelopment of IT or digital strategies and establishing and maintaining a security policy. Even if you have no specific compelling need for this information, it is still essential for the safe and reliable operation of your business network.