A week ago today, the joint hurricane/nor'easter called Sandy slammed into the NY/NJ/CT/MD/DE/PA region, leaving millions without power and vast tracts of low lying areas near the ocean in varying levels of devastation. Much of the less severe damage to the telecommunications infrastructure has now been repaired, and perhaps now it's time to note Sandy has taught us. Here are a few such items from my point of view, I welcome any additions:
Power really is everything. The vast majority of telecommunications outages in Sandy's wake came from loss of power in some form, whether it be at cell towers, data centers, or at consumer homes. Ring topologies, redundant gear, protected circuits - they all did their jobs for the most part, it was the widespread power problems that made it look bad. We've worked so hard to build out fiber and tower nationwide that sometimes we forget that the utility power grid it depends on is so vulnerable to disruption - whether it be the big pipes feeding data centers or the little ones feeding towers or individual homes. Power has been the center of the data center universe for some time, which is why most had adequate diesel backups and prearranged fuel deliveries this past week. But the rest of the infrastructure was not so well prepared.
We need some battery magic. Think about how far batteries have come in the consumer segment over the last few years, powering our laptops and tablets and smartphones for ever more hours for a hundred bucks and a few ounces. And you're telling me that we still can't store more than 24 hours of backup power to a cell tower cost effectively? Ok, so vendors have been working real hard on scaling LTE and backhaul and all that, but now maybe it's time to put some more engineers to work hardening all that infrastructure against simple power failures. Operators need to improve their designs, and vendors need to access the latest in battery technology to give them more options. We need to scale more than just bandwidth.
Raise those pumps and generators. Ok, so without naming names, some of the datacenters that went all the way down last week apparently did so because they got water in their basements. Keeping your backup power in the basement in a low lying area next to the coast is an obvious design flaw whether it's NYC, Florida, Japan, New Orleans, or whatever. Water is the lifeblood of civilization, but we must not forget its potential as a destructive power. Hopefully Sandy's storm surge has sufficiently reminded folks that we don't build backup systems just to satisfy spreadsheets and hedge against hiccups in the power grid but to defend against real physical threats.
The storm gets a few hours blame, service providers will inevitably get the rest. Six, maybe twelve hours after a disaster is your grace period with customers, whether consumer or enterprise. After that the public relations disaster grows by the disconnected hour. It doesn't matter that you have no control over utility power restoration. It doesn't matter that consumer choices themselves have left them so dependent solely on omnipresent wireless connectivity. It doesn't matter that they weren't prepared, only that you weren't.
None of this is an indictment, I happen to believe that overall the industry did quite well this time, all things considered. But whether you chalk it up to global warming or not, the more dependent society gets on rich connectivity the less tolerant customers will be when it comes to their providers' resilience in the face of Mother Nature. Consider it an opportunity rather than a burden, because if you're sufficiently prepared relative to the competition then you will win new customers because of it. It's a good thing for telecom infrastructure that what we provide is so important to people, all we have to do is figure out how to keep them as connected as they want to be.
If you have an item to add to this list that fits, I will happily add it verbatim, with or without attribution (your choice). Either leave a comment below or send it to me via email.