Today’s networks are increasingly complex, and keeping them running at peak efficiency takes a lot of work. So for this week’s spotlight, let’s take a peek behind the curtain of the Network Operations Center. With us today is Hal Baylor, Director of Business Development at INOC. INOC specializes in outsourced NOC solutions for infrastructure of all sizes, and has been seeing rapid growth in recent years.
TR: What are the origins of INOC? What problem were you trying to solve?
HB: The origins were a group of people from different networking backgrounds who were invited to work for the Radiological Society of North America, which has met annually in Chicago at McCormick center for 20+ years. RSNA turns up a network of up to 600 connections to the internet and to each other, and this group started turning up a NOC within that environment for a week to do it. After a while it dawned on them that the NOC service would be a good service to sell. So they started this small company, got a little bit of funding, and started doing it professionally.
TR: What is the essential piece of the puzzle?
HB: The service is a combination of the availability of technical staff working within an operational framework. This framework is a key component -- even with great technology, if you don't have that operational framework then you can't make things run well. Many companies have NOCs but many of them don't necessarily do it efficiently and in some cases effectively. In many industries there's no training for it.
TR: When I envision a NOC, it’s a theatre-like place with huge screens. How does that work when it’s outsourced?
HB: Yes, everyone's been in a show-NOC with the big screens and a stadium environment. But what we've done is modified things a bit to so that it works for a shared environment, although we do also provide dedicated support. We integrate everything into our tools so that it's manageable. We can take monitoring and alarms and do polling not only from our system but also from other management systems, and we funnel all of those alarms from many networks into a single pane of glass. This allows us to correlate alarms, so we see a bunch or alarms come in and they all have color-coded timers. An alarm handler interprets the correlated alarms and then hands the issue off. Then the right engineer or technician starts working the incident. By contrast, most environments will look at a large screen, and when someone starts working on a problem it shows up at the top of the screen. Or in some cases outsourced NOCs will have a screen for every client so you have a room full of screens. We've seen a lot of variants, but we've realized our process works really well, is cost effective and very efficient.
TR: What makes an outsourced NOC more efficient?
HB: It all boils down to the activity, and how much you have people working on a network. For a basic NOC, 11 people will staff a couple of people every hour, 7 days a week including holidays and sick time. Some think they can do it with three, but the service isn't good and they burn out their people. They may take it to 5 and it's still hard. The proper number can range from 7 to 11, but the workload itself might be the equivalent of one or two people. A shared service works well because we're going do it with a lot less people because we don't have to fill the gaps with people that aren't doing anything.
TR: What sort of infrastructure does it take for you to do all this?
HB: All of our software sits on servers and VMs within SOC2 Type 2 data centers, ISO 27001-certified data centers. We connect to the client's network via IPsec VPN tunnels, typically, through which we receive alarms and poll equipment. We have a separate facility with most of our staff, with disaster recovery and business continuity plans to move within a couple hours if needed.
TR: What usage cases are most common for networks to outsource NOC functions for?
HB: There's a sweet spot. The Tier 1 work, i.e. receiving the alarm, correlation, logging in, investigating, and maybe dealing with a carrier for a circuit issue, those things are things many of our customers don't like to do. They really want to hang onto their more technical or networking staff, so they look to outsource that part of it. So we provide that Tier 1 support, and they are able to keep their Tier 2/3 staff on project work and have them jump into a lesser number of issues. But in some cases we're the entire networking arm of the company, designing and implementing and acting as Tier 1, 2, and 3 support staff.
TR: What effect does the underlying technology a network is based on have on the job INOC does?
HB: It really doesn't matter what the technology is. We can do at least Tier 1 work in pretty much every environment, from optical, which we do specialize in a bit, all the way through any kind of networking environment up to supporting servers and applications. We can manage dark fiber as well. Another new technology is just another operational conversation of identifying the steps and procedures needed. The key to it all is the operational framework, the technology is secondary to us.
TR: Are advances in SDN and NFV changing things significantly for you?
HB: We love to see it. It's really nice when someone comes to us and they have a bit of automation because of the data that they have in place. If a client comes in with very little automated and they have a bunch of engineers with spreadsheets, the process can be challenging to implement. But if they have many automated systems, they can just give us access and we can go get the data we need ourselves.
TR: Where do you find the most interesting and challenging parts of your work?
HB: The truly interesting things to be found are the physical environments, the technologies, and the processes of our clients. On our side it's not as exciting putting it all into a project management data base and doing other detail work. The most interesting parts for us are when we go out for training on new, exciting networks that our customers are building that we get to be a part of.
TR: Are you seeing much interest in outsourced NOCs from international customers?
HB: We tend to be US-based just because that's the default of who we are and who we have connections with. However, we have a lot of partners that we work with that sell our services, such as OEM clients and systems integrators. So through them we do have a global footprint and we have been going to more global tradeshows and have been getting a larger audience with international clients. One interesting case is where international clients have a US presence and need a US NOC manage their environment here.
TR: What advice do you have for network operators when it comes to their NOCs?
HB: It's helpful to get an understanding of what's truly going on within a NOC environment, what the metrics are and what to do with the data. Companies build so fast and add so many people so quickly, there's a certain point when they have to stop and review what they're doing and how they're doing it. Everyone has access to big data, but the analysis is too often the missing part.
TR: Thank you for talking with Telecom Ramblings!