Archives

April 2001 Issue
Network Monitoring: Challenges to a Healthy Operation
By Natalia A. Feduschak

Having the correct tools is critical to proper network monitoring. Despite an influx of new methods to help operators maintain properly functioning systems, some hurdles remain.

It’s 3 a.m., and Jeff Campbell, president and CEO of Core Networks, a broadband provisioning and management company, is helping an anxious cable company customer.

The cable operator has strained the capacity of its cable modem termination system (CMTS). "Customers are complaining about the slow throughput," Campbell says. "So the operator buys a new CMTS, plugs it in, recombines the nodes and feeds traffic into the CMTS. They’ve got to be able to move customers rather quickly, or there are going to be those who could get missed in the transition."

Campbell eventually helps resolve the customer’s problem, but the CMTS crisis is evidence of what all cable operators should have quickly learned: The network is their Holy Grail and how well it is monitored will determine the success or failure of their business. In an age where data, telephony and interactive television are changing the way cable companies manage their systems, it is important that networks operate smoothly. To do that, operators must be proactive rather than reactive to potential problems. They need to determine where their thresholds are and where they can expand, Campbell says.

To achieve that goal, companies are increasingly relying on aggressive network monitoring to ensure systems function round-the-clock. Vendors are unveiling a host of tools to meet cable operators’ expanding management needs. Even as operators scramble to keep a competitive edge, they claim to face persistent challenges in monitoring their networks. Convincing management of the importance of customer service, hiring personnel trained in network management and integrating technology from multiple vendors top their list of challenges.

The importance of customer service

The primary challenge operators face in network monitoring is accepting they can better their network by improving customer service.

"Customer service and network monitoring are nonprofit entities in the cable business," says David Fears, director of network services for Cox Communications’ network operations center (NOC). "There’s no immediately recognized payback. Our biggest problem initially was convincing ourselves to release the capital investment so that our customers’ experience would be such that when we win customers over, we keep them."

AT&T Broadband has taken steps to better its customer service, says Charlotte Fields, head of AT&T’s Broadband Network NOC. Top personnel are notified of problems via pagers and cell phones to ensure they are quickly fixed.

In each of its major markets, the company keeps a so-called web format escalation list that identifies who needs to be notified when a problem arises.

"We’re going to page all the way through the vice president of marketing," Fields says. "Everyone knows what’s happening on a critical event." When notified of a potential problem, AT&T also establishes a conference call with NOC personnel regarding the status of a problem.

So far, the system has worked well, Fields says. "Now when a customer wants to find out about an outage... the attendant has the information at his fingertips," she says.

Trained personnel essential

Finding the right people who are well-versed in today’s networks also is a constant challenge.

"If you build a monitoring tool, no matter how good it is, if the people aren’t knowledgeable in what they’re trying to do, the tool gets ignored," says Dennis McClure, staff engineer at C-COR.net.

At the same time, engineers often are too specialized and lack a broad understanding of how a network works. That becomes problematic when troubleshooting network problems.

"We need people who have a broad-based knowledge of RF, digital networks and the whole overall (concept) of what we’re trying to provide," says McClure. "Everyone working (in network monitoring) needs to be trained in a set of standards. However, once they’re trained by a vendor and on specific equipment, they only know how that gear works."

AT&T’s Fields adds, "We need to have NOC personnel using their critical-thinking skills. We need to take information from high-speed data, telephony networks and the common HFC [hybrid fiber/coax] networks and use all of that to resolve problems."

Integration critical

As cable operators move from being simply entertainment companies to full-service providers, their networks have become more complicated. Networks, however, don’t always coordinate well.

"Individual management systems don’t always integrate," says Brigitte McCarthy, product line manager for Acterna, which was formed by the merger of Wavetek Wandel Golterman and TTC and includes the Cheetah product line. "They’re vendor-supplied. When operators are deploying a system, they need to look at solutions that are end-to-end and communicate with other support systems."

Acterna and other network monitoring companies such as Micromuse and Harmonic have introduced programs to help cable operators achieve greater integration.

For instance, Micromuse’s Netcool software suite helps network operators bypass potential problems by flagging, isolating and resolving inconsistencies in the system. The program collects management data from more than 250 management environments and network devices, including voice, data and the Internet. Fault data is filtered and processed by Netcool’s ObjectServer, an in-memory database.

Operators are able to build views of service availability in real time, helping them manage networks more efficiently.

Harmonic recently launched a next-generation network management system aimed at simplifying management of the company’s broadband access equipment by providing management over IP networks. The program enables system operators to remotely configure and control Harmonic’s HFC equipment through a standard Web browser and eliminates the requirement for special element management software.

"We’re struggling to bring [monitoring elements] under a single system," Mark Havlek, Harmonic’s product manager, says. "The most common threads we’re seeing are far more requests to interface systems to a higher level management system."

Those higher-level management systems will allow cable operators to work toward achieving real-time alerts, an important goal for most cable operators.

"Getting connectivity and indications in as real-time as possible was a major objective of ours," Cox’s Fears says. "Our next hurdle is to get over all the legacy equipment out there because... not all equipment is interchangeable." Equipment, therefore, needs to be on a common management platform.

Looking inside out

A critical component of network monitoring is having information about all the software and hardware that make up a system. Unfortunately, not all manufacturers are willing to unveil proprietary information.

"Whenever you can’t see into the equipment, that is a major problem for cable operators," C-COR.net’s McClure says.

Having product specification details included in the network management system makes it easier to determine the scope of a problem.

"You may have one network interface unit (NIU) that doesn’t communicate with the host digital terminal. This may be a singular issue, or numerous NIUs may be affected," Fields says. "We need to correlate information in a quick fashion."

Centralized monitoring

To minimize problems and communicate with vendors and customers, many cable operators have opted to monitor networks from one central location rather than regional hubs.

"What we gained from a central location is a single point of contact for customers and vendors," Fears says. "The two major benefits are we can pool our resources and we have a universal application for fixing problems.... We can fix a bug in the hardware and software in one location."

In addition, one location makes it easier to service a network.

"We maintain a 1 a.m. to 6 a.m. schedule," AT&T’s Fields says. "If the local market says we need to do XYZ... whether it’s upgrades to software, hardware or cleaning out an amplifier with proactive rather than reactive maintenance, we do it."

Ready for disasters

As operators strive to better the performance of their networks, one critical monitoring component is implementing a disaster recovery program.

"We not only need to have a [NOC], but we also need to have the capability to deal with disaster outages," Fields says.

AT&T is specially-training people in disaster recovery, and the company intends to establish a secondary center where operations may be moved in the event of a catastrophe.

Cox, too, is establishing another location should its main operations center shut down. Some of the equipment in the back-up site will mirror that in its main network.

"We have to take up surveillance someplace else in the event of a catastrophe," Fears says. "When we’ve lost all links to the network, we have to have a method of contacting the system."

Changing environment

Networks are different today than they were even five years ago.

"Now we’re moving from that cable guy to an IT [information technology] professional," McClure says. "Not only is there the convergence of hardware, data, television and telephony, but the people who run the network have to be professionals. Now it’s not just a matter of changing this amplifier, but understanding what the ramifications are of each piece of equipment."

In the coming years, the ideal network will have diverse paths to the headend, and a number of hubs that will push fiber closer to the customer.

"The more active devices you can get out of the network, the better and better it becomes," Fields says.

The hope is that as networks and services expand, operators will have a birds-eye view of not only their network, but also customer homes.

Natalia A. Feduschak is senior editor of Communications Technology.

 Back to April 2001 Issue

Access Intelligence's CABLE GROUP
Communications Technology | CableFAX Daily | CableFAX's CableWORLD | CT's Pipeline
CableFAX Magazine | CableFAX databriefs | Broadband Leaders Retreat | CableFAX Leaders Retreat