Archives

June 2003 Issue

Optimize

How to Save Money in the Operation and Maintenance of Your Broadband Networks

Disaster Recovery: Minimize Your Net's Downtime

Disaster recovery is like jogging: Everyone knows it's the right thing to do, but it's often pushed off till tomorrow. That's unfortunate, because outages reduce revenues and increase the possibility of churn.

Clearly, the most important element is to think about ways to minimize downtime before it occurs. "Look at the entire network," says Tony Faccia, vice president of planning for Rogers Cable. "You have to assess the risk associated with any particular loss. Then look at what it would take to mitigate that risk. Then prioritize where the biggest risks are."

It's simple: If nobody knows what to do before the disaster hits, significant downtime and loss of revenue essentially is guaranteed. (See accompanying figure from Rogers.) "The most important thing is to have a plan," says Scott Weber, Comcast's vice president of network services.

There appear to be three areas of preparation:

The basic infrastructure. A certain amount of disaster recovery automatically is gained as ops implement fiber-rich regional designs. Placement of regional headends and hubs on the same ring--done to achieve efficiencies in distribution--by their nature add redundancy.

Using the infrastructure. Simply having a robust infrastructure won't speed disaster recovery unless operators build upon it. For instance, Rogers is digitizing its analog tiers for transport over synchronous optical network (SONET) rings in the metropolitan area network. This enables signals--which will be converted to RF at the hub--to be restored more easily than if they carried end-to-end in an analog format.

Rogers is also researching precisely how to protect its email services. Faccia and Director of Networking Planning Giancarlo Urbani say there are three choices. One--a "hot" secondary site that is kept up and operational all the time--is considered too expensive and is not in the provider's plans. Currently, the MSO is considering outsourcing disaster recovery that backups operations and maintains "warm" sites. Warm sites are pairs of facilities that share responsibilities, with each capable of taking on 100 percent of the responsibilities if the other goes down.

Common Sense. Perhaps this is where the most good can be done with the least investment. Among the suggestions:

Decide what is necessary to back up and what isn't. An operator is far better off proactively deciding not to protect a specific residential pocket than having it happen by default. Operators should keep close records of where backup equipment is. Important spare parts should be warehoused. However, at some point keeping track of each bolt becomes inefficient, so a source for the little items--perhaps the vendor--should be maintained. Also, make certain that backup generators and similar gear are in working order.

Executives need to meet with municipal officials to make sure cable trucks are recognized as emergency vehicles so they can be on the streets. Operators ought to pay special attention to conditions in their area. For instance, operators in Florida need to take precautions for hurricanes, while those San Francisco ought to pay special attention to earthquakes. A plan of which executives to contact in what order--an action plan--must be kept.

The industry hasn't mastered these elements. "The industry is on a learning curve," says Mark Dzuban, Cedar Point Communications' executive vice president of cable telephony deployment. "In all fairness, the migration from being an entertainment to a telecom world is driving their behavior."

--Carl Weinschenk

Backup Power Supplies: A Wasting Asset?

How do the cable industry's power supply maintenance practices look to those with an external point of reference?

"I'm coming from where you have millions and millions of dollars, and you measure stuff in megawatts not watts, and megavolts not volts, with big gas combustion turbines and generators and transformers," says Chuck Roehrig, national operations manager for Alpha Technologies Services, Inc.

Roehrig came to Alpha six years ago, after his first career as an engineer with Westinghouse. His words may sting, but they have accuracy.

"What I've found is this industry can be cheap and dirty," he says.

His larger observation is that it's not enough simply to purchase in backup power supplies and generators. "If you don't maintain them," he says, "don't waste your money."

Even to the consummate insider, the industry's practices don't measure up. "Some (operators) aren't even doing the basics," Gibson Technical Services COO Keith Hayes, says. "I drive by power supplies that are leaning over, having been hit by a car. Others are covered with vines."

Hayes says that operators who have deployed transponders are being lulled into complacency. "They're relying completely on alarms," he says. "There's still a need for eyeballs on the system, if nothing else than to see the dust and blow it off."

Squeezed budgets?

Hidden factors may be contributing to neglect in the field. Mark Ericson, a regional sales director for Alpha, believes that some general managers at the system level use maintenance as a budgetary "safety zone." In other words, these funds are made available when other needs arise that are deemed more pressing.

If workforce turnover is also an issue, then knowledge could be walking away, leaving gaps in the training of new field techs. Whatever the reason, operators may no longer have the luxury of letting these assets waste away in the field.

Roehrig says backup batteries ought to get three-to-five years of operation. But if not properly torqued and lubricated, failure rates accelerate quickly. And that leads to compounding of problems, such as when a tech drops by Kmart to pick up an unmatched replacement battery, which upon installation immediately falls to the level of the worst battery in the string.

As a result of neglect and mishandling, Roehrig says that in some systems as many as 80 percent of batteries need changing out.

That's not to say that all systems, or MSOs, are guilty of such practices. "It varies from operator to operator. Some have stricter rules and guidelines to follow than others," Ericson says. "The systems that are offering high-speed data and telephone are the ones that really have to pay more attention to maintenance."

It's not surprising, therefore, to find Insight Communications in the group of stricter operators. "It would be disastrous not to do regular maintenance," Jerry Knights, Insight's vice president of telephony engineering, says. (For more on operator-of-the-year Insight, see the May issue.)

Routine Check-ups

As it happens, operators such as Insight, which contracts out maintenance service to Alpha, are also the ones most likely to install telemetry to keep track of the many components in their powering systems.

"We employ status monitoring to help in this," Knights says.

"Help" is the operative word here. Highlighting the notion that transponders aren't cure-alls, Anthony Beckers, senior subject matter expert at APC, says current status monitoring systems "have a tendency to be temperamental" and "can feed back incorrect information."

Apart from looking for any obvious problems within the enclosures, techs should use the metering that's built into the power supplies, Beckers says, adding that APC uses true, root mean square (RMS) metering, which reads 10 percent higher than averaging meters.

Beckers says APC recommends quarterly visits to each power supply location, or at least two times a year, and monthly check-ups on standby generators. "Weekly is not a bad idea," he adds.

--Jonathan Tombes

Modeling Data Flows

This firm is profitable, has no debt, and plows 30 percent of revenues into R&D. Its core technology has wide application across communications, enterprise IT and governmental sectors.

"We do a lot of work with the defense space, and have been a part of Iraqi Freedom," says Sue Cole, marketing director for the service provider division at OPNET Technologies.

Some MSOs, including Comcast and Cox, have deployed OPNET's SP Guru intelligent network management software package, a sophisticated tool for modeling high-speed data traffic across their networks.

"Not only do we replicate the network, we actually put the OSPF (open shortest path first) and BGP (border gateway protocol) routing rules that are in our routers into the model," Jay Rolls, vice president of data engineering for Cox. "So it even mimics the way that the routing decisions are made."

Ph.D's in the Sandbox?

Traffic modeling is a relatively new discipline for cable operators. Rick Gaslowli, vice president of engineering for Online and Comcast IP services (CIPS), says that in its migration away from Excite, Comcast has been incorporating this function across various groups:

In the operations group, which looks at network utilization stats day-to-day.

In the deployment group, which takes that data and projects near-term needs.

In Gaslowli's group, which has a dedicated modeling team.

He says they use OPNET and have "several Ph.D.-type engineers that do that modeling.

But a doctorate is not required. "The skill they've got to have is network engineering skill," Cole says. "How you mathematically simulate, that's all under the covers."

The degree of the software's automation impresses Rolls. "You just load it up and it tells you right away, 'Here's your hot spots.'" The kind of modeling Cox does is to load up traffic levels, specify link failures and then to see which links get saturated because they have to absorb additional traffic, Rolls explains.

The software helps operators of a certain size optimize their existing data network, but Cole notes that it also can serve as a "virtual sandbox for planning new service deployment.

-- Jonathan Tombes

Back to June 2003 Issue

Access Intelligence's CABLE GROUP
Communications Technology | CableFAX Daily | CableFAX's CableWORLD | CT's Pipeline
CableFAX Magazine | CableFAX databriefs | Broadband Leaders Retreat | CableFAX Leaders Retreat