In the third quarter of 1995, only weeks after Pacific Bell's huge network operations center in Sacramento was completed, it faced its first major test. A severe nighttime thunderstorm knocked out commercial power across a broad area, putting the NOC on battery backup until the emergency generators could start.

Inside the 120,000-square foot facility, workers were not immediately aware that anything out of the ordinary had happened. The lights never dimmed, even momentarily. Everything kept running as it had. By contrast, all of the four other administration buildings in the company's Watt Avenue Complex were dark.

In designing and building its Sacramento NOC, as well as a sister NOC in San Diego, Pacific Bell had committed itself to centralized control of its entire network (Telephony April 8, 1996, page 30). The idea was to bring the monitoring of switching, transport and signaling under one roof and thus simplify operational control and maintenance. This simplified approach and greater efficiency were vital if Pacific Bell was to better manage the explosion of data and other communications services that customers are demanding.

The result was one of the largest NOCs in the U.S., perhaps the largest among all local exchange carriers. In 1995, for example, the initial responsibilities of the Sacramento NOC included monitoring nearly 800 fiber optic routes spread over an area of 93,560 square miles, 526 local access and tandem switching entities and 436 separate wire centers.

Additionally, the NOC was to manage 790,000 trunk connections to switching equipment and maintain 24-hour monitoring of terrestrial and microwave carrier facilities in 800 company and customer buildings and underground sites. And that's just northern California.

The Sacramento NOC also had to be able to handle its own responsibilities, plus those of its sister NOC in Southern California for the duration of any emergency situation that might render the San Diego facility inoperable. Because the San Diego NOC has similar capabilities, it can handle all Sacramento NOC functions during extended emergencies.

The monitoring and management requirements of the Sacramento NOC have grown since its inception. To oversee all this, the builders of the new NOC had to install an unprecedented number of individual pieces of equipment, including 420 powerful SunSPARC workstations, 18 wall-size theater screens and 12 37-inch monitors.

Perhaps just as important as the equipment itself was the marketing role that Pacific Bell intended to assign the Sacramento NOC. It was designed as a showplace, complete with viewing platform, where the telco could bring current and potential customers to demonstrate why they could count on Pacific Bell under any circumstances. Clearly, no interruption in the network monitoring and management services provided by the center could be tolerated.

In fact, the telco routinely performs responsibility transfer exercises between the NOCs to verify that each one can actually monitor and manage everything in the entire state.

Over the following months, other potentially disastrous tests occurred. In fact, during the two years since Pacific Bell's Sacramento NOC started operations, northern California has experienced some of its worst weather in history-as well as two non weather-related power grid failures of major proportions. Through it all, the Sacramento NOC has functioned without a hitch.

Redefining power backup Coming up with the appropriate amount of power backup required Pacific Bell to overcome two types of hurdles: organizational and technological.

Organizationally, the surveillance task had long been classified by Pacific Bell as an administrative function and was handled by about 20 administrative offices scattered around northern California.

Surveillance equipment at those locations was backed up by uninterruptible power supplies (UPSs) capable of providing only 15 minutes of battery backup. Those UPSs also were unreliable and had other shortcomings. By contrast, the 911 system and critical switches had network classification and were provided with three hours of battery backup plus a travel time allotment.

Personnel at the administrative offices sometimes were unaware of system difficulties until well after they had occurred, causing managers to become increasingly frustrated.

Building a new NOC provided the telco with an obvious opportunity to reclassify surveillance as a network function and move to a more reliable means of providing power backup-preferably DC power. Yet technologically, the best approach to more reliable power was far from clear. The NOC required 20,000 A of backup power, necessitating the creation of one of the largest power plants ever installed in the U.S.

Pacific Bell sought the advice of Reltec and its Lorain Power Systems. Reltec engineers suggested paralleling two 10,000 A power plants to create one large -48 V plant. Such a procedure was once common in the industry, but other power advances had overshadowed it to the point that it verged on becoming a lost art.

In fact, wiring power plants in parallel has become considerably less complicated than it was in the days of Western Electric, when end cell battery strings were in use. Reltec suggested that the NOC power backup could be built with space-saving common battery strings that would service both of the 10,000 A power plants.

By operating the plants in a power-sharing mode, each could carry half the load. Each one would control its own string of rectifiers, yet be able to recognize that its sister plant had power sharing capabilities.

Although such a system can cost three times more than a comparable UPS-supported AC system, the argument that DC power would prove much more reliable in operation carried the day.

Much of the added cost of this DC power system stems from the requirement to first produce DC by rectifying the AC supply and then inverting some of the DC into AC needed by critical items within the NOC. To reduce these inversion costs, Pacific Bell tried to minimize the number of NOC equipment that required AC.

This proved impossible for such core items as the SunSPARC computers and the large overhead theater screens. But a surprising number of devices, including the Cabletron routers, modem shelves, Datakit computers and optoelectronic signal conversion equipment, were available with DC powering. In all, Pacific Bell was able to hold down the number of 40 kVA inverters to 14, saving about $1 million in inverter and extra equipment costs.

In the unlikely event that some parts of the backup system fail, critical AC elements will continue receiving inverted battery power, while less important elements such as some of the lighting will not. The worst that could happen in such an instance would be partial lighting for five minutes until the two 1.75 mW Caterpillar alternators kick in and begin delivering AC power.

As added insurance, Pacific Bell installed a portable AC tap box at a convenient outdoor location near the NOC. If the indoor alternators fail to fire up, technicians can drive up with a huge portable generator, plug it into the tap box and power the building.

Other challenges To monitor power plant operation, the system employs Lorain Smart Data Gathering Units. These units gather intelligence in event of failures and assure better technician response. But Pacific Bell does not wait for an emergency before finding out whether its power backup system will work.

It exercises the system monthly, first going from the utility AC power supply to DC from the battery strings, and then to AC from the alternators. It then follows the reverse procedure until the system once again draws its energy from the power utility. Nothing is left to chance.

In addition to matters of cost and reliability, Pacific Bell emphasized the need for aesthetics. Its own strict seismic installation standards also had to be met.

This meant the cable racks had to be installed with the cable neatly secured and symmetrical with earthquake bracing. It also meant that Lorain would install a single 1/2 inch bus bar rather than two parallel 1/4 inch bus bars to carry much of the electrical load around the two power plants. The one large bus bar not only looked better than two smaller bars, but it also cost less to install.

The opportunity to lay out a new plant from scratch also gave Pacific Bell the rare opportunity to provide and allocate space for growth. The NOC has space designated for six additional rectifiers and inverters and space allocations for four battery strings. There is also room for miscellaneous loads that eventually will be distributed to various parts of the building.

The NOC is a far cry from the cramped spaces many older power plants occupy. Such sites typically require major re-engineering and revamping to accommodate additional power equipment.

In the power backup business, no news is good news. And for the last two years, Pacific Bell has enjoyed considerable good news in Sacramento.

By rethinking its operations so that it could rely on proven technology-including some on the verge of being forgotten-Pacific Bell and its new network surveillance system proved able to withstand some of the most severe tests of any telecommunications system anywhere in the world. As the telecommunications industry faces this period of revolutionary change and rapid introduction of new technologies, it is comforting to know that telcos can keep it all running with the help of a straightforward, well-tested technology like DC power backup.

Laura J. Redeker is Senior Engineer and Bill Catron is Senior Systems Analyst for Pacific Bell in Sacramento.