1/8/2002 -- You probably know that wide area networks (WANs) are more than that opaque cloud shown on user network diagrams -- a lot more. But do you know exactly what's going on under the surface -- how they're structured, how they work?

Even if you're not directly involved in designing these networks, it's interesting to peek under the hood. Plus, as a networking professional -- any kind of networking professional -- the future of WANs will affect you: Many of the technologies first put to use in WANs eventually find their way to local area networks (LANs).

In a couple of previous articles for CertCities.com, I’ve gone into some detail of the workings of fiber optic technology as applied to LANs, typically in a building or on a campus (click here and here for a refresher). Now, let's step back and move up the networking ladder to look at the bigger picture, the architecture of WANs carrying both voice and data traffic. When we're done with that, I'll spend a few words on what we might expect to see in the future.

The Traditional Voice Network
Basic telephone service between your phone and the local central office pretty much still operates as it has for the past hundred-plus years: A copper twisted pair wire carries an analog signal. Back when phone calls were manually connected, operators plugged two wires -- known as "tip" and "ring" -- into a connector. At the central office (CO) these days, the pair of wires coming from each customer is terminated in a Subscriber Line Interface Card. There, the analog signal is converted into an 8-bit digital signal at a bandwidth of 8 kHz. The resulting 64 kilobit/second (kbps) digitized voice signal, known as a DS0, is the basic building block on which the voice network is built.

In the U.S., 24 DS0s are combined with some control information into a DS1 running at 1.544 mbps. A T-1 line provides a DS1 running over two twisted pair wires. T-1s are commonly used from a CO to a customer’s premises to provide either digital voice service to a customer’s PBX (which does the combining of analog voice lines and digitization) or data service when connected to a channel service unit (CSU) and router. Outside the U.S., E-1 lines are used, which are similar to T-1s except that they're comprised of 32 channels rather than 24 and operate at 2.048 mbps.

The next common speed in the hierarchy is the DS3, consisting of 28 DS1s (672 DS0s); T-3s carry DS3 signals over coaxial cable at 44.736 mbps (outside the U.S., E3s carry 16 E1s at 34.368 mbps). After that, fiber is used.

The optical physical layer hierarchy of speeds begins with Optical Channel 1 (OC-1) at 51.84 mbps, though that's rarely used nowadays. The "slowest" commonly used speed is OC-3, at 155.52 mbps; up from there are OC-12, OC-48 and OC-192 at 622.08 mbps, 2.488 gbps and 9.95 gbps, respectively. OC-768 (39.81 gbps) is now being tested, but it will be a couple of years before it's deployed in live networks. Outside the U.S., the same speeds are used with different names: OC-3 is STM-1, OC-12 is STM-4, etc. These are all raw bit rates on the fiber, and don't directly reflect the data rates or number of voice channels being carried.

Data can be carried on OC-n channels directly using asynchronous transfer mode (ATM), a circuit-switched technology using fixed 53-byte packets (called "cells" in ATM terminology.) However, in the voice network, the protocol used is Synchronous Optical Network (SONET) or, outside the U.S., Synchronous Digital Hierarchy (SDH). SONET defines a set of Synchronous Transport Signals (STS) corresponding to the OC-n carriers -- STS-1 is carried on OC-1, STS-3 on OC-3, etc. Each STS is comprised of a number of DS1s or DS3s. Along with carrying large numbers of voice channels, SONET provides a rich set of operations, administration and maintenance (OAM) features. Implemented in rings, the SONET protocol provides for restoration of service in under 50 milliseconds when a fiber is cut. Virtually all of the voice backbone is carried over SONET, both over long distances and in local, metropolitan area rings. Connections to a SONET backbone ring are made over add/drop multiplexers (ADMs) that insert channels into specific time slots in the main carrier. For instance, an OC-48 ring may provide OC-3 or OC-12 drops (carrying STS-3 or STS-12 signals) to a number of central offices along the ring. Each of these channels is assigned a permanent time slot in the backbone, whether or not any information is being carried at a particular point in time. This is known as a time division multiplexed system, and it is well suited to circuit-switched voice connections.

Let's look at the end-to-end workings of a typical phone call. The caller picks up the phone and establishes a circuit to the local central office. Using the information contained in the numbers dialed, the CO decides to send the DS0 digitized signal to a remote location. The CO packages that DS0 up with many more DS0s and DS1s into an STS-3 on its OC-3 to a long-distance carrier’s point of presence (POP). The long distance carrier takes the OC-3/STS-3 to an ADM where it's put on its OC-48 backbone. At the destination’s central office, another ADM picks off the OC-3/STS-3 and delivers it to the local carrier at the POP there. The local carrier then pulls out the DS0 signal and puts it on a pair of copper wires to the called party’s premises.

It's important to note that while many calls are sharing the OC-3 and OC-48 fibers, all getting on and getting off at different points, that little DS0 phone call has the dedicated use of a 64 kbps circuit from end-to-end for the duration of the call. It's as if, before starting off on a cross-country car journey, you have enough pull with the authorities that they ensure in advance that you have an open lane on your local road, then onto the interstate freeway with your own dedicated lane, then off and onto the local road to your destination -- clear sailing the whole trip.

Data over the Voice Network
As legacy fiber backbones are implemented with SONET, and most Internet service providers rely on the local voice carrier’s existing physical infrastructure to customer premises, data connections to ISPs are handled exactly the same way as voice calls. A router takes the Ethernet IP packets destined for the outside world and puts them into something that can be carried on a DS1/T1, typically ATM (although sometimes the IP data may be carried directly using point-to-point protocol, a.k.a. PPP.) Two copper pairs carry the T1 to the local carrier’s central office. There, the DS1 signal may be handed over directly to the ISP’s network -- if the ISP’s data carrier happens to have POP at that CO. More likely, the local carrier packages that DS1 up with other DS1s and DS0s onto its OC-3/STS-3 connection to its own OC-12 or OC-48 backbone, and it's picked off at another location that has a POP for that ISP’s network. From there, the ISP’s network combines it with other data streams onto its backbone and peels it off at its facility. Just like the voice call, a 1.5 mbps circuit is dedicated end-to-end throughout the local carrier and ISP networks; the only difference is that it's "tacked up" all the time rather than established by going off-hook and dialing a number. The actual data coming from the customer is in the form of packets, but the connection to the ISP is circuit-switched. Once the data arrives at the ISP, it is reassembled into (typically Ethernet) packets and either sent to a local application server (e.g., a DNS server to convert a URL to an IP address) via the ISP’s internal LAN or routed to its destination’s ISP via an ATM or IP-Over-SONET backbone network connection.

Now, there are a couple of problems associated with using a network engineered for voice circuits to route data. The use of circuits that dedicate bandwidth from endpoint to endpoint is inefficient because -- unlike a phone call -- the data comes in bursts. The idle time between data packets can't be used for other packets on that same circuit, unlike an Ethernet LAN. Also, the data rates on the traditional voice network don't match up well with the speeds used in LANs. In order to provide a customer with a 10 mbps connection, a carrier must provision an entire 45 mbps DS3 or 52 mbps STS-1 circuit; for a 100 mbps connection, a 155 mbps OC-3/STS-3 must be used, and so on. Also, there's considerable overhead associated with putting IP packets over layers of ATM and SONET protocols. It would be much more efficient if data that starts out as IP remains so throughout the network.

Overlay Data Networks
A number of new carriers have started up based on that premise -- IP all the time, every time -- with Yipes, Cogent and Telseon being the best-known. Rather than use SONET running on OC-n, these carriers use native Ethernet running on fiber in WANs, just as it can be used in a LAN (see sidebar, "Basic Concepts"). Typically employing gigabit Ethernet backbones, these "overlay networks" use fibers totally independent of the existing voice network. Ordinary layer-3 switches can provide customers with connections running at 10,100 or 1,000 mbps at an equipment cost a fraction of that of SONET ADMs. The downside of this architecture is that vanilla Ethernet and IP lack the full set of OAM capabilities supported by SONET. For example, Spanning Tree can take seconds or minutes to recover after a fiber break, compared to the 50 ms SONET maximum. And, of course, an overlay network requires a fiber infrastructure, so that the overlay carrier must either rent dark fiber or lay new fiber.

Voice over the Data Network?
Since packet-switched networks make more efficient use of bandwidth than circuit-switched networks for non-constant data streams, why not just take the digitized voice call, package it up in packets, and throw it on the data network? Then, presumably, all kinds of facilities could be shared and gobs of money saved. You pay for a data link to an ISP to be up all the time; why pay a long distance carrier to route your phone calls to Bangladesh when you can get them there for free posing as data? Who needs a PBX when you can just have a phone that packetizes the voice and puts it on your LAN? In fact, who needs a phone? Just hook a microphone and speaker up to your PC for unlimited, free long distance service.

A few years ago, a lot of people started thinking exactly this. Unfortunately, once you raise your standard of quality for a voice call above that of a first-generation cell phone call routed via satellite, a number of technical problems arise. On a voice call with a dedicated circuit, there's a deterministic (and very small, barring a satellite connection) amount of latency; you speak and a very short time later the other person hears what you said. Packet-switched networks such as Ethernet don't have deterministic latency; if the network is congested, it takes longer to get through than when it's idle. And once those packets leave the LAN and head out to the Internet, they may encounter any number of delays going hither and yon through the route that the carrier views as least costly rather than fastest, a route that may constantly be changing with network congestion. More than about 300 to 500 ms of latency becomes noticeable and annoying. (Remember what it's like to make international calls routed via satellite?) Tricks such as buffering the data can help but, ultimately, what's needed is some sort of guarantee that your voice call’s packets will take priority over that multi-gigabyte file transfer hogging the network. Tremendous amounts of time and effort have been spent over the last few years to develop and standardize Quality of Service (QoS) functions to distinguish between time-critical and background transfers, and equipment implementing these capabilities is now being deployed.

Convergence of Voice and Data Networks
So we now we have packet data or traditional DS0 voice circuits riding on time division multiplexed SONET networks engineered for voice and packetized voice running on data overlay networks. An ideal network would combine the two, taking the best features of both to optimize for the particular type of information to be transported. The easiest solution, already shipping from a number of suppliers, is to use Dense Wavelength Division Multiplexing (DWDM) to allow packet-switched and circuit-switched networks to share the same fiber. DWDM uses different wavelengths (or colors) of light to establish multiple channels on one fiber. These channels are independent of each other and have no knowledge of what's being carried. So you could have a single fiber pair carrying, say, a dozen OC-12 SONET channels, a dozen OC-48 SONET channels and a dozen native gigabit Ethernet/IP channels. DWDM equipment at each end of the link accepts the individual channels from SONET add/drop multiplexers and gigabit Ethernet/IP switches and "grooms" the channels together onto one fiber pair. The reverse happens at the other end of the connection, with the individual electrical signals separated and sent to their respective termination equipment. This increases the capacity of the fiber, but doesn't provide SONET’s OAM capabilities to the data channels, nor does it make the SONET channel more efficient when it comes to carrying packet data.

Future Technology
At this point, there are two main paths being followed to next generation voice and data network convergence. The first is being driven from the traditional telecom side, enhancing the SONET specification to be more data-friendly. A "virtual concatenation" protocol has been proposed that provides for flexibly tailoring bandwidth by combining groups of slower circuits; for example, 10 mbps Ethernet could be carried on a VT-1.5-6 (basically six DS1s) 10.368 mbps link rather than a full 51.84 mbps STS.

Note that although SONET is currently a single-wavelength protocol, efforts are underway to add capabilities tied to spanning multiple wavelengths. And at present, there's no standardized way to pack non-voice data into the SONET frame, so equipment from different manufacturers may not interoperate; however, a standardized data framing protocol is now being developed. For the long term, the next-generation replacement for SONET, the Optical Transport Network (OTN), is now being mapped out by the International Telecommunications Union.

On the other side of the fence, the traditional datacom is seeking to add features from SONET into packet-switched networks so that voice can be better carried there. Focused around gigabit Ethernet and 10 gigabit Ethernet, these standards are being developed under the auspices of the IEEE. QoS provisions are centered around IP with Multi Protocol Label Switching (MPLS); this allows differentiation of traffic types across the internetwork. In the longer term, development of the Resilient Packet Ring standard (IEEE 802.17) brings many of the SONET restoration and other OAM capabilities to IP networks. RPR, however, is close to but not gen-u-wine Ethernet, so there remains a question of whether the economies of scale found in the Ethernet world would be available there. And, finally, to allow another step between the TDM SONET and packet-based Ethernet world, 10 gig Ethernet is, in theory, able to be packed into an OC-192 SONET frame on the physical layer.

So which of these approaches will dominate in the future? Hard to tell at this point, especially with the current business environment in the communications business. Perhaps the current lull in the procurement of WAN equipment by carriers will have an unexpected benefit, in that standards for convergence can be developed in less of a frenzied time-to-market atmosphere. And when carriers do begin buying again, their primary concern will be to deploy the types of gear that best enable them to differentiate higher-value services for increased profits, rather than just throwing bandwidth at the network.

A final thought: It was only 24 years ago that the first live fiber optic connection in the field began carrying phone calls. Given the rate of change in the technologies of physical layers and protocols since then, it will certainly be interesting to see what the networking big picture looks like in 2026!

More articles by Randy Bird:

• Video Extension Technologies 101: A Primer
  
• SONET Architecture 101: A Tutorial for Datacommers
  
• Emerging Optical Technologies: Resilient Packet Ring and Passive Optical Networks
  
• Cranking It Up Again: 10 Gigabit Ethernet Arrives