6/4/2002 -- Just when we all were starting to get comfortable with the idea of Ethernet links running at a billion bits per second, there they go again upping the ante. On June 13, the IEEE is expected to ratify the standard for 10 Gigabit Ethernet, known as IEEE 802.3ae. The standardization effort has been underway for several years, and a number of prestandard products began shipping in 2001. With ratification of the official document, mass production of the various elements will begin, and the technology will begin down the same volume/cost curve that 10, 100 and 1,000 Mbit Ethernet products have seen. This article will outline some of the details of the specification and how it is expected to be applied to both local and wide area networks in the near future.

In a Nutshell
As with earlier Ethernet speeds, 10 Gigabit Ethernet is a blend of old and new technologies, leveraging what has been learned since the original DEC-Intel-Xerox 10 Mbit Ethernet concept. The key concepts of 10 Gigabit Ethernet are:

• Maintain the standard 802.3 Ethernet frame size and format, so that layer 3 and higher protocols are preserved.
• Operates over point-to-point links in full duplex only.
• Defines two types of physical layers: LAN PHY operating at a data rate of 10.000 Gbps, and WAN PHY operating at a data rate compatible with the SONET OC-192c / SDH VC-4-64c (9.5846 Gbps).
• Supports singlemode and multimode fibers over a number of specified distances. No copper version has been defined.

Yep, It's Still Ethernet
Remember, Ethernet defines Layer 1 (Physical) and Layer 2 (Link) layers in the OSI reference model. The theory behind this model is that if the interfaces between layers remains the same, different technologies may be substituted for the various layers without affecting the others. Here, the objective was to provide faster data rates on Layers 1 and 2 without impacting the rest of the protocol stack (IP and TCP, for example). In order to do this, the minimum and maximum frame sizes have been retained. Note that this precludes the use of "jumbo" frames, which have been sort of kicking around in proprietary formats since the Fast Ethernet days. Jumbo frames provide more efficient transport of large blocks of data since fewer frames must be processed by the protocol stack, but are deemed to break the definition of "Ethernet." In addition, the 802.3 frame format remains the same, with 8-byte preamble, 6-byte destination and source addresses, 2-6 byte length/type/tag fields, 46-1500 byte data field, and 4-byte frame check sequence all remaining in their familiar positions. This allows the vast installed base of Layer 3 and higher protocol stacks and tools to be unaffected by the new standard.

Well, Maybe a Little Change
Back in the good ol' days, one of the defining characteristics that made up the Ethernet model was the carrier-sensing multiple-access with collision detection (CSMA/CD) protocol. This was used on half-duplex systems tied together with repeaters. With the advent of full-duplex, switched networks, CSMA/CD and the distance limitations it imposed became unnecessary. While the Gigabit Ethernet standard defined a half-duplex mode of operation for historical consistency, nobody chose to implement it and CSMA/CD became kind of an appendix in the networking world. Since the distance limitation that half duplex operation would have imposed at 10 gigabit speeds would make such systems impracticable, this historical vestige has finally been thrown overboard and only full duplex operation is supported.

Planning for Convergence
When the gigabit Ethernet standard was ratified and the next step was being considered, it was observed that the SONET OC-192 line rate was very close to 10 Gbit/second, and so maybe the 10 Gigabit Ethernet standard should actually be 9.58 Gbit/second for compatibility. Well, this did not go over well with folks that have been used to just multiplying by ten, and so eventually a compromise was reached. 10 Gig Ethernet will support two different physical coding sublayers, a simpler LAN PHY designed to provide a data rate of 10.000 Gbps and a WAN PHY that adds a number of functions to make it compatible to some degree with SONET.

The LAN PHY
The simpler LAN PHY comes in two flavors, a serial version and a 4-channel Wide Wave Division Multiplexing (WWDM) version. Remember, the fast Ethernet and Gigabit Ethernet fiber PHYs used a 8B/10B encoding to provide actual line rates of 125 Mbps and 1250 Mbps, respectively. After some discussion, 10 Gigabit Ethernet settled on a more efficient 64B/66B encoding for the serial PHY, resulting in a line rate of 10.313 Gbps. The various physical media interfaces (optics) are discussed below. The WWDM PHY was added in the anticipation that the slower optical interface could result in a less expensive system; another alternative considered and discarded was using a four-discrete-fiber system. The WWDM interface basically uses four parallel channels running at 2.5 times the Gigabit Ethernet speed, hence each having a line rate of 3.125 Gbps. Again, remember that the actual data rate (MAC to PHY) is 10.000 Gbps; the encoding is used to avoid long runs of ones or zeros that could affect the receiver's ability to recover the clock and data. The LAN PHY is expected to be used in most LAN applications, and in WANs where dark fiber is used to carry 10 Gigabit Ethernet directly.

The WAN PHY
The WAN PHY, new to 10 Gigabit Ethernet, is designed to support connection to SONET/SDH circuit-switched networks. It adds a WAN Interface Sublayer (WIS) to the LAN PHY. The WIS takes the data payload and encapsulates it into a simplified SONET OC-192c (concatenated) frame that carries a payload of 9.58464 Gbps at a physical line rate of 9.95328 Gbps. The structure of a SONET frame is too complicated to go into here, but suffice it to say that it carries Operations, Administration, Maintenance, and Provisioning (OAM&P) and various other overhead bits in certain positions within the frame. The WIS does not support all of these functions, but fills in fixed values for most fields and calculates others, stuffs the Ethernet frame into the SONET data payload, and provides a SONET-compatible polynomial scrambling of the frame. This allows the frame to be recognized and managed by SONET equipment that it passes through, allowing for performance monitoring and fault isolation. Because of this overhead and the 64B/66B encoding, the actual data rate supported is approximately 9.29 Gbps, and so the WIS must have a mechanism to buffer data and adapt to the 10.000 Gbps MAC data rate. This nonetheless is much easier (and hence should be cheaper and faster) to do than the full protocol conversion required for Packet over SONET or other encapsulation schemes.

However, this WAN interface is not true SONET. The "S" in SONET (or the European equivalent, SDH) stands for Synchronous, where all points on the network are synchronized to an excruciatingly accurate master clock. Ethernet is an asynchronous system, where each receiving device derives clock and data from the incoming stream and re-times the outgoing characters with a local clock. Furthermore, SONET has very tight specifications for jitter and the optical interface that add significant cost to those systems. Hence, the 10 Gigabit Ethernet output from a device with a WAN PHY does not connect directly to a SONET ring, but requires an access device. The theory is that use of switches or routers with WAN PHYs connected to the appropriate access devices allows simple extension of Ethernet links over the SONET infrastructure, but there remain some question of how the different timing specifications of the synchronous and asynchronous systems will interact in the real world.

Physical Media
Four different physical media dependent interfaces (optics) have been defined to support singlemode and multimode fibers over defined target distances.

• 850 nm serial, for use on multimode fiber over short distances. The specification calls for this transceiver to support 65 meters over standard 50/125 µm multimode fiber having a bandwidth of 500 MHz-km at 850 nm Note that this does not officially support the more commonly installed 62.5/125 µm fiber with a bandwidth of 160 or 200 MHz-km at 850 nm, though I suspect many installations will try to use it for shorter distances with varying degrees of success. A newer 50 µm fiber with a bandwidth of 2,000 MHz-km is now being marketed by several suppliers, but as the detailed parameters have not been characterized and standardized, there is as yet no official supported length for this media.
• 1310 nm serial, for use on singlemode fiber up to 10,000 meters. Unlike Gigabit Ethernet, there is no provision for using this transceiver on multimode fiber with modal conditioning patch cords.
• 1550 nm serial, for use on singlemode fiber up to 40,000 meters. Gigabit Ethernet did not have a 1550 nm PMD, but many suppliers have been successfully selling nonstandard products with ranges up to 80 km.
• 1310 nm wide wavelength division multiplexed, for use on singlemode or multimode fiber. When used on standard singlemode fiber, the same 10 km distance is specified as for the serial transceiver; the choice would depend on the economics, with the WWDM perhaps being less expensive at first because of the slower lasers. However, the serial transceiver catches up when high-speed 1310 nm VCSELs (vertical cavity surface emitting lasers) are available. For installed 62.5/125 µm fiber with 500 MHz-km bandwidth at 1310 nm ("FDDI-grade" fiber), the WWDM transceiver will support 300 meters. Since the 1000Base-SX specification called for 220 meters, 300m should suffice for most existing installations needing to upgrade from Gigabit Ethernet.

(Editor's Note: For more information on fiber optic types, bandwidths, distances, etc., please see Randy Bird's two fiber optics articles on CertCities.com: "Fiber Optics 101, A Primer" and "Fiber Optics 102, Implementation Considerations.")


The Name of the Game
With all of these permutations of PHY and optics, quite a number of official names are needed to differentiate the interfaces. This seems to be the trend: Fast Ethernet has two, 100Base-TX and -FX; Gigabit Ethernet has three, 1000Base-TX, -SX, and -LX; and 10 Gigabit Ethernet has 10Gbase-x with seven possible suffixes: An "S" indicates 850 nm serial; an "L" indicates 1310 nm serial; an "E" indicates 1550 nm serial; and "LX4" is used for the 4-channel WWDM. Further, for the serial interfaces, the LAN PHY is indicated by an "R" and the WAN PHY by a "W" (the LX4 does not have a WAN PHY defined.) So to summarize:

Applications
So who needs all of this speed? Well, you won't be installing a NIC in your 486 system anytime soon. 10 Gigabit Ethernet is not targeted to the desktop, since not only are there no applications requiring that speed, even if there were, the CPU would be overwhelmed.

The first and most obvious use is in aggregating Gigabit Ethernet segments in large enterprise and service provider data centers. Switch ICs have already been announced with 8 gigabit ports and a 10 gig uplink, and suppliers of large chassis-based switches have been shipping prestandard 10 Gigabit blades since late in 2001. Dark fiber Metropolitan Area Network applications include point-to-point links between customers and service providers and backbone connections for Layer 2 or Layer 3 overlay networks serving a number of customers. And with the WAN interface, 10 Gigabit Ethernet is not limited to dark fiber overlay networks, but could be transported over the existing SONET/SDH infrastructure, sharing fibers and equipment with traditional TDM voice via DWDM. Another MAN/WAN application that is being discussed is supporting remote storage area networks, or remote backups and disaster recovery, using the iSCSI protocol.

What Does It All Mean?
Over the last 20 years, Ethernet has proven to be a flexible, scaleable, and cost-effective technology that has changed the face of computing beyond wildest imagination. I can remember seeing an early wire-wrapped prototype for AMD's first Ethernet chip set that packed two 15"×15" boards; now Gigabit Ethernet NICs that run at 100 times the speed using a couple of chips are available for under $100. As usual, when it first comes out the only ones who will be able to afford the cost premium of 10 Gigabit Ethernet will be those who are faced with even more expensive alternatives to get the bandwidth they need. However, as with its predecessors, when volume increases and costs come down it will become more commonplace. So even though 10 Gigabit Ethernet isn't likely to show up on desktops any time soon, it will be an enabling technology for new applications, such as serverless buildings, that will impact you over the next few years.

Questions or comments about this article? Post 'em below!

More articles by Randy Bird:

• SONET Architecture 101: A Tutorial for Datacommers
• Emerging Optical Technologies: Resilient Packet Ring and Passive Optical Networks
• Understanding WANs: A Technical Primer