9/4/2002 -- Fibre Channel's slow early adoption resulted from high costs and the reluctance of administrators to invest in technology requiring an entirely new skill set. These same factors may accelerate the adoption of iSCSI, the highly anticipated Internet storage protocol, since organizations that have not yet made the leap to Fibre Channel will be able to use their TCP/IP expertise in iSCSI deployment. However, Fibre Channel is the runaway leader in SAN deployments at the moment, and will be until iSCSI products mature. Three standards, hopefully to be ratified later this year by the Internet Engineering Task Force (IETF), are contending in the Internet-connected storage field right now:

• FCIP (Fibre Channel over TCP/IP)
• iFCP (Internet Fibre Channel Protocol)
• iSCSI (Internet Small Computer Systems Interface).

A Little History: SCSI versus IDE/ATA
SCSI (Small Computer Systems Interface) and ATA (Advanced Technology Attachment) are the two prevailing standards for host bus adapters (HBAs). Once installed in a computer, an HBA adds an electrical path known as a "bus" to which other devices can be attached. ATA HBAs became the de facto standard in home computers because of their low cost; but the limitations of ATA were severe: maximum cable lengths of less than two feet meant that ATA devices had to reside close to the computer itself, and only two devices could be attached per bus. While ATA was perfectly suited to home users' requirements, these limitations hastened SCSI's success in the server market, where the need to attach many devices of different types outweighed considerations of cost. Newer SCSI adapters support over a dozen devices, and work reliably over longer distances25 meters or so. SCSI's advantages also spurred the development of a wide range of SCSI peripherals: scanners, digital cameras, and so on, though storage remains SCSI's primary application. The new serial ATA standard (the old standard was parallel) eliminates some of its disadvantages, but its primary advantage is still one of cost.

Fibre Channel
Fibre Channel is a serial interface that encapsulates other storage protocols or channels, including SCSI and HIPPI (High Performance Parallel Interface, a short-distance, high-speed interconnect usually found in supercomputers). SCSI is by far the most common channel used over Fibre Channel, which causes the two to be lumped togetherthus, you'll hear of "Fibre Channel drives," in reference to Fibre Channel devices that encapsulate the SCSI command set. The most common Fibre Channel implementation is the arbitrated loop, also known as FC-AL, which offers several advantages over SCSI:

• Physical separation of up to 10 km between devices, if fiber optic cable is used. Despite its name, Fibre Channel can also be transmitted over copper.
• Path redundancy stemming from the loop design. Interruption of one route to the data does not render it inaccessible.
• Theoretical scalability to 126 devices per loop, although a device population of 50-60 is more realistic.
• Faster throughput.

The New Protocols
Leaving the challenge of Serial ATA aside for now, SCSI and Fibre Channel are the best options for direct attached storage (DAS) in the server market. FCIP, iFCP, and iSCSI are the three primary contenders in the Internet storage race. All three extend the underlying Fibre Channel and SCSI protocols, thereby providing block-level access to data across TCP/IP networks, whereas their predecessors (NFS and CIFS) provided file-level access across the same. Although TCP/IP can be transmitted over many physical media, Gigabit Ethernet is the medium of choice for any of these net-generation protocols. The IP Storage Working Group is expected to ratify all three standards sometime this year. Even without ratified standards, of course, products are already coming to market that support the new protocols.

FCIP
Similar names notwithstanding, FCIP is actually a more distant cousin of iFCP than iFCP is of iSCSI. FCIP is a tunneling protocol that encapsulates whole Fibre Channel frames inside TCP/IP packets, allowing distant Fibre Channel SANs to be transparently connected. FCIP is appropriate for organizations that need to extend a SAN's reach for off-site backups, data mirroring, or other purposes.

FCIP is a gateway protocol; a FCIP gateway sits on each Fibre Channel SAN and performs the necessary encapsulation. The use of tunnelinganalogous to accessing a CIFS or NFS share over a Secure Shell session, for instanceinduces a performance penalty because of the extra overhead involved in extracting and processing the encapsulated frames. Additionally, since FCIP is simply a way to extend a Fibre Channel SAN by distance, latency must be negligible for FCIP installations to functionhence the reliance on Gigabit Ethernet. These limitations cause some pundits to regard FCIP as a temporary solution until iSCSI matures. However, in the three to five years before that happens, FCIP will have substantial impact in the large installed base of Fibre Channel Sans because of its simplicity: assuming sufficient network capacity, the installation of two FCIP gateways can provide immediate connection between distant Sans

iFCP
iFCP is an implementation of the Fibre Channel Protocol (FCP) that uses TCP/IP for transport, instead of tunneling one protocol inside another. This requires some explanation. Fibre Channel provides a type of network in and of itself typically known as the "fabric." Instead of the native Fibre Channel transport, iFCP uses the Internet Protocol, IP, to extend the fabric. Congestion control and error detection and recovery are provided by TCP. This allows individual Fibre Channel devices to be attached directly to iFCP switches without expensive Fibre Channel hubs and switches as intermediariesalthough you still need an iFCP gateway device. Needless to say, Fibre Channel Sans as well as individual devices can be connected to iFCP switches. Although it seems more progressive than FCIP, iFCP lacks widespread vendor support. Its main proponent, Nishan Systems, has trademarked a particular amalgamation of iFCP and other protocols as "Storage over IP," or SoIP. Also, iFCP is positioned as a gateway protocol, perhaps to its detriment; iSCSI, in contrast, will be included in storage devices themselves. With iSCSI, connecting individual hard drives to the Internet is theoretically possible, though the first and second-generation products are disk arrays.

iSCSI
Somewhat similar to iFCP, iSCSI is a method for translating SCSI commands into iSCSI packets that can be transported over TCP/IP, thus eliminating the distance and device quantity limitations of SCSI. Adapters with iSCSI capabilities are already available from QLogic and other vendors; however, iSCSI storage devices are still rare. IBM's TotalStorage IP Storage 200i, the first iSCSI disk array on the market, fell short of expectations. Tape library vendors seem eager to board the iSCSI bandwagon as well, but are also holding back until some improvements are made in iSCSI adapters.

TCP/IP is an extremely complex protocol. Thus, the overhead incurred in mapping every SCSI command onto an equivalent iSCSI transaction is extravagant. iSCSI adapters require extra onboard hardware dedicated to performing all TCP/IP calculations, known as TCP/IP Offload Engines (TOEs), which are just now coming to market. However, this means that a portion of TCP/IP configuration and management is separated from the operating system, which may make management of hosts equipped with iSCSI adapters more complex.

Wanton Forecasting
The iSCSI, FCIP and iFCP protocols are still taking shape. It's expected that the final versions of all three will include provisions for IPSec, the IP Security Protocol, since security of Internet-connected storage is obviously a huge concern; but encrypting IP traffic increases processing overhead yet again, consequently increasing costs. As always, standards compliance is necessary for advertising copy, but when a protocol is too unwieldy or cumbersome to implement, though, the market typically looks elsewhere: the snail's pace of IPv6 adoption and the all-but-complete supplanting of X.500 directory services by the simpler Lightweight Directory Access Protocol (LDAP) are cases in point.

Even so, the promise of iSCSI is intriguing. One of the iSCSI-related RFCs describes a method for booting clients from an iSCSI storage device. This would make deployment of large computing labs astoundingly simple; simply have each computer boot from an iSCSI drive. It also opens up options for service providers that should be frightening to the home user with an Orwellian cast of mind: rather than log on to America Online, you could actually boot into it. The diskless workstation is not a new concept, but iSCSI certainly makes it a more attractive option.

Since the competing standards all serve different purposes, the storage market should remain fragmented for the next few years. Initially, iSCSI should absorb some of the network-attached storage market. Vendors such as NetApp have already promised to include iSCSI support in their NAS devices. Adoption rates for Gigabit Ethernet, the iSCSI medium of choice, may impact the growth of this market, positively or otherwise. Issues of throughput are still contentious, as well. The newer 2GB Fibre Channel technologies mount a serious speed challenge to any of the Internet storage protocols, and with 10 Gigabit Ethernet looming on the horizon, investors are right to wait and see how things will play out.

For more on this topic, check out the following links:

• IP SANS: A Guide to iSCSI, iFCP, and FCIP Protocols for Storage Area Networks (Addison-Wesley Professional, 2001) --- The only comprehensive book to date on the topic.
• The PC Guide: SCSI
• Gary Field's SCSI Info Central -- Home of the venerable SCSI FAQ, and a good set of links.
• IP Storage RFCs
• IP Storage Working Group

• IBM TotalStorage IP Storage 200i -- The first (and only?) iSCSI-capable disk array on the market, now discontinued.

Personally, I'm waiting for the day when I have an Ethernet port on the back of my hard drive, and set the IP address using a row of jumpers; since it'll confirm my suspicion that nothing's really improved since the Macintosh LC.

Next time, we'll discuss the upcoming Serial ATA standard, talk briefly about Infiniband, and see what products are actually out there with Linux/Unix support.

Questions? Comments? Post your thoughts below!