In a Storage Area Network environment most IT companies use Fibre Channel and Fibre Channel over Ethernet architectures for user data access at ease.

Fibre Channel communication can be conducted over copper coax cables, twisted pair cables, or optical fiber. This chapter describes the components used to transform electrical signals to optical signals, and vice versa, and the most common types of optical fibers. It also identifies some of the factors that lead to fiber-optic signal losses.

Table of Contents

• Fibre channel functions levels
• Fibre channel connectors
• Fibre Channel cabling
• Fiber-optic class signal loss — Attenuation
• Cable bending and damage
• Fiber Channel-1 coding layer
• FC-3 common services
• FC-4 ULP mappings
• What is Fibre Channel over Ethernet?
• Fibre Channel Over Ethernet mapping
• Fibre Channel Over Ethernet lossless Ethernet Infrastructure
• Fibre Channel over Ethernet Components
• FCoE advantages and limitations

Fibre Channel Function levels

Fibre Channel is structured as a set of five hierarchical function levels.

• FC-0 is the physical level that defines connectors, cables, and the electrical characteristics of transition.

• FC-1 is the encoding level, which defines the encoding and decoding and the transmission protocol.

• FC-2 is the signaling and framing protocol level. It determines how the data from the upper level is framed for handling by the transport level, and it incorporates the management of frames, flow control, and cyclic redundancy checks.

• FC-3 is the common services level, which is open for future implementation.

• FC-4 is the protocol mapping level. It is usually provided by the device drivers from the different vendors, and it establishes the interface between Fibre Channel and the upper-level protocols.

FC-0—Physical level

Defines the physical link in the Fibre Channel system

• Transceivers

• Connection

• Media type

Available data rates

• 133 Mbit/s

• 266 Mbit/s

• 531 Mbit/s

• 1062 Mbit/s

The lowest architectural level defines the physical links in the system, including the fiber, connectors, optical, and electrical parameters for a variety of data rates.

The physical level is designed for the use of a large number of technologies to meet the widest range of system requirements. An end-to-end communication route can consist of different link technologies to achieve the maximum performance and price efficiency.

This section takes a closer look at the physical link components.

To be able to transmit data, you need transceivers. The most common way of transmitting data is to use light-based fiber optics. The use of electronic signals is the traditional and slower way of transmitting data.

The best modules to use today are the XFP and SFP transceivers. “XFP” stands for “Ten Gigabit Small Form-Factor Pluggable” and “SFP” stands for “Small Form-Factor Pluggable.”

SFP and SFP+ transceivers have the same size and appearance, but they support different standards. As a result, the less expensive SFP supports data rates up to four-point-two-five gigabits and distances up to one hundred fifty kilometers, and the SFP+ supports data rates up to sixteen gigabits and distances up to eighty kilometers.

Fibre Channel connectors

–SFP, SFP+, and XFP transceivers are compatible with the Lucent Connector (LC) type of connectors

–Cables containing LC connectors on both sides are known as LC-LC cables

An optical fiber connector terminates the end of an optical fiber and enables faster connection and disconnection than splicing. The connectors mechanically couple and align the cores of fibers so light can pass through. Better-quality connectors lose little light because of reflection or misalignment of the fibers. In all, about one hundred fiber optic connectors have been introduced to the market.

SFP, SFP+, and XFP transceivers are compatible with the Lucent Connector types of connectors. Cables containing LC connectors on both sides are known as LC-LC cables.

Fibre Channel cabling

Although it was initially designed for use with fiber-optic cable, Fibre Channel works well at shorter distances compared to copper cable in installations like storage area networks. In fact, the specification lists several different types of copper media that can support Fibre Channel.

The most common form of copper for Fibre Channel is shielded, twisted-pair cabling using DB-9 connectors—what looks like shielded telephone wire.

However, it is important to understand that copper cable for Fibre Channel needs to meet higher performance standards than conventional telephone wire. Properly specified and installed copper cable works fine for shorter distances, such as within a building, at speeds up to one hundred megabytes per second.

Common optical (glass fiber) cable types include:

• Sixty-two-point-five-micron multimode,

• Fifty-micron multimode, and

• Nine-micron single-mode.

Multimode Fiber

• Multiple streams of light to travel different paths

• Most popular for networking

• Fibre Channel uses single wavelength

–Example: 850 nm –

Multimode uses a shortwave laser to emit many different light modes. These reflect off the cable cladding at different angles, which causes dispersion. This dispersion reduces the total distance from which the original signal can be reclaimed.

Multimode fiber has a larger core than single-mode fiber. The larger the core, the greater the dispersion factor, hence the reduction in the distance that data, or light, can travel.

Single-mode Fiber

Single-mode is an optical fiber with a core diameter of less than ten microns. Used for high-speed transmission over long distances, it provides greater bandwidth than multimode fiber, but its smaller core makes it more difficult to couple the light source.

Increasingly, single-mode fiber is being used for shorter distances. When single-mode fiber is used in shorter distances, such as a campus or metropolitan area network, step-index fiber is used. For longer distances and for transmitting multiple channels, such as with WDM, dispersion-shifted fiber is used.

Single-mode step-index fiber

When moderate-distance transmission cannot be accomplished with multimode fiber and inexpensive multimode light sources, single-mode fiber is used. This type of fiber is most commonly used in private network, campus, and building applications.

Single-mode fiber is designed for use at both the one-thousand-three-hundred-ten-nanometer and at the one-thousand-five-hundred-fifty-nanometer wavelength windows.

Because the one-thousand-three-hundred-ten-nanometer lasers and detectors are less expensive than one-thousand-five-hundred-fifty-nanometer devices, most of these short-to-moderate distance applications use the one-thousand-three-hundred-ten-nanometer wavelength.

Single-mode fiber is the least expensive fiber available, and is optimized for the lowest dispersion at one-thousand-three-hundred-ten nanometers. It offers the best combination of cost and performance for most short-to-moderate distance private network, campus, and building applications when distances exceed multimode limits.

The information-carrying capabilities of the single-mode fiber are infinite. Single-mode fiber supports speeds of tens of gigabits per second and can carry many gigabit channels simultaneously. Each channel carries a different wavelength of light without any interference.

Fiber-optic class signal loss — Attenuation

Attenuation

• The reduction in power of the light signal as it is transmitted

• Caused by passive media components such as cables, cable splices, and connectors

The correct functioning of an optical data link depends on modulated light reaching the receiver with enough power to be demodulated correctly. “Attenuation” is the reduction in the power of the light signal as it is transmitted.

Attenuation is caused by passive media components such as cables, cable splices, and connectors. Although attenuation is significantly lower for optical fiber than for other media, it still occurs in both multimode and single-mode transmissions.

An efficient optical data link must have enough light available to overcome attenuation.

Fiber-optic class signal loss — Dispersion

Dispersion

• Spreading of the signal over time

• Two types of dispersion can affect an optical data link:

• Chromatic dispersion —Resulting from the different speeds of light rays

• Modal dispersion—Resulting from the different propagation modes in the fiber

Dispersion is the spreading of the signal over time. Two types of dispersion can affect an optical data link:

• The first type is “chromatic dispersion,” which refers to the spreading of the signal that results from the different speeds of the light rays.

• The second type is “modal dispersion,” which refers to the spreading of the signal because of the different propagation modes in the fiber.

For multimode transmission, modal dispersion, rather than chromatic dispersion or attenuation, usually limits the maximum bit rate and link length.

For single-mode transmission, modal dispersion is not a factor; however, at higher bit rates and over longer distances, chromatic dispersion limits the maximum link length.

An efficient optical data link must have enough light to exceed the minimum power that the receiver requires to operate within its specifications.

When chromatic dispersion is at the maximum allowed, its effect can be considered as a power penalty in the power budget.

The optical power budget must allow for the sum of component attenuation, power penalties —including those from dispersion, and a safety margin for unexpected losses.

Cable bending and damage

Bending is one of the primary causes of increases in attenuation in optical fibers. Two types of bending are macro bending and micro bending.

The macro bend has a much larger bend diameter than the fiber diameter. Here, the fiber coating has almost no impact on the optical loss because the light is guided in the core, far from the coating.

The coating cannot protect the glass (core and cladding) from being bent because the bend diameter is much larger than the fiber. 

The situation is the opposite for micro bending. Here the bending is local and the coating can protect the glass from external forces applied on the coating surface.

For this reason, many fibers have a two-layer acrylate coating, where the inner layer is soft and can accommodate for external forces acting on the fiber.

Fibers with a thin and hard coating such as polyimide do not have this protection from local bending and must be handled more carefully to avoid micro bending of the glass.

Fiber Channel-1 coding layer

FC-1 8b/10b encode/decode

• FC-1 defines the transmission protocol including:

• Serial encoding and decoding rules

• Special characters – Error control

• The information transmitted over a fiber is encoded 8 bits at a time into a 10-bit transmission character

Also used in:

• PCI Express

• IEEE 1394b

• Serial ATA

• SSA –Gigabit Ethernet

• Infiniband

FC-1 defines the transmission protocol, including serial encoding and decoding rules, special characters, and error control. The information transmitted over a fiber is encoded eight bits at a time into a ten-bit transmission character.

The primary reason for using a transmission code is to improve the transmission characteristic of information across a fiber. The transmission code must be DC balanced to support the electrical requirements of the receiving units.

FC-2 signaling protocol level

The transport mechanism of fiber channel

• Framing rules
• Payload
• Service classes and controlled mechanisms
• Management of the data transfer sequence

Building Blocks

• Ordered sets
• Frames
• Sequences
• Exchanges

The basic building blocks of a Fibre Channel connection are the frames. The frames contain the information to be transmitted (the payload), the addresses of the source and destination ports, and the link control information. Frames are broadly categorized as data frames and link-control frames.

A sequence is formed by a set of one or more related frames transmitted unidirectionally from one N_Port to another. Each frame within a sequence is uniquely numbered with a sequence count. Error recovery, controlled by an upper protocol layer, is usually performed at sequence boundaries.

An exchange is composed of one or more non-concurrent sequences for a single operation. Exchanges can be unidirectional or bidirectional between two N_Ports.

Within a single exchange, only one sequence can be active at any time, but sequences of different exchanges can be concurrently active.

FC-3 common services

• The FC-3 layer covers functions that can span multiple N-ports

• FC-3 defines the common services necessary for the higher-level capabilities

• FC-3 provides features such as:

• Port striping –RAID –Virtualization

• Compression

• Encryption

• Hunt groups

• Multicast

The FC-3 level of the Fibre Channel standard is intended to provide the common services required for advanced features such as striping, hunt groups, and multicast.

• Striping refers to multiplying the bandwidth by using multiple N_Ports in parallel to transmit a single information unit across multiple links.

• Hunt groups refers to the ability for more than one port to respond to the same alias address. This improves efficiency by decreasing the chance of reaching a busy N_Port.

• Multicast delivers a single transmission to multiple destination ports. This includes broadcasting to all N_Ports on a fabric and sending to only a subset of the N_Ports on a fabric.

FC-4 ULP mappings

• Each upper-level protocol supported by the Fibre Channel transport requires a mapping for its Information Units to be presented to the lower levels for transport

• The FC-4 layer provides these mappings for:

• SCSI-3

• IP

• High-Performance Peripheral Interface (HIPPI)

• FC-AV—A high-bandwidth video link for video networks, up to 500m

• FC-VE—Fibre Channel Virtual Interface Architecture

• FC-AE—Fibre Channel Avionics Environment

• Ficon, IEEE 802.2 LLC, ATM, Link Encapsulation, SBCCS, IPI

• A Fibre Channel SAN is almost exclusively concerned with using the SCSI-3 mapping

Each upper-level protocol supported by Fibre Channel transport requires a mapping for its information units to be presented to the lower levels for transport.

A Fibre Channel SAN uses the SCSI-3 mapping almost exclusively.

What is Fibre Channel over Ethernet?

Fibre Channel over Ethernet is a computer network technology that encapsulates Fibre Channel frames over Ethernet networks. This chapter describes FCoE and explains the benefits of using Converged Network Adapters, which combine the strengths of the Fibre Channel and Ethernet protocols in modern data centers.

What is Fibre Channel over Ethernet?

• Fibre Channel over Ethernet is a mapping of Fibre Channel over selected full-duplex IEEE 802.3 networks

• The goal is to provide I/O consolidation over Ethernet, reducing network complexity in the data center

• Customer benefits of a unified fabric:

• Fewer NICs, HBAs, and cables

• Lower capital expenditures and operating expenses

Fibre Channel Over Ethernet transports the SCSI storage data used in Fibre Channel networks. It uses the Fibre Channel Protocol stack instead of the TCP/IP stack, and it uses the Ethernet infrastructure, which has the NICs, cables, switches, and so on. The goal is to provide I/O consolidation over Ethernet, reducing network complexity in the data center.

Customer benefits of using a unified fabric include needing fewer NICs, HBAs, and cables, and lowering the capital expenditures and operating expenses.

Fibre Channel Over Ethernet I/O consolidation

I/O consolidation enables Ethernet and Fibre Channel to share the same physical cable and still maintain protocol isolation. It also enables you to use and configure the same type of hardware for either network.

Although being simple in concept, this configuration is complex. But the benefits from this idea are significant.

• By leveraging I/O consolidation, that is, by using a combined network interface card and HBA, organizations free up slots, providing a multifunction network and SAN.

• The reduced number of cards reduces power consumption, which in the case of PCI Express is twenty-five watts per card.

• There is also a reduced number of switch ports.

• Less power is consumed in the cooling process, which is the primary barrier to data center expansion and a cause of inefficiency at the present time.

Another advantage of I/O consolidation is that it will give enterprise organizations the means to simplify their cable management. At the moment, twenty gigabits of bandwidth can be provided by two four-gigabit Fibre Channel connections and twelve one-gigabit Ethernet connections.

Fibre Channel and Ethernet can be combined using two ten-Gigabit-Ethernet cables. This maintains the bandwidth but reduces the number of cables being managed by seventy-five percent. This also results in fewer points of management that administrators have to control.

Fibre Channel Over Ethernet mapping

• Fibre Channel Over Ethernet maps the Fibre Channel commands and data directly into Ethernet frames to create Fibre Channel Over Ethernet

• Fibre Channel frames are encapsulated in Ethernet frames

• The mapping is 1:1, meaning there is no segmentation or compression of the Fibre Channel frames

Fibre Channel Over Ethernet maps the Fibre Channel commands and data directly into Ethernet frames. The the mapping is one-to-one, meaning there is no segmentation or compression of the Fibre Channel frames.

But Ethernet is a lossy medium. It provides a single best-effort pipe that drops packets during a network congestion. So in Fibre Channel Over Ethernet , Fibre Channel is encapsulated and run over a lossless Ethernet infrastructure.

Fibre Channel Over Ethernet lossless Ethernet Infrastructure

• Fibre Channel over Ethernet has to create a lossless Ethernet environment to ensure the reliability of large-scale data transportation

• Two standards enable lossless Ethernet

• Data Center Bridging (DCB) –Converged Enhanced Ethernet (CEE)

• In addition to DCB and CEE, the standard introduces three enhancements to
  the Ethernet to make it lossless:

• Priority Flow Control (IEEE 802.1Qbb)

• Congestion Notification (IEEE 802.1Qau)

• Enhanced Transmission Selection (IEEE 802.1Qaz)

FCoE has to create a lossless Ethernet environment to ensure the reliability of large-scale storage data transportation. The two standards that enable this are Data Center Bridging and Converged Enhanced Ethernet.

A few of the enhancements to make Ethernet lossless are listed in this slide.

Priority Flow Control

Priority Flow Control (IEEE 802.1Qbb)

• IEEE 802.1Qbb is an enhanced QoS service

• Traffic is classified in 8 lanes, each of which could be assigned a priority level

• Priority Flow Control issues a “Pause” command to manage and prioritize traffic when there is congestion

• The administrators can create lossless (virtual) lanes for FCoE traffic and lossy (virtual) lanes for normal IP traffic

The Institute of Electrical and Electronics Engineers had defined the means to categorize traffic according to its priority in the Quality-of-Service standard IEEE 802.1p.

So the new standard IEEE 802.1bb takes advantage of the earlier standard by first classifying the traffic into eight “lanes,” each of which can be assigned a priority level.

Priority Flow Control issues a Pause command that halts FCoE traffic during congestion so the losses can be minimized. It uses the priority level to distinguish FCoE traffic from other types of traffic. This means that administrators can create lossless virtual lanes for FCoE traffic and lossy virtual lanes for normal IP-based traffic.

Congestion Notification

Congestion Notification (IEEE 802.1Qau)

• Congestion is measured at the congestion point, but link rate limiting is taken at the point of origin

• Example: An aggregation switch can ask an edge switch to stop (or limit) its traffic from a particular port, if congestion occurs

Congestion is measured at the congestion point in the network, wherever it is happening, but the action is taken at the reaction point, which is the originating point.

For example, an aggregation switch can ask an edge switch to stop or limit its traffic from a particular port if congestion is encountered.

Enhanced Transmission Selection

Enhanced Transmission Selection (IEEE 802.1Qaz)

• High-priority traffic such as FCoE is allocated with a minimum guaranteed bandwidth

• If the FCoE traffic does not fully utilize its reserved capacity, the extra bandwidth can be used by other types of traffic, and this can be controlled dynamically

High-priority traffic like FCoE can be allocated with a minimum guaranteed bandwidth so that all the other traffic on the network does not congest the path with its high volumes.

However, if the FCoE traffic does not fully utilize the path, its “reserved capacity,” then the extra bandwidth can be used by other types of traffic. The protocol can control this dynamically.

Fibre Channel over Ethernet Components

An FCoE configuration includes several components.

In this diagram, the first key component is the Converged Network Adapter. The CNA is a single adapter in the server that attaches to PCI Express slot. It can provide the functionalities of both Ethernet NICs and Fibre Channel HBAs virtually.

That means the server still sees two interfaces, and it sends the IP traffic to the NIC and the Fibre Channel traffic to the HBA. But the CNA collects from both of them and transports the data over a single Ethernet cable, after wrapping all the Fibre Channel frames to Ethernet frames.

The second key component in the diagram is the FCoE link. The FCoE infrastructure uses the same Ethernet infrastructure as the TCP/IP network. It uses UTP copper cables, optical fiber cables, and even the low-cost cables that use the SFP+ interface to carry ten-Gigabit-Ethernet over short distances.

The third component identified in the diagram is the set of FCoE switches and network switches that support the FCoE protocol. Fibre Channel SANs only understand the Fibre Channel protocol and only recognize Fibre Channel interfaces, so there needs to be an intermediary that separates the FCoE traffic from the regular IP traffic and that connects to the Fibre Channel SANs directly.

This intermediate functionality is provided by FCoE switches or network switches with Fibre Channel ports that support the FCoE protocol. The HBAs from the servers connect to the FCoE switch, which in turn connects to the SAN network using Fibre Channel ports and to the IP network using IP ports.

FCoE advantages and limitations

What are the advantages of FCoE?

• FCoE simplifies the network by reducing the two individual cables from each server and the two network adapters, which are the HBA for storage connectivity and the NIC for computer network connectivity, to just one.

• FCoE can carry traffic over the Ethernet medium and uses the familiar and easily available copper UTP cables and optical fiber cables.

• FCoE uses one network adapter instead of two, which results in some power savings for the server.

• Some I/O virtualization solutions support FCoE, which enables you to reduce the total number of server adapters for a group of servers by consolidating them onto an I/O virtualization appliance and allowing the servers to share the common pool of adapters. The servers themselves connect to the I/O virtualization appliance through interfaces like PCI Express and the appropriate cables from there. You should note that certain proprietary, vendor-based drivers might have to be installed to complete this setup.

• The performance of an FCoE network is comparable to that of Fibre Channel and IP networks, with FCoE currently supporting the speeds of Ethernet networks up to one, ten, or more gigabits per second. This speed is expanding to forty gigabits per second and one hundred gigabits per second.

• FCoE can be used in virtualized environments (server virtualization) and is quite advantageous in such circumstances. –FCoE, unlike iSCSI, is a reliable storage transportation protocol. It can scale up to thousands of servers.

• Because FCoE encapsulates the Fibre Channel data onto Ethernet frames for transportation only, all the existing administration tools and workflows for Fibre Channel remain intact.

Hence, the existing investment in Fibre Channel storage is preserved and the reliability of Fibre Channel is also maintained. The support for FCoE from network switch vendors strengthens the case of FCoE. These vendors are offering converged switches with both Ethernet and Fibre Channel ports.

Some disadvantages and limitations of FCoE include:

• The only Ethernet component that is currently compatible with FCoE is the cables. Everything else has to change to implement FCoE. This means that the actual savings would only be the amount and cost of cables.

• The cost of a Unified CAN, although it is coming down, might be more than the cost of the HBA and NIC combined.

• FCoE is currently restricted to access networks only (server-to-switch connections). The distribution and core storage networks are still in Fibre Channel and will continue to be in Fibre Channel until the FCoE technology matures enough that its own FCoE SAN networks can be created.

• iSCSI proponents might still argue that changing one disparate network into another does not amount to convergence of the storage and network infrastructures. –Security on FCoE networks might have to be re-evaluated because the network is now running over Ethernet, which is more easily accessed than Fibre Channel.