By Dave Fredricks, Data Center Infrastructure Architect, CABLExpress
Published March 2019
Data speeds continue to increase. The collective appetite for bandwidth is seemingly insatiable, and networks are rarely seen as “fast enough.” Many entities are awaiting the next big jumps in speed: 400 Gb/s Ethernet and 128 Gb/s Fibre Channel. Supporting these speeds requires critical fiber cabling considerations.
From a fiber optic cabling perspective, rather than reacting to each upgrade of hardware (that comes with higher transmission speeds) by implementing a new cabling system, a planned and properly implemented structured cabling solution can reduce personnel hours and investment costs.
For this to be done properly, one must understand three concepts:
1. How the current and future generation of optics work.
2. The pros and cons of each fiber optic polarity option.
3. The cabling design options for your data center.
The purpose of this paper is to provide this information in a consolidated format, allowing you to make educated decisions when choosing a structured cabling solution that will last for multiple generations of hardware upgrades.
There are several key considerations to understand when relating the transmission speed of optics to data center cabling. First is that the optics (the active hardware component that cabling plugs into) are what drives the data transmission. In other words, the cabling is a passive component, with the role of carrying the data transmission from sender to receipt.
This partnership between optics and cabling needs to work in concert. There are standards bodies that provide a roadmap for the hardware and cabling manufacturers in order to accomplish this.
The second consideration is that there are two primary data transmission protocols used in the active hardware for data center connectivity. Ethernet is widely used in network connectivity, and Fibre Channel is used for data storage (SAN) connectivity.
The last key consideration to understand is “parallel” versus “serial” transmission. Serial (sometimes referred to as “duplex”) transmission uses a single fiber to transmit data, and a single fiber to receive. Serial transmission has been widely used for decades in the data center. As multi-mode, single fiber properties have a limited amount of data that can be transmitted without errors occurring, necessitating the development of another transmission method.
Parallel transmission divides the signal at transmission and aggregates it at reception (multi-plexing), thus allowing higher transmission speeds over multi-mode fiber. This is currently not necessary for singlemode fiber, as it’s carrying capacity for data is much higher.
The Institute of Electrical and Electronics Engineers (IEEE) released the 802.3bs for the Ethernet protocol as Physical Layers and Management Parameters for 200 Gb/s and 400 Gb/s Operation. The new standard replaced the existing one that was published April 2015. This new standard increased the speed per fiber strand from 25 Gb/s to 50 Gb/s on multi-mode glass.
It also introduced a new singlemode optic that reduced the maximum distance from 2 kilometers to 500 meters. This reduction in maximum distance also reduced the cost of the optic.
The T11 Technical Committee for Fibre Channel protocol released Generation 7 optics doubling the speed of Generation 6 speeds going from 32 Gb/s to 64 Gb/s on duplex (or two fibers) and 128 Gb/s to 256 Gb/s on parallel (or eight fibers). Additional information on parallel transmission will be provided in Concept 2.
A third organization to consider when designing a structured cabling system is from the Telecommunications Industry Association (TIA). The TIA has many published standards that will aid with design (see Figure 1).
The TIA-942-B standard (named Commercial Building Telecommunications Cabling Standard) shows how to deploy a structured cabling solution with the use of patch panels, conveyance pathways, and proper cabinets (see Figure 2).
Another standard is the TIA-568.3-D. This standard shows the different polarity (light path) options when using an MPO/MTP connector, which is used in the new Ethernet and Fibre Channel optics. Understanding these light path options is an important aspect, and we’ll cover it in detail in the next section.
The key takeaways from the latest IEEE 802.3 Ethernet, Gen7 Fibre Channel, and TIA-942-B standards are:
Plan for a structured cabling design to work with both duplex (LC connector) and parallel (MPO connector) optics. This will be needed to migrate to higher speeds that require parallel transmission for multi-mode fiber, and it’s an important consideration that affects both cost and flexibility. For distances under 100 meters, use at least OM4-rated multi-mode glass. For distances over 100 meters, consider singlemode glass to distances up to 500 meters.
Plan for patch panels at the Core, Director or Spine switches to have the ability to plug in both duplex and parallel optics. The break the patch panels provide in the link near these switches allows the horizontal or backbone cabling to remain in place for multiple generations of optics moving forward. Having a break in the link at the other end, typically a Server connection, allows the ability to plug in both duplex and parallel optics on the end. It also allows the user to breakout connections on the Server side, ie. 100G into 4 x 25G.
An optical loss budget is the minimum amount of light loss allowable between two connected transceivers. This loss amount is defined by the standards bodies as well as the manufacturers of the electronics.
Optical loss is measured in decibels (dB). This loss occurs in three ways:
Patch panels add flexibility to the structured cabling design, but inherently introduce optical loss at each interconnect point. Having these interconnect points is essential to support duplex and parallel optics. The optical loss incurred from the use of patch panels should not exceed allowable light budgets in order to support current and next generation optics. To ensure this, it is important to understand the maximum allowed loss of cabling products that are to be specified for the cabling infrastructure.
For more information on this subject, see the following white paper: Specifying Fiber Infrastructure as a Critical Network Component.
Figure 3 shows the latest specifications from IEEE 802.3bs to support speeds of 200 Gb/s and 400 Gb/s. Notice how OM4 multi-mode glass at a distance of 100 meters features a maximum channel loss of 1.9 dB and connector loss of 1.5 dB. The difference between the 1.9 and 1.5 is 0.4 dB, which is attenuation loss associated with the horizontal or backbone distance of the fiber trunking.
Figure 4 shows Fibre Channel specs for Gen6. Gen7 will carry the same distance and loss specifications. Both Ethernet and Fibre Channel are aligned for distance and loss specification budgets, allowing easier planning to the higher speeds.
Knowing that we want patch panels in the design, and the ability to connect both duplex and parallel optics, will provide some challenges to be aware of when it comes to interconnect loss considerations. We need to keep the number of the interconnect loss below 1.5 dB to keep the loss within specification for both the latest Ethernet and Fibre Channel standards.
We must also consider the TIA-568.3-D document to decide on what MPO/MTP polarity option to select to best support the new design. Polarity is critical in cabling systems in order to assure that data transmissions are received by the appropriate receivers. There are three methods described in the standard as Method A, Method B, and Method C. All three have their advantages and disadvantages.
Method A was the first MPO/MTP polarity option introduced. It was simple to install but had two different polarity duplex jumpers on each end to manage. Imagine the signal beginning in an LC to LC jumper as straight. It then enters an LC to MTP cassette module as straight, into an MTP to MTP trunk as straight, then into the other MTP to LC cassette module as straight light path. At this point, the LC to LC jumper on the other cassette module would then cross the signal to correct the light path as transmit to receive.
Method B became an option when the data center market was anticipating parallel optics that would need more than two fiber strands to increase the speed of the link. Going from duplex (single fiber for transmission and reception) to parallel signals (multiple fibers for transmission and reception) would be needed as an option in the structured cabling. Just recently, the transceiver industry was able to increase the transmission speed over one fiber to keep duplex connections as the dominant type in the data center space.
The main advantage of Method B over Method A was the MTP to MTP trunks were flipped polarity (send to receive), where Method A was straight polarity. The flipped polarity better served parallel optics.
One of the challenges with Method B, however, was that it required two types of MTP to LC cassettes as alpha and beta. Also, if singlemode fiber was used, a large cabling company owned intellectual property on MTP connections at the trunk to cassette. This would require every organization using Method B to purchase related patch panel products and cable assemblies from a single manufacturer, reducing consumer options and autonomy.
Method C was introduced with the thought that duplex connections were the most common in the data center space. It improved on Method A by using the same LC to LC jumper on each end. The cross (transmit to receive) would occur in the MTP to MTP trunk. The MTP to LC cassette modules were the same as straight polarity, an improvement over Method B which uses an alpha and beta cassette. The downside to Method C was that it didn’t easily work with parallel optics as the MTP to MTP trunk was crossed pairs and not flipped polarity
Today most data center operators are managing switches and devices that use both duplex and parallel optics. When deciding on which of the three polarity methods in TIA-568.3-D to use, no single method supports both optics and changes effectively.
For this reason, CABLExpress introduced a new polarity method called Multi-Path (learn more in our white paper Simplify Fiber Optic Cabling Migrations with a Multi-Path System). Multi-Path makes use of the best parts of the three approved polarity methods.
It uses duplex jumpers as A-to-B on both ends. It also uses MTP jumpers as female or un-pinned with flipped polarity. Both of these jumpers will work when plugging into active optics on each end. It has the same MTP to LC cassette modules at each end.
The MTP to MTP trunk is male or pinned with flipped polarity to best support parallel optics. Lastly, Multi-Path has the option to break out an optic like 40G to 4 x 10G, 100G to 4 x 25G and 128GFCp to 4 x 32GFC. The use of patch panels in all the polarity methods allows for modularity and scalability in a structured cabling solution. In other words, patch panels will make any moves, adds, and changes (MAC’s) in the data cabling infrastructure much easier.
At this point in the structured cabling design we have decided to use patch panels over the Core, Director or Spine switches. We’ve also chosen to use patch panels on the other side of the link or Server side to best serve needed ports in a cabinet, pod, or row, and we’ve chosen an MPO/MTP polarity method to support duplex and parallel optics.
If distances over 100 - 150 meters are needed, look to singlemode fiber and the PSM-4 optic. Consider pricing singlemode optics to determine if installing singlemode fiber will set the data center up for next generation gear seamlessly.
Selecting the best size and port count patch panel for the switch greatly helps manage the connectivity. There are two schools of thought when it comes to patch panel sizes. The first is to pack as many ports as possible into a rack unit. This option works well when at the initial time of installation all the ports are connected and the patch panel can be closed and the cabling can be dressed once and left alone.
The second school of thought for the size of patch panels is to use port replication of the active hardware to be cabled. Port replication is “mirroring” the ports of active fiber optic hardware in a passive component (fiber patch panel). This creates a direct, one-to-one relationship between the active hardware ports and the passive structured cabling environment, thus simplifying the cabling process as all numbers on the hardware directly correspond to the numbers on the patch panel. Learn more about the advantages of port replication in our white paper Passive Optical Port Replication in the Data Center.
An example of port replication would be in the cabling of a new Arista 32- or 36-port network chassis that features four line cards. As we covered earlier, having a break (allowed by the use of patch panel) in the fiber link at the Core, Director, or Spine provides the ability to connect a duplex or parallel optic. Having the patch panel here replicate the switch to be connected as one-to-one helps manage the connections (see Figure 5).
Notice that a 1U enclosure can exactly replicate a 32-port line card with LC or MTP connections. The same goes for a 36-port line card. The horizontal or backbone cabling would plug into the back of these patch panels.
Another example of port replication comes when connecting a new Brocade/Broadcom Gen6 X6-8 Director switch featuring eight 64-port blades. The 64 ports from each blade can be cabled into a replication enclosure that can hold all 512 ports. This allows the cabinet with the new Director switch to be locked and all active ports are replicated into a patch cabinet or cross-connect cabinet.
Another option is one blade can be replicated 1U at a time with 64 ports out on the data center floor (see Figure 6).
When we use the term “design,” we are referring to the overall layout of the fiber cabling in the data center. Traditionally, a “three-tier” design has been the standard. More recently, a “spine-and-leaf” design is being utilized.
The traditional, three-tier layout has been dominant for decades. However, a spine-and-leaf design removes a hop that allows for a myriad of benefits. It now becomes more scalable and solves the east-west traffic problem. For more information on this topic, please see our white paper Cabling Designs for Hyperconvergence.
As an example of what a typical spine-and-leaf deployment might look like, see Figure 7.
Notice the replication panels over each four-slot chassis. This allows either an LC to MTP or MTP to MTP cable to be installed to connect a duplex or parallel optic.
There are enough ports in the patch panels to support full growth of the chassis. The MTP to MTP fiber trunking goes out into the data center floor and can stay in place when new switches are installed, and the old switches can be removed one fabric at a time.
Instead of an enclosure at the Server side or Leaf switch, a Zero U bracket can be installed. These brackets, like their name states, do not take up a rack unit and can be installed anywhere there is space, such as the back of the cabinet. Not only are the Zero U brackets easy to install, they are cost-effective.
A typical Storage or SAN installation might look like Figure 8.
Each 384-port SAN chassis sits in a dedicated cabinet. This allows for growth of an additional chassis or the addition of edge switches.
Having space also allows easy migration when the time comes to refresh to a new Fibre Channel switch. You can simply install the new switch above the old, and then migrate off the old switch one fabric at a time.
Notice the bulk of the cabling in a cross-connect cabinet by Fabric A and Fabric B. This means there will be less cabling congestion in front and around your active hardware.
Each 384-port SAN chassis is replicated in the cross-connect cabinet by fabric adjacent to the chassis. There is no need to connect into active ports as they are moved into the cross-connect cabinet.
At the top of the cross-connect cabinet there are ports replicated from the data center floor. Notice the 6U enclosures at the top of those cabinets. These enclosures are designed to replicate needed ports on the data center floor and room for growth.
Lastly, notice the 6U enclosure on the far right of the drawing. This enclosure has 72 ports from each SAN chassis or fabric. It can be mounted middle-of-row (MoR) or split up and mounted at each end-of-row (EoR).
There is also room for growth to double the ports into the enclosure. The total link loss for this design is a maximum of -1.05 dB, but designs typically see a loss of only -0.69 dB. These numbers are far below the standard of -1.50 dB as a maximum, but when looking at the typical loss it’s half the allowance to support Gen7 fibre channel speeds.
The TIA-942 standard shows us how and why to use a structured cabling solution, as it avoids congestion, confusion, and other issues that can lead to downtime. Now more than ever, structured cabling provides a modular, scalable solution in order to cost-effectively keep up with changing hardware components. This will allow you to quickly manage the layer one infrastructure during transitions to higher speeds, and still maintain your budget.
The IEEE standard for Ethernet, and T11 standard for Fibre Channel, define the amount of allowable loss in the cabling system. Loss must be under the allowed budget in order to maintain proper operation. Loss amounts must be considered when choosing cabling and patch panel products, as the highest loss amounts occur here. Compare the amount of allowable loss with the maximum loss amount of your selected cabling products in order to ensure those budgets are met.
The TIA-942 standard offers multiple polarity options, which provide guidance for proper signal transmission and reception. With the current direction of hardware offerings, using both LC and MTP ports, the ideal option will allow for a seamless and cost-effective transition between the two connector types.
The overall design will be a key consideration. Whether you choose a traditional three-tier topology, or a spine-and-leaf design, will depend on your overall hardware deployment strategy. This choice will dictate how your structured cabling solution will be implemented.