A CABLExpress educational document written by Rick Dallmann, Director, Data Center Architecture, CABLExpress and Ryan Steele, Strategic Account Executive, CABLExpress
For even more insights, check out our white papers.
As a derivative design characteristic of the TIA-942B Data Center Standard, port replication™ is the duplication and relocation of the SAN director’s active fiber optic ports on any of its line cards, including ICL’s or Flex ports. Invented by CABLExpress, and successfully deployed over several years, port replication creates a direct, port-for-port, 1:1 relationship between these active ports and their replication passive enclosure “twin” in the structured cabling environment.
Specific and exact to the Brocade director switch’s design, line card and HBA/SFP’s, these port replication patch panels/enclosures allow the data center client to connect into or perform moves, adds, and changes in an environment away from the active switch ports and other components.
This drastically minimizes the accidental chance of failures, by decreasing congestion and wear-and-tear on active connection points. Also, since the screened ports on the director’s line card port directly correspond to the screened port numbers and line card positions on the CABLExpress replication patch panel, this greatly simplifies the cabling connection process.
As port density per director and per rack increases, port replication provides great flexibility in connecting any blade port to any HBA/SFP, switch, or storage port within the data center. Also, while allowing the use of any desired link configuration, port replication, by default, implements an important cable management plan that becomes essential during servicing or scaling of the fabric.
Port replication also significantly reduces cabling bulk in the switch rack by utilizing specifically engineered harnesses or trunks in front of the active switch ports. The end result is the potential reduction of cordage bulk by up to 66 percent in the director.
Lastly, due to this duplication, port replication helps minimize user error and eliminates mistakes. This is important for establishing a reliable, stable Storage Area Network. Figures 1 and 2 show how cabling can be implemented correctly using port replication techniques, while Figure 3 shows how easily a cabling mess can happen without port replication.
In 2015, the introduction of Brocade’s Gen 6 Fibre Channel platform of director and switch offerings led the industry with breakthrough 32 Gbps performance. This level of performance accelerated process response time by up to 71 percent, thereby eliminating data bottlenecks.
By doing this, they unleashed the full performance of flash and next-generation Non-Volatile Memory Express (NVMe)-based storage, in contrast to spinning drives. Now the virtualized, all-flash data center has become a reality with the support of 1st generation NVMe.
According to the Fibre Channel Industry Association (FCIA), “Gen 6 doubles 16GFC to 32GFC and also provides the option to quadruple to the performance of 128GFC, all based upon the seamless inter-and backward-compatibility of Fibre Channel technology that data center customers expect. The newest innovation includes parallel Fibre Channel which stripes four simultaneous lanes of 32GFC to create a single link with speeds up to 128 Gbps, initially expected to provide the perfect ISL (Inter-Switch Link) connectivity for 32GFC edge connectivity.”
While Gen 6 Fibre Channel introduced speeds of 32G over 2 fibers, it also specified speeds of 128GFCp Fibre Channel over 8 fibers(SR4), MPO/MTP parallel optic (128GFCp). The 128GfCp has dual purpose on this platform, being used as an Inter-Chassis Link (ICL) and in a 64-port, 32Gb line card, named the FC32-64. As Brocade’s Gen6 hardware began to enter the data center space in large numbers in 2017, the ability to simplify management, monitor storage performance, speed up troubleshooting, and optimize performance was a game-changer for consistent and stable storage operation in any data center’s SAN environment.
Seen in Figures 4, 5, and 6 are Brocade’s Gen 6, next-generation Directors and Switches. The X6 is capable of configurations of 256 to 512 port at 32gb/s options and the multi-flexible G620 series is capable of 24 to 64 ports at 32gb/s.
The Brocade G620 is built for maximum flexibility, scalability, and ease of use. Storage networks can easily scale from 24 to 64 ports with 48 duplex ports and assign the (4) Q-Flex ports (QSFPSR4) to either function as (4) ICL 128gb/s or (16) additional 32gb/s duplex ports. This enables a pay-as-you-grow scalability and unmatched on-demand flexibility.
Three years after Gen 6 product was deployed in the market, Fibre Channel Gen 7 was released. It features several key benefits over Gen 6 Fibre Channel, but two benefits stand out as the most notable upgrades.
The first is doubling the speeds of Gen 6, going from duplex 32GFC and parallel 128GFCp to 64GFC and 256GFCp respectfully. The second is advancing NVMe from generation 1 to generation 2 (now known as FC-NVMe).
These new faster speed specifications of Gen 7, along with Gen 2 NVMe, reduce latency and increase output in the compute cycle. They also work best with newer cloud-based software applications.
As you can see in Table 1, the Fibre Channel roadmap will support customer SAN applications well into the future, doubling data rate speed with each generation. It will utilize both duplex and parallel optics as its transmission platforms.
The sixth generation of Fibre Channel showed this, when it was designed to deliver 128 Gb through four transmit lanes of 32 Gb and four receive lanes of 32 Gb (MPO-based parallel optics). Although not specified in the Fibre Channel standard, break-out capability to 32 Gb will be supported by OEM and cabling vendors. Other lower-cost modules based on SWDM (OM5 MMF) and PSM4 and CWDM4 (SMF) are also available for 128 GFC and Fibre Channel roadmap items such as 256 Gb Fibre Channel (and beyond).
It is easy to understand that the most obvious benefits of upgrading networking technology are to increase the speed of data transfers and decrease the number of link devices required to accomplish network tasks. New server and storage technology advancements, such as flash-based storage, are driving up storage network bandwidth demand well beyond current capabilities. Additionally, requirements for higher-density server virtualization, new latency-sensitive applications, dynamic workloads, and overall application growth all are placing unprecedented demands on the network.
The total loss budget is defined as the total acceptable amount of optical/signal power loss (expressed in decibels) a fiber optic link can sustain and still operate error-free. Operating within the optic link loss budget is critical for an error-free, reliable, SAN network that delivers 100% uptime. The ‘link’ is defined as the path of the cable between the transmitter and the receiver.
Link loss budgets are measured based on the amount of attenuation, or signal reduction, related to the distance and number of connectors in the link. Prior to designing or installing a fiber optic link, the path and connections should be calculated for maximum or worst-case loss.
This is especially true for Short Reach or Multimode-based Fibre Channel links because there are limits to just how many connections and just how long a distance a link may support. Fibre Channel has historically promoted the fewest amount of connections whenever possible as their recommended link-design philosophy.
Theoretically, the fewer mated pair connections in any link, the less loss that link will have. Usually a link will perform better with less loss. However, with the average number of Fibre Channel duplex ports in a typical data center ranging between 768-1024, managing this mass amount of fiber becomes very challenging. This is especially true in the dense 14U to 28U space that the directors occupy.
Without some sort of infrastructure or cabling design plan, chaos can result. This will significantly impact performance or cause downtime, which is what limiting the number of interconnects was supposed to prevent.
In addition, Fibre Channel’s latest Gen 7 256Gb blistering speed specifications, and the associated miniscule fiber optic loss budgets standards, have become a challenge for data center (DC) managers looking to implement and maintain a manageable SAN cabling infrastructure (see Table 2). Another challenge is building this structured SAN cabling system to support not only current Gen 5 or Gen 6 speeds, but the increased data rates required for these Gen 7 Fibre Channel and beyond.
To emphasize how critical loss budget is to the health of your SAN fabrics, there are several factors to consider.
Let’s imagine we implement a brand new 16G SFP which produces approximately 550uW + of Tx Power. On the other side of the OM3/OM4 run we would like to see enough Rx power to have a clean link.
An Rx power of greater than 200uW is optimal. An Rx power between 100uW and 200uW is marginal. But with an Rx power of less than 100uW we will start seeing large numbers of ITW and CRC link errors on the port.
With a new SFP producing 550uW of Tx power, we have an attenuation budget of approximately 350uW (to keep our Rx light level above 200uW). So if a fiber run has an attenuation of 250 uW (550 – 250 = 300uW Rx power on the link) it should work fine.
Or will it? SFPs age but rarely die. This SFP produced 550uW Tx power when new, but four years later it might only be producing 400uW of Tx Power. Now if it’s only producing 400uW, the equation works out to: 400 uW (Tx) – 250 uW (attenuation) = 150uW Rx power on the other side of the link. At 150 uW Rx we are in the danger zone and could be producing ITWs and CRCs resulting in a problematic link and a slow drain device.
So attenuation matters! And the lower the attenuation of a fiber run, the longer your SAN fabric runs in an optimal/healthy state.
Is it possible to design and implement a structured SAN fiber optic cabling infrastructure to support a life cycle of ten plus years without having to significantly reconfigure or redo the physical layer infrastructure? Can we achieve this flexibility of “any port to any port” patching in an MDA through multiple connections? It is possible, but there are several factors to consider in order to ensure success.
The approach to designing this can be accomplished in two basics steps, but even these can be difficult to implement for some organizations. The first is to specify and only use what the standard refers to as Grade A (-0.25dB/mated pair or better), low loss connector components. By utilizing these, we can limit the sum of the connectors’ mated pair insertion loss less than 1.5dB.
The second requires specifying a patch panel or replication enclosure solution to manage the mass amounts of fibers in such a way that it will not impact the link’s performance, but will allow easy management and identification of the active ports in a passive environment. It should also allow easy migration of the link to future generations and topologies of Fibre Channel speeds.
To reiterate, Fibre Channel MMF links are standardized with 1.5dB of total connector loss, except for 128GFC-SW4 which has a connector loss of 1.0dB. While supporting a higher total connector loss is possible, the supported link length will be sacrificed.
It is also critical we do not use the statistical values of connector mated pair loss, but only the stated and documented maximum loss specifications. That will be the only true modeling of the designed SAN infrastructure. Standardizing our design links by utilizing the worst case (max) values of the connector loss is the only true way to predict performance of the link.
With these design criteria concepts defined, we can now deploy a Fibre Channel solution with a cross-connect/central patching and director replication in any advantageous or strategic positions within our data center and within the specified distance. Remember, according to the TIA-942 Data Center Standard, bringing ports off the switch and mapping them in the cross-connect/MDA and port replication panel is best practice.
When planning to deploy and cable up Brocade’s X6 director in either the 4-slot or 8-slot versions, it is critical to understand what kind of transmission type the optic on the line card is utilizing. The FC32-48 line card offers the SAN user 48 duplex 32Gb ports of Fibre Channel. The FC32-64 line card offers 16 QSFP SR4 ports of 128GB, which then coverts into 64 duplex ports of 32Gb Fibre Channel.
Each of these options are shown in Figure 7 below. Once the line card and transmission optic options are chosen and understood, the SAN manager can now move onto designing exactly how these “monsters” will be cabled up.
Fully loaded with 8 line cards, the X-6 Director has a minimum capacity of 384, or a maximum of 512, duplex, 32Gb ports. Needless to say, the fiber optic cable solution will need to be precisely planned in order to manage the bulk of fiber optic cable that these blades, in a 14U package, will create.
Not only does the solution need to manage the fiber bulk of hundreds of ports, but it is critical that the solution performs optimally. These 16, 32, and 128Gb high-speed Fibre Channel link loss budgets are miniscule, so any interconnects implemented into the solution need to perform even better than what we specified earlier in Grade A type connector performance (-0.25dB/mated pair or better).
The CABLExpress port replication solution achieves just that. First as shown in Figure 7, the CABLExpress 10U enclosure produces an exact image of Brocades X6-8 director and does all this in 4 fewer rack units of space.
By taking advantage of this more compact design, CABLExpress is able to increase the spacing between the LC adapter in the replication enclosure. This helps make 384 or 512 ports easy to connect, manage, and deploy.
Second, by utilizing specifically staggered and labeled trunk cables according to the line card in use, cabling up a fully loaded 384- or 512-port active director can be amazingly easy, and dressed properly. This virtually eliminates any cause for concern over fiber optic performance or reliability. See Figure 8.
These Brocade-specific staggered trunking cables, reduced on the opposite end by a factor of six, utilize MTP connectors and technology, and can be easily connected into any CABLExpress enclosure to achieve port replication. The active ports are then managed from there, away from the active director.
CABLExpress is able to accomplish this with a link loss budget significantly lower than the maximum amount allowed by the 64GB-256Gb Fibre Channel Standard. Let us also not forget that as we progress up into the 256 Gb Fibre Channel spectrum, these allowable loss budgets drop to 1.9 dB total link loss with the connector loss dropping to 1.0 dB.
These higher speeds leave very little margin for error in fiber optic infrastructure components, and insertion loss is a critical performance parameter in data center SAN fiber infrastructure. It should be noted that CABLExpress features a maximum insertion loss of 0.2dB for the MTP connector and 0.15dB for the LC connector, which are both industry-leading.
But what if your SAN doesn’t require the sheer number of ports as an X6 director provides? Then Brocade’s G620 and G630 switch platforms offer the SAN manager a one- or two-rack unit option, allowing for port availability and flexibility, and the ability to expand as you go.
Offering up to 48 duplex 32Gb LC ports, the G620 is a one rack unit switch that provides the SAN user with an additional 4 QSFP “flex ports” that can be configured as 128Gb FC ICL or 16 additional 32Gb LC ports. The G630 doubles this in a two rack unit package. Even though these two switches are smaller in footprint, the “horsepower” is no less and requires the same type of high-performing fiber optic cabling highway and port replication solution.
As shown in Figure 9, not only does the CABLExpress 1U replication enclosure provide an exact copy of the G620 port screen, but it performs a very important and seamless conversion of the 4 QSFP 128Gb flex ports to 16 additional 32Gb duplex LC ports. Again, all the magic is done inside the replication enclosure, making this high-performing switch solution a simple plug and play. Pair this with our specifically staggered G620/G630 harnesses and trunk cables, and this switch is dressed and replicated according to the standards.
Rounding out the latest technology to hit the data center is the release of IBM’s newest generation z15 mainframe in conjunction with Brocade’s latest generation FICON director. To provide high-performing cabling options for these, CABLExpress designed a new port replication solution in accordance with IBM and Brocade (shown in Figure 10).
As discussed previously, this greatly reduces installation time and helps mitigate the risk of costly mistakes. Such mistakes can lead to downtime from plugging into incorrect ports or misidentifying port locations.
As always, it is imperative to utilize high-performance, low-loss replication hardware or patch panels that enable a one-to-one relationship from the z15 mainframe to the FICON director ports. This not only makes the cross-connect between the z15 mainframe and the FICON Director simple, but it is also a TIA-942 standard compliant solution.
In conclusion, following the Fibre Channel industry’s best practices when designing and installing the fiber optic infrastructure design for Brocade’s latest generation of hardware will help ensure optimal performance and reliability of your SAN network.
Utilizing CABLExpress’ industry-leading, highest-performing fiber optic components, and our port replication fiber optic management, will allow data center managers to stay well within and under the maximum allowable link budgets up to and including 256GB Fiber Channel. This will ensure that your data center infrastructure will perform at the highest levels possible and last for generations.
Published August 2020