Achieving 400G Ethernet
The move to new optics

The Race to 400G Ethernet Part 3

By: Dave Fredricks | Data Center Infrastructure Architect


This post is part three of a series meant to track the technology advancements in the move from 100G to 400G Ethernet speeds and their effects on structured cabling. If you’re just joining me now, feel free to check out Part 1 and Part 2 before reading on.

The TIA-942-B data center cabling standard was released in August 2017. This standard includes the 16-fiber and 32-fiber MPO-style connector, as well as the standard TIA-492-AAAE. The TIA-492-AAAE introduces wide band multi-mode fiber (WBMMF), also known as OM5.

Another standard included with the TIA-942-B publication is the TIA-606-C Administration Standard for Telecommunications Infrastructure. The TIA-606-C addresses cable labeling and identifiers along with other topics, such as automated systems approaches.

There are two main multi-source agreement (MSA) groups working to introduce 200G and 400G optics into the upcoming IEEE 802.3bs standard. One of these groups is the Octal Small Form Factor Pluggable (OSFP) driven by Arista Networks. The other is the Quad Small Form Factor Pluggable Double Density (QSFP-DD).

Warning - Shameless Plug! CABLExpress is a member of the OSFP MSA and has specific products and solutions to match the new optics and port counts on the line cards and switches they will plug into.

OSFP and QSFP-DD working together

The OSFP and QSFP-DD standards will operate the same from a cabling perspective. Both MSA groups have agreed to design similar fiber lane assignments so that compatibility between the two optics will be seamless. The move to these new transceivers will change the required structured cabling plant to best maintain and support the equipment running on these optics.

Both MSA groups are planning optics that will use LC connectors (duplex) using 2 fibers and MPO/ MTP® connectors (parallel) using 8 and 16 fibers. There will be a mix of both multi-mode and single-mode glass solutions to include an in-cabinet copper offering.

From a data center perspective, the majority of the optics used will be for distances of 500 meters or less. For up to 500 meters, the optic will use 8 single-mode fibers with 4 fibers for transmit and 4 fibers for receive with an MPO-style connector.

Below is a drawing from a light path perspective, showing a basic link with MTP® jumpers on each end into MTP® coupler panels - with an MTP® trunk in the middle.

Direct Attach Copper Cabling

At CABLExpress, we have been deploying this type of connection link for many organizations running new spine-and-leaf equipment with SR-4 and PSM-4 100G optics. The 200G and 400G will use this same cabling solution.

One of the potential roadblocks using these components in a single-mode link is intellectual property owned by one of the large cabling providers. This provider has a patent with aligned MPO/MTP® couplers that use angled MPO/MTP® ferrules when mating together. To bypass this roadblock CABLExpress uses only opposed, or Key Up to Key Down, MPO/MTP® couplers.

The fiber connector for a distance up to 100 meters will use the MPO/MTP® connector running on 16 multi-mode fibers. This will have eight fibers transmit and eight fibers receive.

This application will look similar to the above light path drawing, but with two rows of 8 fibers from the MPO/MTP® ferrule of the connector. Also planned in the new standard will be Direct Attach Copper (DAC) cabling for up to 3 meters for in-cabinet connections.

To summarize, applications now running 100G on PSM-4 (single-mode) optics and applications for 200G and 400G will use mostly MPO/MTP® connectors into the optics and switches.

Increasing the speed between spine and leaf is essential to support current and upcoming equipment like Ethernet-based storage architectures and software-defined storage (SDS).

A look into the future of new optics

From a cabling perspective, when connecting MPO/MTP® optics (parallel) versus LC optics (duplex or serial) the link doesn’t require a MPO/MTP® to LC cassette module. This is beneficial because modules can be expensive.

In place of the cassette module is an MPO/MTP® coupler panel, which costs much less. For example, if the application requires breaking out the signal for 100G to 4x 25G, or 400G to 4x 100G, a cassette module can simply be installed on one end of the link.

Looking at different scenarios in the future, the move to the new optics will reduce the costs but also increase the needed amount of individual fibers to support 8-fiber single-mode and 16-fiber multi-mode connections. Again, the diagram above replicates what the fiber optic links will look like connecting current 100G PSM-4 (single-mode) and 100G SR-4 (multi-mode) optics and the upcoming 200G and 400G optics.

Also to be considered is port replication on the spine side of the link. As a data center grows, leaf switches will be added. These additional leaf switches will require a connection to each spine switch with two separate connections for redundancy likely.

To best replicate a 1U spine or chassis spine switch using QSFP or parallel optics, specific port count MPO/MTP® coupler panels are needed.

For the 32-port line card, two 16-port MPO/MTP® coupler panels are used. The numbering on the panels matches the numbering on the switch. This means the top left is port #1, bottom left port is #2, and so on to the second panel to finish with port #32. The same idea of port replication can be attributed to the 36-port line card with two 18-port MPO/MTP® coupler panels.

Look for higher speeds

Lastly, the connection from spine to leaf will require higher speeds to support current and future equipment. We are looking at speeds from the current 100G to upcoming 400G Ethernet.

Single-mode PSM-4 optics have come down in cost and are able to carry distances from 100 to 500 meters. Single-mode glass begins to gain more acceptance in the data center space.

The MPO/MTP® connectors will increase in usage in the data center as the optics require a higher number of fibers to transmit and receive signals. Also, plans for port replication on the spine side of the link for growth and cable management needs to be considered.  

The standard from Institute of Electrical and Electronics Engineers (IEEE) 802.3bs 200 Gb/s and 400 Gb/s Ethernet is still scheduled for release in December of 2017. When this standard is released, the fourth and last part of this blog series will be completed with a final update.

About the Author

Dave Fredricks is a data center infrastructure architect at CABLExpress with more than 20 years of experience in fiber optic and structured cabling solutions.

He is a Certified Data Centre Designer (CDCD) and an active member of AFCOM since 2007. He authored the white papers Conflicts in Data Center Fiber Structured Cabling Standards, Meeting 40/100G Ethernet and 32GFC, 28GFCp Light Budgets with a Structured Cabling Design and Cabling Designs for Hyperconvergence. He has comprehensive knowledge of the TIA and IEEE industry standards for network and storage connectivity.