By Dave Fredricks, Data Center Infrastructure Architect, CABLExpress
and Atif Wasi, Manager - Systems Engineering, Arista Networks
Published December 2019, Updated May 2020
This technical brief demonstrates different ways to cable up the new 100G and 400G line card ports in the Arista 7800R3 chassis. The intent is to describe how to replicate ports into patch panels and break out ports into patch panels to best utilize their higher-speed capabilities.
Arista recently introduced its next-generation Spine routers: the 7800R3 series along with updates to its flagship 7500R router, the 7500R3. The newly released 7800R3 is the highest- density router in its class, supporting up to 288 x 400G ports in a compact 8-slot chassis.
The new 7800R3 series router comes in 4- and 8-slot versions today. The 7800R3 line card modules support multiple speed options such as 10G, 25G, 40G, 50G, 100G, and 400G.
When cabling up the new 400G line card systems, it helps to understand the nomenclature and the technology. The terms QSFP, QSFP-DD, and OSFP refer to the form factor of the optical module that plugs into the ports of a router, switch, or server. The type of optical connector on the module (such as LC, MPO-12, etc.) is determined by the optical media type of the module, which can be determined from the module part number (such as CWDM4, PSM4, DR4 etc.).
The 7800R3 48-port 100G QSFP line card accepts optics that support both multi-mode fiber (MMF) and single-mode fiber (SMF). Available optics support duplex (LC) and parallel (MPO/MTP).
The 100G multimode optics support distances up to 300 meters over OM4 MMF. Supported multimode optics include: 100GBASE-SR4, 100GBASE-XSR4, 100GBASE-SWDM4 and 100GBASE-SRBD. The 100G-BASE-SR4/XSR4 modules use an MPO/MTP connector while the 100GBASE-SWDM4 and SRBD use duplex LC connectors.
Available single-mode optics support distances up to 40 kilometers over SMF. In the data center space, commonly used 100G single-mode optics include the 100GBASE-PSM4 (up to 500 meters over parallel SMF), the 100GBASE-CWDM4 (2km), and the 100GBASE-LR4 (10km). The 100GBASE-PSM4 uses a parallel MPO/MTP connector, while the 100GBASE-CWDM4 and LR4 optics use duplex LC connectors.
Can I have an OSFP on one end of a 400G link, and a QSFP-DD on the other end?
The OSFP and QSFP-DD describe the physical form factor of the module. As long as the Ethernet media types are the same (i.e. both ends of the link are 400G-DR4, or $00G-FR4 etc.), OSFP and QSFP-DD modules will interoperate with each other.
The 7800R3 also supports 36-port 400G line cards which comes in two flavors supporting either OSFP or QSFP-DD optics. 400G optics are available in OSFP and QSFP-DD form factors, and Arista fully supports both form-factors.
The OSFP stands for “Octal Small Form-Factor Pluggable,” and is a new form-factor optimized for 400G and 800G modules. The term “octal” refers to the 8 lanes present in the OSFP electrical connector, with each lane running at 50G/s, for an aggregate of 400G bandwidth.
The QSFP-DD stands for “Quad Small Form-Factor Pluggable – Double Density.” The QSFP-DD electrical connector also has 8 lanes, each running at 50G/s. The QSFP-DD is similar to the existing QSFP form-factor, except that a second row of electrical connectors has been added to allow for 400G bandwidth. Most 400G optical media types are available in both the OSFP and QSFP-DD.
The QSFP-DD form-factor provides for strict backwards compatibility with existing 100G QSFP modules. That is, a 100G QSFP module can be directly inserted into a QSFP-DD port, and the port run at 100G (instead of 400G).
The OSFP allows for backward compatibility by using a simple mechanical adapter that allows the use of existing 100G QSFP optics on OSFP ports and line cards. OSFP modules have an integrated heatsink making them easier to cool, and they are also designed to be forward-compatible with future 800G systems to ensure future-proofing.
Figure 1 shows replication and breakout of both the 100G and 400G line cards using multimode and single-mode optics as in-cabinet connectivity.
The 36-port 400G line card accepts both multi-mode and single-mode optics. 400G multi-mode options include the 400GBASE-SR8 and the 400GBASE-SRBD (future availability), supporting distances up to 100 meters over OMF MMF.
The 400GBASE-SR8 optic runs on 16 individual fibers (8 fibers transmit and 8 fibers receive) and uses an MPO-16 optical connector. The 400GBASE-SRBD will use a total of 8 fibers, and will use the same type of MPO-12 connectors used for 100G-SR4 optics.
Figures 2 and 3 show cabling designs in both the 100G and 400G with multimode fiber. Figure 2 features 48 x 100G port line cards with SR4 optics while Figure 3 features 36 x 400G port line cards with SR4 and SR8 optics.
400G single-mode optics media types include the 400G-DR4, XDR4 and FR4. Probably the most common in the data center space is the 400GBASE-DR4 (500M) and 400GBASE-XDR4 (2km), which use a parallel MPO/MTP connected with 8 individual fibers (4 fibers transmit and 4 fibers receive). The MPO-MTP connector is the same as used on the 100G PSM4 optic.
The 400GBASE-DR4 optic can be broken out into 4 x 100G with the 100G end using the 100GbASE-DR4 QSFP optical modules, over a distance of up to 500M. The 400GBASE-XDR4 breaks out into 4x100GBASE-FR QSFP modules, over a distance of up to 2km. The 100GBASE-DR / FR QSFPs use a duplex LC connector.
Figure 4 shows a cabling design with 400G 36-port line cards using 400GBASE-DR4 optics.
Figure 5 shows a design with using 400GBASE-FR4 optics.
Typical structured cabling designs have the Spine switch’s (Arista 7800R3) active ports broken out into passive patch panel(s). This allows the Spine router to be isolated from day-to-day connections and maintenance activity and be protected in its own cabinet or rack.
This cabling infrastructure creates a direct relationship between the active hardware ports and the passive structured cabling environment. This practice is known as Port Replication™. Simply put, when cabling up such a high-speed switch, Port Replication reduces the likelihood of downtime.
The Spine replication patch panels can be mounted above the Spine router if breakouts are required (for example, 1x 400G port into 4x 100G ports). Spine ports can also be replicated or broken out into zone cabinets (ZDA), middle of row (MoR), end of row (EoR), or top of rack (ToR) locations depending upon the application.
Can I plug a 100G QSFP module into a QSFP-DD port?
A QSFP (40G or 100G) module can also be inserted into a QSFP-DD port (without a mechanical adapter). When using a QSFP module in a QSFP-DD port, the QSFP-DD port must be configured for a data rate of 100G (or 40G), instead of 400G).
As Leaf switches or routers are added into the network they are connected back to each Spine switch or router. Having the Spine router replication patch panels near the Leaf switches or routers can help make connections easier and quicker. It also reduces disruption to existing cabling pathways that could cause connection failures.
Looking into the future for Spine and Leaf and Two-Tier cabling designs, single-mode fiber will dominate the connection between Spine and Leaf switches. Single-mode fiber can easily support 400G today and future-proof the cabling infrastructure for 800G and beyond.
The connection from the Leaf switch to the server will move from 10G/25G today to 50G and 100G. These connections will likely remain on multimode fiber as the shorter distance (in-cabinet or in-row) will support the higher speeds.
In conclusion, cabling the Arista 7800R3 series switch begins with understanding the optic types available for both 100G and 400G. Consider 400G over 100G because the price of breaking out 400G ports into 4x 100G ports can be more cost-effective versus supporting individual 100G optics at the Spine.
Consider 400G single-mode optics (DR4) as the cabling infrastructure will work with next-generation 800G optics. Plan to replicate or breakout the 400G ports in-cabinet, in-row, or at a desired location on the data center floor to have Spine ports near equipment that needs to be connected.