Previous Reports
Research Reports

HPC User Site Census: Interconnects and Networks

by Christopher G. Willard, Ph.D., Addison Snell, Laura Segervall
for Intersect360 Research
April 2011

This report, part of the Intersect360 Research Site Census series, provides an examination of the server interconnect and network characteristics found at a sample of HPC user sites. Intersect360 Research surveyed a broad range of users about their current computer system installations, storage systems, networks, middleware, and software supporting these computer installations.

The goal in this analysis of computer system interconnect and network implementation is to examine the suppliers, type, and purpose of the network within the HPC user communities, and to explore how this usage varies between computer systems interconnects, storage networks, and local area networks.

Key findings of the survey include the following:

  • Ethernet and InfiniBand are the two primary standards that compete across all three locales: system interconnects, storage networks, and LANs. Fibre Channel was also used at all three locales, but represented only about 1% of the interconnects and LANs mentioned. Ethernet, including all data rates, has the greatest representation for all locales with 54% of mentions for system interconnects, 62% for storage networks, and 83% for LANs. For performance usages, InfiniBand has the greatest representation for system interconnects, 10 Gigabit Ethernet has more representation for LANs, and InfiniBand has slightly greater representation than 10 Gigabit Ethernet for storage networks.
  • Use of Ethernet and InfiniBand standards at other locales appeared to be dependent on the site heritage. That is, use of Ethernet for a system interconnect or storage network at a site was more likely if the site had an Ethernet LAN. Use of InfiniBand for storage networks and LANs was more likely if it was being used as a system interconnect at that site.
  • Mellanox is the supplier used most often for system interconnects, with 9% of mentions. Cisco is the supplier used most often for storage networks, with 18% of mentions and for LANs, with 51%. Myricom is the supplier used most often for 10 Gigabit Ethernet system interconnects.
  • Most NAS storage systems (60%) are connected to 1 Gigabit Ethernet networks. SAN storage systems are evenly divided with 40% connected to a Fibre Channel network and 46% connected to an Ethernet (all data rates) network. InfiniBand accounted for 17% of the SAN storage networks.

Ethernet, as expected, is the dominant LAN network implemented at these sites, with 91% of the 242 sites providing network data having some type of Ethernet installed. Considering all 308 installations, some with more than one type installed, approximately 82% were using Ethernet (all data rates). 1 Gigabit Ethernet is still the primary type, with 49% of the mentions. Cisco is the primary supplier for Ethernet and InfiniBand implementations.

For full access to our HPC research studies, analysis, custom research and consulting, contact us directly.

» 888-256-0124
» Subscribe to our mailing list

Sign in to view premium content
Remember me on this computer
Become a client
Get access to our full reports