Leading ESnet to the Next Level of Innovation:
Steve Cotter

As science becomes more data-intensive, whether due to massive amounts of data collected by experimental facilities or increasingly detailed simulations on supercomputers, that same research is in turn increasingly reliant on networking. For most of its 25 years, the Department of Energy’s Energy Sciences Network, better known as ESnet, has been a critical but relatively quiet component in supporting DOE’s research missions.

That began to change about four years ago when ESnet deployed its Science Data Network, a system that could be configured to provide dedicated end-to-end virtual circuits to accommodate the direct transfer of massive datasets. Other innovative services followed, including perfSONAR, a set of tools for diagnosing and defeating network bottlenecks, and OSCARS, an online system for easily reserving these end-to-end connections for large-scale data transfers.

The stage was set for the next level of innovative networking in August 2008, when Steve Cotter left Google to become just the third person to lead ESnet.

“What brought me here was the opportunity to contribute to helping solve some of society’s major challenges,” Cotter said. “Networking in this day and age is such an enabler of scientific research, especially as science is increasingly data-driven. We work directly with people who are developing cleaner energy, studying the Earth’s climate and increasing our understanding of diseases.”

ESnet is now deploying DOE’s Advanced Networking Initiative (ANI), a prototype network linking three of the nation’s top supercomputing centers at the unprecedented capability of 100 gigabits per second. The centers — the National Energy Research Scientific Computing Center at Lawrence Berkeley National Laboratory, and the Leadership Computing Facilities at Argonne National Laboratory and Oak Ridge National Laboratory — also will be linked to a key network peering point in New York.

While ANI is the first step toward a planned terabit-scale network serving all of DOE’s research facilities and university researchers, it also provides an opportunity to improve the energy efficiency of networking, an issue Cotter has championed for several years.

“At Google, I noticed that the company put a lot of time and effort into making their data centers energy efficient and they made tremendous strides in reducing energy use and seeking out clean energy sources,” Cotter said. “And many other data centers are also looking to cut energy usage. But not much has been done about the consumption of energy by the networks connecting those centers.”

One reason, Cotter offers, is that the data centers are large entities and the power use can be more easily tracked — often in the form of a power bill. Networks, on the other hand, are small compared to the centers, with equipment installed in many locations, which makes it harder to measure energy use.

“In aggregate, a lot of energy is being used by networks, but no one is really looking at that,” Cotter said. “This is a field in which we can have a real impact and kickstart a new way of thinking.”

The ANI infrastructure has been designed to gather information about energy use from the routers and switches along the network, providing a wealth of real-world data. By studying the data, engineers may learn how to design more efficient equipment as well as more efficient networks, including how major carriers design their facilities.

And the savings can be significant. Cotter says he’s seen several studies showing that the information and computing technology industry consumes about two percent of the world’s energy — equivalent to the entire airline industry — and it’s growing much faster than other industries. In fact, it could double in a few years. At the same time, more and more pieces of critical infrastructure now depend on networking, so if the increasing power demands lead to outages, the effects can be devastating, Cotter said.

On the flip side, networking also can enable energy savings in a wide range of areas, from “smart buildings” that can be remotely regulated in terms of heating, cooling and lighting to real-time transportation management to improve management of the nation’s power grid. Data processing could also be switched to “greener” data centers.

“If we design and build dynamic, intelligent networks, there is a lot more we can do,” Cotter said. “Although we are a Department of Energy facility, I’d like us to become the Department of Less Energy.”

One area where Cotter has expended significant energy is the recruiting of new staff, though the entire operation is maintained by 37 full-time employees. While there are still one or two who have been part of ESnet from its launch in 1986, Cotter is casting a wide net for new hires, seeking people from industry, academia, research and production networking.

“We want to build a very diverse team — the last thing our organization needs is people who all think alike,” he said. “And, while we have a culture of inclusiveness and respect with high professional standards, we also need to be able to respectfully challenge one another when it comes to our ideas.”

Early on, Cotter encouraged his staff to write on the walls of their office areas at Lawrence Berkeley National Laboratory, turning the hallways into jumbo whiteboards. This helps hallway conversations generate ideas that can be quickly written down, seen by others and germinate and grow, he said.

One of those ideas became MyESnet, a user portal that provides extensive information on network traffic, detailed looks at various connections and reports on upcoming maintenance. In its first four months, MyESnet had three upgrades as new features were added and user feedback was incorporated. “We have a quick, innovative cycle and that feeds the momentum. People see the benefits of providing good feedback and that makes them want to contribute more.”

Although ESnet has won a handful of awards for its efforts over the past three years, Cotter still sees room for improvement. Asked to grade ESnet, he thinks for a moment and then awards his team a solid B.

“It took awhile to bring in the right skills to build on the strong foundation that had been established and now we are really hitting our stride,” Cotter said. “I give us a B because we have a lot more potential. When we hit that potential, we’ll have earned an A.”

Scientific Computing Highlights

Conference Dates:
November 12-18, 2011

Exhibition Dates:
November 14-17, 2011

• Technical Program: This flagship program will feature hundreds of sessions from luminaries and leading experts focusing on ongoing research and new developments for industry – and as featured throughout this exclusive edition

• Communities: Complimenting this year’s SC11 Theme – “Connecting Communities through HPC” – Scientific Computing is pleased to be a proud sponsor of the conference and contribute to the collaborative spirit of the global community.

• News & Press: As part of this annual conference exclusive and as you begin to plan your Seattle SC itinerary we have provided you an advanced guide to help map your sessions and exhibit plans -  stop by and meet our showcase individuals, conference thrusts and tracks and prepare to get and make the most out of your Seattle sessions.

www.scientificcomputing.com

HPC Source

HPC Webcasts

Subscribe to SCIENTIFIC COMPUTING