‘Leap second’ confuses some Internet routers, bringing down a small slice of networks

Stacks of networking switches in one rack of the DETER testbed at the USC Information Sciences Institute.
Stacks of networking switches in one rack of the DETER testbed at the USC Information Sciences Institute.

Sometimes, the littlest changes can have big effects. This week, a “leap second” — an occasional one-second adjustment that keeps the world’s official clock in time with the Earth’s rotation — caused intermittent outages to more than 2,000 Internet networks around the globe.

Dyn Inc., a Manchester, N.H.-based company that monitors Internet performance, reported the outages occurred just after midnight, around the time the extra second was added. Generally, when service outages are reported, it’s the fault of a single internet service provider, or ISP — however, in this case, because the outages were so widespread, Dyn concluded that time change was the most likely reason.

“It seems like a couple of different types of routers couldn’t handle that, and their software crashed,” said Doug Madory, Dyn’s director of Internet analysis. Madory said that software engineers were discussing the problem on forums across the Web, and found that the crash most likely occurred because of rusty software in a variety of routers, including ones by MikroTik.

Most networks were back online within a few minutes, although others took a few hours to reset themselves. Madory said that the fix was simple: restart the router.

Leap seconds are fairly unusual, with only 26 implemented since 1972. That means these outages are hard to prepare for, since it’s difficult to know what a system can handle in such an unlikely scenario. Madory said some engineers recommend inserting a leap second every couple of months, so they can test whether their software can handle it.

However, it doesn’t seem as if this latest outage spike has had any verifiable lasting consequences. The number of routed networks affected, 2,096, is only a fraction of a percentage of the total worldwide.

“Most models survived this with no issue,” Madory said. “By the numbers, this wasn’t a global catastrophe.”