On May 8, when the Federal Communications Commission website failed and many people were prevented from submitting comments about net neutrality, the cause seemed obvious. Comedian John Oliver had just aired a segment blasting FCC Chairman Ajit Pai's plan to gut net neutrality rules, and it appeared that the site just couldn't handle the sudden influx of comments.
But when the FCC released a statement explaining the website's downtime, the commission didn't mention the Oliver show or people submitting comments opposing Pai's plan. Instead, the FCC attributed the downtime solely to "multiple distributed denial-of-service attacks (DDoS)." These were "deliberate attempts by external actors to bombard the FCC's comment system with a high amount of traffic to our commercial cloud host," performed by "actors" who "were not attempting to file comments themselves; rather, they made it difficult for legitimate commenters to access and file with the FCC."
The FCC has faced skepticism from net neutrality activists who doubt the website was hit with multiple DDoS attacks at the same time that many new commenters were trying to protest the plan to eliminate the current net neutrality rules. Besides the large influx of legitimate comments, what appeared to be spam bots flooded the FCC with identical comments attributed to people whose names were drawn from data breaches, which is another possible cause of downtime. There are now more than 2.5 million comments on Pai's plan. The FCC is taking comments until August 16 and will make a final decision some time after that.
The FCC initially declined to provide more detail on the DDoS attacks to Ars and other news organizations, but it is finally offering some more information. A spokesperson from the commission's public relations department told Ars that the FCC stands by its earlier statement that there were multiple DDoS attacks. An FCC official who is familiar with the attacks suggested they might have come either from a DDoS or spam bots but has reason to doubt that they were just spam bots. In either case, the FCC says the attacks worked differently from traditional DDoSes launched from armies of infected computers.
A petition by activist group Fight for the Future suggests that the FCC "invent[ed] a fake DDoS attack to cover up the fact that they lost comments from net neutrality supporters."
But while FCC commissioners are partisan creatures who are appointed and confirmed by politicians, the commission's IT team is nonpartisan, with leadership that has served under both Presidents Obama and Trump. There's no consensus among security experts on whether May 8 was or wasn't the result of a DDoS attack against the FCC comments site. One security expert we spoke to said it sounds like the FCC was hit by an unusual type of DDoS attack, while another expert suggested that it might have been something that looked like a DDoS attack but actually wasn't.
Breaking the silence
FCC CIO David Bray offered more details on how the attack worked in an interview with ZDNet published Friday. Here's what the article said:
According to Bray, FCC staff noticed high comment volumes around 3:00 AM the morning of Monday, May 8. As the FCC analyzed the log files, it became clear that non-human bots created these comments automatically by making calls to the FCC's API.
Interestingly, the attack did not come from a botnet of infected computers but was fully cloud-based.
By using commercial cloud services to make massive API requests, the bots consumed available machine resources, which crowded out human commenters. In effect, the bot swarm created a distributed denial-of-service attack on FCC systems using the public API as a vehicle. It's similar to the distributed denial of service attack on Pokemon Go in July 2016.
This description "sounds like a 'Layer 7' or Application Layer attack," Cloudflare Information Security Chief Marc Rogers told Ars. This is a type of DDoS, although it's different from the ones websites are normally hit with.
"In this type of [DDoS] attack, instead of trying to saturate the site's network by flooding it with junk traffic, the attacker instead tries to bring a site down by attacking an application running on it," Rogers said.
"I am a little surprised that people are challenging the FCC's decision to call this a DDoS," Rogers also said. Cloudflare operates a global network that improves performance of websites and protects them from DDoS attacks and other security threats.
When asked if the FCC still believes it was hit with DDoS attacks, an FCC spokesperson told Ars that "there have been DDoS attacks during this process," including the morning of May 8. But the FCC official we talked to offered a bit less certainty on that point.
"The challenge is someone trying to deny service would do the same thing as someone who just doesn't know how to write a bot well," the FCC official said.
FCC officials said they spoke with law enforcement about the incident.
Spam bots and DDoS could have same effect
DDoS attacks, according to CDN provider Akamai, "are malicious attempts to render a website or Web application unavailable to users by overwhelming the site with an enormous amount of traffic, causing the site to crash or operate very slowly." DDoS attacks are "distributed" because the attacks generally "use large armies of automated 'bots'—computers that have been infected with malware and can be remotely controlled by hackers." (Akamai declined to comment on the FCC downtime when contacted by Ars.)
In this case, the FCC's media spokesperson told Ars the traffic did not come from infected computers. Instead, the traffic came from "cloud-based bots which made it harder to implement usual DDoS defenses."
The FCC official involved in the DDoS response told us that the comment system "experienced a large number of non-human digital queries," but that "the number of automated comments being submitted was much less than other API calls, raising questions as to their purpose."
If these were simply spammers who wanted to flood the FCC with as many comments as possible, like those who try to artificially inflate the number of either pro- or anti-net neutrality comments, they could have used the system's bulk filing mechanism instead of the API. But the suspicious traffic came through the API, and the API queries were "malformed." This means that "they aren't formatted well—they either don't fit the normal API spec or they are designed in such a way that they excessively tax the system when a simpler call could be done," the FCC official said.
Whether May 8 was the work of spam bots or DDoS attackers, "the effect would have been the same—denial of service to human users" who were trying to submit comments, the FCC official said. But these bots were submitting many fewer comments than other entities making API calls, suggesting that, if they were spam bots, they were "very poorly written."
The official said a similar event happened in 2014 during the previous debate over net neutrality rules, when bots tied up the system by filing comments and then immediately searching for them. "One has to ask why a bot would file, search, file, search, over and over," the official said.
If it was just a spam bot, "one has to wonder why, if the outside entity really wanted to upload lots of comments in bulk, they didn't use the alternative bulk file upload mechanism" and "why the bots were submitting a much lower number of comments relative to other API calls," the official said.
The FCC says it stopped the attacks by 8:45am ET on May 8, but the days that followed were still plagued by intermittent downtime. "There were other waves after 8:45am that slowed the system for some and, as noted, there were 'bots' plural, not just one," the FCC official said. On May 10, "we saw other attempts where massive malformed search queries also have hit the system, though it is unclear if the requestors meant for them to be poorly formed or not. The IT team has implemented solutions to handle them even if the API requests were malformed."
reader comments
55