​CIO Diary: Lessons from the FCC bot-swarm attack

No one ever said that being a Chief Information Officer is easy. As if CIOs don't have enough to worry about, you can now add API-driven bot attacks to the list.

One of the most highly-regarded CIOs in the world, David Bray from the FCC (who is also a frequent guest on CXOTALK), faced exactly this problem. I spoke with him at length today to learn just what happened.

On Sunday, May 7, the FCC started receiving an unusually large number of public comments and document requests on its Electronic Comment Filing System. The ECFS is where members of the public, including companies and researchers, can search the full range of FCC archives. To put the numbers into context, Bray explained, "It took about 110 days in 2014 to reach 2+ million comments. Here we received 2+ million comments in just ten days."

According to Bray, FCC staff noticed high comment volumes around 3:00 AM the morning of Monday, May 8. As the FCC analyzed the log files, it became clear that non-human bots created these comments automatically by making calls to the FCC's API.

Interestingly, the attack did not come from a botnet of infected computers but was fully cloud-based.

By using commercial cloud services to make massive API requests, the bots consumed available machine resources, which crowded out human commenters. In effect, the bot swarm created a distributed denial of service attack on FCC systems using the public API as a vehicle. It's similar to the distributed denial of service attack on Pokemon Go in July 2016. The FCC moved to cloud services in 2015.

Bray said his team could not just disable the API to solve the problem because the FCC has an obligation to offer public access to the data. That's why the ECFS is "open by design."

The emphasis on data accessibility also means that spam fighting systems, like CAPTCHA, are not an option because they may interfere with access from legitimate users. For example, these tools can stop some users, who may possess disabilities, from accessing the site. Importantly, public stakeholders also want to allow users to submit comments on behalf of others using automation. Again, this goal is in conflict with CAPTCHA.

The bot swarm did not affect an alternative mechanism the FCC maintains for uploading comments. This alternate user interface is a bulk uploading facility that lets users upload comments with a standard CSV file.

Implications for Chief Information Officers

The CIO role must always refer to strategic organizational mandates that come from the board and CEO. The fundamental challenge, therefore, facing any CIO is figuring out how to maintain technology systems while remaining true to core business strategy.

The FCC situation offers a straightforward illustration of what we might call "mandate conflict:"

1. Openness to the public is an organizational imperative
2. This openness, in the form of a public API, created the conditions for API-based denial of service attack
3. Replacing the open system with something more closed is not a possibility

The simple question becomes, "What's a CIO to do?" Mandate conflict is one of the most difficult challenges any CIO will ever face. In truth, the way we manage those conflicts can decide the course of our future career. Push too hard in one direction or the other, and the juggling act falls apart.

In a situation of conflicting or unclear priorities, the only solution is huddle with stakeholders and decide the most important order of priority.

In this specific case of the FCC, Bray and team decided that data openness was a top priority. As a result, all decisions in handling the situation reflected this emphasis.

Another organization may have handled the situation by shutting down the public API and restricting access. For a private company, that solution may be correct. But not at the FCC.

I asked Bray being a non-partisan, government CIO:

I was involved in the responses to 9/11 and anthrax in 2001. I also volunteered to deploy to Afghanistan in 2009. While these events created some long hours for the team, no one is shooting at us, and this is part of the challenge of doing IT a digital world where almost anyone can write a bot for whatever purpose fairly easily. Public service has a mandate to be open wherever possible and where it does not conflict with individual privacy or security.

In summary, the best defense against mandate conflict is being clear about the mandate and communicating with stakeholders to ensure there is consensus on priorities.

Yeah, no one ever said being a CIO is easy.