The 2016 Cyber Grand Challenge (CGC) was a challenge created by The Defense Advanced Research Projects Agency (DARPA) in order to develop automatic defense systems that can discover, prove, and correct software flaws in real-time.
The event placed machine versus machine (no human intervention) in what was called the "world's first automated network defense tournament."
The final event was held on August 4, 2016 at the Paris Hotel & Conference Center in Las Vegas, Nevada within the 24th DEF CON hacker convention.
It resembled in structure the long-standing "capture the flag" (CTF) security competitions, and the winning system indeed competed against humans in the "classic" DEF CON CTF held in the following days. The Cyber Grand Challenge featured, however, a more standardized scoring and vulnerability-proving system: all exploits and patched binaries were submitted and evaluated by the referee infrastructure
Background
Races develop between criminals attempting to abuse vulnerabilities and analysts who assess, remediate, check, and deploy a patch before significant damage can be done. Experts adhere to a process that involves complicated reasoning followed by manual creation of each security signature and software patch, a technical process that requires months and dollars. This has resulted in various software insecurities favoring attackers. Devices such as smart televisions, wearable technologies, and high-end home appliances that are connected to the internet aren't always produced with security in mind and moreover utility systems, power grids, and traffic lights could be more susceptible to attacks, says the DARPA.
To help overcome these challenges, DARPA launched in 2014 the Cyber Grand Challenge: a two-year competition seeking to create automatic defensive systems capable of reasoning about flaws, formulating patches and deploying them on a network in real time. The competition was split into two main events: an open qualification event to be held in 2015 and a final event in 2016 where only the top seven teams from the qualifiers could participate. The winner of the final event would be awarded $2 million and the opportunity to play against humans in the 24th DEF CON capture the flag competition.
Technology
Challenge binaries
Challenge Binaries ran on the full 32-bit Intel x86 architecture, albeit with a simplified ABI.
Reducing external interaction to its base components (e.g., system calls for well-defined I/O, dynamic memory allocation, and a single source of randomness) simplified both modeling and securely running the binaries in isolation to observe their behavior.
Internal complexity was however unrestricted, with challenges going as far as implementing a particle physics simulator, chess, programming/scripting languages, parsing of huge amounts of markup data, vector graphics, just-in-time compilation, VMs, etc.
The challenge authors were themselves scored based on how well they distinguished the players' relative performance, encouraging challenges to exercise specific weaknesses of automatic reasoning (e.g., state explosion) while remaining solvable by well-constructed systems.
Player systems
Each playing system -- a fully-automated "Cyber Reasoning System" (CRS) -- had to demonstrate ability in several areas of computer security:
Automatic vulnerability finding on previously-unknown binaries. Automatic patching of binaries without sacrificing performance. Automatic exploit generation within the framework's limitations. Implementing a security strategy: balancing resource-assignment among the available servers (a variation of the multi-armed bandit problem), responding to competitors (e.g., analyzing their patches, reacting to exploitation), evaluating own action's effect on the final score, ...
Teams described their approach in various venues.
Additionally, the third-place finisher (Shellphish) released their entire system's source code.
Due to the complexity of the task, players had to combine multiple techniques and do so in a fully-unattended and time-efficient fashion. For instance, the highest attack score was reached by discovering vulnerabilities via a combination of guided fuzzing and symbolic execution -- i.e., an AFL-based fuzzer combined with the angr binary analysis framework, leveraging a QEMU-based emulation and execution-tracing system.