Introduction
Fuzz testing is a software testing technique used to find security and stability issues by providing pseudo-random data as input to the software.
Rust is a high performance, safe, general purpose programming language.
This book demonstrates how to perform fuzz testing for software written in Rust.
There are two tools for fuzzing Rust code documented in this book: afl.rs and cargo-fuzz.
The source of this book is available on GitHub at https://github.com/rust-fuzz/book.
Fuzzing with cargo-fuzz
cargo-fuzz is the recommended tool for fuzz testing Rust code.
cargo-fuzz is itself not a fuzzer, but a tool to invoke a fuzzer. Currently, the only fuzzer it supports is libFuzzer (through the libfuzzer-sys crate), but it could be extended to support other fuzzers in the future.
Setup
Requirements
libFuzzer needs LLVM sanitizer support, so this only works on x86-64 Linux, x86-64 macOS and Apple-Silicon (aarch64) macOS for now. Requires a C++ compiler with C++11 support. Rust provides multiple compilers. This project requires the nightly compiler since it uses the -Z
compiler flag to provide address sanitization. Assuming you used rustup to install Rust, you can check your default compiler with:
$ rustup default
stable-x86_64-unknown-linux-gnu (default) # Not the compiler we want.
To change to the nightly compiler:
$ rustup install nightly
$ rustup default nightly
nightly-x86_64-unknown-linux-gnu (default) # The correct compiler.
Installing
cargo install cargo-fuzz
Upgrading
cargo install --force cargo-fuzz
Tutorial
For this tutorial, we're going to be fuzzing the URL parsing crate rust-url. Our goal here is to find some input generated by the fuzzer such that, when passed to Url::parse
, it causes some sort of panic or crash to happen.
To start, clone the rust-url repository and change directories into it:
git clone https://github.com/servo/rust-url.git
cd rust-url
Although we could fuzz the latest commit on master
, we're going to checkout a specific revision that is known to have a parsing bug:
git checkout bfa167b4e0253642b6766a7aa74a99df60a94048
Initialize cargo-fuzz:
cargo fuzz init
This will create a directory called fuzz_targets
which will contain a collection of fuzzing targets. It is generally a good idea to check in the files generated by init
. Each fuzz target is a Rust program that is given random data and tests a crate (in this case, rust-url). cargo fuzz init
automatically generates an initial fuzz target for us.
Use cargo fuzz list
to view the list of all existing fuzz targets:
cargo fuzz list
The source code for this fuzz target by default lives in fuzz/fuzz_targets/<fuzz target name>.rs
. Open that file and edit it to look like this:
#![no_main]
#[macro_use] extern crate libfuzzer_sys;
extern crate url;
fuzz_target!(|data: &[u8]| {
if let Ok(s) = std::str::from_utf8(data) {
let _ = url::Url::parse(s);
}
});
libFuzzer
is going to repeatedly call the body of fuzz_target!()
with a slice of pseudo-random bytes, until your program hits an error condition (segfault, panic, etc). Write your fuzz_target!()
body to hit the entry point you need.
Since the generated data is a byte slice, we'll need to convert it to a UTF-8 &str
since rust-url expects that when parsing.
To begin fuzzing, run:
cargo fuzz run <fuzz target name>
Congratulations, you're fuzzing! The output you're seeing is generated by the fuzzer libFuzzer. To learn more about what the output means see the 'output' section in the libFuzzer documentation.
If you leave it going for long enough you'll eventually discover a crash. The output would look something like this:
...
#56232 NEW cov: 2066 corp: 110/4713b exec/s: 11246 rss: 170Mb L: 42 MS: 1 EraseBytes-
#58397 NEW cov: 2069 corp: 111/4755b exec/s: 11679 rss: 176Mb L: 42 MS: 1 EraseBytes-
#59235 NEW cov: 2072 corp: 112/4843b exec/s: 11847 rss: 178Mb L: 88 MS: 4 InsertByte-ChangeBit-CopyPart-CopyPart-
#60882 NEW cov: 2075 corp: 113/4953b exec/s: 12176 rss: 183Mb L: 110 MS: 1 InsertRepeatedBytes-
thread '<unnamed>' panicked at 'index out of bounds: the len is 1 but the index is 1', src/host.rs:105
note: Run with `RUST_BACKTRACE=1` for a backtrace.
==70997== ERROR: libFuzzer: deadly signal
#0 0x1097c5500 in __sanitizer_print_stack_trace (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x62500)
#1 0x108383d1b in fuzzer::Fuzzer::CrashCallback() (fuzzer_script_1:x86_64+0x10002fd1b)
#2 0x108383ccd in fuzzer::Fuzzer::StaticCrashSignalCallback() (fuzzer_script_1:x86_64+0x10002fccd)
#3 0x1083d19c7 in fuzzer::CrashHandler(int, __siginfo*, void*) (fuzzer_script_1:x86_64+0x10007d9c7)
...
#33 0x10838b393 in fuzzer::Fuzzer::Loop() (fuzzer_script_1:x86_64+0x100037393)
#34 0x1083650ec in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (fuzzer_script_1:x86_64+0x1000110ec)
#35 0x108396c3f in main (fuzzer_script_1:x86_64+0x100042c3f)
#36 0x7fff91552234 in start (libdyld.dylib:x86_64+0x5234)
NOTE: libFuzzer has rudimentary signal handlers.
Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 2 InsertByte-EraseBytes-; base unit: 3c4fc9770beb5a732d1b78f38cc8b62b20cb997c
0x68,0x74,0x74,0x70,0x3a,0x2f,0x2f,0x5b,0x3a,0x5d,0x3a,0x78,0xc5,0xa4,0x1,0x3a,0x7f,0x1,0x59,0xc5,0xa4,0xd,0x78,0x78,0x3a,0x78,0x69,0x3a,0x0,0x69,0x3a,0x5c,0xd,0x7e,0x78,0x40,0x0,0x25,0xa,0x0,0x29,0x20,
http://[:]:x\xc5\xa4\x01:\x7f\x01Y\xc5\xa4\x0dxx:xi:\x00i:\\\x0d~x@\x00%\x0a\x00)
artifact_prefix='/private/tmp/rust-url/fuzz/artifacts/fuzzer_script_1/'; Test unit written to /home/user/rust-url/fuzz/artifacts/fuzzer_script_1/crash-e9b1b5183e46a288c25a2a073262cdf35408f697
Base64: aHR0cDovL1s6XTp4xaQBOn8BWcWkDXh4OnhpOgBpOlwNfnhAACUKACkg
The line in the output that starts with http
is the input that causes a panic in rust-url.
Guide
All available commands available for cargo-fuzz:
cargo fuzz --help
Run a target:
cargo fuzz run <fuzz target name>
Cargo features
It is possible to fuzz crates with different configurations of Cargo features by using the command line options --features
, --no-default-features
and --all-features
. Note that these options control the fuzz_targets
crate; you will need to forward them to the crate being fuzzed by e.g. adding the following to fuzz_targets/Cargo.toml
:
[features]
unsafe = ["project/unsafe"]
#[cfg(fuzzing)]
Every crate instrumented for fuzzing -- the fuzz_targets
crate, the project crate, and their entire dependency tree -- is compiled with the --cfg fuzzing
rustc option. This makes it possible to disable code paths that prevent fuzzing from working, e.g. verification of cryptographic signatures, with a simple #[cfg(not(fuzzing))]
, and without the need for an externally visible Cargo feature that must be maintained throughout every dependency.
libFuzzer configuration options
See all the libFuzzer options:
cargo fuzz run <fuzz target name> -- -help=1
For example, to generate only ASCII inputs, run:
cargo fuzz run <fuzz target name> -- -only_ascii=1
Structure-Aware Fuzzing
Not every fuzz target wants to take a buffer of raw bytes as input. We might
want to only feed it well-formed instances of some structured data. Luckily, the
libfuzzer-sys
crate enables us to define fuzz targets that take any kind of type,
as long as it implements the Arbitrary
trait.
libfuzzer_sys::fuzz_target!(|input: AnyTypeThatImplementsArbitrary| {
// Use `input` here...
})
The arbitrary
crate implements Arbitrary
for nearly all the types in std
,
including collections like Vec
and HashMap
as well as things like String
and PathBuf
.
For convenience, the libfuzzer-sys
crate re-exports the arbitrary
crate as
libfuzzer_sys::arbitrary
. You can also enable #[derive(Arbitrary)]
either by
- enabling the
arbitary
crate's"derive"
feature, or - (equivalently) enabling the
libfuzzer-sys
crate's"arbitrary-derive"
feature.
See the arbitrary
crate's documentation for more details.
This section concludes with two examples of structure-aware fuzzing:
Example 1: Fuzzing Color Conversions
Let's say we are working on a color conversion library that can turn RGB colors into HSL and back again.
Enable Deriving Arbitrary
We are lazy, and don't want to implement Arbitrary
by hand, so we want to
enable the arbitrary
crate's "derive"
cargo feature. This lets us get
automatic Arbitrary
implementations with #[derive(Arbitrary)]
.
Because the Rgb
type we will be deriving Arbitrary
for is in our main color
conversion crate, we add this to our main Cargo.toml
.
# Cargo.toml
[dependencies]
arbitrary = { version = "1", optional = true, features = ["derive"] }
Derive Arbitrary
for our Rgb
Type
In our main crate, when the "arbitrary"
cargo feature is enabled, we derive
the Arbitrary
trait:
// src/lib.rs
#[derive(Clone, Debug)]
#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
pub struct Rgb {
pub r: u8,
pub g: u8,
pub b: u8,
}
Enable the Main Project's "arbitrary"
Cargo Feature for the Fuzz Targets
Because we made arbitrary
an optional dependency in our main color conversion
crate, we need to enable that feature for our fuzz targets to use it.
# fuzz/Cargo.toml
[dependencies]
my_color_conversion_library = { path = "..", features = ["arbitrary"] }
Add the Fuzz Target
We need to add a new fuzz target to our project:
$ cargo fuzz add rgb_to_hsl_and_back
Implement the Fuzz Target
Finally, we can implement our fuzz target that takes arbitrary RGB colors,
converts them to HSL, and then converts them back to RGB and asserts that we get
the same color as the original! Because we implement Arbitrary
for our Rgb
type, our fuzz target can take instances of Rgb
directly:
// fuzz/fuzz_targets/rgb_to_hsl_and_back.rs
libfuzzer_sys::fuzz_target!(|color: Rgb| {
let hsl = color.to_hsl();
let rgb = hsl.to_rgb();
// This should be true for all RGB -> HSL -> RGB conversions!
assert_eq!(color, rgb);
});
Example 2: Fuzzing Allocator API Calls
Imagine, for example, that we are fuzzing our own malloc
and free
implementation. We want to make a sequence of valid allocation and deallocation
API calls. Additionally, we want that sequence to be guided by the fuzzer, so it
can use its insight into code coverage to maximize the amount of code we
exercise during fuzzing.
Add the Fuzz Target
First, we add a new fuzz target to our project:
$ cargo fuzz add fuzz_malloc_free
Enable Deriving Arbitrary
Like the color conversion example above, we don't want to write our Arbitrary
implementation by hand, we want to derive it.
# fuzz/Cargo.toml
[dependencies]
libfuzzer-sys = { version = "0.4.0", features = ["arbitrary-derive"] }
Define an AllocatorMethod
Type and Derive Arbitrary
Next, we define an enum
that represents either a malloc
, a realloc
, or a
free
:
// fuzz_targets/fuzz_malloc_free.rs
use libfuzzer_sys::arbitrary::Arbitrary;
#[derive(Arbitrary, Debug)]
enum AllocatorMethod {
Malloc {
// The size of allocation to make.
size: usize,
},
Free {
// Free the index^th allocation we've made.
index: usize
},
Realloc {
// We will realloc the index^th allocation we've made.
index: usize,
// The new size of the allocation.
new_size: usize,
},
}
Write a Fuzz Target That Takes a Sequence of AllocatorMethod
s
Finally, we write a fuzz target that takes a vector of AllocatorMethod
s and
interprets them by making the corresponding malloc
, realloc
, and free
calls. This works because Vec<T>
implements Arbitrary
when T
implements
Arbitrary
.
// fuzz/fuzz_targets/fuzz_malloc_free.rs
libfuzzer_sys::fuzz_target!(|methods: Vec<AllocatorMethod>| {
let mut allocs = vec![];
// Interpret the fuzzer-provided methods and make the
// corresponding allocator API calls.
for method in methods {
match method {
AllocatorMethod::Malloc { size } => {
let ptr = my_allocator::malloc(size);
allocs.push(ptr);
}
AllocatorMethod::Free { index } => {
match allocs.get(index) {
Some(ptr) if !ptr.is_null() => {
my_allocator::free(ptr);
allocs[index] = std::ptr::null();
}
_ => {}
}
}
AllocatorMethod::Realloc { index, size } => {
match allocs.get(index) {
Some(ptr) if !ptr.is_null() => {
let new_ptr = my_allocator::realloc(ptr, size);
allocs[index] = new_ptr;
}
_ => {}
}
}
}
}
// Free any remaining allocations.
for ptr in allocs {
if !ptr.is_null() => {
my_allocator::free(ptr);
}
}
});
Code Coverage
Visualizing code coverage helps you understand which code paths are being fuzzed and — more importantly — which aren't. To help the fuzzer exercise new code paths, you can look at what it is failing to reach and then either add new seed inputs to the corpus, or tweak the fuzz target. This chapter describes how to generate coverage reports for your fuzz target and its current corpus.
Prerequisites
First, install the LLVM-coverage tools as described in the rustc book.
If you are using a non-nightly toolchain as your default toolchain, remember to
install the rustup components for the nightly toolchain instead of the default
(rustup component add --toolchain nightly llvm-tools-preview ...
).
You must also have cargo fuzz
version 0.10.0
or newer to use the cargo fuzz coverage
subcommand.
Generate Code-Coverage Data
After you fuzzed your program, use the coverage
command to generate precise
source-based code coverage information:
$ cargo fuzz coverage <target> [corpus dirs] [-- <args>]
This command
-
compiles your project using the
-Cinstrument-coverage
Rust compiler flag, -
runs the program without fuzzing on the provided corpus (if no corpus directory is provided it uses
fuzz/corpus/<target>
by default), -
for each input file in the corpus, generates raw coverage data in the
fuzz/coverage/<target>/raw
subdirectory, and -
merges the raw files into a
coverage.profdata
file located in thefuzz/coverage/<target>
subdirectory.
Afterwards, you can use the generated coverage.profdata
file to generate
coverage reports and visualize code-coverage information as described in the
rustc book.
Example
Suppose we have a my_compiler
fuzz target for which we want to visualize code
coverage.
-
Run the fuzzer on the
my_compiler
target:$ cargo fuzz run my_compiler
-
Produce code-coverage information:
$ cargo fuzz coverage my_compiler
-
Visualize the coverage data in HTML:
$ cargo cov -- show fuzz/target/<target triple>/release/my_compiler \ --format=html \ -instr-profile=fuzz/coverage/my_compiler/coverage.profdata \ > index.html
There are many visualization and coverage-report options available (see
llvm-cov show --help
).
Targets
A collection of community maintained cargo-fuzz compatible fuzz targets can be found here.
Fuzzing with afl.rs
American fuzzy lop (AFL) is a popular, effective, and modern fuzz testing tool. afl.rs allows one to run AFL on code written in the Rust programming language.
Setup
Requirements
Tools
- C compiler (e.g. gcc or clang)
- make
Platform
afl.rs works on x86-64 Linux, x86-64 macOS, and ARM64 macOS.
cargo install cargo-afl
Alternatively, cargo-afl
can be installed from source.
Upgrading
cargo install --force cargo-afl
Tutorial
For this tutorial, we are going to fuzz the URL parser rust-url. Our goal here is to find some input generated by the fuzzer such that, when passed to Url::parse
, it causes some sort of panic or crash to happen.
Create a fuzz target
The first thing we’ll do is create a fuzz target in the form of a Rust binary crate. AFL will call the resulting binary, supplying generated bytes to standard input that we’ll pass to Url::parse
.
cargo new --bin url-fuzz-target
cd url-fuzz-target
We’ll need two dependencies in this crate:
url
: the crate we’re fuzzingafl
: not required, but includes a couple utility functions to assist in creating fuzz targets
So add these to the Cargo.toml
file:
[dependencies]
afl = "*"
url = "*"
Now we’ll need to write the source for the fuzz target in src/main.rs
:
#[macro_use]
extern crate afl;
extern crate url;
fn main() {
fuzz!(|data: &[u8]| {
if let Ok(s) = std::str::from_utf8(data) {
let _ = url::Url::parse(&s);
}
});
}
fuzz!
is a utility macro provided by the afl
crate that reads bytes from standard input and
passes the bytes to the provided closure.
In the body of the closure, we call Url::parse
with the bytes that AFL generated. If all goes well, url::Url::parse
will return an Ok
containing a valid Url
, or an Err
indicating a Url
could not be constructed from the String
. If Url::parse
panics while parsing the String
, AFL will treat it as a crash and the AFL UI will indicate as such.
One important detail about the fuzz!
macro: if a panic occurs within the body of the closure, the panic will be caught and process::abort
will be subsequently called. Without the call to process::abort
, AFL would not consider the unwinding panic to be a crash.
Build the fuzz target
Normally, one uses cargo build
to compile a Cargo-based Rust project. To get AFL to work with Rust, a few extra compiler flags need to be passed to rustc during the build process. To make this easier, there is an AFL cargo subcommand (provided by the afl
crate) that automatically passes these rustc flags for us. To use it, you’ll do something like:
cargo afl <cargo command>
Since we want to build this crate, we’ll run:
cargo afl build
Provide starting inputs
AFL doesn't strictly require starting inputs, but providing some can make AFL’s job easier since it won’t need to ‘learn’ what a valid URL looks like. To do this, we'll create a directory called in
with a few files (filenames don’t matter) containing valid URLs:
mkdir in
echo "tcp://example.com/" > in/url
echo "ssh://192.168.1.1" > in/url2
echo "http://www.example.com:80/foo?hi=bar" > in/url3
Start fuzzing
To begin fuzzing, we’ll run:
cargo afl fuzz -i in -o out target/debug/url-fuzz-target
The fuzz
subcommand of cargo-afl
is the primary interface for fuzzing Rust code with AFL. For those already familiar with AFL, the fuzz
subcommand of cargo-afl
is identical to running afl-fuzz
.
The -i
flag specifies a directory full of input files AFL will use as seeds.
The -o
flag specifies a directory AFL will write all its state and results to.
The last argument target/debug/url-fuzz-target
specifies the fuzz target binary AFL will call, supplying random bytes to standard input.
As soon as you run this command, you should see AFL’s interface start up:
For more information about this UI and what each of the sections mean, see this resource hosted on the AFL website.
AFL will run indefinitely, so if you want to quit, press CTRL-C
.
Reproducing
Once you have a few crashes collected from running your fuzzer, you can reproduce them by passing them in manually to your test case. This is typically done via stdin
. E.g. for url-fuzz-target
the command would be:
cargo afl run url-fuzz-target < out/default/crashes/crash_file
where out
is the -o
parameter from your fuzz command and crash_file
is an arbitrary file in the crashes
directory.
Installing from source
First, clone afl.rs:
git clone https://github.com/rust-fuzz/afl.rs
cd afl.rs
Next, checkout afl.rs's submodule (AFL++). Note that --recursive
is not required.
git submodule update --init
Finally, install cargo-afl
:
cargo install --path cargo-afl
Troubleshooting
If cargo-afl
is panicking, consider installing with --debug
and running cargo-afl
with RUST_BACKTRACE=1
, e.g.:
cargo install --path cargo-afl --debug
...
RUST_BACKTRACE=1 cargo afl ...
Adding --debug
to the cargo install
command causes cargo-afl
to produce more elaborate backtraces.
Trophy Case
A collection of bugs found in Rust code through fuzz testing can be found in the trophy-case repository.