John Regehr talk

Fuzzer Types

generative fuzzers
comes up with these things out of nowhere to try test inputs

mutation fuzzers
tweak an existing corpus to develop new examples

The structure of programs is often very ill-defined and vague. Fuzzingis useful because it allows you to rigorously build a justified sense ofconfidence that your code doesn't suck.

Code Differences

what's different about approaching the code?

for one compiler, the generated code could stack overflow. forces you tobe super tight about catching corner cases earlier in the start ratherthan actually just putting up with them without the fuzzing testing.gets you to dig into these test cases more easily.

outgrowth of test driven development - it's about being able to defineclasses of test cases you care about and iterate on testing those cases

you want to look for anything - crashes, things going wrong, etc… asyou're writing the code, you embed test oracles (assertions, primarily)within the code to dynamically check this! the weaker the language themore you need.

property testing and fuzzing are related; property testing is thought ofas fuzzing with more oracles - it allows you to describe the properties,whereas fuzzing is thought of as more 'black box'

adjusting to languages with fewer guardrails, or type safety; ships theresponsibility to the testing, mostly. this is why gradual typing are soexciting! you can shift towards a fully typed environment and ensurethat you add the types later, saving you a ton of the trouble oftesting.

don't invest a lot of work into generating test cases if you alreadyhave a super rich corpus of text to have your fuzzer draw from

c had lots of undefined behavior, so a very strong property had to beenforced on the test code by csmith to ensure that the code generatedwas from a defined subset of the c specification.

need to impose structure on the testing - valid c++ is one of the mostdifficult file formats.generators are so hard to write for not muchbenefit!

what should be fuzzed? what doesn't matter? ideal workflow with fuzzinginvolved

• project started from scratch write genreators as you write the code.okay to throw away them if they aren't scalable, continue to iterate
• project you pick up: heavily depends on use case obviously

don't fuzz if your code doesn't yet have the maturity to do it!

fuzzing is just weak verificatio: it's a weak quantifier over allpossible inputs in the space of inputs. it's one of hte closest ways wecan get to formal verification without the effort to do it! most thingsare mostly wrong and you need to fuzz them before proving them correct!only after fuzzing does verification make any sense. always fuzz withassertions, too, now have a decent specification for the system alreadyafter doing this fuzzing campaign!

almost everything he does is about people and making thingsbetter for people! great programming language trend towardsinteractivity and ways to work with you the way you want to work incode!

Insight that makes creduce works across languages - alcohol and lisp?shared syntax of block structure language - understands all of thebraces at some rudimentary level. has tokenizatio npass based on ctokenizer, but it works half decently for other things?

creduce:

• smtlib files? works fine . there is another smtlib reducer. creduce ispluggable and modular, plugs in a lot of reduction passes. could hackit to delete a bunch of passes from c and replace them with other onesas needed. can't work super well with other things, as it tries tolabel everything with ssa values - very hard coded for algol, lisp orsimilar.
• JITs?

• what keeps us from doing automated testing in industry today? why doyou think people aren't using it more?

  focus fuzzing on hot paths and common paths rather than all of thepaths. bad things: not prioritizing bugs, not being able to focus onthings that fuzzers care about. fuzzers can be dynamic - peopledon't necessarily want static checking!

DSL to specify what you care about - take advantage of what you careabout in testing! in general this is an incredible view on this program

best advice for producing qulaity software, in general: use modernlanguages, test rigorously, write proper fuzzer generators

Resources

Delta debugging was mentioned as an interesting tactic to use. What isit? AFL-Fuzz :: a fuzzerwritten in OCaml Big list of naughtystrings:Useful utilities for fuzzing. Fuzzing with Grammars - The FuzzingBookCsmith: Generate random C programsthat satisfy the C99 standard. GitHub - csmith-project/creduce:C-Reduce, a C programreducer: Take a largeprogram with an interesting property and reproduce a minimal test casefor it. FuzzOS \| Gamozo LabsBlog: Anoperating system designed and optimised specifically for fuzzing - sovirtual memory management are incredibly fast.