Fuzzing is a well-established software testing methodology, particularly effective for examining smart contracts. Traditionally, testers have primarily focused on Black Box Fuzzing and, for those more experienced, Property-Based Fuzzing. However, a novel and creative approach known as Manually Guided Fuzzing enhances both efficiency and effectiveness, filling the voids left by conventional fuzzing methods. This article will delve into how Guided Fuzzing contrasts with Black Box Fuzzing, Property-Based Fuzzing, Differential Fuzzing, and other strategies.
Understanding the Fundamentals: Stateful vs. Stateless and Black Box vs. Grey Box vs. White Box Fuzzing
Fuzzing techniques can be divided based on their statefulness and the extent of guidance provided:
Stateful vs. Stateless Fuzzing:
Stateless Fuzzing: In stateless fuzzing, each test case operates independently, without any memory of previous states or inputs, which restricts its capacity to identify bugs tied to the sequence of actions.
Stateful Fuzzing: This technique takes into account the state of the system, meaning the result of one test case can affect those that follow. Stateful fuzzing proves more useful for testing intricate systems like smart contracts, where state transitions hold significant importance. For clarification, refer to this code example created using the Wake Framework.
Black Box vs. Grey Box vs. White Box:
Black Box Fuzzing: This method operates without any insight into the internal mechanisms of the system being tested. While it must understand the interface, it depends on random inputs and heuristic strategies to explore the state space to unveil unexpected behaviors or crashes. Though black box fuzzing can be effective in certain situations, it tends to be resource-intensive and often falls short of achieving comprehensive coverage.
Grey Box Fuzzing: The fuzzer has partial knowledge of the internal workings of the system under test. This category is broad, with examples such as Invariants (refer to Property-based Fuzzing).
White Box: In contrast, this approach provides the fuzzer with complete knowledge of the system’s internal workings. This ability allows for more focused and efficient fuzzing, concentrating on critical code paths and specific scenarios, thus leading to faster and more reliable bug detection (see Manually Guided Fuzzing).
Diving Deeper: Advanced Fuzzing Techniques
Additional advanced techniques facilitate testing in even more edge-case scenarios:
Differential Fuzzing: This method compares the outputs of the system under test with those from a reference model. The reference could be an implementation written in a high-level language such as Python. The subject of testing could also be a single function analyzed against an established library that performs the same operation. Differential fuzzing is a sophisticated method primarily for mathematical functions that can unveil numerous rounding and precision errors associated with Solidity and gains additional strength when paired with stateful fuzzing. A notable example of differential fuzzing is the IPOR audit, where Ackee Blockchain Security compared the implementation of continuous compound interest rate calculations within contracts to a custom Python model, which uncovered one critical and two high-severity issues.
Fork Testing: This technique is used for projects that have complex dependencies or integrations (e.g., Aave, Chainlink). Fork testing entails executing tests on a development chain (like Anvil) that forks from a live network. This allows the system to interact with authentic data and states without the necessity of redeployment and mocking contract storage. Leveraging real-world data ensures more precise and comprehensive testing when compared to data mocking (a common approach in Formal Verification). A practical instance is the Lido Stonks audit, during which Ackee Blockchain Security forked the Ethereum mainnet to test the protocol in realistic conditions. The integration of a forked USDT contract led to the discovery of a medium severity integration issue; see the complete source code here.
Understanding Property-Based Fuzzing
Property-based fuzzing, also known as Invariant testing, occupies a space between black box and white box fuzzing. In this method, the tester (with some familiarity with the system) identifies specific properties known as invariants that must always hold true, regardless of the inputs. The fuzzer’s task is to generate inputs that violate these properties. With the tester’s introduction of invariants, this approach provides the “gray-box” designation. While it is more structured than black box fuzzing, property-based fuzzing still encounters challenges in fully exploring intricate state spaces, especially within complex smart contracts.
An invariant refers to a test executed following each state change (transaction).
@invariant()
def invariant_balance(self) -> None:
assert self.token.balanceOf(self.admin) == self.balances[self.admin]Introducing Manually Guided Fuzzing
Manually Guided Fuzzing merges the benefits of Stateful Fuzzing and White Box Fuzzing, incorporating the concept of Flows to create a systematic approach to testing. This method necessitates the tester to thoroughly understand the system under scrutiny and direct the fuzzing process, ensuring thorough examination of critical code sections.
A Flow denotes a series of actions or transactions within the system. For example, a flow could encompass the transfer of tokens from one account to another. The tester outlines these flows to guarantee that the fuzzer targets essential code paths.
A flow is a singular test step executed within a test sequence. Flows are established using the @flow annotation:
@flow(precondition=lambda self: self.count > 0)
def flow_decrement(self) -> None:
self.counter.decrement(from_=random_account())
self.count -= 1How Manually Guided Fuzzing Operates
Utilizing the Wake Framework, Manually Guided Fuzzing involves multiple stages:
- Defining Invariants: As in property-based fuzzing, Manually Guided Fuzzing necessitates defining properties or invariants that should hold true after a flow is executed. For instance, following a token transfer, an invariant could be that the total token supply remains unchanged and that balances are updated accurately.
- Defining Flows: The tester now specifies how the fuzzer should interact with the contract. Continuing with the previous example, this would involve initiating the token transfer process.
- Combining Flows and Invariants: Manually Guided Fuzzing makes for a more effective fuzzing methodology by integrating random flows, with each flow followed by comprehensive invariant checks. Instead of haphazardly exploring the state space, this technique allows the tester to concentrate on targeted areas of code, thereby significantly minimizing the time and computational resources required.
(Dis)advantages of Manually Guided Fuzzing
- Efficiency: By steering the fuzzer with designated flows and properties, Manually Guided Fuzzing drastically reduces the explored state space, consequently shortening the time needed to identify bugs.
- Flexibility: Testers wield complete control over the fuzzing procedure, allowing them to examine specific situations, including stateful interactions, cross-chain transactions, and intricate contract dependencies, which are frequently challenging to address using black box fuzzing or formal verification.
- With great power comes great responsibility: The tester must recognize potential red flags within the code and determine how to address them with fuzz tests. Typically, the tester defines flows for all public functions that alter state and selects appropriate input values for these functions. If the tester overlooks these red flags or lacks a clear idea for attack vectors, the fuzzing campaign may not uncover potential vulnerabilities.
Conclusion
Manually Guided Fuzzing signals a shift in responsibility from testers employing brute force or heuristic algorithms back to auditors and security researchers, who must guide fuzz tests toward potential attack vectors to uncover vulnerabilities. This approach provides a more efficient, precise, and scalable method for assessing complex systems such as integrated smart contracts. Explore the techniques discussed utilizing the Wake framework, which supports Manually Guided Fuzzing, fork testing, and differential testing.
Additional Resources
The following examples demonstrate the real-world application of various types of fuzz tests:
