In relation to EIP-4844, Ethereum clients must be equipped to compute and verify KZG commitments. Instead of each client developing its own cryptographic solutions, researchers and developers collaborated to create c-kzg-4844, a compact C library with bindings for higher-level programming languages. The objective was to establish a secure and efficient cryptographic library available for all clients. The Protocol Security Research team at the Ethereum Foundation assessed and enhanced this library. In this blog post, we will cover several methods we utilize to enhance the security of C projects.
Fuzz
Fuzzing is a dynamic code testing approach that involves injecting random inputs to find bugs within a program. LibFuzzer and afl++ are two widely used fuzzing frameworks for C-based projects. Both are coverage-guided, in-process evolutionary fuzzing engines. For c-kzg-4844, we opted for LibFuzzer, as we were well-aligned with other offerings from the LLVM project.
Below is the fuzzer for verify_kzg_proof, a function in c-kzg-4844:
#include "../base_fuzz.h" static const size_t COMMITMENT_OFFSET = 0; static const size_t Z_OFFSET = COMMITMENT_OFFSET + BYTES_PER_COMMITMENT; static const size_t Y_OFFSET = Z_OFFSET + BYTES_PER_FIELD_ELEMENT; static const size_t PROOF_OFFSET = Y_OFFSET + BYTES_PER_FIELD_ELEMENT; static const size_t INPUT_SIZE = PROOF_OFFSET + BYTES_PER_PROOF; int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) { initialize(); if (size == INPUT_SIZE) { bool ok; verify_kzg_proof( &ok, (const Bytes48 *)(data + COMMITMENT_OFFSET), (const Bytes32 *)(data + Z_OFFSET), (const Bytes32 *)(data + Y_OFFSET), (const Bytes48 *)(data + PROOF_OFFSET), &s ); } return 0; }
Upon execution, the output appears as follows. Should an issue arise, the input would be recorded to disk, and execution would cease. Ideally, the problem should be reproducible.
Differential fuzzing is also a technique that fuzzes two or more implementations of the same interface and compares their outputs. If the output for a given input is inconsistent with expected results, it indicates a potential issue. This technique is particularly favored in Ethereum because multiple implementations help safeguard against flaws in any single implementation.
For KZG libraries, we developed kzg-fuzz, which differentially fuzzes c-kzg-4844 (via its Golang bindings) and go-kzg-4844. No differences have been detected so far.
Coverage
Next, we employed llvm-profdata and llvm-cov to create a coverage report based on the tests we executed. This serves as an effective method to confirm which code paths are executed (“covered”) and tested. Refer to the coverage target in the Makefile of c-kzg-4844 for an example of generating this report.
When this target is executed (for instance,, make coverage) it generates a table providing a high-level review of function execution counts. Exported functions appear at the top, followed by static (non-exported) functions at the bottom.
The table above contains substantial green, though some yellow and red indicators exist as well. To ascertain which elements are executed versus those that are not, refer to the generated HTML file (coverage.html). This webpage highlights the entire source file and includes non-executed code in red. In this particular project’s context, most of the non-executed sections pertain to intricate error scenarios like memory allocation failures. For instance, the following is an example of code that was not executed:
Initially, this function verifies that the trusted setup is sufficiently sized for a pairing check. Since there’s no test case providing an invalid trusted setup, this portion remains unexecuted. Furthermore, as we only verify with the correct trusted setup, the output of is_monomial_form invariably returns the same value and does not trigger an error response.
Profile
We do not advocate this for all projects, but given that c-kzg-4844 is a performance-sensitive library, profiling its exported functions to monitor their execution time is crucial. This practice identifies inefficiencies that might lead to denial-of-service scenarios for nodes. We utilized gperftools (Google Performance Tools) over llvm-xray due to its rich feature set and enhanced user-friendliness.
The example below profiles my_function. Profiling functions involves periodically checking which instructions are being executed. If a function executes quickly, the profiler may miss it, so it’s prudent to call the function multiple times. In this case, we invoke my_function 1000 times.
#includeint task_a(int n) { if (n <= 1) return 1; return task_a(n - 1) * n; } int task_b(int n) { if (n <= 1) return 1; return task_b(n - 2) + n; } void my_function(void) { for (int i = 0; i < 500; i++) { if (i % 2 == 0) { task_a(i); } else { task_b(i); } } } int main(void) { ProfilerStart("example.prof"); for (int i = 0; i < 1000; i++) { my_function(); } ProfilerStop(); return 0; }
Utilize ProfilerStart(“
The following graph was generated using the above command:
Here’s a larger example from one of the functions in c-kzg-4844. The image portrays the profiling graph for compute_blob_kzg_proof. As evident, 80% of the execution time of this function is consumed by Montgomery multiplications, which aligns with expectations.
Reverse
Next, analyze your binary using software reverse engineering (SRE) tools like Ghidra or IDA. These tools assist in understanding how high-level constructs are translated into low-level machine code. Reviewing your code in this manner can provide new perspectives, akin to reading a paper in a different font. It is also beneficial to observe the compiler’s optimization strategies. Occasionally, the compiler might eliminate what it considers unnecessary components. Monitoring for such occurrences is advisable, as realized within c-kzg-4844 where some tests were eliminated during optimization.
A decompiled function will lack variable names, complex types, or comments, as this information isn’t retained in the binary after compilation. Reverse engineering this information becomes your responsibility. Functions may be inlined into a single operation, multiple variables may be consolidated into a single buffer, and the arrangement of verifications could vary due to compiler optimizations, which are generally acceptable. Compiling your binary with DWARF debugging information may aid most SRE tools in providing enhanced results.
For instance, this is how blob_to_kzg_commitment appears initially in Ghidra:
With some modifications, you can rename variables and add comments for better readability. After a few minutes, it might look like this:
Static Analysis
Clang includes the Clang Static Analyzer, a commendable static analysis tool adept at identifying numerous issues overlooked by compilers. This “static” analysis examines code without executing it. It tends to be slower than compilers but significantly faster than “dynamic” analysis tools that do execute code.
Here’s a basic illustration where a failure to free arr (coupled with another issue which we will address later) persist. Even with all warnings enabled, the compiler does not flag this, as it technically qualifies as valid code.
#includeint main(void) { int* arr = malloc(5 * sizeof(int)); arr[5] = 42; return 0; }
The unix.Malloc checker will identify that arr was not freed. While the warning message may be somewhat misleading, it becomes clearer upon reflection; the analyzer arrives at the return statement without freeing the memory.
However, not all findings are this straightforward. The Clang Static Analyzer discovered the following in c-kzg-4844 upon its initial introduction into the project:
An unexpected input could induce a 32-bit shift in this value, leading to undefined behavior. To resolve this, input validation was incorporated using CHECK(log2_pow2(n) != 0) to prevent this scenario from arising. Kudos to the Clang Static Analyzer!
Sanitize
Sanitizers are dynamic analysis tools that instrument (add code) to programs, allowing for the identification of issues during runtime. They are particularly effective at uncovering common mistakes involving memory management. Clang includes several sanitizers, and below are the four we find most beneficial and accessible.
Address
AddressSanitizer (ASan) is a rapid memory error detector capable of identifying out-of-bounds accesses, use-after-free issues, use-after-return situations, use-after-scope incidents, double-free issues, and memory leaks.
Using the earlier example again, where arr is not freed and the sixth element is set in a five-element array, demonstrates a classic heap buffer overflow:
#includeint main(void) { int* arr = malloc(5 * sizeof(int)); arr[5] = 42; return 0; }
When this is compiled with -fsanitize=address and executed, it will yield an error message directing you to an issue (a 4-byte write in main). The binary can be analyzed in a disassembler to pinpoint the exact instruction (at main+0x84) responsible for the error.
In a similar vein, here’s an instance where it detects a heap-use-after-free:
#includeint main(void) { int *arr = malloc(5 * sizeof(int)); free(arr); return arr[2]; }
This indicates a 4-byte read from freed memory at main+0x8c.
Memory
MemorySanitizer (MSan) detects uninitialized reads. Here’s a simple example that reads (and returns) an uninitialized value:
int main(void) { int data[2]; return data[0]; }
When it is compiled with -fsanitize=memory and executed, the following error message appears:
Undefined Behavior
UndefinedBehaviorSanitizer (UBSan) identifies undefined behavior, referring to situations where a program’s behavior becomes unpredictable and is not specified by the language standard. Common examples include accessing out-of-bounds memory, dereferencing invalid pointers, reading uninitialized variables, and overflowing signed integers. For instance, incrementing INT_MAX is considered undefined behavior.
#includeint main(void) { int a = INT_MAX; return a + 1; }
When compiled with -fsanitize=undefined and executed, the output provides specific details about the location and conditions of the issue:
Thread
ThreadSanitizer (TSan) detects data races, which can arise in multi-threaded applications when two or more threads attempt to access a common memory location simultaneously. This context introduces unpredictability and may lead to undefined behavior. The following example depicts two threads attempting to increment a global counter variable concurrently without any locking or semaphores in place.
#includeint counter = 0; void *increment(void *arg) { (void)arg; for (int i = 0; i < 1000000; i++) counter++; return NULL; } int main(void) { pthread_t thread1, thread2; pthread_create(&thread1, NULL, increment, NULL); pthread_create(&thread2, NULL, increment, NULL); pthread_join(thread1, NULL); pthread_join(thread2, NULL); return 0; }
When compiled with -fsanitize=thread and executed, it will generate an error message indicating a data race:
This message indicates there is a data race when the increment function is writing to the same 4 bytes simultaneously in two threads, specifically the memory associated with the variable counter.
Valgrind
Valgrind is a powerful instrumentation framework for developing dynamic analysis tools, primarily known for uncovering memory errors and leaks through its built-in Memcheck tool.
The following image illustrates the output from running c-kzg-4844’s tests with Valgrind, highlighting a valid finding regarding a “conditional jump or move [that] depends on uninitialized value(s)” within the red box.
This unearthed an edge case in expand_root_of_unity. If incorrect root unity or width values were supplied, the loop could conclude prior to the initialization of out[width]. In such cases, the final check would rely on an uninitialized value.
static C_KZG_RET expand_root_of_unity( fr_t *out, const fr_t *root, uint64_t width ) { out[0] = FR_ONE; out[1] = *root; for (uint64_t i = 2; !fr_is_one(&out[i - 1]); i++) { CHECK(i <= width); blst_fr_mul(&out[i], &out[i - 1], root); } CHECK(fr_is_one(&out[width])); return C_KZG_OK; }
Security Review
After the development phase concludes, thorough testing occurs, and the codebase undergoes multiple manual reviews by the team, seeking a security review from a credible security group is prudent. This does not equate to a guarantee of safety, but it indicates that your project has at least some level of security. Always bear in mind that perfect security is an illusion, and vulnerabilities can still exist.
For c-kzg-4844 and go-kzg-4844, the Ethereum Foundation engaged Sigma Prime to execute a security audit. They provided this report detailing eight findings. One critical vulnerability identified in go-kzg-4844 was particularly noteworthy: the BLS12-381 library employed by go-kzg-4844, gnark-crypto, contained a flaw that allowed invalid G1 and G2 points to be decoded successfully. If unaddressed, this could have led to a consensus issue (disagreement between implementations) in Ethereum.
Bug Bounty
If your project has exploit vulnerabilities, akin to Ethereum, implementing a bug bounty program is advisable. This enables security researchers, or indeed anyone, to report vulnerabilities in exchange for compensation. Typically, this program focuses on findings proving the possibility of an exploit. If the bug bounty payouts are within reason, individuals discovering bugs will usually alert you rather than exploit them or sell the information elsewhere. We suggest initiating your bug bounty program once the findings from the initial security review have been addressed; ideally, the cost of security review should be less than the bug bounty payouts.
Conclusion
Developing robust C projects, especially in the blockchain and cryptocurrency sector, requires a comprehensive approach. Due to the inherent vulnerabilities associated with the C language, employing a blend of best practices and tools is vital for crafting resilient software. We hope that the insights and experiences from our work with c-kzg-4844 will offer valuable guidance and best practices for others undertaking similar projects.
