MACHINE LEARNING-GUIDED GREYBOX FUZZING FOR ADAPTIVE DISCOVERY OF DEEP-STATE SOFTWARE VULNERABILITIES

Abstract

Coverage-guided greybox fuzzing (CGF) is the de-facto choice for automated discovery of memory safety errors and security vulnerabilities in production software. On the other hand, classical CGF makes use of pseudo-random mutation operators, has a hard time to break multi-byte magic values, cryptographic checksums or nested structures and state machines that restrict the input space. The chance of meeting a complicated 32-bit or 64-bit predicate randomly is so low that vanilla fuzzers are found wanting at easily reaching deep into programs. In this survey, we review ML-guided greybox fuzzing as an adaptive, data-driven optimization process that replaces blind heuristics with program surrogates learned from historical mutation data. We propose a novel five-stage architectural taxonomy which aligns machine learning paradigms such as deep neural networks, contextual multi-armed bandits, reinforcement learning and LLMs with the phases of the fuzzing lifecycle: seed generation, seed scheduling and energy allocation, mutation and operator selection, exploration of deep-state paths, and crash triage post-fuzzing. We study state-of-the-art solutions in depth, including the variant adversarial bandit formulation of EcoFuzz, the contextual LinUCB scheduler of LinFuzz, the neural program smoothing with gradient-guided mutation of NEUZZ and MTFuzz, taint-based solutions like Angora and GreyOne, as well as generative fuzzying driven by LLMs such as Fuzz4All, TitanFuzz, ChatAFL, and StateLens. In addition, we evaluate benchmarking approaches, comparing traditional synthetic suites like LAVA-M to contemporary ground- truth platforms, for example FuzzBench, Magma and FIRMRE- B... Finally, some promising research directions for addressing the open challenges of model obsolescence and retraining overhead, scalability in multithreaded and monolithic systems, and overfitting and hallucination in LLM-guided fuzzers are discussed to enable future work toward adaptive, generalizable deep-state vulnerability discovery.