Design and implementation of control flow deobfuscation framework based on IDA microcode

The malware confused by O-LLVM has some analysis problems such as chaotic control flow and poor readability, which seriously affects the quality and efficiency of reverse analysis by security personnel. Therefore, O-LLVM deobfuscation has become one of the important research directions in the field of binary security research. In order to solve the problem that the existing O-LLVM deobfuscation framework is only suitable for standard obfuscator and supports a single instruction set architecture, a real block recognition algorithm and a multiplex block segmentation algorithm compatible with customized obfuscation are proposed. Then, by combing the previous two algorithms with the intermediate language IDA microcode, an deobfuscation framework BinDeob is designed and implemented. The adoption of intermediate languages brings the advantage of architecture independence, and so BinDeob supports multiple instruction set architectures such as ARM32, ARM64, x86, x64, etc. Experiments on C/C++ classic confusion benchmarks and public high-risk security vulnerability dataset show that the average similarity of the control flow chart between the BinDeob deobfuscation program and the original unobfuscated program is 98.9%, which has better performance than the deobfuscation framework in the references. In addition, pseudo-code similarity is introduced as an evaluation index, and the average pseudo-code similarity between the original program source code and the pseudo-code obtained by BinDeob's deobfuscation is 97.6%.