In the context of malware analysis, the Reverse engineering Custom ciphers present a challenge: when an algorithm combines multiple layers of permutation, key mixing, and substitution, manual analysis slows down and quickly becomes more complex and inaccurate.
The new study from Malware Lab show how to overcome this limitation by using Z3, an SMT solver developed by Microsoft Research that, instead of manually inverting the cipher, models the function in symbolic form, leaving the solver to automatically reconstruct its inverse.
Based on a real case, starting from a Substitution-Permutation Network from KALMAR CTF 2024, this analysis shows how Z3's methodology is directly applicable also to the analysis of loader, ransomware e packer.
After introducing the operating principles of Z3, the study illustrates a complete solving script, delves into the topic of symbolic indexes and concludes with an account of the main difficulties encountered in its initial applications.
If you wish to learn more, here is the link to our comprehensive study.
In addition, you can subscribe to the specific mailing list Cyber Studios by T-Defence, to receive updates on upcoming research:
