devirtualizing a toy VM obfuscator

The first time you hit a VM-protected binary it feels like the author cheated. There's no real control flow — just one enormous function that reads bytes from an array and switches on them forever. Your decompiler gives up. Your breakpoints land in the same handler every time.

Then it clicks: it's an interpreter. Someone wrote a tiny CPU in software and compiled the real logic down to bytecode for it. Reverse the CPU, and the bytecode is just another program waiting to be disassembled. This is the writeup of doing exactly that against a toy I built to teach myself the workflow.

recognizing the shape#

VM obfuscators all have the same skeleton, and once you've seen it you can't unsee it:

1
2
3
4
5
6

while (vm->running) {
    uint8_t op = vm->code[vm->pc];      // fetch
    vm->pc += 1;
    handler = dispatch_table[op];       // decode
    handler(vm);                        // execute
}

Three tells, every time: a program counter that only ever indexes one buffer, a dispatch table or giant switch, and a struct that's clearly a register file. Find those and you've found the machine. The "code" it runs is data sitting somewhere in the binary.

mapping the registers#

Before the opcodes mean anything, you need the VM's state layout. I rename fields in the disassembler as I confirm them — guesswork at first, then pinned down by watching the handlers use them.

1
2
3
4
5
6
7
8

offset   field        evidence
------   ----------   --------------------------------------------
0x00     pc           incremented every dispatch iteration
0x08     sp           moved by push/pop-looking handlers
0x10     stack[32]    sp indexes into here
0x90     regs[8]      handlers read/write by a 3-bit operand
0xB0     code*        the fetch reads code[pc]
0xB8     running      cleared by exactly one handler (that's HALT)

The handler that clears running is your HALT. The handlers that move sp are your stack ops. You're not guessing anymore — you're reading.

recovering the opcode table#

Now the grind: open each handler, figure out what it does, write it down. Most are two or three instructions. I keep a table as I go.

writing the disassembler#

Once the table is solid, the bytecode disassembler is fifty lines of Python. The point isn't elegance — it's turning an opaque blob into something you can read.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

OPS = {
    0x01: ("PUSH", 1),   # mnemonic, immediate byte count
    0x04: ("ADD",  0),
    0x05: ("XOR",  0),
    0x0A: ("STORE", 1),
    0x0B: ("LOAD",  1),
    0x10: ("JNZ",  0),
    0xFF: ("HALT", 0),
}

def disasm(code):
    pc = 0
    while pc < len(code):
        op = code[pc]; pc += 1
        name, imm = OPS.get(op, ("DB", 0))
        operand = code[pc:pc + imm]; pc += imm
        arg = f" {operand.hex()}" if imm else ""
        print(f"{pc - 1 - imm:04x}: {name}{arg}")

Run it over the code buffer you found at 0xB0 and the obfuscation evaporates. What was one unreadable interpreter loop becomes a linear listing of a stack machine doing — in my toy — a rolling XOR check against an embedded key. The "protection" was a layer of indirection, and you just removed it.

the takeaway#

VM obfuscation trades one hard problem (reading native code) for two easier ones (reverse the interpreter, then disassemble the bytecode). It looks scarier than packing or anti-debug, but it's more mechanical: there's a finite set of handlers, each one is small, and once you've named them all the program has nowhere left to hide.¹

Build your own VM obfuscator before you fight someone else's. Twenty handlers and an afternoon, and the genre stops being scary forever.