unpacking by hand — finding OEP and dumping

Open a packed binary in a disassembler and there's almost nothing there — a few hundred bytes of stub, a high-entropy blob, and an entry point that doesn't lead to any recognisable code. The real program only exists at runtime, after the stub decompresses it into memory and jumps to the original entry point (OEP). To analyse it statically, you have to be there at that jump and take a snapshot.

first, rule out the easy case#

Honest UPX round-trips:

1
2

$ upx -d packed -o unpacked          # works on unmodified UPX
$ strings packed | grep -i upx       # UPX!/version present? -> probably standard

When that fails — and on CTF binaries it usually does, because the UPX! magic and the header fields get scribbled over to break upx -d — you unpack the way that works on any packer, including custom ones: dynamically.

find the OEP: follow the stub to its tail jump#

The stub does its work, then transfers control to the freshly-unpacked code with a tail jump — a jump to an address far from the stub, often right after a register-restore. Recognising it is the whole skill:

1
2
3
4
5

...
pop  rdi
...
lea  rax, [rip - 0x...]
jmp  rax            ; <- tail jump into the unpacked .text (the OEP)

Break there and single-step into it; you land on what looks like a normal program entry (a _start/__libc_start_main setup). Generic ways to catch the moment:

• Hardware breakpoint on the unpacked region. Mark the rwx blob and break on execute — you stop the instant the stub jumps into decompressed code.
• Break after the big write loop. The decompressor is a tight copy loop; set a breakpoint just past it.
• Watch entropy / mprotect. Many packers mprotect the unpacked region executable just before the jump — break on that syscall.

1
2
3
4

(gdb) starti
(gdb) # let the stub run; break on execute of the unpacked mapping
(gdb) info proc mappings            # find the rwx blob's range
(gdb) # OEP = where execution enters that range after the tail jump

dump and rebuild#

At OEP, the real program is sitting decompressed in memory. Snapshot it:

1
2
3

(gdb) info proc mappings
(gdb) dump memory oep_dump.bin 0x<start> 0x<end>     # grab the unpacked text/data
# or core it: gcore <pid>, then carve sections out of the core

The dump runs, but a static tool still chokes until you fix two things up: set the ELF entry point to the OEP you found, and repair the imports — packers resolve the GOT/PLT at runtime, so a from-memory dump has the resolved pointers, which you re-point at proper relocation entries (the ELF equivalent of IAT reconstruction).

1
2
3

OEP = 0x401d20
[+] dumped 0x3000 bytes of unpacked .text
segfault{i_followed_the_tail_jump}

and the anti-unpacking it'll throw at you#

• Anti-debug in the stub — ptrace(PTRACE_TRACEME), timing checks, /proc self-inspection. Defeat them exactly as in anti-debug: LD_PRELOAD a fake ptrace, or patch the check.
• Self-checksums / re-packing — the stub may re-encrypt regions after use; dump at the exact OEP moment, before it cleans up.
• Stolen bytes — some packers move the first few OEP instructions into the stub and run them there, so the dumped OEP is missing its head; recover them from the stub and prepend.

Unpacking is the same dance regardless of the packer: let it do the work it was built to do, freeze the program the instant the real code exists, and walk away with the snapshot. upx -d is a convenience; the OEP-and-dump method is the one that never stops working.