anti-debug bingo: five checks and how they fall

Anti-debug on Linux is mostly theatre. Almost every check reduces to "am I being ptraced," asked five different ways, and once you've seen each one it's a pattern match — spot the tell, patch one branch, move on. Here's the bingo card.

#	check	what it measures	the tell	cheapest defeat
1	`PTRACE_TRACEME`	is a tracer already attached	a `ptrace` syscall, ret in EAX	force the return value
2	`TracerPid`	the kernel's own bookkeeping	an `open("/proc/self/status")`	hook the read, or patch
3	`rdtsc` timing	wall-clock across a region	two `rdtsc` and a `sub`	don't single-step
4	breakpoint scan	`0xCC` bytes in `.text`	a self-checksum loop	hardware breakpoints
5	parent identity	who launched me	a `readlink` on `/proc/.../exe`	rename your debugger

The checks are independent, so they fold independently — open the one you need.

1 · ptrace(PTRACE_TRACEME)

A process can have exactly one tracer. The program calls TRACEME on itself first, so that when a debugger later tries to attach, the kernel refuses — or, run the other way, TRACEME returns -1 because the debugger got there first.

check 1c

if (ptrace(PTRACE_TRACEME, 0, 0, 0) < 0) {
    puts("nice try");
    _exit(1);
}

Defeat. In gdb, intercept the syscall and zero its result:

txt

catch syscall ptrace
commands
  set $rax = 0
  continue
end

Or kill it permanently — flip the js/jne that acts on the result to an unconditional fall-through:

patch the branchdiff

-    test eax, eax
-    js   fail            ; jump if ptrace returned < 0
+    test eax, eax
+    nop                  ; eax ignored
+    nop

2 · TracerPid in /proc/self/status

The kernel records the tracer's PID and exposes it; 0 means nobody's watching. No syscall to catch this time — just a file read and a parse.

check 2c

char *p = strstr(status_buf, "TracerPid:");
if (p && atoi(p + 10) != 0) _exit(1);

Defeat. LD_PRELOAD an open/read shim that rewrites the line, or — since you're already in a debugger — set a breakpoint after the atoi and clobber its return to 0. The file read is the tell; nothing else opens /proc/self/status in a hot path.

3 · rdtsc timing

Single-stepping is slow. The program reads the cycle counter, runs a short region, reads it again, and if the delta is enormous it assumes a human is stepping through.

check 3nasm

    rdtsc
    mov   ebx, eax        ; t0
    ; ... guarded region ...
    rdtsc
    sub   eax, ebx        ; delta
    cmp   eax, 0x100000   ; "too slow" threshold
    ja    fail

Defeat. Don't step the guarded region — set a breakpoint past it and continue, so the whole thing runs at native speed between two stops. Timing checks only catch you if you're stepping the instructions they bracket.

4 · breakpoint / 0xCC scan

A software breakpoint is the byte 0xCC (int3) patched over an instruction. The program defends by checksumming its own .text, or literally scanning for 0xCC, and bails if it finds one it didn't put there.

check 4c

for (uint8_t *p = text_start; p < text_end; p++)
    if (*p == 0xCC) _exit(1);

Defeat. Use hardware breakpoints (hbreak in gdb) — they live in the debug registers and never touch the instruction bytes, so a .text scan sees nothing. Four of them is usually enough.

5 · parent identity

The crudest one: walk up to the parent and see if it's a known debugger.

check 5c

char exe[256];
readlink("/proc/<ppid>/exe", exe, sizeof exe);
if (strstr(exe, "gdb")) _exit(1);

Defeat. It's a string compare against a filename — rename the binary, or launch the target from a neutral wrapper so the parent is sh, not gdb. Cheap check, cheaper bypass.

Run the card top to bottom and the target usually goes quiet:

ptrace self-attach — return value forced
TracerPid — read hooked
rdtsc — breakpoint moved past the region
0xCC scan — hardware breakpoints only
parent check — still renaming gdb to g, every single time¹

find them before they find you

Most of these announce themselves statically. ptrace, rdtsc, and a /proc/self/status string are all visible in a flat disassembly — grep the symbols and the constants first, and you'll know which cards are on the board before you ever hit a single trap.

There is no good reason I haven't just aliased this. The wrapper is three lines. I will do it next CTF. (I have said this for four CTFs.) ↩