After releasing the x86obf tool for free I received quite a few requests for the source code. It was planned for the future, but I've decided to release it sooner.
The source code has been slightly stripped. Junk code generators are removed (they emit NOP only), data encryption is removed and so is bytecode encryption/obfuscation.
Virtualization handlers have been removed for some important and some not so important instructions. One of the important handlers were Jxx instructions which cannot be executed by the VM in its native form. In fact, every instruction that somehow changes execution flow (e.g. CALL, RET, JMP, Jxx, etc) must be handled manually.
I am releasing this as a learning material, not a ready to use compile-my-own-protector source. It will successfully virtualize and produce working binaries for short blocks of code which don't have conditional jumps. If you want to build a real protector on top of this, you will have to develop some code yourself.
x86obf was initially meant to be a commercial application, but it didn't work out. It had two versions, local and remote, which is why it's split into two projects and still has context data transferring code which is not needed in its current state.
If you have any questions, feel free to send me an email!
The public version of x86obf will continue to be developed, in fact it's being rewritten slightly to provide better functionality and also to compensate for this source code release - I am changing its internal VM design.
You're free to use this source code in free and commercial projects, but please do credit me.
Writing virtualization handlers is straightforward:
// ----------------------------------
// push*
// ----------------------------------
XL_DEF_INS_VIRT(push, 3)
if (no == 1) {
UINT32 size;
if (TOP0 == UD_OP_IMM) {
xl_virt_load_imm(vm, VR[0], &OP0);
size = 4;
} else if (TOP0 == UD_OP_REG) {
size = xl_virt_load_reg(vm, VR[0], ROP0);
} else if (TOP0 == UD_OP_MEM) {
size = xl_virt_load_x86_mem(vm, VR[0], &OP0);
I(RD, 0, 0, size, VR[0], VR[0]);
} ELSE_UNSUPPORTED_OPERAND
// sub esp, size
I(MOV, 0, 0, VR[1], NR(ESP), 0);
xl_virt_asgn_imm(vm, VR[2], -size, false);
I(ADD, 0, 0, 4, VR[1], VR[2]);
I(MOV, 0, 0, NR(ESP), VR[1], 0);
// write value to [esp]
I(WR, 0, 0, size, VR[1], VR[0]);
} ELSE_UNSUPPORTED_OPERAND
XL_END_INS_VIRT
DOWNLOAD: x86obf_source.zip