Some noise to C1 JIT compiler for x86_64 (OpenJDK JVM – Hotspot)

Quarkslab folks proposed a challenge:  modify any compiler to generate code with some obfuscation but valid semantics. In other words, add some noise but keep the program logic valid.

Time to have some fun with C1: OpenJDK’s 1st tier JIT compiler.

Obfuscation “noise” can be added at different stages of the compilation process. Adding it early may be useful to target multiple architectures but it’s necessary to check if following optimization passes revert the effect.

The basic flow for C1 is the following: Java bytecodes -> IR (Intermediate Representation) -> LIR (Low-level Intermediate Representation) -> assembly code (architecture specific). A few key functions involved in these transformations: GraphBuilder::iterate_bytecodes_for_block (for IR generation), LIRGenerator::block_do (for LIR generation) and LIR_Assembler::emit_lir_list (for assembly generation).

This is my basic pseudo-random Java test code -don’t even try to make sense of it!-:

Function Main::f is called many times in a loop so it will be compiled by C1 first.

This is Main::f in bytecodes:

My obfuscation noise will be before any addition operation. In this case, before the iadd instruction.

This will be my x86_64 innocuous noise:

I’m saving the EFLAGS, RAX and RDX registers first because their values will be destroyed -restoring original values is mandatory to keep the state integrity after the obfuscation code-. Then I get the timestamp with RDTSC and compare the lower and higher halves. They will likely be different. In the unrealistic case that they are equal, we try again. Then we restore original values and continue execution as if nothing happened.

In x86_64, this is Main::f JIT compiled method:

I then decided to add some unaligned jumps as an anti-disassembly technique. Main::f now looks a bit more obscure:

Download patch here (based on JDK rev 89111a0e6355).

Leave a Reply

Your email address will not be published.