DLL Injection and API Hooking

The goals of this lab:

• Understand the concept of DLL injection — running your own code inside another process's address space.
• Learn the classic CreateRemoteThread + LoadLibrary injection technique and recognize the Win32 APIs that enable it.
• Understand API hooking — both IAT hooks and inline (trampoline) hooks — and why malware and defenders both rely on it.
• Use x64dbg, Process Hacker, and Ghidra to observe, patch, and reason about a live 64-bit Windows process.

This lab modernizes the classic reverse-engineering exercise from the course's reference material. The concepts (injection and hooking) are unchanged and remain the heart of both offensive malware and defensive EDR products; only the platform and tools are brought up to date — Windows 10 (x64), x64dbg, and Ghidra instead of Windows XP and OllyDbg.

Background

Why inject a DLL?

A running Windows process has its own private virtual address space. If an attacker can force a victim process to load an attacker-controlled DLL, the attacker's code then runs with the identity and privileges of the victim process. This is attractive because it lets malware hide inside a trusted process (e.g., explorer.exe), bypass application allow-listing, hook APIs from the inside, and steal data the victim already has access to. The same mechanism is used legitimately by debuggers, overlays, and endpoint security agents.

The classic technique: CreateRemoteThread + LoadLibrary

The most-taught injection method uses only documented Win32 APIs. The injector performs these steps against a target process:

• OpenProcess — obtain a handle to the target with rights to allocate memory and create threads.
• VirtualAllocEx — allocate a small buffer inside the target to hold the full path of the DLL to load.
• WriteProcessMemory — copy the DLL path string into that buffer.
• GetProcAddress(GetModuleHandle("kernel32"), "LoadLibraryW") — find the address of LoadLibraryW. Because kernel32.dll loads at the same base in every process, this address is valid in the target too.
• CreateRemoteThread — start a new thread in the target whose start routine is LoadLibraryW and whose argument is the buffer holding the DLL path.
• The target's new thread executes LoadLibraryW("...payload.dll"); the loader maps the DLL and calls its DllMain, and the payload runs.

Other techniques you should be able to name (covered in lecture): SetWindowsHookEx, AppInit_DLLs (legacy), QueueUserAPC (APC injection), Reflective DLL injection (no LoadLibrary, no file on disk), Thread Execution Hijacking, and Process Hollowing (RunPE).

API hooking

API hooking means intercepting calls to a function so your code runs first, letting you observe or alter the call. Two common approaches:

• IAT hook. Each loaded module has an Import Address Table — an array of pointers the loader fills in with the real addresses of imported functions. If you overwrite the IAT entry for, say, MessageBoxW with the address of your handler, every call made through that module's IAT jumps to you instead. Simple, but it only affects one module and misses functions resolved dynamically via GetProcAddress.
• Inline (trampoline) hook. You overwrite the first few bytes of the target API itself with a jmp to your handler. To still be able to call the original, you save the overwritten prologue bytes into a small trampoline that executes them and then jumps back past the patch. This catches all callers, but is easier to detect (the API prologue no longer matches the on-disk file). Libraries such as Microsoft Detours and MinHook implement this correctly, handling instruction lengths and relocations.

MessageBoxW(HWND hWnd, LPCWSTR text, LPCWSTR caption, UINT type) hWnd → RCX text → RDX caption → R8 type → R9

Experiment Setup

• Log in to the Windows 10 analysis VM through the Guacamole web portal (link and credentials on the course website). All work happens in this VM.
• Copy the provided files from the shared Lab3 folder to your desktop: injector.exe, payload.dll, and hookdemo.exe (a small program that calls MessageBoxW). The reference source lab3_hookdemo.c is also provided for reading.
• Launch x64dbg (64-bit), Process Hacker, and Ghidra.
• Read the questions in each part below and record your answers, with screenshots, in your report.

Part A: DLL Injection (5 points)

• Start notepad.exe. In Process Hacker, find its PID.
• Run the injector against it: injector.exe notepad_PID payload.dll A message box should pop from inside Notepad reading ``Injected!! -- CSC 471 -- Si Chen''.
• In Process Hacker, open Notepad's Properties $\rightarrow$ Modules and confirm payload.dll is now loaded in Notepad's address space.
• Your task: change the message text from ``Injected!! -- CSC 471 -- Si Chen'' to "Hello World — Your Name". First do it dynamically: attach x64dbg to Notepad (or to payload.dll in a debuggee), locate the wide-character string in memory, and patch its bytes. As a stretch goal, patch the string on disk in the DLL and save a new copy, then re-inject to confirm.

Part B: API Hooking (5 points)

• Run hookdemo.exe and observe that it displays a message box via MessageBoxW.
• Attach x64dbg to hookdemo.exe. Set a breakpoint on MessageBoxW (Symbols tab → user32.dll → MessageBoxW, or bp MessageBoxW in the command bar).
• Trigger the message box so the breakpoint hits. Inspect the registers: under the Windows x64 convention the arguments arrive in RCX, RDX, R8, R9. Follow RDX in the dump to read the message text as a Unicode string.
• Your task: demonstrate a hook effect without recompiling the program. For example, at the breakpoint change the pointer in RDX to point at a different wide string you place in memory, or edit the string bytes in place, so the displayed text changes. Screenshot the altered message box.

Bonus (2 points)

Open hookdemo.exe in Ghidra. Statically locate the call site of MessageBoxW and the message string in the binary. Confirm that what you found statically matches what you observed dynamically in x64dbg (same string, and the call reached through the IAT). Include the relevant Ghidra decompiler/listing screenshot.

Hint

Review the lecture slides — Class 6 DLL Injection and Class 7 API Hooking. For the x64 calling convention and reading arguments in the debugger, revisit Class 2 and Class 3.

Deliverables

A detailed project report in PDF format documenting both parts, including: your answers to all questions; screenshots of the injected module in Process Hacker, the patched message boxes (Parts A and B), and the x64dbg register/memory views at the MessageBoxW breakpoint; and the address/bytes you patched. Also submit any modified files you produced.

Submission

• The lab due date is available on our course website. Late submissions will not be accepted.
• Submit your assignment directly to D2L.
• Your submission should include a detailed project report in PDF format describing what you have done, including screenshots and code snippets.
• No copy or cheating is tolerated. If your work is based on others' or AI, please give clear attribution. Otherwise, you WILL FAIL this course.