CSC 471 · Lab 2

Goals

Read and reason about x86-64 (64-bit) assembly produced by a real compiler.
Understand the general-purpose CPU registers (RAX..R15) plus RIP, RSP, RBP, and RFLAGS.
Understand the function-call mechanism: the stack, the stack frame, the prologue and the epilogue.
Learn the System V AMD64 calling convention (Linux) and briefly contrast it with the Microsoft x64 convention (Windows).
Practice with GDB and Ghidra on the Badger CTF Linux server (and optionally x64dbg on the Windows 10 VM).

Introduction and Background

This lab modernizes our classic "stack frame" exercise. In earlier semesters we

poked at 32-bit programs in OllyDbg; today the malware you will meet in the wild is

overwhelmingly 64-bit, so we work directly in x86-64. The good news is that once you

understand registers, the stack, and the calling convention,

64-bit code reads just like a story: values move into registers, a function is called,

the stack frame is built, work happens, a result comes back in RAX, and the frame is

torn down.

Registers

x86-64 gives you sixteen 64-bit general-purpose registers. The first eight have

historical names; the rest are simply R8 through R15. Each register can be accessed at

smaller widths: for example the low 32 bits of RAX are called EAX, the low 16 bits AX,

and the low 8 bits AL. When a 32-bit result (an int) is written to EAX, the

upper 32 bits of RAX are automatically zeroed.

RIP always points at the next instruction to execute; branches and calls work by

changing RIP. RFLAGS holds condition bits such as the Zero Flag (ZF) and Sign Flag (SF)

that comparisons set and conditional jumps read.

The Stack Grows Down

The stack is a region of memory used for return addresses, saved registers,

and local variables. On x86-64 the stack grows toward lower addresses: pushing

a value decreases RSP, and popping increases it.

RSP (stack pointer) always points at the current top of the stack (the lowest in-use address).
RBP (base/frame pointer) points at a fixed anchor inside the current function's frame, so local variables and arguments can be addressed at constant offsets like [rbp - 4].

Stack Frame: Prologue and Epilogue

Each function typically builds a stack frame on entry and tears it down on exit.

A standard prologue looks like this (Intel syntax):

push rbp        ; save the caller's frame pointer
mov  rbp, rsp   ; establish our own frame pointer
sub  rsp, N     ; reserve N bytes for local variables

The matching epilogue restores everything and returns:

leave           ; equivalent to: mov rsp, rbp ; pop rbp
ret             ; pop the return address into RIP

The call instruction pushes the return address (the RIP of the instruction

following the call) onto the stack and jumps to the target. ret pops that

address back into RIP. leave is a compact way to undo push rbp; mov rbp,rsp.

Calling Convention (System V AMD64, Linux)

A calling convention is the contract for how arguments are passed and who is

responsible for preserving each register. On Linux (and macOS) the standard is the

System V AMD64 ABI. The first six integer/pointer arguments go in registers,

in this order:

Any further arguments are passed on the stack. The integer {return value comes

back in RAX} (or EAX for a 32-bit int).

Caller-saved vs.\ callee-saved. Some registers may be clobbered by a called

function, so the caller must save them if it still needs their values across a

call; these are caller-saved (volatile): RAX, RCX, RDX, RSI, RDI, R8—R11.

Other registers must be preserved by the callee (the function itself saves and

restores them); these are callee-saved: RBX, RBP, R12—R15, and RSP.

Contrast: Microsoft x64 (Windows). Windows uses a different order: the first

four integer arguments go in RCX, RDX, R8, R9, the return value is still in RAX,

and the caller must reserve 32 bytes of "shadow space" on the stack for the callee.

This is why the same C code disassembles differently on Linux and Windows. In this lab

we focus on the Linux/System V convention.

Experiment Setup

Connect to the Badger CTF Linux server over SSH: ssh <your_username>@badger.cs.wcupa.edu
Copy the provided lab2.c into your working directory (or paste it from the course site). Confirm it is there: ls -l lab2.c
Compile with debug symbols and no optimization so the assembly stays readable: gcc -g -O0 lab2.c -o lab2 (Optional: if you want to compare against the old 32-bit world, also build a 32-bit binary with gcc -g -O0 -m32 lab2.c -o lab2_32 and disassemble both.)
Launch the debugger: gdb ./lab2
Inside GDB, switch to Intel syntax (much closer to what Ghidra and most malware write-ups use): set disassemble-flavor intel

Lab Exercise

Work through the questions below. Use GDB to disassemble both functions:

disas main disas compute

For each answer, paste the relevant assembly lines and explain them in your own words.

Question 1 (1 pt): Look at the very beginning of main. Identify the exact instructions that build main's stack frame (the prologue). Which instruction saves the caller's frame pointer, which one establishes the new frame pointer, and which one reserves space for local variables? What value is subtracted from RSP, and why that number?

Question 2 (1 pt): Find where the local variables x, y, and z are initialized to 7, 3, and 10. How are these locals addressed —- relative to RBP or to RSP? Give the offset used for each variable (for example [rbp - 4]), and explain why they are at negative offsets from RBP.

Question 3 (1 pt): Immediately before the call compute instruction, how are the three arguments passed? Name the specific registers used for the first, second, and third arguments, and match each one to x, y, and z. Which calling convention does this confirm?

Question 4 (1 pt): In compute, the source computes a * 6. Look at the emitted assembly for that multiplication. With GCC at -O0 you will typically see a short sequence of lea/add/shl instructions (for example, a lea that computes 3 * a followed by an add that doubles it) rather than a single imul. Write out the exact instructions you see and explain why a compiler prefers lea/shift/add here. (If your compiler happened to emit imul instead, show that and explain the trade-off.) Also note how b * 2 is compiled.

Question 5 (1 pt): Which register holds compute's return value when it executes ret? Back in main, after the call compute returns, trace where that register's value is used —- which local variable (answer) does it get stored into, and how is it then passed to printf?

Question 6 (1 pt): Open the same lab2 binary in Ghidra (File > Import File, then let the auto-analyzer run) and view the decompiler output for compute and main. Compare Ghidra's reconstructed C to the real lab2.c source. What did the decompiler get right (control flow, argument count, the arithmetic)? What did it get wrong or represent differently (variable names, types, the a * 6 expression, temporary variables)? Include a screenshot of the decompiler window.

Bonus (2 pts): Watch the stack frame come alive at runtime. In GDB, set a breakpoint just after main's prologue and inspect the frame: break main run info registers rsp rbp rip x/16xg $rsp Step a few instructions (stepi) until the prologue has finished, then re-run info registers and x/16xg $rsp. Explain how RSP and RBP changed, and identify the saved return address and saved RBP on the stack. Include a screenshot of the frame right after the prologue.

Hints

Review Class 2: x86-64 Assembly Primer for the register set, Intel vs. AT&T syntax, and common instructions (mov, lea, add, shl, imul, call, ret).
Review Class 3: The Stack and Stack Frames for the prologue/epilogue pattern, how call/ret use the stack, and how locals are addressed off RBP.
In GDB, disas /r <func> also shows the raw opcode bytes; layout asm gives a live disassembly view. Use tbreak for a one-shot breakpoint.
Remember: on x86-64 the stack grows down, so sub rsp, N makes room and locals sit at negative offsets from RBP.
lea does address arithmetic without touching memory, which makes it a handy, fast way to compute scale * x + y in one instruction.

Deliverables

A single PDF report answering Questions 1—6 (and the Bonus, if attempted).
For each question, include the relevant assembly snippet (copied from GDB) and/or a screenshot, plus your explanation in your own words.
Include your Ghidra decompiler screenshot for Q6 and your GDB stack screenshot for the Bonus.
Attach or inline the lab2.c source and the exact gcc command you used.

Submission

Due date: See the course website for the exact due date and time. Late submissions are NOT accepted. Where: Submit your report on D2L as a single PDF containing your written answers, screenshots, and code snippets. Academic integrity: No copy or cheating is tolerated. If your work is based on others' work or on AI, give clear attribution. Otherwise, you WILL FAIL this course.