Memory in Computer Architecture

From tiny registers to massive hard disks: how computers organize memory and why speed always costs more than size.

Every computer must remember things. Your code. Your data. The screen you're looking at. But remembering comes with a cost: memory can be fast or cheap, rarely both. This tension shapes everything about how computers work.

The memory hierarchy

The answer: we don't pick one. We use many. Computers are built in layers, from tiny fast memories near the CPU to vast slow memories on disk.

Think of a library. The book you're reading right now? On your desk. Reference books you might need today? On the shelf. Archived books you might need someday? In the basement. As you move further down, storage gets cheaper but access takes longer.

Each level caches copies of data from the level below, making frequently accessed data available faster. Your CPU has a few kilobytes of registers, some megabytes of cache, gigabytes of RAM, and terabytes on disk.

Primary memory: RAM and ROM

Primary memory is what the CPU can directly address. This is where your program runs, where variables live, where the stack and heap exist.

RAM (Random Access Memory)

RAM is readable, writable, and volatile: turn off power and it all vanishes. Each byte has an address. Access any address and you get the data in roughly the same time (hence "random" access).

A 32-bit address space gives you 2^32 = 4 billion addresses, each holding 1 byte, for 4GB total. A 64-bit address space is astronomically larger.

How does RAM actually work?

RAM is typically built using capacitors, two metal plates with insulation between them. Apply a voltage and the plates store charge. Remove the voltage and the charge leaks away.

This is why DRAM must be "refreshed" constantly. Every ~64 milliseconds, the memory controller reads each bit and writes it back.

Static RAM

Cache memory uses a different technology: SRAM. Instead of capacitors, SRAM uses flip-flops (the same circuits that make CPU registers). This makes SRAM much faster but also much more expensive.

ROM (Read-Only Memory)

ROM is permanent. Turn off the power and it survives. Its contents are baked in at the factory and can't be changed (or at least, not easily). Your computer's boot firmware lives in ROM.

Modern machines use flash memory and EEPROMs, which can be rewritten. Your SSD is basically a huge, fast-ish flash memory.

Memory addressing

How do computers distinguish between different memory locations? They use addresses, just like houses on a street. But there's a quirk in how those addresses work.

Modern computers use byte addressing: each byte (8 bits) has its own address. When you store a multi-byte value like 0x12345678, which byte goes where? There are two conventions.

Big endian stores the most significant byte first. It reads naturally, like Western writing. Little endian stores the least significant byte first. It looks backwards, but arithmetic operations find the units place immediately at offset 0.

// Big endian: most significant byte first
// Address 0x1000: 0x12  0x1001: 0x34  0x1002: 0x56  0x1003: 0x78
 
// Little endian: least significant byte first
// Address 0x1000: 0x78  0x1001: 0x56  0x1002: 0x34  0x1003: 0x12

Intel chose little endian. Most networking protocols chose big endian. When two machines try to talk, they have to convert. Little endian won on desktops because Intel dominated, not because it's inherently better.

Caches: the middleman

Keep a small amount of fast memory near the CPU and store copies of frequently used data there. This is a cache.

The strategy hinges on locality. Programs don't access memory randomly. They reuse data (temporal locality), access nearby data (spatial locality), and repeat patterns (sequential locality). Caches exploit these patterns by keeping copies of recently accessed data nearby.

Cache organizations

There are three main ways to organize a cache, each making different tradeoffs.

Direct mapped maps each address to exactly one cache line. Simple and fast, but inflexible: two addresses that hash to the same slot constantly evict each other.

Fully associative lets any address go in any line. Flexible but expensive: checking every line takes hardware.

Set associative is the compromise: divide the cache into sets. Any address can go in any line within its set, but you only check one set, not the whole cache.

// Direct mapped: hash(address) determines the cache line
cache_line = hash(address) % cache_size
 
// Problem: address1 and address2 both hash to line 0
// They fight for the same slot in tight loops

How cache writes work

Reading from cache is straightforward: if it's there, great. If not, fetch it. Writing is trickier. When you store a value in cache, you also need to eventually update main memory.

Write-through writes to both cache and RAM on every store. Simple, always consistent, but every write waits for slow RAM.

Write-back writes to cache only and marks the line as "dirty". The data gets flushed to RAM later, when the cache needs space. Faster writes, but memory can be stale. This is why sudden power loss can corrupt data: unflushed writes never made it to disk.

// Write-through: always write to both
cache[address] = value;
mainMemory[address] = value;
 
// Write-back: write to cache only, flush later
cache[address] = value;
cache.markDirty(address);
// On eviction: if dirty, write back to mainMemory

Modern CPUs use write-back because the speed gains outweigh the complexity.

Secondary storage: disks

Everything below main RAM is secondary storage. It's not directly accessible by the CPU. To use it, the CPU has to load it into RAM first.

Hard Disk Drives (HDDs)

An HDD is a spinning magnetic platter with a read-write head on an arm. To access a specific bit: move the arm to the right track (seek time), wait for the platter to spin the right sector under the head (rotational latency), then read the data.

SSDs are faster because there's no spinning, no seeking, just electrical access. But they're still slow compared to RAM.

Memory performance in practice

Understanding the hierarchy helps explain real-world performance mysteries.

Cache lines

Caches don't store individual bytes. They store cache lines, typically 64 bytes. When you access one byte, the whole 64-byte line comes into cache. This is why sequential access is so efficient: you pay for the first byte, but get 63 free neighbors.

Sequential vs random access

Sequential access exploits cache lines. Adjacent elements share lines, so after the first miss, the next several accesses are free hits. Random access jumps around, wasting cache lines and forcing more memory fetches.

// Sequential: ~95% hit rate, ~1 second
for (int i = 0; i < 1000000; i++)
  sum += array[i];
 
// Random: ~10% hit rate, ~100 seconds
for (int i = 0; i < 1000000; i++)
  sum += array[random() % 1000000];

The same 1 million accesses, 100x slower. The only difference is the access pattern.

Prefetching

Modern CPUs notice patterns (like sequential access) and fetch the next cache line before you ask for it. If they predict correctly, you get a free hit. If they predict wrong, the prefetched data just takes up space.

Virtual memory

Modern operating systems let programs pretend they have infinite memory. The OS maps virtual addresses to physical RAM, and when RAM runs out, it swaps data to disk.

This is transparent to the program. You allocate memory, use it, and the OS figures out where it actually lives. Each program has its own virtual address space.

But if a program accesses memory that's on disk instead of RAM, the OS has to fetch it. This is a page fault. It takes millions of cycles. If your program page faults frequently, it becomes glacially slow.

Why this matters

Memory is a hierarchy because of physics: fast is small, slow is big, and you can't have both.

Your code has to navigate this hierarchy carefully. Keep working data small (in caches). Access it sequentially (to exploit spatial locality). Reuse it often (to exploit temporal locality). When you violate these rules, your program becomes orders of magnitude slower.