Before you start: the words
Primer · read once, come back when a term bitesThis page uses about twenty words that every systems programmer takes for granted and nobody else has a reason to know. Here they are, in the order they build on each other. If you know Python or JavaScript and have used a Linux shell, you have everything you need.
What a program actually is
- machine code, instruction
- A program on disk is not the source you wrote; it is a long list of instructions — tiny operations the chip performs directly: "add these two numbers", "copy 8 bytes from this address", "jump to that spot if the last result was zero". Each is a few bytes. Your Python code is turned into these by the interpreter as it runs; a compiled C or Go program is turned into them once, ahead of time, and the file is the instructions.
- CPU, x86-64, arm64
- The chip that executes instructions. Different chip families understand different instruction languages. Intel Macs (2006–2020) speak x86-64; Apple silicon (M1 and later) speaks arm64. They are as different as two human languages — an x86-64 program is gibberish to an arm64 chip. That single fact is the reason this project exists.
- register
- A CPU has a handful of named, ultra-fast storage cells — sixteen general ones on x86-64 (
rax,rdi,rsp…), thirty-one on arm64 (x0–x30). Nearly every instruction reads or writes registers; memory is where data lives when it is not being worked on. Think of them as the variables the chip can touch without a memory access. - flags
- A few bits the CPU sets after arithmetic — was the result zero, negative, did it overflow — that the next conditional jump reads. x86 sets six of them after almost every instruction; arm64 sets four and only when asked. Faking x86 flags cheaply on arm64 is a recurring theme.
How a program lives in memory
- process, thread
- A process is one running program with its own memory. A thread is one line of execution inside it; a process can have many threads working at once on the same memory. In Go terms, roughly a program versus its goroutines — except OS threads really do run in parallel.
- address, address space
- To a process, memory is one enormous numbered byte array, and an address is an index into it —
0x7ff8033d7000is just a big number naming one byte. Each process has its own private numbering, its address space; the same number means different bytes in different processes. - page
- The operating system hands out and protects memory in fixed-size chunks called pages. Intel Macs used 4 KB pages; Apple silicon uses 16 KB. A program built for 4 KB pages asks for things a 16 KB machine cannot literally do, which is a whole stage of this atlas.
- stack, heap
- Two regions of a process's memory. The stack grows and shrinks with function calls — each call pushes its return address and local variables, each return pops them. The heap is where
mallochands out long-lived memory. - fault, signal
- Touching an address you may not touch (unmapped, or read-only) makes the CPU stop the instruction and tell the operating system, which tells the process through a signal (
SIGSEGV,SIGBUS). Programs can catch signals. ocerz relies on this constantly: it deliberately protects memory so that a fault tells it something happened.
The operating system's side
- kernel, syscall
- The kernel is the core of the operating system: it owns files, memory, processes and devices. A program cannot touch those directly; it asks, using a special instruction called a system call (syscall) — "open this file", "give me 1 MB", "start a thread". On macOS there are two families: the Unix-style BSD calls, and Mach, an older message-passing interface where you send a message to a port and get a reply. Both appear in this tree.
- executable file, Mach-O
- The file format of a macOS program or library (Linux has ELF, Windows has PE). It holds the instructions, the data, and a table of load commands that tell the loader what it needs — including which libraries. A fat or universal file holds one copy per chip type, called slices; on macOS 27 most Apple-supplied files carry only an arm64 slice.
- library, linking, loader
- A library is code shared by many programs —
libSystemis macOS's C library plus threads, sockets and more. Linking is connecting a program's references ("callprintf") to the library's actual code. When that happens at run time, the program is dynamically linked and the piece of the operating system that does it is the loader — dyld on macOS,ld.soon Linux. On macOS, every real program is dynamically linked. - shared cache
- Apple pre-combines every system library into one enormous file at fixed addresses, so the loader maps one file instead of hundreds. The x86-64 version of it is what cache mode uses — and it ships as part of Rosetta.
- framework, Objective-C
- macOS's libraries above the C level — Foundation, AppKit, Metal — are called frameworks, and most are written in Objective-C: an object-oriented language whose method calls are messages looked up by name (a selector) at run time. Its runtime (
libobjc) keeps the tables of classes and methods, and it talks to the loader constantly. Newer frameworks are in Swift, which is the project's biggest coverage gap.
Running one machine's programs on another
- emulation, interpretation
- Reading the foreign program's instructions one at a time and doing, in your own code, what each one would have done. Simple, exact, and about thirty times slower than the real chip. ocerz's interpreter is this.
- translation, JIT
- Converting a run of foreign instructions into equivalent native instructions and then running those. Done ahead of time it is a compiler; done while the program runs, on demand, it is a JIT ("just in time"). ocerz's JIT translates x86-64 into arm64 a block at a time and caches the result.
- guest, host
- The words this whole page hangs on. The guest is the Intel program and everything it believes: its addresses, its registers, its idea of the kernel. The host is the real arm64 machine ocerz runs on. Every mechanism in ocerz is a place where guest and host disagree and something has to give — a guest address becomes a host address, a guest syscall becomes a host syscall with translated arguments, a guest register lives in a host register. Whenever a sentence here says "guest", read "what the program thinks"; "host", read "what is actually there".
- Rosetta 2
- Apple's own translator, built into macOS since 2020, and the thing ocerz stands beside and — once Apple removes it — in for.
- byte for byte, bit for bit
- Exactly equal output, compared as raw bytes. Used here about the tests: the interpreter and the JIT must produce identical bytes, not "the same answer".
Self-check: in the sentence "the guest asks for 4 KB at a guest address the host maps in 16 KB pages", which words describe belief and which describe reality?
"Guest asks", "4 KB", "guest address" are the program's beliefs — what an Intel Mac would have done. "Host maps in 16 KB pages" is reality. The mechanism that reconciles them is the slot table in mem.c (Stage 4).
What ocerz is
Stage 0 · orientationNew to words like instruction, syscall or guest? The primer before this stage defines each one in a paragraph.
An Intel Mac program is a Mach-O file full of x86-64 instructions that expects three things from the machine: a CPU that executes those instructions, a kernel that answers its system calls, and a loader that puts the system libraries it links against into its address space. On Apple silicon none of the three exist for it. ocerz fakes all three, in one process, so the program cannot tell.
If you know Linux: this is the job qemu-user does for foreign binaries (with binfmt_misc), and the job Apple's Rosetta 2 does on macOS. ocerz is the open one, and the one whose native mode is designed for the day Rosetta is gone.
Two axes you will see everywhere
Engine: interpreter or JIT
The interpreter executes one x86 instruction at a time in C, exactly. The JIT translates whole basic blocks into arm64 machine code, once, and runs that. Same program, same results, one is ~30× faster. -no-jit picks the interpreter; the differential gates prove the two agree byte for byte.
Mode: cache or native
Cache mode (the default) maps Apple's x86-64 dyld shared cache — one prelinked blob holding every Intel system library — and translates those libraries like any other x86 code. Native mode (-native) maps no cache at all: it synthesizes fake x86 system libraries whose every export is a doorway into the host's real arm64 framework. Cache mode needs Rosetta's files to exist on the Mac; native mode does not.
Think of the two engines as two ways to follow a recipe. The interpreter reads each line aloud and does it — slow, but it never skips a step, so it is the reference. The JIT reads the whole recipe once, writes it out in the cook's own shorthand, and from then on cooks from the shorthand. The differential gate is two cooks who must produce the identical dish from both versions.
Cache mode vs native mode is a hotel. An Intel program packs a list of what it needs from the hotel — libSystem, Foundation, AppKit. Cache mode is a hotel that still stocks the Intel versions of everything (Apple's shared cache). Native mode is a hotel that stocks only arm64 versions and puts an interpreter at every counter: the guest asks in x86, the interpreter relays it to the arm64 staff, and hands back the answer. Apple is about to stop stocking the Intel versions.
The shape of the tree
65,305 lines of C and assembly in 29 files, flat, one directory, a 60-line Makefile. Sizes matter: jit.c alone is 17,509 lines, a quarter of everything. That is a chosen cost, not drift.
| File | Lines | Fake | Owns (the file's own words, shortened) |
|---|---|---|---|
jit.c | 17509 | CPU | guest basic blocks translated to native arm64, the interpreter as fallback |
syscall.c | 7868 | kernel | the guest-to-host syscall boundary: x86-64 syscalls onto the arm64 kernel |
dyld.c | 5172 | loader | mini-dyld: loads, links and launches a dynamic executable against the shared cache |
decode.c | 4440 | CPU | x86-64 and i386 decoder: bytes in, one X86Insn out |
vm.c | 3632 | kernel | the master run loop and crash containment |
interp_sse.c / interp.c / interp_ext.c | 2536 / 1893 / 1197 | CPU | the reference interpreter: SSE/AVX, core integer, the long tail |
dyldapi.c | 2436 | loader | the dyld API surface libdyld's trampolines dispatch through |
objcbridge.c / objcclass.c | 2421 / 1357 | native | Objective-C messages crossed into the native runtime; guest-defined classes made native |
bridge.c / sysbridge.c / abi.c | 2402 / 1488 / 1744 | native | what a synthesized export does; the libSystem calls only ocerz can answer; the ABI crossing itself |
mem.c | 1867 | kernel | guest memory: mapping tables, 4 KB-in-16 KB slots, host protections |
vdylib.c / apidb.c | 1228 / 1108 | native | building the synthesized x86 libraries; reading the API database that describes them |
cache.c | 898 | loader | maps the x86-64 shared cache and resolves symbols out of it |
a64emit.c | 878 | CPU | arm64 encoder: one function per instruction, no scheduling, no allocation |
blocks.c · leaf.s · loader.c · flags_live.c · flags.c · main.c · stack.c · cpu.c | 909 · 598 · 378 · 283 · 230 · 280 · 137 · 163 | blocks both ways · in-place string routines · static Mach-O loading · flag def/use table · the bit-exact flag spec · CLI · the initial stack · CPU reset and dumps |
Self-check: which fake does mem.c belong to, and why isn't it part of the CPU?
The kernel boundary. The CPU fake only needs to load and store bytes at guest addresses; which host memory a guest address means, and what protection it has, is the operating system's business. mem.c is ocerz's virtual-memory manager: it decides how a guest address becomes a host address and keeps the 4 KB/16 KB bookkeeping the kernel will not do for it.
Self-check: what does "the same program gives byte-identical output under -no-jit and under the JIT" buy the project?
An oracle. The interpreter is simple enough to be right by inspection; the JIT is fast and complicated. Because every guest test must produce identical bytes under both, any JIT bug shows up as a difference, with the interpreter as the reference — that is what the differential gates in make check are.
The life of a process
Stage 1 · from./ocerz app to a windowThe best way to hold the whole system in your head is to follow one launch from the shell to the first frame on screen. Every stage after this one zooms into a step of this path.
What happens at each step
- main.c parses flags (
-native,-cache,-no-jit,-path), applies Wine defaults, and asksocerz_peek_dynamicone question: does this binary link libraries at all? A static binary (the guest test programs) goes toloader.cand a plain offset arena; everything real is dynamic and goes todyld.c. Since 2026-09-23 a binary with no x86-64 slice is told apart here — "it is not an Intel program" — instead of dying as unreadable. ocerz_dyld_runreserves the guest address space (identity mode: guest addresses are host addresses, Stage 4) and, in cache mode, maps the shared cache (cache.c). On macOS 27 that cache lives inside the Rosetta cryptex.- Loading the image.
select_slicepicks the x86-64 slice of a fat file; segments are copied into the arena; eachLC_LOAD_DYLIBis resolved — a hit in the cache costs nothing (the cache is already mapped), a dylib on disk becomes aDynImageing_dimgs. - Fixups. Chained fixups or classic binds: every pointer the linker left symbolic is written with a real address. Thread-local variables get their descriptors rewritten (two layouts exist; Stage 6).
- The initial stack is laid out exactly as XNU's exec path does it, because libSystem finds
apple[]by walking off the end ofenvp— a contract, not a convenience. - The initializer phase runs libSystem's initializer, then every library's
+loadand constructors in dependency order. This is the most invariant-laden code in the tree (Stage 6); getting the order wrong fails far away from the cause. ocerz_vm_callbuilds a guest frame and entersmain. From here the thread runs guest code and only comes back to C at syscalls, faults, and untranslated instructions.
If you know Linux: steps 2–6 are what ld.so does for a dynamic ELF binary, and step 5 is what execve does before it. ocerz has to be both, because the real dyld is an x86 program too and there is nobody to run it.
Picture the loader as a stage crew before a play. Step 3 carries the set pieces onto the stage (segments into memory). Steps 3–4 fetch the props the script names from the prop house (libraries) and tape the cue sheet to every entrance so each actor knows where to go (fixups: rebase = "we're on stage B tonight, shift every mark", bind = "the prop the script calls sword is on that table"). Step 5 lays out the programme exactly as the audience expects it (the initial stack). Step 6 is the warm-up, in a strict order — the orchestra tunes before the singers rehearse. Only then does the curtain rise (step 7, main).
Static vs dynamic is packing for a trip. Static: you packed everything, the suitcase is heavy, you need nothing from the hotel — that is a Go binary, and it is ocerz's guest test programs. Dynamic: you packed a list and are counting on the hotel to have a hair dryer — that is a Python script and its imports, and it is every real macOS program.
Where to look
src/main.c— 280 lines, read all of it; it is the table of contents of a launch.src/dyld.cprose block, thenocerz_dyld_runtop to bottom; skim the calls, not their bodies.src/stack.c— 137 lines; the contract is the comment block.
main runs, ocerz has already executed a great deal of Intel code — every library initializer — so the loader and the CPU are not phases, they are two fakes working at once.Self-check: a program prints nothing and dies with an unrecognized selector inside Foundation before its own main. Which step is suspect?
Step 6, the initializer order. An unrecognized selector or a "static initializer ran before its library" message is the classic fingerprint of an initializer running before something it depends on — see the five invariants in Stage 6. The bytes and the syscalls are probably fine.
Executing x86: decode, interpret, translate
Stage 2 · the CPU fakeThree layers, and the contract between them is one struct.
If you know Python: decode.c is the parser producing an AST node; interp.c is CPython's eval loop, a big switch that does exactly what each node says; jit.c is PyPy — it compiles hot straight-line runs to machine code and keeps them in a cache keyed by where they came from.
The interpreter (the oracle)
One function, ocerz_interp_step, executes one instruction against an OcerzCPU — the struct holding all sixteen general registers, the flags, the xmm registers, the instruction pointer (cpu.h; its field order is load-bearing because the JIT reaches into it at fixed offsets). It is slow and it is right. flags.c is the bit-exact specification of the x86 flags; the JIT's cleverer flag handling must agree with it, and the differential gates check that it does.
The JIT (the engine)
A block is a straight run of instructions ending in a branch, call or return. The JIT translates a block into arm64 words appended to a 1 GB MAP_JIT arena, records it in a hash keyed by guest address, and jumps into it. The next time the guest reaches that address, no translation happens. The tricks that make the translated code fast are Stage 3's subject; three ideas belong here:
- Register pinning. In pin class 3 (the default) every one of the sixteen x86 registers lives permanently in an arm64 register —
rdiinx28,rsiinx27, and so on — andxmm0–15live inv16–v31. Nothing is loaded or stored just to execute an instruction. Because every block shares this one layout, blocks can jump directly into each other's bodies. - Analogy: a chef's mise en place. Every ingredient is already in a bowl within arm's reach, so no step of the recipe begins with a walk to the pantry. And because every station in the kitchen is laid out the same way, a cook can move from one station to the next mid-recipe without looking.
- Lazy flags. x86 sets six flags after almost every instruction; computing them costs more than the instruction.
flags_live.cknows which flags each instruction reads, so the JIT computes only what a later instruction will consume, and fusescmp+jccinto one arm64 compare-and-branch. - Analogy: a restaurant that only totals the bill when someone asks for it. Every dish "sets the flags" — but if nobody looks at the total before the next round, why add it up? The JIT keeps the receipts (a deferred-flags record) only when it cannot prove the next block will never ask.
- Fault recovery. When translated code faults (a guest bug, or a page ocerz deliberately protected), the fault happens at an arm64 address, but the guest must see exact x86 state at an x86 address.
vm.c'scrash_handlerand the JIT's recovery recipes reconstruct it. This is the subtle part of the whole tree. - Analogy: a flight recorder. The plane (translated arm64) crashed at some GPS point; the report must say exactly which line of the flight plan (x86 instruction) it was on and what every gauge read. The recipes are how the recorder reconstructs the gauges from the wreckage.
X86Insn → either the exact interpreter or the fast JIT; the two must agree byte for byte, and that agreement is tested, not assumed.Self-check: you suspect the JIT mistranslates one instruction. What is the cheapest experiment, before reading any code?
Run the program with -no-jit. If the symptom vanishes, it is the JIT; if it persists, the interpreter has it too and the bug is in the decoder, memory, syscalls or the loader — anything but translation. Then bisect the JIT with its OCERZ_NO_* switches (Stage 9).
The run loop: blocks, chains, trips
Stage 3 · where the time goesOnce main starts, a thread lives inside translated blocks and comes out only when it must. Coming out is called a trip, and most of the performance work is about avoiding trips or making them cheap.
The mechanisms, each with the problem it solves
Chaining
A block's exit is a b patched to the successor's body once that exists. arm64's b reaches ±128 MB; the arena is 1 GB, so a late-translated caller could not reach an early callee and the edge was silently dropped. Since 2026-09-22 a veneer pool every 64 MB (ldr x15, literal; br x15) makes every chain activate.
Analogy: connecting doors between hotel rooms, so you walk room to room without going down to the lobby (the dispatcher) each time. A veneer is a relay runner for a door that is too far to reach in one stride: you hand the message to someone standing halfway, who carries it the rest of the way.
Return-address stack (RAS)
A guest ret needs the arm64 address of the block after the call. Calls push {guest return, host continuation} on the host machine stack and use real bl/ret, so the hardware predictor works. Every trip resets the stack and loses the entries; a miss used to exit through the epilogue and cascade. Now a miss resolves in-body through the same per-site cache indirect calls use.
Analogy: a stack of bookmarks. Every time you open a chapter you drop a bookmark so you can flip straight back. A trip is someone bumping the table: all the bookmarks fall off, and until you have re-placed them every "go back" means searching the index. The fix stopped each search from also bumping the table again.
Per-site caches (PSC / IC)
An indirect jmp/call keeps a small {target, body} cache at the site, then an inline hash probe, before ever tripping. In the game profile these missed zero times.
Analogy: speed-dial on a phone. An indirect call is "dial whoever is in this variable"; the per-site cache remembers the last few numbers this exact phone dialed, so the common case never opens the phone book (the hash), let alone calls the operator (the dispatcher).
Superblocks and flipping
Translation continues past a forward conditional branch, up to six side exits. If the taken side turns out hot, the block is retired and retranslated following that edge. Decisions are per branch, counted in the arena.
Analogy: a road built along the route the planner assumed people take. Superblocks keep paving past the fork. Flipping is noticing, from the counters, that traffic actually turns left at that fork, and rebuilding the road so left is the straight-through direction.
Invalidation
Programs write code (JITs inside the guest, like Adobe AIR's ActionScript engine). Pages holding translations are "armed"; a write faults, the blocks are retired, and regions that thrash are blacklisted to run interpreted.
Analogy: cooking from a recipe you memorized while someone keeps editing the recipe card. Arming the page is putting a bell on the card: any edit rings it, you throw away what you memorized and re-read. A card edited every minute is not worth memorizing — read it aloud each time instead (interpret).
In-place leaf routines
strlen, memcpy and friends are called constantly and do almost nothing. leaf.s holds arm64 versions under a private register contract, reached with one branch; a fault inside declines and the x86 code takes the same fault, so handlers see what they would have seen.
Analogy: a vending machine beside the kitchen door for the ten things people ask for a hundred times a minute. If the machine jams (a fault), it hands you nothing and you walk into the kitchen and order the normal way, so the kitchen sees the same order it always would have.
If you know Go: a trip is a goroutine switch that also throws away the CPU's branch history. Chains are direct calls the compiler resolved; a RAS miss or a dropped chain is an interface call through a lookup table where a direct call would do.
The numbers from 2026-09-22, as a worked example
Brawlhalla's main thread was pegged at 100.8% of a core. The attributed profile said: 42% in instruction bodies, 26% in prologues and epilogues (trips), 11% jcc, 8% returns. Three fixes, in order: return misses no longer exit through the epilogue (+33–48% throughput); veneers made every chain activate (prologue+epilogue fell to 9%, the core dropped to 71% while doing 60% more work); constant-address memory accesses lost a 7-word guard (+13% work per second). Each one was found by measurement, and each one is a mechanism in this stage.
Self-check: PERFSTAT reports jit_step/cache_lookup = 1.2 M/s. Is that good or bad, and what would you compare it with?
It is the rate of C-level dispatches — trips that reached the slowest path. Bad in absolute terms if the code should be chained; but compare it with block_execs: 1.2 M trips against 460 M block executions a second is one trip per ~380 blocks, which is fine. The ratio is the number, not the count.
Memory: two address spaces
Stage 4 ·mem.c and the guards in jit.cThe guest has addresses; the host has addresses; they are not the same space. Everything about memory in this tree follows from how one maps to the other.
jgb_usable(), mem_guard_needed() and stack_identity().The address map is a hotel's room numbers versus the building's floor plan. The guest knows rooms by number (guest addresses); the building has physical positions (host addresses). Identity mode: the room number is the position — no lookup at all. Offset mode: every room is on a floor that starts at some fixed offset — one addition. Low shadow: two wings with their own numbering, and every visit first checks which wing you meant — the expensive one. The arena is renting the whole floor up front so nobody else's luggage ever ends up in the guest's rooms.
The 4 KB / 16 KB problem
Intel Macs used 4 KB pages; Apple silicon uses 16 KB. A guest may protect, map or unmap a 4 KB range that is a quarter of a host page. mem.c therefore tracks slots — 4 KB pieces inside each 16 KB host page — with their own owner and protection, and the host page's real protection is the union of its four slots. Consequences: guest protection changes resolve on 16 KB boundaries; a file is mapped directly only when its offset and guest address agree modulo 16 KB; and any path that hands a guest address straight to the kernel (a forwarded Mach vm_map) can destroy the neighbouring 12 KB unless the fragments are saved and restored around the call. That last one cost Safari a Swift precondition failure nineteen million instructions after the damage.
Slots: one key per room, but the locks are per floor. The guest wants to lock a single room (a 4 KB page); the building can only lock a whole floor of four rooms (a 16 KB host page). So mem.c keeps its own ledger of which of the four rooms are meant to be locked and sets the floor lock to the strictest of them — and any request that bypasses the ledger and talks to the building directly can lock, or wipe, the neighbours by accident.
Two guards the JIT emits around a memory access
- The commpage guard. The x86 commpage — a kernel-provided page of timers and feature bits at
0x7fffffe00000— cannot exist there on an arm64 kernel, so ocerz keeps its own and redirects accesses to it. Every access whose address is not provably a stack slot used to compare against that range at run time (7 words). For a rip-relative or absolute operand the address is a constant, so since 2026-09-22 that is decided at translation time. - Analogy for the commpage guard: mail forwarding for one address that moved. Every letter used to be checked against the forwarding card at the post office. If the address is printed on the envelope (a constant), you can check it once, when the envelope is written.
- Ordered mode. An x86 CPU promises stronger memory ordering (TSO) than arm64. A single-threaded program runs in plain mode; the first thread, fork or shared mapping switches the process to ordered mode for good, where scalar loads and stores use acquire/release forms — and those fault on an access crossing a 16-byte granule, hence a small alignment test before each. Measured cost on the game: about 5%.
- Analogy for ordered mode: a shared kitchen. With one cook, put dishes down wherever, whenever. The moment a second cook walks in, a house rule applies to everyone for the rest of the night: tap the counter before you set anything down, so nobody grabs a half-plated dish. It slows every cook a little, and you cannot revoke a rule some cooks have already stopped following.
If you know Linux: mem.c is a tiny mm/: it keeps the VMA-like region table and decides what to mmap/mprotect on the host. The slot table is what you would need if the kernel's page size were four times your process's.
Self-check: why does a program that creates a thread get slightly slower on every load and store for the rest of its life?
Because ordering only matters when another thread can observe it. With one thread, plain loads and stores are correct; with two, x86 programs rely on TSO, so ocerz switches the process to ordered mode and emits acquire/release accesses (with their granule test) from then on. It never switches back, because a block translated in plain mode would be wrong once a second thread exists.
The kernel boundary
Stage 5 ·syscall.c, 7,868 linesWhen guest code executes syscall, the JIT trips to ocerz_handle_syscall. Three classes arrive: BSD calls (class 2: files, sockets, memory, processes), Mach traps (class 1: ports, messages, threads), and machdep. Most are forwarded to the real kernel — but not blindly.
The syscall table is a switchboard operator with an index card per extension. A card says how many arguments the call takes and which of them are addresses the guest wrote down in its room numbers, so the operator can translate them to the building's before putting the call through (passthrough). Some cards say "the concierge answers this one himself" (intercept) — because the building's honest answer would be wrong for this guest, as with vm_region. And a Mach message is a letter the operator opens on the way back, rewriting the reply into the guest's room numbers before handing it over.
Spawning a child is hiring an interpreter for the new arrival — unless the new arrival already speaks arm64, in which case sending an interpreter along would only get in the way. That is the rule that landed on 2026-09-23.
What the intercepts have to fake
- Memory calls (
mmap,mprotect,shmat) go throughmem.c, because the host kernel's view of ocerz's arena is not the guest's view. A Machvm_regionreply is rewritten to say what the guest declared, not what the host has — Electron checks that and executesint3if__DATA_CONSTis not read-only, which is why Discord's renderers died until 2026-09-22. - Processes.
posix_spawnandexecveof an Intel binary rewrite the child toocerz -path child -- argv…with the mode and bases inherited through the environment. A child with no x86-64 slice runs natively, as Rosetta would run it. - Signals are delivered at guest instruction boundaries with an exact x86 frame; the sigaction struct has different in and out layouts (a real bug once).
- Threads and workqueues. Each guest thread is a host thread running its own
OcerzCPU. libdispatch's workqueue threads are bridged to the host's — without that, the firstdispatch_syncin any Cocoa app parked its main thread forever. - Identity.
proc_pidpathof the own pid answers the program, notocerz;sysctl.proc_translatedanswers 1, so programs take the same compatibility paths they take under Rosetta.
If you know Linux: read this file the way you read strace output. OCERZ_SYSFAIL=1 prints the failures a healthy program almost never earns (ENOMEM, EFAULT), and -strace prints everything.
Self-check: the guest calls a syscall ocerz has never seen. What happens, and why not abort?
It logs once and returns ENOSYS. Aborting kills the guest thread where it stands, and under Wine that is often inside the loader with the loader lock held, deadlocking every other thread. OCERZ_STRICT_SYSCALL restores the abort for bring-up work.
Cache mode: the loader and the dyld shim
Stage 6 ·dyld.c, cache.c, dyldapi.cApple ships every Intel system library prelinked into one file, the dyld shared cache, with every library at a fixed address inside it. ocerz maps that file and the libraries simply exist. What it must still do is everything the real dyld would have done around them — and answer the questions the libraries ask dyld at run time.
The shared cache is a library whose books were all pre-shelved on one floor, in fixed positions. Cache mode does not fetch books one at a time; it opens the floor and every book is already where its catalogue number says. dyldapi.c is the librarian's desk — the books themselves walk up and ask "how many books are on the floor?", "which book is this page from?", "tell me when a new one arrives". A librarian who forgets to list a book that is plainly on the shelf is exactly the GPU bug: Metal asked for the driver's book and was told there was none.
The initializer phase and its five invariants
After libSystem's initializer, run_init_phase walks the dependency graph bottom-up and, per image, delivers the Objective-C load notification and runs the constructors. Each rule below was learned from a real failure, and each failure surfaced far from its cause:
- Upward links are not ordering edges — they declare the back edge of a cycle. Follow them for membership, never for order (and still initialize them, at the end).
- Prune on already-done images, or libc++abi's walk descends through libxpc into Foundation.
- Load notifications and initializers interleave per image; two global phases put the wrong things first and blew the stack inside malloc.
- The eager set is closed under dependencies.
- Decode the new
dylib_use_commandflags, where upward and delay-init live now; reading only the command type pulled 532 images into CoreFoundation's subtree.
The initializer order is seating a dinner party where some guests must be seated before others. Dependencies sit first. An upward link is a guest saying "I know her, but do not make me wait for her" — you follow the acquaintance to decide who is invited, never to decide who sits first. Interleaving means each guest is introduced (+load) and then seated (initializer) before the next, not everyone introduced and then everyone seated. Every one of the five rules was learned from a party that went wrong.
Thread-locals and Objective-C
Thread-local variables have two descriptor layouts (classic on disk, packed in the cache); reading the wrong one aliased every thread-local in an image. Objective-C needs categories attached to classes in late-loaded images, and — the 2026-09-22 lesson — an image dlopened out of the cache must join the image list after libobjc has mapped it, or libobjc treats it as already handled and drops its categories.
Analogy for TLV: a coat check ticket per guest — every thread gets its own coat, and the ticket says where it hangs. Two ticket formats exist; read a packed ticket as a classic one and every guest is handed the same coat.
If you know Linux: the shared cache is prelink taken to its conclusion; dyldapi.c is the part of ld.so that answers dl_iterate_phdr, dladdr and the loading hooks — except here the libraries were built to talk to Apple's dyld through an object, and ocerz has to be that object.
Self-check: _dyld_image_count() returns 45 under ocerz and 348 under Rosetta after the same dlopen. Name the class of bug and what it breaks.
A dyld-API answer that is incomplete: the library is loaded, its symbols resolve, but it is absent from the list. It breaks anything that walks the list — libobjc's per-image class queries, CFBundle, plug-in hosts. That exact gap made Metal report no GPU device, which took OpenGL's accelerated renderer with it.
Native mode: the bridge
Stage 7 ·-native, the future defaultNative mode maps no shared cache. The program still links /usr/lib/libSystem.B.dylib, Foundation, AppKit — so ocerz builds each of them, on demand, as a real x86-64 Mach-O in memory whose every export is twelve bytes: load my id, jump to the bridge. The program's own code runs through the same decoder and JIT as always; only the universe it binds against has changed.
Native mode is an embassy. The guest walks up to what looks like the Intel library's front desk (the synthesized library: a facade, every door leads to a booth). Behind each door sits an interpreter with a phrasebook (the API database: how this exact request translates), who rearranges the sentence into the other language's grammar (abi.c: registers to a frame, structs reshaped) and relays it to the real arm64 official (the host framework). A "stub" is a door with a sign reading not yet staffed. Callbacks are the guest leaving a phone number: the official cannot speak x86, so the embassy keeps 65,536 numbers that all ring an interpreter who fetches the guest.
The parts
vdylib.c
Builds the fake library as a real Mach-O with a real export trie (a prefix trie — a flat one made _writev consume the _write edge). The loader takes it as an ordinary image; the decoder and JIT were never touched.
apidb.c + the .api files
Fifty-nine libraries have a database. A record says how an export crosses: a plain function with a signature, a native variable, a variable ocerz fills itself (page size, stack bounds — answers the guest must not see as they are), a function ocerz answers itself, a stub that stops with a named message.
abi.c + abicall.s
The ABIs differ: System V passes variadic doubles in registers, Apple arm64 puts every variadic argument on the stack; small structs are classified by eightbyte on one side and by float count on the other. Signatures drive the move in both directions.
objcbridge.c + objcclass.c
objc_msgSend from the guest is answered by asking the native runtime for the method's type encoding and crossing under that signature. Classes the guest defines are built in the native runtime as themselves, so AppKit can draw a guest NSView subclass.
sysbridge.c
The libSystem calls a generic crossing cannot make: variadic file calls, memory mapping, non-local jumps, process and thread state the guest must own.
Why it exists
Rosetta is scheduled to leave; macOS 27's bundled apps already ship no Intel code and the x86 cache lives inside Rosetta's cryptex. Native mode is the path that survives that. Steam's UI ran in native mode on 2026-09-21.
If you know Python or Go: this is an FFI — ctypes, or cgo — built at the scale of an entire operating system's frameworks, with the bindings written as data files instead of code, and with the extra twist that the "foreign" side is the process's own frameworks.
Self-check: a program in native mode exits with code 72 and names one symbol. What happened, and how is it different from exit 71?
72: everything bound and the program reached main, then called an export whose record is a stub — the bridge behind that one function is missing. 71: nothing bound because an import had no record at all, and the guest never ran. They are kept distinct on purpose so a log says which kind of gap to fill.
How it stays correct
Stage 8 · the gates and the rulesThe tree is kept honest by one command and a few rules. make check runs the gates in order and takes about twenty minutes; nothing is committed until it is green.
| Gate | What it proves |
|---|---|
make unit | the C-level harnesses in tests/unit, one per subsystem, driving functions directly |
| guest tests, interpreted | ~100 freestanding x86 programs against golden outputs, under -no-jit |
| guest tests, JIT | the same under the JIT |
| x86-64 differential | every guest binary: interpreter output and JIT output must match byte for byte |
| i386 differential | 32-bit sequences run interpreted and translated, comparing every register, flag and touched byte |
| dynamic tests | real dylib-linked programs against the shared cache — where the compatibility fixes get their tests |
| native tests | the bridge: C, Objective-C, Cocoa fixtures in native mode, most compared against an arm64 build of the same source |
The gates are two accountants who must produce the same totals. One (the interpreter) works slowly with a pencil and a printed rulebook; the other (the JIT) uses every shortcut she knows. The goldens are the answer key from a real Intel machine. If the pencil matches the key and the shortcuts match the pencil, the shortcuts are right — for every ledger you gave them. A ledger you never gave them proves nothing, which is why real programs keep being added.
If you know git: "gate before every commit" is a pre-commit hook that takes twenty minutes; "a test that pins every change" is refusing to merge a fix without the regression test in the same commit; and the commit subject stating the rule is writing the message for the person running git blame in a year.
The rules, and why each one exists
- Every behavioural change gets a test that pins it, named in the commit message. A fix without a test is a fix that can silently return.
- Gate before every commit, then commit code and tests, then a separate
tests: rebuild unit binariescommit, because the test binaries are tracked. - No inline comments; one prose block per file, present tense. The why goes in the commit message and the file's prose. For a reader without C, that prose block is the whole point: each file explains itself in paragraphs before the first line of code.
- Commit subjects state the rule, not the activity: "jit: a chain that cannot reach goes through a veneer instead of being dropped" tells you the new law of the system.
- The README must stay wholly true. A row measured on an older macOS says so.
Ways a gate can lie
macOS has no timeout, so a sweep that used it "passed" everything; a benchmark binary in the wrong directory becomes a slow extra test case; a stale object file turns the i386 gate into a measurement of an older tree; the dynamic suite flakes under load, so nothing else runs while a gate runs. Each of these was found by being burned.
Self-check: a change passes the whole gate, and Photoshop still crashes. What does that tell you about the gate?
That the gate proves what it tests and nothing more. It covers ocerz's own programs and a set of real dynamic ones; it does not run Photoshop. The right response is the one used for every compatibility fix this month: reproduce the failure in the smallest program that shows it, add that program to the dynamic tests, then fix it.
Reading a debugging session
Stage 9 · the knobs and the numbers I quote at youAbout 290 OCERZ_* environment variables exist. They are not ad hoc; there are three families, and they are how this codebase is debugged.
1 · OCERZ_NO_<FEATURE>
One switch per optimization: NO_CHAIN, NO_RAS, NO_HOST_RAS, NO_FULLPIN, NO_SUPERBLOCK, NO_LAZYFLAGS, NO_LEAF_INPLACE, NO_UPWARD_INIT… The workflow: a wrong result under the JIT is bisected by turning transforms off one at a time until it goes away. Yesterday NO_HOST_RAS=1 was the experiment that proved the return-stack hypothesis.
2 · OCERZ_<AREA>LOG / STAT / DUMP
Diagnostics: DLOPENLOG (every dlopen and its answer), SYSFAIL (syscall failures a healthy program rarely earns), MACHMSG, OBJCLOG, PERFSTAT (the JIT's report at exit), JITDIS (+_BRIEF: dump every translated block), ALLMISS, IMGLOG.
3 · range selectors
INTERP_LO/HI force an address range to run interpreted; NOJIT_EXE=<substr> interprets one process out of Steam's nineteen; JIT_CODE_KB shrinks the arena to force the fallback paths.
Read once, at first use
Nearly all are cached in a static on first read, so changing one mid-process does nothing, and one that guards a translation decision must be set before that code is translated.
The NO_* switches are git bisect over features instead of commits. Turn one optimization off; if the symptom vanishes, the bug is inside that optimization. PERFSTAT is the car's dashboard — it will not tell you why the engine is loud, but it tells you the RPM, the temperature and the gear, and the gear is usually the answer. sample is walking the factory floor: count only the people with tools in their hands, not the ones standing around holding a clipboard (parked threads). kill -INFO two hundred times is asking every worker "what are you doing right now?" every tenth of a second.
PERFSTAT, line by line
PERFSTAT t=60.1s blocks=372106 static_insns/blk=8.51 block_execs=27796591150 (462166628/s) jit_step/cache_lookup=87149024 (1449004/s) avg EXECUTED insns per block = 11.89 insns/s = 5496038395 HOTBLOCK # 1 rip=0x8027f67d10 execs=255960085 insns=20 words=209 … SLOWOP # 1 imul 3975959 23.23% of slow (0.01% of ALL) RAS misses=883386978 (stale=883385785) align-hotpatches=2832 ras_slots=0/262144 CHAIN activated=635725 veneered=40697 out_of_range=0 (0.00% dropped)
- blocks translated so far; static_insns/blk their average length.
- block_execs/s how many blocks ran per second — the "work" rate; insns/s the same in guest instructions. At a saturated core these are the throughput; once the thread has headroom they measure how much the program chose to do.
- jit_step/cache_lookup trips that reached the C dispatcher. Judge it as a ratio to block_execs.
- HOTBLOCK the busiest blocks: address, executions, length, and words — arm64 instructions emitted, whose static count includes miss paths that rarely run.
- SLOWOP instructions the JIT hands to the interpreter, ranked. "% of ALL" is what matters; 0.01% is nothing.
- RAS misses returns that missed the host return stack (stale = the entry was there but wrong or gone). 36 million a second was the game's problem; 11 million after the fixes, and now cheap.
- CHAIN edges patched, edges patched through a veneer, edges dropped for distance. Dropped must read 0.
The other three instruments
sample <pid>— the host profiler. It counts parked threads as fully as running ones, so keep only leaf frames and drop the blocking syscalls; ocerz's owndispatch_bsd_at/dispatch_machleaves are parked threads, not hot code. Leaf addresses in JIT code map ontoJITDIS_BRIEF's block table to give cycles per prologue, epilogue, instruction and miss path.kill -INFO <pid>— ocerz dumps every guest thread: its x86 rip, the syscall it is blocked in, a return-address chain. Two hundred of them at 0.1 s is a guest-level PC sampler for free.- The probe ladder — for a compatibility failure, write the smallest C program that asks the next question down: which CGL call fails → which renderer is missing → does Metal see a device → does IOKit answer the same as native → does the driver bundle load and its symbols resolve → does objc list its classes → is the image in the dyld list. Each step halves the search; the native arm64 control, not just Rosetta, is what rules out red herrings.
NO_*; read PERFSTAT as ratios; profile leaves, not stacks; and reproduce in the smallest program before touching the code.Self-check: PERFSTAT says RAS misses=3.2 G, stale=3.2 G and ras_slots=0/262144. Which of two hypotheses does the second number kill?
Slot exhaustion. If the software slot pool were the problem it would be used; zero used means every call takes the host-stack path, so the misses come from that path losing its entries — which is what a trip does to it. Reading one more field on the line saved an evening.
Case files: seven fixes, one method
Stage 10 · 2026-09-22, mapped onto the modelEvery fix that landed on 2026-09-22 followed the same shape — symptom, measurement, mechanism, fix, test — and each one lives in a stage of this atlas. Read them as the model in motion.
| Symptom | Measurement that named it | Mechanism (stage) | Fix · test |
|---|---|---|---|
| Discord: renderers "gone" at startup | Electron asks mach_vm_region about __DATA_CONST and executes int3 unless it is read-only | a rewritten Mach reply answered from the host's view, not the guest's (5) | reply from ocerz's own map in every mode; every segment's declared protection applied, after objc is done with the image (6) |
| Brawlhalla: SIGSEGV at 0 after Steam init | unimplemented BSD syscall num=262 — shmat | System V shared memory absent from the table (5) | attach where the host wants, mach_vm_remap into guest space — sharing, proven by a child process · sysv_shm |
| No GPU: Metal nil, GL software only, the game draws nothing | 7-step probe ladder → _dyld_image_count 45 vs 348 | a dlopened cache image never joined the image list (6) | register it after the objc mapping — order matters, or categories vanish · dlopen_image_list |
| Main thread pegged at 100.8% | PERFSTAT: 36 M RAS misses/s, all stale, slots unused; NO_HOST_RAS A/B | host return stack wiped by every trip; a miss exited through the epilogue and cascaded (3) | a miss resolves in-body through the per-site cache · xbench unchanged |
| Still 26% in prologues/epilogues | attributed profile: entries are function starts; exits are direct calls with unpatched chain edges; CHAIN 5% dropped | b reaches 128 MB in a 1 GB arena (3) | veneer pools every 64 MB; dropped = 0; core 100.8% → 71% |
| Global loads cost 18 host words | JITDIS statistics over the engine: mov r,[rip+…] = 18 words | commpage guard + granule test paid at run time for a constant address (4) | decide both at translation time; 18 → 4 words; +13% work/s |
cannot read /usr/bin/open, three times a Steam launch | file: arm64e only | a spawned child with no Intel code run under ocerz anyway (5) | such a child runs natively, as under Rosetta · spawn_arm64_only |
The method, stated once
- Reproduce in the smallest program that shows it, under ocerz and under Rosetta or native — the control is what turns a guess into a fact.
- Measure with the instrument that names the mechanism (Stage 9), never with a hypothesis.
- Fix the mechanism, not the symptom; document the why in the file's prose and the commit message.
- Pin it with a test that would have failed before; gate; commit.
Self-check: I tell you "the thread is saturated inside translated code, and ocerz's own runtime is under 1%". Which stage's mechanisms are the only candidates, and what would you ask me for next?
Stage 3 (how blocks chain, return and trip) and Stage 4 (what a memory access costs) — the cost is in the emitted code or between blocks, not in C. Ask for the attributed profile: cycles by prologue, epilogue, instruction bodies and miss paths. That single table is what turned "codegen is slow" into three specific fixes.
Glossary
the words that recur- guest / host
- The x86 program and its view of the world / the arm64 machine and ocerz's view.
g2hmaps one to the other. ≈ the traveller and the country: the traveller's expectations versus what is actually there. - block
- A straight run of x86 instructions ending in a branch, translated as one unit and cached by its address. ≈ one paragraph of a recipe, up to the next "then go to…".
- chain
- A patched direct branch from one block's exit into the next block's body, so no C runs between them. ≈ a connecting door between two hotel rooms.
- trip
- Any exit from translated code back to the C dispatcher: syscall, unchained edge, slow op, fault, interrupt. Costs the epilogue, the dispatcher and the prologue, and wipes the host return stack. ≈ going back down to the lobby between rooms — and the bump that knocks the bookmarks off the table.
- dispatcher
- The C loop (
ocerz_vm_run_cpu→ocerz_jit_exec_one) that finds or translates the block for the current rip. ≈ the lobby desk that finds your next room, or has it built if it does not exist yet. - pin class
- The register-assignment convention a block was built under. Class 3 pins all sixteen guest registers permanently, which is what makes body-to-body chaining legal. ≈ the kitchen's station layout; class 3 means every station is laid out identically.
- RAS
- Return-address stack: {guest return, host continuation} pairs pushed at a call so a
retcan branch straight to the right body. Kept on the host machine stack with realbl/ret. ≈ the stack of bookmarks for "go back to where I was". - PSC / IC
- Per-site cache / inline cache: a few {target, body} entries kept next to an indirect branch site, then an inline hash probe, before tripping. ≈ speed-dial at one phone.
- veneer
- A 16-byte
ldr; br; addressstub in a pool near the source block, used when a chain'sbcannot reach its target. ≈ a relay runner for a door too far to reach in one stride. - superblock / flip
- A block translated past forward branches with side exits; flipping retranslates it following a taken side that proved hot. ≈ paving past the fork; rebuilding the road when traffic turns the other way.
- lazy flags
- Computing only the x86 flags a later instruction will read, using the def/use table in
flags_live.c. A deferred-flags record is stored at a block edge when the next block might read them. ≈ totalling the bill only when someone asks for it. - slow op
- An instruction the JIT declines to encode; the block trips and the interpreter executes that one step. ≈ a step the shorthand cannot express, so the cook reads that one line aloud.
- arena
- Either the guest memory reservation (
mem.c) or the 1 GBMAP_JITregion holding translated code (jit.c); context tells which. ≈ renting the whole floor up front and handing out rooms yourself. - slot
- A 4 KB guest page inside a 16 KB host page; the unit
mem.ctracks ownership and protection in. ≈ one room of the four behind a single floor lock. - identity / offset / low shadow
- The three guest→host maps (Stage 4). ≈ room number equals position / one fixed offset per floor / two wings, check which first.
- commpage
- The kernel-provided page of timers and feature bits at a fixed x86 address; ocerz keeps its own and redirects accesses to it, hence the guard. ≈ the notice board that moved; every visit is forwarded.
- ordered mode / TSO
- The memory model x86 programs rely on; entered for good at the first thread, fork or shared mapping; loads and stores then use acquire/release forms. ≈ the shared-kitchen rule that applies to everyone once a second cook arrives.
- shared cache
- Apple's single prelinked file of every x86-64 system library at fixed addresses; on macOS 27 it lives inside the Rosetta cryptex. ≈ a library floor with every book pre-shelved at a fixed position.
- closure
- The set of images reachable from the main executable through its load commands — also the list the dyld API reports as "loaded images". ≈ the catalogue of what is on the floor.
- upward dependency
- A load command declaring the back edge of a dependency cycle; followed for membership, never for initialization order. ≈ "I know her, but do not make me wait for her" at the dinner party.
- init phase
- The dependency-ordered run of every image's
+loadand constructors beforemain. ≈ the seating order. - TLV
- Thread-local variable; its descriptor has two layouts (classic on disk, packed in the cache). ≈ each guest's coat-check ticket.
- DynImage
- The loader's record of an image loaded from disk (or synthesized); the cache's images have no DynImage. ≈ a book brought in from outside the library and given a card of its own.
- vdylib
- A synthesized x86-64 system library whose exports are doorways into the bridge (native mode). ≈ the embassy's facade of the Intel library: every door is a booth.
- API database
- The
.apifiles describing how each synthesized export crosses: function, data, var, special, stub, shape, struct, inplace. ≈ the interpreter's phrasebook. - bridge / crossing
- The path from a guest call of a synthesized export to the real arm64 function and back, including argument placement between the two ABIs. ≈ the interpreter relaying a request to the arm64 official and back.
- trampoline bank
- 65,536 assembled arm64 entry points, one per guest function that native code must be able to call. ≈ the 65,536 phone numbers the guest can leave for a call back.
- MIG reply
- The reply half of a Mach message; several are rewritten so the guest sees its own memory map and port names. ≈ the letter the operator opens and rewrites on the way back.
- ulock
- Darwin's futex; a wait on a stack address means "waiting for a dispatch block to run elsewhere". ≈ waiting by a mailbox for a note that another thread will drop in.
- unstick
- The watchdog that nudges a thread blocked too long in a kickable syscall so it re-checks for signals and suspend requests. ≈ the supervisor who taps a worker who has been waiting too long: "check your messages".
- gate
- One of the test suites
make checkruns; green before every commit. ≈ the twenty-minute pre-commit hook.