The BEAM File

What a .beam file is for

Every Erlang module you have ever loaded arrived as a .beam file. The compiler turns hello.erl into hello.beam; code:load_file/1 hands those bytes to the runtime; a few microseconds later hello:greet/1 can be called. The file is the contract between two programs that are developed together but run at very different times: the compiler, written in Erlang, and the loader, written in C inside ERTS.

So the obvious job is to carry code. But look at what a module really is when it reaches the runtime. It isn't just instructions. It's instructions plus a set of names those instructions talk about:

• the atoms it mentions (ok, error, its own module name);
• the functions it exports, and where each one starts;
• the functions it calls in other modules, which may not be loaded yet;
• the constant terms it uses, such as #{a => [1,2,3]} or a long binary;
• the funs it creates, which must survive hot code upgrades;
• the line numbers that make stack traces readable;
• its attributes, compile options, documentation, and, optionally, enough debug information to rebuild the source.

The runtime needs some of these to execute anything. It never needs others. Tools such as xref, dialyzer, the debugger, the shell's h/1 and cover need yet others. A .beam file is the single artefact that carries all of them.

Who reads it

It helps to list the consumers, because each one wants something different from the same bytes:

The constraints

Combine those readers with how Erlang systems actually live, and a handful of hard requirements falls out:

1. Portable. One file must load on any word size and endianness, and on both the JIT and the interpreter.
2. Separable. Optional information has to be removable without touching, or even understanding, the parts that matter.
3. Extensible. New releases add things: UTF-8 atoms, type hints, feature flags, native records, debugger metadata. Old tools must keep working on new files wherever that is possible.
4. Refusable. When old runtimes can't run new code, they must say so clearly, not crash halfway through loading.
5. Identifiable. Hot code loading and release handling need to know whether two files contain "the same module", even if one was compiled on another day or another machine.
6. Cheap. Releases boot thousands of modules. Parsing must be close to free.

Keep this list in mind. Every design decision that follows answers one of these six points.

Other ways to do it

Imagine you are designing this from scratch in the late 1990s. You have several reasonable options:

What you actually want is somewhere in the middle: a sequence of independent, named, length-prefixed blobs. A reader can look at a blob's name, decide whether it cares, and jump past it in O(1) if it doesn't. Writing a new kind of blob doesn't break anyone. Removing an optional blob is cutting bytes out of a file.

That idea already existed, fully specified, and was in the public domain.

Enter EA IFF 85

In January 1985, Jerry Morrison at Electronic Arts published "EA IFF 85: Standard for Interchange Format Files". It was written so that independently developed Amiga programs, such as paint programs, music editors and word processors, could exchange data. Its building block is the chunk:

A file is a FORM chunk whose data begins with a four-character form type (such as ILBM for images or AIFF for audio) followed by more chunks. The spec's central rule for readers is the reason the format is still around 40 years later:

"It should skip over all unrecognized chunks."EA IFF 85, on readers

The spec also includes a design philosophy, borrowed from Alan Kay: "Simple things should be simple and complex things should be possible." Its descendants are everywhere: AIFF audio, Microsoft's little-endian clone RIFF (WAV, AVI, WebP), and the chunk-based design of PNG.

BEAM uses IFF almost unchanged, with two deliberate deviations:

The 4-byte alignment exists because almost every BEAM table is made of 32-bit words. With 4-byte alignment, a C loader can read them from aligned addresses, and a .beam file can be mmaped and walked directly.

Now go back to the six constraints. Portable: nothing in the container depends on the host machine. Separable: chunks are independent. Extensible: unknown chunk IDs are skipped. Cheap: finding a chunk means reading 8-byte headers and adding offsets. The other two, refusable and identifiable, are solved inside the chunks, as we'll see.

What it costs

Every format is a set of trade-offs. Here are the ones BEAM accepts:

• No directory. To find a chunk you scan from the start. With roughly 10–16 chunks, this costs nothing.
• 32-bit sizes. Each chunk, and the whole file, is limited to 4 GiB. No real module comes close.
• No per-chunk integrity check. The module MD5 covers meaning, not transport. If you need a checksum on the bytes, the tools gzip the whole file instead (which also adds a CRC).
• Chunk IDs are a shared namespace. Anyone can add a chunk with the {extra_chunks, …} compile option, so choose IDs that won't collide.

1 · A flat bag of chunks

A .beam file is FOR1, a size, BEAM, and then chunks one after another. Each chunk has a four-letter ID that roughly describes it: AtU8 holds atoms in UTF-8, Code is code, ImpT and ExpT are the import and export tables, and LitT is the literal table. The chunks fall into three groups:

• Required by the loader: AtU8, Code, StrT, ImpT, ExpT.
• Used by the loader when present: FunT, LitT, Line, Type, Meta, Recs, DbgB, Attr, CInf, LocT.
• For tools only: Dbgi, Docs, and anything else.

Order doesn't matter to the loader, because it scans the whole file and picks out the IDs it knows. The compiler still writes chunks in a stable order, and it writes the essential ones first.

2 · Name things once, refer to them by number

Instructions never contain an atom's text, a function name, or a constant term. Instead the compiler builds tables: atoms, imports, literals, strings, line locations, types and funs. The code holds small integers that index into those tables.

This one idea gives you several properties. The code stream is small, because most operands fit in a single byte. It is position-independent and identical on every machine. And the loader's job becomes clear: it resolves each table once into runtime values (interning atoms, finding export entries, copying literals into the literal area), then patches the code as it translates it.

3 · Compact terms: a variable-length integer with a type tag

Operands in the code stream use a small custom encoding. The low three bits of the first byte are a tag that says what kind of operand this is: a plain number (u), an integer literal (i), an atom index (a), an X register, a Y register, a label (f), or "extended" (z). The remaining bits hold the value. Values from 0 to 15 fit in the same byte, values up to 2047 take two bytes, and anything larger carries its own byte count.

Most operands are small register numbers, atom indices and labels, so the common case is one byte. The encoding is self-delimiting and carries its type with it. A reader can walk the instruction stream and know what kind of value each operand is without decoding anything else first. The format is so convenient that newer chunks (atom lengths in OTP 28, DbgB, Recs, Line) reuse it instead of inventing their own.

4 · When it's rich, use the External Term Format

Attributes, compile info, debug info, docs, feature metadata and every literal are stored as ordinary term_to_binary/1 output. Erlang already has a precise, versioned serialisation for arbitrary terms, so there's no reason to invent another. The rule of thumb:

• Hot path (read on every load, consumed by C): fixed 32-bit big-endian words and compact terms.
• Rich or cold data (arbitrary terms, consumed by Erlang tools): External Term Format.

5 · Generic instructions, and a way to refuse

The opcodes in a .beam file come from lib/compiler/src/genop.tab. Each one has a number and a fixed arity, for example 1: label/1, 64: move/2 and 164: put_tuple2/2. New opcodes are only ever appended, and old ones are never renumbered. Opcodes the compiler stopped using are kept for compatibility, or marked as removed.

The Code chunk header records the highest opcode number used. A runtime that doesn't know that opcode refuses the module up front, with a clear message about recompiling, instead of failing in the middle of loading. That covers the refusable constraint. The header also has a format number, which has been 0 since the format was created and would only change if existing instructions changed meaning.

6 · Identity is a hash of the meaning

Is hello.beam today the same module as hello.beam last week? The answer is the module MD5: an MD5 over the payloads of the significant chunks (atoms, code, strings, imports, exports, funs, literals, features, records and debug metadata), in a fixed order. Headers, padding, timestamps, docs and debug info are excluded. If you don't set -vsn(...), this MD5 becomes your module's version. It's what Mod:module_info(md5) returns, and it's what lets beam_lib:strip/1 remove half a file without changing the module's identity.

The container

The whole writer, from beam_asm.erl:

%% Build an IFF form.
build_form(Id, Chunks0) when byte_size(Id) =:= 4, is_list(Chunks0) ->
    Chunks = list_to_binary(Chunks0),
    Size = byte_size(Chunks),
    0 = Size rem 4,                         % Assertion: correct padding?
    <<"FOR1",(Size+4):32,Id/binary,Chunks/binary>>.

chunk(Id, Contents) when byte_size(Id) =:= 4, is_binary(Contents) ->
    Size = byte_size(Contents),
    [<<Id/binary,Size:32>>,Contents|pad(Size)].

And the reader side, from beam_lib:scan_beam/5. Note the rounding to the next multiple of four:

{NFD, {ok, <<IdL:4/binary, Sz:32>>}} ->
    Id = binary_to_list(IdL),
    Pos1 = Pos + 8,
    Pos2 = (4 * trunc((Sz+3) / 4)) + Pos1,

The loader (iff_init and read_beam_chunks in beam_file.c) does the same: it checks FOR1, takes FormSize bytes, checks BEAM, and then loops over chunk headers. Every chunk ID it doesn't recognise is skipped, just as the IFF spec says.

A whole file, annotated

Here is a complete module:

-module(hello).
-export([greet/1]).

greet(Name) ->
    {hello, Name}.

Compiled with erlc +deterministic hello.erl, it is 528 bytes. Click a chunk name, or any byte, to see what it is. Chunk headers are in bold and padding is greyed out.

Six chunks are worth a closer look by hand.

AtU8: the atom table

00000005                          count = 5 (OTP 27 style: positive)
05 68 65 6c 6c 6f                 len 5, "hello"            → atom 1
05 67 72 65 65 74                 len 5, "greet"            → atom 2
0b 6d 6f 64 75 6c 65 5f 69 6e 66 6f   "module_info"   → atom 3
06 65 72 6c 61 6e 67              "erlang"                   → atom 4
0f 67 65 74 5f 6d 6f 64 75 6c 65 5f 69 6e 66 6f  → atom 5
00                                padding (51 → 52)

OTP 28 and later write the same table like this:

ff ff ff fb                       count = −5  ⇒ "lengths are compact terms"
50 68 65 6c 6c 6f                 len: u-tagged 5 = (5 bsl 4) bor 0
50 67 72 65 65 74
b0 6d 6f 64 75 6c 65 5f 69 6e 66 6f
…

The negative count tells the reader which encoding follows. Old files still load, and atoms longer than 255 bytes become possible. To produce the old layout from a new compiler, use the no_long_atoms option.

ImpT and ExpT: who we call, and who may call us

ImpT  00000002                     2 imports
      00000004 00000005 00000001   erlang:get_module_info/1   → import 0
      00000004 00000005 00000002   erlang:get_module_info/2   → import 1

ExpT  00000003                     3 exports
      00000003 00000001 00000006   module_info/1 at label 6
      00000003 00000000 00000004   module_info/0 at label 4
      00000002 00000001 00000002   greet/1       at label 2

Each entry is three words: atom index, arity, and (for exports) the entry label. module_info/0,1 are generated by the compiler and are just tail calls into an import, which is why the import table isn't empty even though hello calls nothing itself. BIFs are imports too: erlang:length/1 ends up in ImpT like any other remote function.

Code: the header and the instructions

00000010    sub-header size = 16 bytes (the four words below)
00000000    format number   = 0
000000ab    highest opcode  = 171  (OTP 30 forces ≥ 183)
00000007    label count     = 7    (labels 1..6 used, 0 = "no label")
00000003    function count  = 3
01 10 99 10 02 12 22 10 01 20 10 30 10 a4 03 17 20 12 03 13 …

The first word gives the size of the rest of the header. A future release can add header fields, and older readers will still find where the code begins. The decoded instructions are below.

Attr: where your vsn comes from

83 6c 00000001 68 02 77 03 "vsn" 6c 00000001 6e 10 00 b3 36 14 … 31 6a 6a
= term_to_binary([{vsn, [66337080814856080590574765180527589043]}])

That big integer is the 128-bit module MD5. It's ordinary External Term Format (83 is the version byte, 6c a list, 68 a small tuple, 77 a small UTF-8 atom, 6e a small bignum).

Line: locations

00000000 00000000    version 0, flags 0
00000003             3 line instructions in the code
00000001             1 location item
00000000             0 extra file names (index 0 is implicitly "hello.erl")
41                   i-tagged 4: "line 4"
00 00 00             padding

Dbgi: debug info, without debug info

term_to_binary({debug_info_v1, erl_abstract_code, {none, [deterministic]}})

We didn't compile with +debug_info, so the chunk still names the backend but contains none. With +debug_info it would hold the full abstract code, which is enough to reconstruct the source. That's why releases strip it, or encrypt it.

Compact term encoding

This is the part of the format that's least like IFF, and the part you need to understand to read code. The definitive implementation is beam_asm:encode/2. Every operand starts with a byte whose low three bits are the tag:

The two bits above the tag (bits 3 and 4) choose how big the value is:

Here is a complete decoder in Erlang. The bit syntax maps almost directly onto the diagrams:

decode(<<V:4, 0:1, Tag:3, Rest/binary>>) ->              % small
    {Tag, V, Rest};
decode(<<Hi:3, 0:1, 1:1, Tag:3, Lo, Rest/binary>>) ->     % medium
    {Tag, (Hi bsl 8) bor Lo, Rest};
decode(<<7:3, 3:2, Tag:3, Rest0/binary>>) ->              % huge
    {_, N, Rest1} = decode(Rest0),
    big(Tag, N + 9, Rest1);
decode(<<L:3, 3:2, Tag:3, Rest/binary>>) ->               % large
    big(Tag, L + 2, Rest).

big(Tag, N, Bin) ->
    <<V:N/signed-unit:8, Rest/binary>> = Bin,
    {Tag, V, Rest}.

Try it

Extended (z) forms

When the tag is z, the value is a sub-type and more compact terms follow:

Chunk reference

These are the chunks the OTP 30 compiler can write, in the order beam_asm:build_file/9 emits them. The MD5 column marks chunks included in the module MD5. With the slim option, the compiler writes only the essential chunks plus Attr and Type.

Notice the trick that Recs and DbgB share. Instead of inventing a new serialisation, beam_asm writes their contents as if they were instructions, so the loader can reuse its operand decoder, along with the atom and literal tables. Atoms used only by those chunks still go in AtU8. That's why beam_asm builds them before the atom table.

Reading the code

An instruction is a one-byte opcode followed by exactly as many compact-term operands as genop.tab says. A z1 list can add more operands at the end. Here is the full code stream of hello.beam:

That's 56 bytes for three functions. Compare it with erlc -S hello.erl, which prints the same program as Erlang terms before beam_asm encodes it. Two consequences worth remembering:

• The loader is a translator, not a copier. It turns a1 into the runtime atom hello, u0 after call_ext_only into a pointer to an export entry, and labels into addresses. It then rewrites generic instructions into specialised ones using the rules in ops.tab, or into native code.
• Everything is in the tables. You can't understand an instruction without AtU8, ImpT and LitT. That's why those chunks are essential and part of the module MD5.

The module MD5

beam_lib:md5/1, code:module_md5/1 and the compiler all compute the same value:

md5_chunks() ->
    ["Atom", "AtU8", "Code", "StrT", "ImpT", "ExpT",
     "FunT", "LitT", "Meta", "Recs", "DbgB"].

%% MD5 = erlang:md5([Data || Id <- md5_chunks(), present, in this order]),
%% with every FunT entry's OldUniq word replaced by 0.

Only the chunk payloads are hashed. Headers and padding are not. For hello.beam, hashing AtU8 ++ Code ++ StrT ++ ImpT ++ ExpT ++ Meta gives exactly the integer stored in Attr as vsn. There's a circular dependency here, and the compiler breaks it the same way every time: it builds the MD5 chunks with OldUniq = 0, hashes them, then writes 27 bits of the hash into FunT. Everyone who re-hashes zeroes those words first.

Because CInf, Dbgi, Docs and Line aren't hashed, you can change docs or line numbers without changing the module's identity. Release upgrades and beam_lib:cmp/2 depend on that.

Versions, and how they show up in the bytes

The format number has never changed. All evolution happens through new chunks, new opcodes and small in-chunk markers. Ignoring the inside of individual instructions, the history looks like this:

The compiler options no_long_atoms and compressed_literals write the older forms, so files built by a new compiler can be read by older tools. The loader, however, always accepts both forms.

Tools of the trade

%% Which chunks, where, how big
beam_lib:info("hello.beam").

%% Decoded tables
beam_lib:chunks("hello.beam", [atoms, imports, labeled_exports, attributes]).

%% Raw bytes of any chunk, by ID
beam_lib:chunks("hello.beam", ["Code", "Line"]).

%% Instructions, as the compiler saw them
beam_disasm:file("hello.beam").
%% shell: erlc -S hello.erl   → hello.S (before assembly)

%% Instructions, as the loader produced them
erts_debug:df(hello).          % writes hello.dis

%% Strip, compare, rebuild
beam_lib:strip("hello.beam").
beam_lib:cmp("a/hello.beam", "b/hello.beam").
beam_lib:build_module([{"AtU8", A}, {"Code", C}, ...]).

%% Add your own chunk at compile time
compile:file(hello, [{extra_chunks, [{<<"Xmpl">>, <<"hi">>}]}]).

Finally, here is the whole container in ten lines, because Erlang's bit syntax was made for formats like this:

chunks(File) ->
    {ok, <<"FOR1", Size:32, "BEAM", Rest/binary>>} = file:read_file(File),
    Size = byte_size(Rest) + 4,
    split(Rest).

split(<<>>) -> [];
split(<<Id:4/binary, Len:32, Data:Len/binary, Rest0/binary>>) ->
    Pad = (4 - Len rem 4) rem 4,
    <<_:Pad/binary, Rest/binary>> = Rest0,
    [{Id, Data} | split(Rest)].

From here, the best next step is to read the sources this page is built on. beam_asm.erl is under 900 lines and writes everything. beam_lib.erl reads it back from Erlang. erts/emulator/beam/beam_file.c is the loader's strict view of the same bytes, and genop.tab is the instruction set.