Mix.install([{:kettlecorn, "~> 0.1.0"}])About this notebook
Inspect the runtime's files, native launcher command line, and VM's private directory. Trace the frames between host and guest and examine how the host decodes guest data.
Requires Elixir 1.20 or later on x86_64 or ARM64 Linux or Apple silicon macOS. Run the cells in order. Some cells use internal modules and data structures, including the protocol decoder and VM state, which can change between versions; the getting started notebook covers the public API.
{:ok, runtime} = Kettlecorn.Install.runtime()The runtime
flowchart LR
subgraph host [your BEAM]
owner[owner process] --- vm[Kettlecorn.VM]
end
vm -- "port: stdin frames" --> wasmtime
wasmtime -- "local socket: output records" --> vm
subgraph wasmtime [launcher process]
beam[beam.wasm: BEAM emulator] --- root["/ = a private copy of root/"]
endA runtime is a directory. beam.wasm and beam-http.wasm are two emulator
variants, each a WebAssembly component; beam-http.wasm also imports
wasi:http. root/ is the guest's file system: OTP, Elixir, Req and the
kettlecorn_guest application, as .beam files.
runtime = Kettlecorn.Runtime.dir()
File.ls!(runtime) |> Enum.sort()manifest.json records the versions, the protocol version the library
checks before it starts a VM, and the WASI interfaces each emulator imports
(read from the components when the runtime was built):
manifest = Kettlecorn.Runtime.load!(runtime).manifest
Map.drop(manifest, [:imports])manifest.imports["beam-http.wasm"] -- manifest.imports["beam.wasm"]The Rust library reads fixed guest arguments, environment, and emulator names
from launch/ for both host VMs and the script CLI. Argument files hold one
argument per line:
for file <- File.ls!(Path.join(runtime, "launch")) |> Enum.sort(), into: %{} do
{file, File.read!(Path.join([runtime, "launch", file])) |> String.split("\n", trim: true)}
end-Mea min makes ERTS use plain malloc. Without it, ERTS's own allocators
fragment WebAssembly memory, which never shrinks, and a VM that keeps
building large heaps runs out of memory with little live data.
The command line
Kettlecorn.Sandbox.new/1 turns options into a sandbox and fails on an
unknown or invalid option, before anything starts. prepare/1 creates the
VM's private directory through the native root populator, and command/1 is
what the VM runs:
sandbox =
Kettlecorn.Sandbox.new(max_memory: 256 * 1024 * 1024, max_lifetime: 60_000, network: true)
instance = Kettlecorn.Sandbox.Instance.prepare(sandbox)
Kettlecorn.Sandbox.Instance.command(instance)The host passes limits, grants and guest options to the bundled launcher.
The Rust library reads the fixed guest arguments and environment from launch/.
Kettlecorn.VM adds a private --output-socket path when opening the port and
requests one record of at most 64 KiB at a time, before decoding guest frames.
The private directory has an unguessable name and mode 0700. It holds the guest's root and, out of the guest's reach, the launcher's stderr:
%File.Stat{mode: mode} = File.stat!(instance.dir)
{instance.dir, Integer.to_string(Bitwise.band(mode, 0o777), 8), File.ls!(instance.dir)}Kettlecorn.Sandbox.Instance.cleanup(instance)
File.exists?(instance.dir)A running VM
Start a VM and look at its state. It owns the launcher process through a port, and tracks unanswered requests in a request registry. This interactive VM expires after ten minutes; restart from this cell if it expires:
{:ok, vm} =
Kettlecorn.start_link(
owner: self(),
max_fuel: 1_000_000_000_000,
max_lifetime: 600_000
)
%{boot: :ready, requests: requests} = state = :sys.get_state(vm)
%{limit: 1_024} = requests
true = requests.entries == %{}
Map.take(state, [:boot, :requests, :max_message_size])Each VM has its own launcher process. All of the guest's BEAM schedulers run on one OS thread.
{:os_pid, os_pid} = Port.info(state.port, :os_pid)
{ps, 0} = System.cmd("ps", ["-o", "pid,command", "-p", Integer.to_string(os_pid)])
IO.puts(ps)Frames
The guest reads frames from its stdin and writes frames to its stdout. Each
frame has a 4-byte big-endian length, a type byte, and the payload (the protocol specification
has the full list). Kettlecorn.send/4 becomes a send
frame (type 16): a request id, a target and the payload as external term
format. The host sends it through port stdin. The launcher transports guest
output over the local socket in records of at most 64 KiB; a record may contain
partial or multiple guest frames:
prepared = Kettlecorn.Protocol.prepare("echo", {:ping, 1})
frame = Kettlecorn.Protocol.send_frame(7, prepared)
frame |> Kettlecorn.Protocol.frame() |> IO.iodata_to_binary()The guest answers with a send_result frame (type 5) carrying the same id
and a status. The host splits frames off the stream itself, and refuses a
frame whose declared length exceeds :max_message_size.
The next stream concatenates two frames: a six-byte body containing
send_result (type 5, request 7, status 0), followed by a three-byte body
containing stdout (type 1, text hi). Both four-byte length prefixes
count the type byte and payload. Split the stream inside the first frame and
pass the decoder state into the next call:
stream = <<0, 0, 0, 6, 5, 0, 0, 0, 7, 0, 0, 0, 0, 3, 1, ?h, ?i>>
<<first::binary-size(7), second::binary>> = stream
{:ok, [], decoder} = Kettlecorn.Protocol.split(first, 1_024)
{:ok, bodies, {:header, ""}} = Kettlecorn.Protocol.split(second, 1_024, decoder)
[{:send_result, 7, :ok}, {:stdout, "hi"}] =
Enum.map(bodies, &Kettlecorn.Protocol.decode(&1, 1_024)){:error, {:frame_too_large, 1_000_000_000}} =
Kettlecorn.Protocol.split(<<1_000_000_000::32, 1>>, 64 * 1024 * 1024)Kettlecorn.Protocol.decode/2 returns decoded events. For message frames,
its second argument limits the ETF decoder's memory use.
Kettlecorn.call/4 isn't a frame type of its own: it's a send to the
guest's Kettlecorn.Proxy, which makes the GenServer.call in the guest
and answers with an event. The host sends the proxy timeout in the call
payload. For evaluations it also sends the evaluation and capture-finish
budgets. The guest applies these budgets directly.
The guest's view
Inside, the VM is a regular OTP system on a WebAssembly architecture, with
one scheduler and the arguments Kettlecorn.Sandbox passed:
{:ok, %{"value" => view}} =
Kettlecorn.eval(vm, """
%{
architecture: :erlang.system_info(:system_architecture),
schedulers: :erlang.system_info(:schedulers),
root: :code.root_dir(),
applications: Enum.map(Application.started_applications(), &elem(&1, 0)),
registered: Enum.filter(Process.registered(), &(&1 in [:kettlecorn_proxy, :kettlecorn_evaluator]))
}
""")
viewEach evaluation runs in a new process whose group leader captures its
standard output. Writes to stderr bypass this capture and arrive as :stderr
console events. Kettlecorn.take_output/1 combines console stdout and stderr
in one buffer without stream labels. A second process traces descendants and
attempts to kill them on timeout. After success, the group leader forwards
their later output to the VM's stdout. Ask the evaluation who its tracer is:
{:ok, %{"inspected" => inspected}} =
Kettlecorn.eval(vm, """
{:tracer, tracer} = :erlang.trace_info(self(), :tracer)
{is_pid(tracer) and tracer != Process.group_leader(), :erlang.trace_info(self(), :flags)}
""")
IO.puts(inspected)A process started by an evaluation outlives it. Its output arrives at the
owner as a :stdout event:
{:ok, _} =
Kettlecorn.eval(vm, ~S"""
spawn(fn -> Process.sleep(100); IO.puts("later") end)
:ok
""")
receive do
{:kettlecorn, ^vm, {:stdout, text}} ->
Kettlecorn.acknowledge(vm)
text
after
5_000 -> :timeout
endDecoding what the guest sends
The guest is untrusted, so the host never calls binary_to_term/1 on its
data. Kettlecorn.ETF decodes plain data only:
etf = :erlang.term_to_binary({:ok, self(), make_ref(), "data"})
{:ok, {:ok, %Kettlecorn.Pid{vm: nil}, %Kettlecorn.Opaque{type: :reference}, "data"}} =
Kettlecorn.ETF.decode(etf)This synthetic input contains the host's pid. Standalone decoding produces an
unbound Kettlecorn.Pid; it cannot address a guest. Pids received through a
VM are bound to that VM and can be used with Kettlecorn.send/4. The reference
becomes an opaque placeholder. Unknown atoms become
{:unknown_atom, name} tuples, so the guest cannot grow the host's atom
table. Funs are rejected:
name = "an_atom_the_host_lacks_#{System.unique_integer([:positive])}"
{
Kettlecorn.ETF.decode(<<131, 118, byte_size(name)::16, name::binary>>),
Kettlecorn.ETF.decode(:erlang.term_to_binary(fn -> :evil end))
}Structs are maps with a :__struct__ key, so the guest can name any host
module. Decoded as a struct, this map would be a File.Stream whose path
the guest chose, and host code that enumerates it would read that file. It
decodes as a Kettlecorn.Struct instead, which only holds the data:
forged = %{
__struct__: File.Stream,
path: "/etc/passwd",
line_or_bytes: :line,
modes: [],
raw: true
}
{:ok, decoded} = Kettlecorn.ETF.decode(:erlang.term_to_binary(forged))
{decoded, Enumerable.impl_for(decoded)}Small inputs can decode into large terms: each empty list in a list takes
one byte to encode and 16 bytes on a 64-bit host heap. Compression shrinks
the input further. The decoder charges the decoded data and each node's
estimated heap size, and stops at a budget (the VM uses :max_message_size):
bomb = :erlang.term_to_binary(List.duplicate([], 4_000_000), compressed: 9)
{byte_size(bomb), Kettlecorn.ETF.decode(bomb, max_bytes: 16 * 1024 * 1024)}VM termination
If a limit stops a VM, every pending request fails with
{:error, {:vm_down, reason}}. The reason identifies the limit:
{:ok, fuel_vm} = Kettlecorn.start(max_fuel: 3_000_000_000, max_lifetime: 30_000)
Kettlecorn.eval(fuel_vm, "Stream.repeatedly(fn -> 1 end) |> Stream.run()", timeout: 60_000)The fuel budget includes booting, and Wasmtime stops the guest when it
runs out. :max_lifetime stops it after the wall-clock limit
({:vm_down, :timeout}). :max_memory caps WebAssembly memory growth;
an allocation failure can terminate ERTS.
Kettlecorn.stop(vm)