Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added DWARF debug information support for JIT-compiled code
- Added I2C and SPI APIs to rp2 platform
- Added `code:get_object_code/1`
- Added Erlang distribution over serial (uart)

### Changed
- ~10% binary size reduction by rewriting module loading logic
Expand Down
150 changes: 149 additions & 1 deletion doc/src/distributed-erlang.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,15 @@ Distribution is currently available on all platforms with TCP/IP communication,
- ESP32
- RP2 (Pico)

Two examples are provided:
Distribution over serial (UART) is also available for point-to-point
connections between any two nodes, including microcontrollers without
networking (e.g. STM32). See [Serial distribution](#serial-distribution).

Three examples are provided:

- disterl in `examples/erlang/disterl.erl`: distribution on Unix systems
- epmd\_disterl in `examples/erlang/esp32/epmd_disterl.erl`: distribution on ESP32 devices
- serial\_disterl in `examples/erlang/serial_disterl.erl`: distribution over serial (ESP32 and Unix)

## Starting and stopping distribution

Expand Down Expand Up @@ -94,6 +99,149 @@ fun (DistCtrlr, Length :: pos_integer(), Timeout :: timeout()) -> {ok, Packet} |

AtomVM's distribution is based on `socket_dist` and `socket_dist_controller` modules which can also be used with BEAM by definining `BEAM_INTERFACE` to adjust for the difference.

## Serial distribution

AtomVM supports distribution over serial (UART) connections using the
`serial_dist` module. This is useful for microcontrollers that lack
WiFi/TCP (e.g. STM32) but have UART, and for testing distribution
locally using virtual serial ports.

### Quick start

```erlang
{ok, _} = net_kernel:start('mynode@serial.local', #{
name_domain => longnames,
proto_dist => serial_dist,
avm_dist_opts => #{
uart_opts => [{peripheral, "UART1"}, {speed, 115200},
{tx, 17}, {rx, 16}],
uart_module => uart
}
}).
```

On Unix, the `peripheral` is a device path such as `"/dev/ttyUSB0"` and
the `uart_module` is `uart` from the `avm_unix` library.

### serial\_dist options

- `uart_opts` — proplist passed to `UartModule:open/1` (see `uart_hal`
for common parameters: `peripheral`, `speed`, `data_bits`, `stop_bits`,
`parity`, `flow_control`)
- `uart_module` — module implementing the `uart_hal` behaviour. Defaults
to `uart`.

### Wire protocol

Serial distribution layers three protocols on the raw UART byte stream:

**1. Sync frames (link layer)**

Both sides periodically send 2-byte sync frames (`<<16#AA, 16#55>>`) on
the UART. These serve two purposes:

- **Liveness detection**: a node knows its peer is alive when it receives
sync frames.
- **Garbage collection**: after a failed handshake attempt, stale bytes
from the old handshake remain in the UART buffer. Sync frames are
stripped from all received data, so stale data that happens to contain
`16#AA 16#55` pairs is harmlessly consumed, and non-sync stale bytes
eventually get discarded when the next handshake attempt interprets
them as an invalid packet.

The sync magic (`16#AA 16#55`) is chosen so it cannot appear as the
first two bytes of a valid length-prefixed handshake message (handshake
messages have a 16-bit big-endian length prefix, and the longest
handshake message is well under 100 bytes, so the first byte is always
`0x00`).

**2. Handshake packets (2-byte length prefix)**

During the Erlang distribution handshake, messages are framed as:

```
<<Length:16/big, Payload:Length/binary>>
```

This matches the TCP distribution handshake format. The handshake
follows the standard Erlang distribution protocol (send\_name,
send\_status, send\_challenge, send\_challenge\_reply,
send\_challenge\_ack).

**3. Data packets (4-byte length prefix)**

After the handshake completes, distribution data packets use a 4-byte
length prefix:

```
<<Length:32/big, Payload:Length/binary>>
```

Tick (keepalive) messages are sent as `<<0:32>>` (4 zero bytes).

### Peer-to-peer connection model

Unlike TCP distribution which uses a client/server model (one side
listens, the other connects), serial is point-to-point: both nodes
share a single UART link.

A **link manager** process on each node is the sole owner of UART
reads. On each iteration it:

1. Checks its mailbox for a `setup` request from `net_kernel`
(non-blocking).
2. Sends a sync frame.
3. Reads from the UART with a short timeout.
4. Strips sync frames from received data.
5. If handshake data remains, enters the **accept** path (responder).
6. If a `setup` request was pending, enters the **setup** path
(initiator).
7. Otherwise, loops.

This design ensures only one process reads from the UART at any time,
avoiding the race condition that would occur if separate accept and
setup processes competed for the same byte stream.

If a handshake fails (the distribution controller process exits), the
link manager flushes stale `setup` messages from its mailbox and
restarts the loop, allowing retries.

### Testing with socat

On Unix, `socat` can create virtual serial port pairs for testing:

```bash
socat -d -d pty,raw,echo=0 pty,raw,echo=0
```

This creates two pseudo-terminal devices (e.g. `/dev/ttys003` and
`/dev/ttys004`) connected back-to-back. Each AtomVM node uses one side:

```erlang
%% Node A
{ok, _} = net_kernel:start('a@serial.local', #{
name_domain => longnames,
proto_dist => serial_dist,
avm_dist_opts => #{
uart_opts => [{peripheral, "/dev/ttys003"}, {speed, 115200}],
uart_module => uart
}
}).

%% Node B (separate AtomVM process)
{ok, _} = net_kernel:start('b@serial.local', #{
name_domain => longnames,
proto_dist => serial_dist,
avm_dist_opts => #{
uart_opts => [{peripheral, "/dev/ttys004"}, {speed, 115200}],
uart_module => uart
}
}).

%% From Node B, trigger autoconnect:
{some_registered_name, 'a@serial.local'} ! {self(), hello}.
```

## Distribution features

Distribution implementation is (very) partial. The most basic features are available:
Expand Down
1 change: 1 addition & 0 deletions examples/erlang/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ pack_runnable(network_console network_console estdlib eavmlib alisp)
pack_runnable(logging_example logging_example estdlib eavmlib)
pack_runnable(http_client http_client estdlib eavmlib avm_network)
pack_runnable(disterl disterl estdlib)
pack_runnable(serial_disterl serial_disterl eavmlib estdlib DIALYZE_AGAINST avm_esp32 avm_unix)
pack_runnable(i2c_scanner i2c_scanner eavmlib estdlib DIALYZE_AGAINST avm_esp32 avm_rp2 avm_stm32)
pack_runnable(i2c_lis3dh i2c_lis3dh eavmlib estdlib DIALYZE_AGAINST avm_esp32 avm_rp2 avm_stm32)
pack_runnable(spi_flash spi_flash eavmlib estdlib DIALYZE_AGAINST avm_esp32 avm_rp2)
Expand Down
127 changes: 127 additions & 0 deletions examples/erlang/serial_disterl.erl
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
%
% This file is part of AtomVM.
%
% Copyright 2026 Paul Guyot <pguyot@kallisys.net>
%
% Licensed under the Apache License, Version 2.0 (the "License");
% you may not use this file except in compliance with the License.
% You may obtain a copy of the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS,
% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
% See the License for the specific language governing permissions and
% limitations under the License.
%
% SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later
%

%% @doc Example: distributed Erlang over serial (UART).
%%
%% This example starts distribution using a UART connection instead of
%% TCP/IP. It works on ESP32 and Unix (using a real serial device or a
%% virtual serial port created with socat).
%%
%% <h3>ESP32 wiring</h3>
%%
%% ```
%% ESP32 TX (GPIO 17) -> Peer RX
%% ESP32 RX (GPIO 16) -> Peer TX
%% ESP32 GND -> Peer GND
%% '''
%%
%% <h3>Unix with socat</h3>
%%
%% Create a virtual serial port pair:
%% ```
%% socat -d -d pty,raw,echo=0 pty,raw,echo=0
%% '''
%% Then set the SERIAL_DEVICE environment variable to one of the pty
%% paths before running this example. The peer node uses the other pty.
%%
%% <h3>Connecting</h3>
%%
%% The node name is derived from the serial device and hostname, e.g.
%% `ttys003@myhost' or `uart1@esp32-abc'. Once both nodes are running,
%% trigger autoconnect from either side:
%% ```
%% {serial_disterl, 'ttys003@myhost'} ! {hello, node()}.
%% '''
-module(serial_disterl).

-export([start/0]).

start() ->
UartOpts = uart_opts(),
NodeName = make_node_name(UartOpts),
{ok, _NetKernelPid} = net_kernel:start(NodeName, #{
name_domain => longnames,
proto_dist => serial_dist,
avm_dist_opts => #{
uart_opts => UartOpts
}
}),
io:format("Distribution started over serial~n"),
io:format("Node: ~p~n", [node()]),
net_kernel:set_cookie(<<"AtomVM">>),
io:format("Cookie: ~s~n", [net_kernel:get_cookie()]),
register(serial_disterl, self()),
io:format("Registered as 'serial_disterl'. Waiting for messages.~n"),
io:format("From the peer:~n"),
io:format(" {serial_disterl, '~s'} ! {hello, node()}.~n", [node()]),
loop().

%% Build a node name from the serial device.
%% e.g. "UART1" -> 'uart1@serial'
%% "/dev/ttys003" -> 'ttys003@serial'
make_node_name(UartOpts) ->
Peripheral = proplists:get_value(peripheral, UartOpts, "serial"),
BaseName = basename(Peripheral),
list_to_atom(string:to_lower(BaseName) ++ "@serial.local").

basename(Path) ->
case lists:last(string:split(Path, "/", all)) of
[] -> Path;
Name -> Name
end.

%% Platform-specific UART configuration.
uart_opts() ->
case erlang:system_info(machine) of
"ATOM" ->
case atomvm:platform() of
esp32 ->
[{peripheral, "UART1"}, {speed, 115200}, {tx, 17}, {rx, 16}];
generic_unix ->
Device = os:getenv("SERIAL_DEVICE"),
case Device of
false ->
io:format("Error: set SERIAL_DEVICE env var to a serial port path~n"),
io:format(" e.g. /dev/ttyUSB0 or a socat pty~n"),
exit(no_serial_device);
_ ->
[{peripheral, Device}, {speed, 115200}]
end;
Other ->
io:format("Error: unsupported platform ~p~n", [Other]),
exit({unsupported_platform, Other})
end;
"BEAM" ->
io:format("Error: this example requires AtomVM~n"),
exit(beam_not_supported)
end.

loop() ->
receive
quit ->
io:format("Received quit, stopping.~n"),
ok;
{hello, From} ->
io:format("Hello from ~p!~n", [From]),
loop();
Other ->
io:format("Received: ~p~n", [Other]),
loop()
end.
3 changes: 3 additions & 0 deletions libs/estdlib/src/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ set(ERLANG_MODULES
maps
math
net
os
proc_lib
sys
logger
Expand All @@ -67,6 +68,8 @@ set(ERLANG_MODULES
queue
sets
socket
serial_dist
serial_dist_controller
socket_dist
socket_dist_controller
ssl
Expand Down
5 changes: 4 additions & 1 deletion libs/estdlib/src/net_kernel.erl
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,10 @@ handle_cast(_Message, State) ->
{noreply, State}.

%% @hidden
handle_info({accept, AcceptPid, SocketPid, inet, tcp}, #state{proto_dist = ProtoDist} = State) ->
handle_info(
{accept, AcceptPid, SocketPid, _Family, _Protocol},
#state{proto_dist = ProtoDist} = State
) ->
Pid = ProtoDist:accept_connection(AcceptPid, SocketPid, State#state.node, [], ?SETUPTIME),
AcceptPid ! {self(), controller, Pid},
{noreply, State};
Expand Down
Loading
Loading