feat: numerous tools additions + firm make support + fixes!

* tools: init pica disasm dvl/raw, now dvl (shbin's) and raw instruction
  streams can be disassembled.
* tools: init firm dump and firm make, firm make has been tested and has
  built sucessfully a Luma3DS firm (even booted!)
* tools/Ncch: more misc changes, still not finished.
* hardware/pica: finish shader assembler, only missing literally *1*
  diagnostic.
* horizon/fmt: finish dvl format description.
* docs/zpsh: document the zpsh format, still not final.
* ci: update to 0.15.2

Author: GasInfinity

.github/workflows/cd.yaml

@@ -22,7 +22,7 @@ jobs:
       - name: Setup zig
         uses: mlugg/setup-zig@v2
         with:
-          version: 0.15.1
+          version: 0.15.2
 
       - name: Build docs
         run: zig build docs

.github/workflows/ci.yaml

@@ -18,7 +18,7 @@ jobs:
       - name: Setup zig
         uses: mlugg/setup-zig@v2
         with:
-          version: 0.15.1
+          version: 0.15.2
 
       - name: Test
         run: zig build test

README.md

@@ -80,6 +80,10 @@ Currently there are multiple examples in the `demo/` directory. To build them, y
 - [flappy](demo/flappy) is a simple fully functional flappy bird clone written entirely with software blitting.
 - [gpu](demo/gpu/) is a playground for [mango](src/mango.zig), bleeding edge features are tested there. Not really an example per-se.
 
+--- 
+
+You can (and are encouraged) to look at the `tools` directory as it is a good example of how to use the API's `zitrus` provides outside (and inside!) of a 3DS environment. Almost all tools are self-contained and span 50-300 LOC.
+
 # Coverage
 
 ### Legend
@@ -109,8 +113,9 @@ Currently there are multiple examples in the `demo/` directory. To build them, y
 - 🟢 3dsx (tools/3dsx): Make / Dump
 - 🟢 Pica (tools/Pica): Assemble / Disassemble
     - 🟢 Assemble: Only **Z**itrus**P**ica**Sh**aders are implemented as an output format.
-    - 🟢 Disassemble: Outputs **Z**itrus**P**ica**A**sse**m**bly, only ZPSH's can be disassembled currently.
-- 🟡 Firm (tools/Firm): Info
+    - 🟢 Disassemble: Outputs **Z**itrus**P**ica**A**sse**m**bly. Either RAW instructions, ZPSH's or DVL's (.shbin) can be disassembled.
+- 🟢 Firm (tools/Firm): Make / Info / Dump
+    - 🟢 Make: Confirmed to build (and boot!) Luma3DS from source, however needs more testing as the firm is not 1:1.
 - 🟡 Ncch (tools/Ncch): Dump (/ Info)
     - 🟢 ExeFS (tools/ExeFs): Make / List / Dump
     - 🟢 RomFS (tools/RomFs): Make / List / Dump
@@ -197,7 +202,7 @@ Whether register bits are present and/or relevant tooling (assemblers, disassemb
 
 ## Why?
 I wanted to learn arm and always wanted to develop 3DS homebrew, also I searched and I haven't found any kind of zig package that doesn't use libctru, so I just started reading a lot and doing things. Furthermore, using only zig has a lot of advantages:
-- Really simplified and easy development. You don't need complex toolchains, you just need the `zig` executable, that's it! (However, obviously it is recommended that you use devkitPRO's tools as I'm sure you'll need them. You want to use gdb, don't you?)
+- Really simplified and easy development. You don't need complex toolchains, you just need the `zig` executable, that's it. The tools `zitrus` provides also have no dependencies, they'll work on any platform that zig supports! You can still use devkitPRO's binutils if you need.
 - Safety in `Debug` and `ReleaseSafe` modes. Zitrus currently uses the `ErrDisp` port to report panics and returned errors. The only missing thing is reporting return traces with debugging symbols (Currently only addresses are logged)
 - Really useful and simple build-system (as you've seen the example `build.zig` is really small and makefiles are really arcane)
 

docs/pica/zpsh.md

@@ -1,8 +1,115 @@
-# Zitrus PICA200 shader
+# PICA200 shader
 
-This file format is very minimalist and is the default output of a zpsm when assembled by zitrus.
+**This document is marked as a DRAFT, it is NOT final**
+
+This new file format is very minimalist and is the default output format for a file assembled by zitrus.
+
+It has been designed from scratch with iterative changes based on real world use-cases (mango)
 
-## Format
+All fields are little endian unless explicitly told otherwise and all types are described with `zig` syntax.
+
+## Header
+
+The main header of the binary, starting at offset `0x00`
+
+| Field     | Type                     | Notes                      |
+|:---------:|:------------------------:|----------------------------|
+| `magic`   | `[4]u8`                  | Must be `ZPSH`             |
+| `shader`  | `bitpacked struct(u32)`  | Starting from the LSb, the first `u12` is the number of entrypoints, the next `u12` is the number of instructions **minus one**, a valid PICA200 shader must have at least one instruction, `end`. The last `u8` is the number of instruction operand descriptors, **the value must be no larger than 128**|
+| `entry_string_table_size` | `u16`    | Size **in `u32`s** of the entrypoint string table |
+| `flags`   | `bitpacked struct(u8)`   | Reserved                   |
+| `header_size` | `u8`                 | Real size **in `u32`s** of the **Header**, allows for *forward compatibility* |
+
+#### `shader` bit layout
+
+| 31...24 | 23...12 | 11...0 |
+|:-------:|:-------:|:------:|
+| descriptors (`u8`) | instructions_minus_one (`u12`) | entrypoints (`u12`) |
+
+## Instruction & Operand Descriptors
+
+Instructions (`shader.instructions_minus_one` + 1) and operand descriptors (`shader.descriptors`), starting at offset `header_size * @sizeOf(u32)`
+
+## Entrypoint string table
+
+This string table stores unique `0`-terminated `UTF-8` strings for entrypoint names, starting at offset `(header_size + shader.instructions_minus_one + 1 + shader.descriptors) * @sizeOf(u32)`
+
+## Entrypoint Header
+
+Entrypoints follow, they're dynamically sized as constants follow this header.
+  
+Starting at offset `(header_size + shader.instructions_minus_one + 1 + shader.descriptors + entry_string_table_size) * @sizeOf(u32)`
+
+| Field                | Type                     | Notes                                                |
+|:--------------------:|:------------------------:|------------------------------------------------------|
+| `name_string_offset` | `u32`                    | Offset of the name in the entrypoint string table    |
+| `instruction_offset` | `u16`                    | Entry instruction offset, must be less than 4096     |
+| `info`               | `bitpacked struct(u16)`  | Shader type and parameters, see below for the layout |
+| `boolean_constant_mask` | `bitpacked struct(u16)`  | Each bit represents the state of the constant boolean register `bX` |
+| `integer_constant_mask` | `bitpacked struct(u16)`  | Each bit represents whether a constant for the integer register `iX` follows, the remaining 12-bits are reserved and must be zeroed. |
+| `floating_constant_mask` | `extern struct`  | Each bit represents represents whether a constant for the floating register `fX` follows, the remaining bits are reserved and must be zeroed. |
+| `output_mask` | `bitpacked struct(u16)`  | Each bit represents represents whether an output map for the output register `oX` follows. |
+
+#### `boolean_constant_mask` bit layout
+
+Each bit represents the constant value for the register at that bit index.
+
+| 16  | ... | 0  |
+|:---:|:---:|:--:|
+| b15 | bN  | b0 |
+
+#### `integer_constant_mask` bit layout
+
+Each bit represents whether a constant value for the register at that bit index follows after the header.
+
+Constants are sorted for each bit, that is, if bits `0` and `3` are set, the first constant will be for register `i0`
+and the second for `i3`.
+
+|  16...4  | 3  | 2  | 1  | 0  |
+|:--------:|:--:|:--:|:--:|:--:|
+| reserved | i3 | i2 | i1 | i0 |
+
+#### `floating_constant_mask` layout
+
+Each bit represents whether a constant value for the register at that bit index follows after the header.
+
+The `extern struct` has this layout:
+
+| Field                | Type                     | Notes                                                      |
+|:--------------------:|:------------------------:|------------------------------------------------------------|
+| `low`                | `bitpacked struct(u32)`  | Whether a constant value follows for registers `f0`-`f31`  |
+| `mid`                | `bitpacked struct(u32)`  | Whether a constant value follows for registers `f32`-`f63` |
+| `high`               | `bitpacked struct(u32)`  | Whether a constant value follows for registers `f64`-`f95` |
+
+Bit layout for each field follows the same structure as `boolean_constant_mask` and `integer_constant_mask`.
+
+#### `output_mask` bit layout
+
+Each bit represents whether an output map for the register at that bit index follows after the header.
+
+| 16  | ... | 0  |
+|:---:|:---:|:--:|
+| o15 | oN  | o0 |
+
+### Entrypoint constants and output maps
+
+Each entrypoint can be considered as a different `shader module` that shares instructions with others. As such, they have different sets
+of constants and output maps.
+
+The layout is the following:
+- Entrypoint Header
+- Integer constants, one for each bit set in `integer_constant_mask`
+- Floating constants, one for each bit set in `floating_constant_mask`
+- Output maps, one for each bit set in `output_mask`
+
+#### Integer constant
+
+A `[4]u8` representing a 4-component vector with layout `xyzw`.
+
+#### Floating constant
+
+A packed 4-component `F7_16` vector in the format required by the PICA200. It can be seen as storing all `F7_16` components packed in an `[3]u32` and doing a `swap` of the first and last `u32`.
+
+#### Output map
 
-You can see the format types in `fmt/zpsh.zig` and how it is written in `tools/pica/main.zig`.
-If you've never read zig, [here's the official reference](https://ziglang.org/documentation/master/)
+An map describing output semantics for each component of the specified output. They're in the format required by the PICA200.

docs/pica/zpsm.md

@@ -23,15 +23,15 @@ A `Limited source register` or `src_limited` refers to an `Input` or `Temporary`
 
 A `Destination register` or `dst` refers to an `Output` or `Temporary` register.
 
+A `Floating constant` register may be addressed at runtime starting at a base register fX, in that case you may use `a.x`, `a.y` or `a.l` to do so. E.g: `fX[a.x]`
+
 ## Basic Syntax
 
 Each basic unit is a line, operands must be in the same line as the mnemonic/directive, 
 one exception to this rule is an instruction preceded by a label, as it is perfectly valid. 
 
 Comments start with ';', only single line comments are supported.
 
-There's one TODO left, relative `Floating constant` source addressing with the address register.
-
 ### Directives
 
 `.entry <label> <shader> [parameters]`

src/fmt.zig

@@ -5,3 +5,22 @@ pub const @"3dsx" = @import("fmt/3dsx.zig");
 
 pub const zpsh = @import("fmt/zpsh.zig");
 pub const z3ds = @import("fmt/z3ds.zig");
+
+pub fn fixedArrayFromSlice(comptime T: type, comptime n: usize, slice: []const T) [n]T {
+    std.debug.assert(slice.len <= n);
+    var buf: [n]T = undefined;
+    @memcpy(buf[0..slice.len], slice);
+    @memset(buf[slice.len..], std.mem.zeroes(T));
+    return buf;
+}
+
+comptime {
+    _ = firm;
+
+    _ = code;
+    _ = @"3dsx";
+    _ = zpsh;
+    _ = z3ds;
+}
+
+const std = @import("std");

src/fmt/3dsx.zig

@@ -4,6 +4,21 @@
 
 pub const magic = "3DSX";
 
+const Segment = enum {
+    text,
+    rodata,
+    data,
+
+    pub fn fromCodeSegment(seg: code.Segment.Kind) Segment {
+        return switch (seg) {
+            .text => .text,
+            .rodata => .rodata,
+            .data => .data,
+            else => unreachable,
+        };
+    }
+};
+
 pub const Header = extern struct {
     magic: [magic.len]u8 = magic.*,
     header_size: u16,
@@ -47,13 +62,25 @@ pub const MakeOptions = struct {
 /// Asserts that the `text` segment address is the base address and entrypoint, the segments are sequential,
 /// and that the only segment with differing file/memory sizes is `data`.
 pub fn make(writer: *std.Io.Writer, reader: *std.fs.File.Reader, info: code.Info, gpa: std.mem.Allocator, options: MakeOptions) !void {
-    std.debug.assert(info.findNonSequentialSegment() == null);
-    std.debug.assert(info.findNonDataSegmentWithBss() == null);
+    std.debug.assert(info.findNonSequentialPhysicalSegment(.fromByteUnits(zitrus.horizon.heap.page_size)) == null);
 
-    // They may be sorted already but we never know...
-    std.mem.sort(u32, info.relocations.items, {}, comptime std.sort.asc(u32));
+    const segments = info.segments;
+    const header_size: u16 = if (options.smdh != null or options.romfs != null) @sizeOf(Header) + @sizeOf(ExtendedHeader) else @sizeOf(Header);
 
-    var processed_relocations = try processRelocations(info, gpa);
+    const base_address = segments[0].virtual_address;
+    const text_size = segments[0].memory_size;
+    const rodata_size, const data_size, const bss_size = switch (segments.len) {
+        1 => .{ 0, 0, 0 },
+        2 => switch (segments[1].kind) {
+            .rodata => .{ segments[1].file_size, 0, 0 },
+            .data => .{ 0, segments[1].memory_size, segments[1].memory_size - segments[1].file_size },
+            else => unreachable,
+        },
+        3 => .{ segments[1].file_size, segments[2].memory_size, segments[2].memory_size - segments[2].file_size },
+        else => unreachable, // NOTE: Cannot happen
+    };
+
+    var processed_relocations = try processRelocations(info, rodata_size, (data_size - bss_size), gpa);
     defer {
         var it = processed_relocations.iterator();
 
@@ -62,13 +89,6 @@ pub fn make(writer: *std.Io.Writer, reader: *std.fs.File.Reader, info: code.Info
         }
     }
 
-    const header_size: u16 = if (options.smdh != null or options.romfs != null) @sizeOf(Header) + @sizeOf(ExtendedHeader) else @sizeOf(Header);
-
-    const base_address = info.segments.get(.text).?.address;
-    const text_size = info.segments.get(.text).?.memory_size;
-    const rodata_size = if (info.segments.get(.rodata)) |rodata| rodata.memory_size else 0;
-    const data_size, const bss_size = if (info.segments.get(.data)) |data| .{ data.memory_size, data.memory_size - data.file_size } else .{ 0, 0 };
-
     try writer.writeStruct(Header{
         .header_size = header_size,
         .relocation_header_size = @sizeOf(RelocationHeader),
@@ -84,7 +104,7 @@ pub fn make(writer: *std.Io.Writer, reader: *std.fs.File.Reader, info: code.Info
         const executable_end: u32 = @sizeOf(Header) + @sizeOf(ExtendedHeader) + (3 * @sizeOf(RelocationHeader)) + text_size + rodata_size + (data_size - bss_size) + (tot_reloc: {
             var total: u32 = 0;
 
-            for (std.enums.values(code.Segment)) |segment| {
+            for (std.enums.values(Segment)) |segment| {
                 total += @intCast(@sizeOf(Relocation) * processed_relocations.get(segment).items.len);
             }
 
@@ -102,21 +122,19 @@ pub fn make(writer: *std.Io.Writer, reader: *std.fs.File.Reader, info: code.Info
         }, .little);
     }
 
-    for (std.enums.values(code.Segment)) |segment| {
+    for (std.enums.values(Segment)) |segment| {
         try writer.writeStruct(RelocationHeader{
             .absolute_relocations = @intCast(processed_relocations.get(segment).items.len),
             .relative_relocations = 0,
         }, .little);
     }
 
-    var info_rw = info;
-    var segment_it = info_rw.segments.iterator();
-    while (segment_it.next()) |seg| {
-        const segment_relocs = processed_relocations.get(seg.key);
+    for (segments) |seg| {
+        const segment_relocs = processed_relocations.get(.fromCodeSegment(seg.kind));
 
         var patched: usize = 0;
 
-        try reader.seekTo(seg.value.file_offset);
+        try reader.seekTo(seg.file_offset);
         for (segment_relocs.items) |rc| {
             try reader.interface.streamExact(writer, rc.words_to_skip * @sizeOf(u32));
 
@@ -129,10 +147,10 @@ pub fn make(writer: *std.Io.Writer, reader: *std.fs.File.Reader, info: code.Info
             patched += (rc.words_to_skip + @as(usize, rc.words_to_patch)) * @sizeOf(u32);
         }
 
-        try reader.interface.streamExact(writer, (seg.value.file_size - patched));
+        try reader.interface.streamExact(writer, (seg.file_size - patched));
     }
 
-    for (std.enums.values(code.Segment)) |segment| {
+    for (std.enums.values(Segment)) |segment| {
         for (processed_relocations.get(segment).items) |reloc| {
             try writer.writeStruct(reloc, .little);
         }
@@ -147,26 +165,26 @@ pub fn make(writer: *std.Io.Writer, reader: *std.fs.File.Reader, info: code.Info
     }
 }
 
-fn processRelocations(info: code.Info, gpa: std.mem.Allocator) !std.EnumArray(code.Segment, std.ArrayList(Relocation)) {
-    var processed: std.EnumArray(code.Segment, std.ArrayList(Relocation)) = .initFill(.empty);
+fn processRelocations(info: code.Info, rodata_file_size: u32, data_file_size: u32, gpa: std.mem.Allocator) !std.EnumArray(Segment, std.ArrayList(Relocation)) {
+    var processed: std.EnumArray(Segment, std.ArrayList(Relocation)) = .initFill(.empty);
     errdefer {
         var it = processed.iterator();
         while (it.next()) |relocs| {
             relocs.value.deinit(gpa);
         }
     }
 
-    const text, const text_size = .{ info.segments.get(.text).?.address, info.segments.get(.text).?.memory_size };
-    const rodata, const rodata_size = if (info.segments.get(.rodata)) |rodata| .{ rodata.address, rodata.memory_size } else .{ text + text_size, 0 };
-    const data, const data_size = if (info.segments.get(.data)) |data| .{ data.address, data.memory_size } else .{ rodata + rodata_size, 0 };
-    const top = data + data_size;
+    const text, const text_size = .{ info.segments[0].virtual_address, info.segments[0].memory_size };
+    const rodata = std.mem.alignForward(u32, text + text_size, zitrus.horizon.heap.page_size);
+    const data = std.mem.alignForward(u32, rodata + rodata_file_size, zitrus.horizon.heap.page_size);
+    const top = std.mem.alignForward(u32, data + data_file_size, zitrus.horizon.heap.page_size);
 
     const base_addresses: []const u32 = &.{ text, rodata, data, top };
 
     var last_relocation_address: u32 = text;
     var current_base: u8 = 0;
 
-    const relocs = info.relocations.items;
+    const relocs = info.relocations;
 
     // NOTE: relocations are already sorted
     var current_absolute: usize = 0;