Bootstrap JSCL

[davazp]

Compile JSCL with itself!

Lot of changes were needed. The PR is not the cleanest. But it's in a good state, workflow wise.

The build scripts now support both SBCL and JSCL as host compilers:

• ./make.sh — Cross-compiles JSCL using SBCL (default) or a JSCL binary (--jscl=). Produces dist/jscl.js and REPL variants.
• ./run-tests.sh — Runs the test suite in JSCL (default) or SBCL (--sbcl).
• ./bootstrap.sh — Two-stage bootstrap: SBCL builds a stage-0 JSCL (out/), then that JSCL builds the final stage-1 (dist/). CI runs tests against
  both stages.

Key changes

• Introduce jscl-xc package and jscl-target feature flag to cleanly separate host/target code paths
• Test suite moved to jscl-tests package and made portable across JSCL and SBCL (JSCL-specific tests gated with #+jscl)
• Bug fixes: BOA constructor supplied-p parameters, read-from-string second return value, defstruct type/predicate, set-dispatch-macro-character,
  various loop/sort/package fixes

[kchanqvq]

Thanks for the write up! This clears up a misunderstanding from me: I thought using JSCL instead of other CL to compile JSCL is rather an exercise on conformance than bootstrap (because any conforming CL should be able to compile JSCL). But turns out it is indeed an exercise on bootstrap because we need to handle clashing between host/target JSCL package, etc.

Introduce a feature #+jscl-target. Equivalent to our existing jscl-xc. But I wanted to avoid xc (crosscompile). Because from jscl -> jscl is not technically cross compiling, but bootstraping. Anyway. I can rename it later.

I think xc is fine. I think technically this is usually still cross compiling because we will be compiling from a different (earlier) version of JSCL. Anyway, SBCL use sb-xc.

Macro readers

At some point we need to overhaul the reader, but I repeat myself :) A conforming reader is required to be thoroughly read-table based.

[vlad-km]

Any suggestions for compiling on windows?

[davazp]

Any suggestions for compiling on windows?

What are the main limitations now? the lack of shell? I guess WLS or just doing the build in sbcl directly like

$ sbcl
(load "lisp.js")
(jscl-xc:bootstrap "...") 

Or is there any additional limitations? I imagine paths might be quite broken now. But something we should fix.

Let's think about it when I open a PR to improve the build system to allow specify jscl as a host.

[davazp]

A mismatch in bootstrapping was caused by this

(read-from-string "|toString|") => <#FUNCTION>

Kind of funny! Thank you JS prototype chain. Prelude.js was good but packages created from package.lisp weren't.

Probably good to migrate packages to use hash tables /map too at some point.

[davazp]

Claude was almost completely useless for these bootstrapping issues. Can't grasp the root cause of the issues ever. It always suggests to add #+jscl-target when loading jscl.lisp.

It only helped me to compare jscl-xc.js and jscl.js, looking at the huge diff and categorizing what kind of differences are there.

It makes it more fun though. It's quite cool to hack inside out/jscl-xc-node.js vs sbcl vs dist/jscl-node.js and think of code to trigger differences and examine the state.

[davazp]

Ok! jscl.lisp is compiling itself now 🎉

There were quite a few hacks. Especially defstruct.lisp vs structures.

Many tests are still failing

1824/1873 test(s) passed successfully.

but it's at the point where I build jscl from jscl and I get a pretty close jscl.js and a functioning REPL.

It might be a lot of effort to recreate individual PRs. Maybe it's better to refine this one and fix or create issue to solve all the hacks that were needed.

[kchanqvq]

It might be a lot of effort to recreate individual PRs. Maybe it's better to refine this one and fix or create issue to solve all the hacks that were needed.

Does it make sense to split out the part that implements functionality (i.e. improve conformance, like the time related functions) from the part that make JSCL-from-JSCL work? It looks like there are quite a few such things, which we can merge cleanly to get immediately more functionalities.

I think conceptually the following goals are in order:

1. Build JSCL from any other mostly conforming CL (done).
2. Make JSCL a mostly conforming CL.
3. Build JSCL from JSCL.

Improvements in 2 will probably reduce hacks needed in 3. And it does seem you completed some tasks in 2 in order to make 3 work! I think if we separate them out and consider it under premise of 2, it's easier to review and produce good code. No hacks or magic required in 2!

[davazp]

Let me work on rebasing the PR and try to clean up the commits a bit to try to match that order, and make sure tests (at least the --prebuilt one as default) work.

Some of my fixes can be merged but others are very hacky, def!struct and reader for instance. And I'd prefer if we do them properly.

I'll try to extract the good stuff from it.

[kchanqvq]

I took a peek, one reminder: be aware that if JSCL host uses some JSCL specific feature (rather than using it as "any other CL"), this is less "sanely" bootstrappable, because future changes in these feature might break bootstrap.

[davazp]

I took a peek, one reminder: be aware that if JSCL host uses some JSCL specific feature (rather than using it as "any other CL"), this is less "sanely" bootstrappable, because future changes in these feature might break bootstrap.

I know! Thanks.

This is my plan for now: I'm just trying to get one clean bootstrap + tests running. Just to make sure the approach is doable. All hacks allowed.

Then I'll open a PR for the package + feature only.

Then we can properly fix the the missing features. We can divide some work if you have time. And I'll incrementally rebase this PR to make sure I can still get tests passing, until they do pass in master, and we can make them a required step in the CI.

[kchanqvq]

This is my plan for now

Great plan! Thanks for the hard work in this PR :)

[davazp]

Ok! I'm happy where this is now.

Here is the status:

• ./make.sh and ./run-tests.sh does the same as in master, and all tests pass.

The bootstrap

./bootstrap.sh does build jscl with sbcl in out/, and uses this to build the regular dist/ with make.sh --jscl=out/jscl-node.js.

It mostly works, but it fails some web REPL build later. I haven't tried to fix that. As I do successfully end in dist/ with jscl.js, jscl-node.js and jscl-tests.js, more than enough for now.

out/jscl.js and dist/jscl.js differ by a diff of 600 lines (out of 60_000, not bad!). I'll attach a summary below of the differences.

The tests

The "prebuilt" tests (built during bootstrap works). I prioritized these, because it's what we have in master now.

But I changed the test suite to be usable at runtime too. My goal is to run tests.lisp as a regular CL application. Usable in SBCL (maybe subset of tests), as well as out/jscl.js and dist/jscl.js too.

It's a bit fragile, a bit of a mess with the packages. We'll have to fix it, probably puttin all tests in a new jscl-tests package instead of jscl or jscl-xc.

But if I get inside the host jscl package, then they do work:

$ node dist/jscl-node.js

CL-USER> (in-package :jscl) ;; hack
CL-USER> (load "tests.lisp")
CL-USER> (run-tests)

; ...

Finished. The execution took 9.193 seconds.
1775/1776 test(s) passed successfully.

Everything is quite slow. But it'll be fun to optimize later.

So this is a great branch to have as a reference as we go. I'll try to open the PRs. Let's try to not diverge too much from this one. It's pretty hard to debug issues. It'll get better once we have the full ./bootstrap.sh passing.

[davazp]

Claude analysis on diff out/jscl.js dist/jscl.js:

Differences between out/jscl.js and dist/jscl.js

1. Floating-point precision

Constants are full double-precision in dist/ but single-precision truncated in out/:

• pi: 3.1415927 vs 3.141592653589793
• e: 2.718282 vs 2.718281828459045
• 10.0 / 1.0 / 1000.0 → 10 / 1 / 1000 (integer literals instead of floats in a few places)

2. Symbol naming for format directive expanders

The biggest category of changes. In out/, format directive names use the raw character (e.g., A-FORMAT-DIRECTIVE-EXPANDER, $-FORMAT-DIRECTIVE-EXPANDER). In dist/, they use Unicode character names (e.g., LATIN_CAPITAL_LETTER_A-FORMAT-DIRECTIVE-EXPANDER, DOLLAR_SIGN-FORMAT-DIRECTIVE-EXPANDER).

This affects ~30 format directives: A, S, C, W, D, B, O, X, R, P, F, E, G, T, I, $, %, &, |, ~, _, *, ?, (, ), [, ], ^, {, }, <, >, /, ;.

This means dist/ is using char-name to print characters in symbol names, while out/ is printing the character directly. The out/ behavior is correct — the symbol name should contain the literal character, not the Unicode name.

3. Package-qualified dump-loop-* functions

Four loop-related dump functions have different package qualifiers:

• out/: DUMP-LOOP-MINIMAX, DUMP-LOOP-UNIVERSE, etc. (in JSCL-USER)
• dist/: JSCL-XC::DUMP-LOOP-MINIMAX etc. (package-qualified with JSCL-XC)

The JS function names also reflect this (JSCL_USER_DUMPLOOPMINIMAX vs JSCL_USER_JSCLXCDUMPLOOPMINIMAX).

4. Gensym counter offset

Gensym counters are consistently offset (e.g., G66049 vs G66118, G73996 vs G74066). This is a ~69 count shift, likely from more gensyms being generated during compilation in the dist/ build.

5. Version string

• out/: 'dev-1ec25eb5' (git-derived dev version)
• dist/: 'stage1-3979836526' (stage1 build identifier)

6. Build date string

• out/: 'built on 11 February 2026'
• dist/: 'built on NIL #j:undefined NIL' — the date is broken in the dist/ build, suggesting get-decoded-time or the date formatting code isn't working correctly during that build phase.

7. One actual code difference

Line 51456: v15876 vs l3.value — a variable reference changed from a local to a global lookup. This looks like it could be a meaningful behavioral difference in format directive processing (inside the </justification handler).

[kchanqvq]

Finished. The execution took 9.193 seconds.
1775/1776 test(s) passed successfully.

Everything is quite slow. But it'll be fun to optimize later.

Interesting, why would the version loaded in target slower than cross-compiled one? Or do even the cross-compiled one take 9.193 seconds? I think they should be mostly identical.

[davazp]

It’s loading and running tests so they include compilation time.

comoilation from inside sbcl is much faster but the running time only is identical I think. I’ll double check tomorrow.

[davazp]

I'm not sure how to proceed with the plan for this.

Splitting this PR as I did for bootstrap 01 but not making changes to them is ab it pointless. But it's going to take quite some effort to rebase and keep everything working and up to date.

It might just be easier to polish this PR enough to get it merged.

[davazp]

@kchanqvq ok. I got it to finally cleanly bootstrap itself and pass all tests. Please check the updated description of the PR.

Time to go through the diff and find all the hacks that were needed, and see which ones are easy to fix!

I'll go through it myself tonight or tomorrow. If you have time to have a look, or checkout the branch locally and try the updated scripts or give any other feedback, it'd much appreciated.

[davazp]

Hm. We do not have HTML tests right now!

I think it's nice to have these to make sure we do not use things browser do not support.

I have to think of a solution.

[davazp]

I think I'll merge this if there is no big concerns with it, and keep working on recovering the old tests.html file and a few other fixes later.

The PR is already huge.

[kchanqvq]

Hmm, I thought I spotted some possible improvements but missed the deadline for review! Anyway, I'll start separate issues/PRs once I get to it. A bit busy this week :)