Preload en locale in __locales__/Component.ts files, and allow building MathJax with any locale as default.

[dpvc]

Note: while there are a lot of files touched by this PR, most come in batches that all work the same (e.g., removal of imports of en.js from the component definitions, and deletion of en.js itself, or addition of preload.cts files to the __locales__ directories, plus modifications to Component.ts files to import preload.cjs and changes to component config.json files to exclude copying preload.cts to the bundle directories. There are 30 __locales__ directories, so this changes are repeated 30 times each, leading to 150 file changes just for these changes alone, the vast majority of the files changed here.)

This PR updates how the English locale is included into MathJax by default. In an earlier PR, this was done through the component files, so worked for in-browser use, and for node applications that used components, but not for those that used direct importing of the MathJax modules.

This PR moves the loading of English localizations into the __locales__/Component.ts files themselves, so that even direct loading of the modules will have the English messages available by default. Applications can still use Locale.setLocale() to change the locale and load the appropriate message files.

Because import ... with {type: 'json'}; reports an error for our CJS compilation configuration, I needed to work out an arrangement that works for both MJS and CJS files. To do that, I added preload.cts files in the __locales__ directories, which use require() to load the en.json file and export that data back to the Component.ts files. They also export he en string, and Components.ts uses that to set the initial messages. This makes it possible to have webpack replace the en.json and en string by any of the other locales so that you can build a vision of MathJax that preloads a different locale (without having the English). The build scripts are modified to allow the locale to be specified, so it is easy to build MathJax with a different default locale.

That means we could potentially make packages like mathjax-de, mathjax-fr, etc., that have the components built with each of the available languages. Then a web page could load https://cdn.jsdelivr.net/npm/mathjax-de@4/tex-chtml.js to get a version with German localization that does not include the English messages. Because the messages are webpacked into the components, this would decrease the number of files needed for a German page, as otherwise, all the German localization JSON files would need to be loaded individually for each component. So that might be a useful thing to provide (though then a release would need to build a dozen separate packages).

Details

The components/bin/makeAll and components/bin/pack files are modified to handle passing the locale.

The various component config.json files are modified to exclude the preload.cts file from being copied to the bundle __locales__ directories.

The original en.js files from the components have been removed, and the definition files modified to remove the importing of en.js, since that is now done in the ts modules directly. That accounts for a large number of the changes in this PR.

The messages from ts/util/Locale.ts are moved to a new ts/util/Locale/__locales__ directory, and a new locale component is created just to do the copying of those files into the bundle directory.

The components/wepack.common.cjs file is modified to handle changing the default locale. It does this by performing a string substitution on the preload.cjs files to convert en to whatever locale is requested. This changes the en string that is passed back to Component.ts (so that the proper locale is initialized) as well as the en in en.json so that the proper messages are loaded. The Components.ts file doesn't really know which locale is being loaded, so whichever one it ends up being is OK with it.

The other webpack.config.* files are modified to pass the local on to the main webpack common file.

Because the preload.cts files use require(), which is not allowed in the lab, a new lab/preload.js file that returns an empty locale and empty message data object is added so that the lab can redirect the __locales__/preload.cjs files to lab/preload.js, this avoiding the errors that would occur from trying to load the actual preload.cjs files. Because the lab includes a Locale.setLocale() call, it will end up loading the needed localizations itself, so there is no problem with undefined messages in the lab. The only complication is that you do need to redirect every preload.cjs file to lab/preload.js in the lab's importmap script. So you need to add

    "./mjs/a11y/aria/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/a11y/explorer/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/a11y/semantic-enrich/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/a11y/speech/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/adaptors/linkedom/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/core/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/mathml/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/ams/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/base/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/bbox/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/begingroup/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/bussproofs/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/cases/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/color/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/colortbl/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/empheq/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/extpfeil/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/html/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/mathtools/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/mhchem/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/newcommand/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/physics/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/require/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/setoptions/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/textmacros/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/unicode/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/input/tex/verb/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/ui/menu/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",
    "./mjs/util/Locale/__locales__/preload.cjs": "./node_modules/@mathjax/src/lab/preload.js",

to the import map, and if any new components get locales, those would have to be added as well. There doesn't seem to be any way to use wildcards in the import map, so you need an individual line for each preload file. Sigh.

The package.json file is modified to pass the locale information on as needed, and a few additional -s options are added to prevent output from some commands when using current versions of pnpm which seem to have changed how output is produced from nested scripts. it also loads the string-replace-loader dependency that is needed for the web packing of alternate locales.

You can now do

pnpm -s build <locale>

to build all of MathJax with a given locale, or

pnpm -s make-mjs-components <locale>

to just webpack everything with the given local, or

pnpm -s make-one <component> mjs --locale <locale>

to build a specific component with the given locale. We don't usually build the cjs bundle, but you can do that with

pnpm -s make-cjs-components --locale <locale>

to pack with a specific locale.

Some tests were modified to accommodate the changes or add coverage.

The various __locales__/Component.ts files were modified to load the locale and message data from __locales__/preload.cjs, and the preload files were added. This is another big chunk of this PR.

The default locale in ts/a11y/semantic-enrich.ts is initialized by the Locale.default rather than an explicit 'en', so that if MathJax is webpacked with another default locale, that will be used here instead. Similarly in the ts/a11y/speech/SpeechUtil.ts and ts/ui/menu/Menu.ts files.

The ts/utile/Locale.ts file is modified to move its messages into separate locale files in ts/util/Locale/__locales__, so that it operates the same as other components, without needing to have its messages for all the locales be hard-coded into it, and so that MathJax can be built with other default locales using the same mechanism as with other components. We don't use a Component.ts file here, however, because that file calls Locale.registerLocaleFiles() and Locale.registerMessages(), and these can't be called before the Locale class is defined, so we could not import Component.ts in Locale.ts. The actions that are usually performed in Component.ts are done in Locale.ts instead.

In addition, there are some small changes made to not overwrite the component data if a component is registered twice (as the a11y/aria component may be, if the a11y/explorer and ui/menu components are loaded separately rather than as part of a combined component), and to produce an error message if messages are being registered to a component that hasn't been defined.