Add support for hebrew translation

[BLooperZ]

Another attempt at supporting hebrew translation following #7512
This attempt is based on the new BiDi library (SheenBiDi) added by @glebm (Thanks!)

Few notes:

• I haven't found documentation and example of using SheenBiDi, but the AI generated code in unicode-bidi.cpp seems to work well
• The most prominent change (and enabler) is having the text drawn line by line, it seems to work correctly but I would like more eyes on it, should be tested to not affect english, mostly cursor and highlights
• The hebrew translation itself is only the main menu at the moment, I can exclude it until translation is complete if you so wish
• I should open another PR to devilutionx-assets with the fonts
• BiDi behaviour for the cursor and highlights are not yet implemented, I did came up with a neat trick to support BiDI cursor while experimenting few years ago for another project, hope the same method will fit here as well, need to research more regarding highlights, but we should make sure they still work with LTR language before even starting.

Thank you

[glebm]

I think the AI-generated code here went for a rather complicated solution.

Instead, the rendering can be done directly without any text copying or extra work, which is a very nice feature of SheenBidi.

You can do this by iterating over the "runs" (SBParagraphCreateLine, SBLineGetRunsPtr, SBLineGetRunCount).

Within a run, all characters have the same direction, which can be determined by looking at whether the run_level field of the run is even. Then, for reverse runs, you can use SBCodepointDecodePreviousFromUTF8 to get the code point in the reverse direction.

You can look at other projects using SheenBidi (many mentioned in SB's readme here https://github.com/Tehreer/SheenBidi) for inspiration but please be careful with the license if copy-pasting code from other projects.

SheenBidi lacks documentation but it implements the Unicode Bidirectional Algorithm precisely and uses the same names for things as in the algorithm description, so reading this document should give you a pretty good idea of how to use SB: https://unicode.org/reports/tr9/

[glebm]

It'd be nice to also add tests to text_render_integration_test (including a test with highlighting).

For creating the Hebrew fonts, if we don't already have them, I have a basic font converter here https://github.com/diasurgical/devilutionx-font-converter, with an example of how to use it here diasurgical/devilutionx-assets#18

[AJenbo]

We currently only have hebrew (font page 0x05) in the small font used on error messages and the credit screen. the upside is that we don't have a renderer script for that so it does allow use to pretty easily create a fonteset for Hebrew.

Thaana and N’Ko should also work once we get this PR merged (font page 0x07) .

For other RTL scripts we would also need support for shaping in the renderer so probably it would be to early to worry about generating the font pages (0x06, 0x07, 0x08) for them.

[BLooperZ]

@AJenbo Great, I drew hebrew fonts back then but I just the larger ones (missing only the 2 smallest sizes)

[AJenbo]

Fun fact, emojis also require shaping, and Google kind of require this feature for chat applications.

[glebm]

The good news is that the converter from diasurgical/devilutionx-assets#18 works pretty well for large font sizes. It doesn't work that well for small font sizes yet.

[BLooperZ]

I opened PR for the fonts: diasurgical/devilutionx-assets#21

[AJenbo]

diasurgical/devilutionx-assets#23

[AJenbo]

I added the Hebrew font so if you update your PR it should now render correctly ingame.

[strich]

Hey @BLooperZ, just checking in on the status of this PR. Are you still planning to get it in? I'd love to see this move forward!

[BLooperZ]

yes, trying to figure out cursor behaviour with BiDi