Screen size error (Assertion tlPixelBuffer.cc:260 n < m_height was not true)

[martinjankoehler]

as found by @mgolser

Context

• OpenSUSE amd64 host (single laptop display 1920x1200)
• Docker using IIC-OSIC-TOOLS/next with X11 client

Steps to reproduce

• Open KLayout
• Make window tiny
• Close KLayout to save window size setting
• Open KLayout
• Menu action New Layout, press OK
• EXPECTED: now layout to appear
• OBTAINED: dialog with fatal error

ERROR: ../../../src/tl/tl/tlPixelBuffer.cc,260,n < m_height
ERROR: Internal error: ../../../src/tl/tl/tlPixelBuffer.cc:260 n < m_height was not true
ERROR: Signal number: 11
Address: 0x5612489130d0
Program Version: KLayout 0.30.7 (2026-04-10 r6dd3e6b1)

Backtrace:
/foss/tools/klayout/libklayout_lay.so.0 +0x30e1ff lay::enable_signal_handler_gui(bool) [??:?]

[klayoutmatthias]

How do you always manage to trigger these bugs? :(

I tried to reproduce with IIC-OSIC-TOOLS, "next_release" from April 10th. It gives me KLayout 0.30.6 and this is the smallest Window I can produce:

With this, I don't see this crash. I assume it's a sign overflow or something (negative view port size). But a stack trace was helpful.

Matthias

[klayoutmatthias]

I think I was able to reproduce and fix the assertion - although it did not crash for me.

The commit is this: 5ccf626

I have placed it into master already since I want to release tomorrow. The effect was in the code that was drawing the ruler - if the ruler became larger than the image size, this assertion happened. However, that just triggered an exception, not a segfault, but maybe the behavior is different on other systems.

We will see if that patch fixes the problem :)

Matthias

[mgolser]

This issue also reproduces with the latest Docker image (IIC-OSIC-TOOLS/2026.04) and KLayout 0.30.8.

I was able to narrow down the root cause, the bug only occurs at a display scaling factor of 125%. At all other scaling factors tested, KLayout behaves correctly.

[klayoutmatthias]

Thanks a lot. With that info I was able to reproduce the problem under these conditions:

• Fractional scaling 125% and
• Oversampling with factor 3

It's not difficult to fix. There is a rounding bug under those conditions.

Matthias