fix(linux/pipewire): restore dummy buffer for software encoders and fix buffer ownership

[luanweslley77]

Description

the problem (software encoder + KWin/portal crashes at startup with "Couldn't scale frame"; and SIGABRT during teardown), the cause (regression from PR #5360, which removed the dummy_img allocation and added a destructor with delete[] despite lacking ownership), and what the PR does (restores the dummy buffer with data_owned; adds NV12 support to the software encoder; adds a unit test).

Screenshot

Issues Fixed or Closed

Roadmap Issues

Type of Change

• feat: New feature (non-breaking change which adds functionality)
• fix: Bug fix (non-breaking change which fixes an issue)
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
• refactor: Code change that neither fixes a bug nor adds a feature
• perf: Code change that improves performance
• test: Adding missing tests or correcting existing tests
• build: Changes that affect the build system or external dependencies
• ci: Changes to CI configuration files and scripts
• chore: Other changes that don't modify src or test files
• revert: Reverts a previous commit
• BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

• Code follows the style guidelines of this project
• Code has been self-reviewed
• Code has been commented, particularly in hard-to-understand areas
• Code docstring/documentation-blocks for new or existing methods/components have been added or updated
• Unit tests have been added or updated for any new or modified functionality

AI Usage

See our AI usage policy.

• None: No AI tools were used in creating this PR
• Light: AI provided minor assistance (formatting, simple suggestions)
• Moderate: AI helped with code generation or debugging specific parts
• Heavy: AI generated most or all of the code changes

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[luanweslley77]

Thanks for the review — and agreed on both the destructor ownership and the
fill_img() points. For the dummy image, I dug into what you linked and I think
I can show where the root cause actually is, because it ends up supporting your
preference to keep dummy_img() as a no-op.

The reason kmsgrab and wlgrab work with a no-op dummy_img() is not the dummy
itself — it's that their alloc_img() already hands out a buffer:

// kmsgrab.cpp:1702
img->data = new std::uint8_t[height * img->row_pitch];

// wlgrab.cpp:351
img->data = new std::uint8_t[height * img->row_pitch];

So when validate_config() runs alloc_img() → dummy_img() → convert(),
the image already has valid data. PipeWire's alloc_img() is the odd one out:

// pipewire.cpp:939
img->data = nullptr;

That's why the same no-op dummy breaks software encoding only on PipeWire
captures (kwin/portal): the software encoder's sws_scale_frame() gets a null
pointer and fails with EINVAL. The ValidateEncoder/software gtest reproduces
it — it fails on master and passes with the fix.

To keep dummy_img() a no-op everywhere (which I agree is the right thing for
maintenance), the fix would move into alloc_img() instead: allocate the
buffer there like kmsgrab/wlgrab, and have fill_img()'s memory path copy the
frame into that owned buffer instead of pointing img->data at the staging
vector. That removes the front_buffer aliasing entirely (no heap corruption
by construction), and there's no leak (the buffer is owned by the image,
recycled through the pool, and freed by the destructor).

One trade-off to be transparent about: that means one extra memcpy per frame
in the memory-buffer path (the staging copy already exists in on_process).
For 1080p60 NV12 that's roughly 180 MB/s of additional copying — a couple
percent of CPU on software encoding. If that cost is acceptable, I'll rework
the PR along those lines. If you'd rather keep the allocation in
dummy_img() as it is now (which doesn't add per-frame copies and doesn't
leak), the current version already addresses the crash on master.

---

[Kishi85]

Thanks for the review — and agreed on both the destructor ownership and the fill_img() points. For the dummy image, I dug into what you linked and I think I can show where the root cause actually is, because it ends up supporting your preference to keep dummy_img() as a no-op.

The reason kmsgrab and wlgrab work with a no-op dummy_img() is not the dummy itself — it's that their alloc_img() already hands out a buffer:

// kmsgrab.cpp:1702
img->data = new std::uint8_t[height * img->row_pitch];

// wlgrab.cpp:351
img->data = new std::uint8_t[height * img->row_pitch];

So when validate_config() runs alloc_img() → dummy_img() → convert(), the image already has valid data. PipeWire's alloc_img() is the odd one out:

// pipewire.cpp:939
img->data = nullptr;

That's why the same no-op dummy breaks software encoding only on PipeWire captures (kwin/portal): the software encoder's sws_scale_frame() gets a null pointer and fails with EINVAL. The ValidateEncoder/software gtest reproduces it — it fails on master and passes with the fix.

To keep dummy_img() a no-op everywhere (which I agree is the right thing for maintenance), the fix would move into alloc_img() instead: allocate the buffer there like kmsgrab/wlgrab, and have fill_img()'s memory path copy the frame into that owned buffer instead of pointing img->data at the staging vector. That removes the front_buffer aliasing entirely (no heap corruption by construction), and there's no leak (the buffer is owned by the image, recycled through the pool, and freed by the destructor).

One trade-off to be transparent about: that means one extra memcpy per frame in the memory-buffer path (the staging copy already exists in on_process). For 1080p60 NV12 that's roughly 180 MB/s of additional copying — a couple percent of CPU on software encoding. If that cost is acceptable, I'll rework the PR along those lines. If you'd rather keep the allocation in dummy_img() as it is now (which doesn't add per-frame copies and doesn't leak), the current version already addresses the crash on master.

Thanks for clarifying the differences. I wasn't aware that both kmsgrab and wlgrab handle the alloc earlier already.

In that case I'm good with your PR changes on pipewire.cpp the way they are right now. No need to rework that especially when that would add unnecessary overhead.

Also feel free to mark my earlier comments as resolved (for some reason github won't allow me to do that myself due to missing permissions).

P.S.: I'll apply this PR to my personal setup build and do some testing

UPDATE: No issues on my system running this so far.