feature: Allow use of secondary stream for analysis/motion detection#4965
feature: Allow use of secondary stream for analysis/motion detection#4965madcamel wants to merge 4 commits into
Conversation
…Secondary)
Add a per-monitor SecondStreamThread sidecar that decodes the configured
SecondPath substream in its own thread and publishes each frame as a
motion-analysis image. When AnalysisSource=Secondary and a SecondPath is set,
Analyse() scores against the substream instead of the primary, and the primary
is decoded on-demand (viewer-driven) rather than always, so the full-resolution
stream is not software-decoded while nobody is watching live.
- zm_second_stream_thread.{h,cpp}: sidecar owning its own FFmpeg_Input, with
reconnect backoff (which also keepalives the substream RTSP session) and a
latest-image mailbox holding the frame at native substream resolution.
- FFmpeg_Input::Open overload taking an AVDictionary so the sidecar can set
rtsp_transport and a socket read timeout.
- Monitor::getMotionSourceImage() resolves the analysis image (substream mailbox
vs primary Y/colour) and upscales the substream to camera dimensions only when
a frame is actually scored, so zones keep working in primary coordinates with
no changes. effective_decoding downgrades DECODING_ALWAYS to on-demand under
secondary analysis. Sidecar lifecycle wired into PrimeCapture()/Pause().
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FFmpeg_Input::Open() selects a hardware decoder whenever get_decoder_data returns one (VAAPI, etc.) and get_frame() returns the frame without transferring it to system memory. The substream sidecar then received an AV_PIX_FMT_VAAPI frame whose data[0] is a GPU surface handle, and ProduceImage() dereferenced it as a CPU Y plane, segfaulting zmc when VAAPI was enabled. Add FFmpeg_Input::set_no_hwaccel() to skip hardware decoders (default off, so other callers are unchanged) and use it in the sidecar: the D1 substream is cheap to decode on the CPU and this also avoids contending with the primary over the VAAPI device. ProduceImage() additionally rejects any hardware frame instead of dereferencing it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ilbox copies - The RTSP read timeout was set only via `stimeout`, which ffmpeg 5.0 renamed to `timeout`; on the target's ffmpeg 7.x the option was ignored, so a stalled substream could block av_read_frame indefinitely and hang shutdown in Join(). Set both option names. - getMotionSourceImage() called GetLatestImage() every analysed packet, deep copying the mailbox image under the mutex even when the frame had not advanced and the copy was discarded. Add SecondStreamThread::PeekLatest() to check sequence/age cheaply and only copy when the frame will actually be used. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The substream images were built with the Image(w,h,colours,subpix) constructor, which sets linesize = width*colours (unaligned). But Image::Scale and all of ZM's swscale paths treat buffers as align-32: SWScale::Convert fills the source frame with av_image_fill_arrays(..., 32), i.e. stride FFALIGN(width,32). For a width that is not a multiple of 32 (e.g. a 720-wide D1 substream) the scaler read each row at the aligned stride while the data was width-packed, drifting every row and shearing the analysis image into horizontal bands. Build the images with the align-32 constructor (buffer=nullptr, allocation=0) and copy the decoded Y plane with av_image_copy_plane, which bridges the decoder's stride (frame->linesize[0]) to the image's align-32 stride. Applies to both the YChannel and FullColour paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| // that DetectMotion and the reference image match the primary coordinate | ||
| // space and zones need no changes. | ||
| secondary_image.Assign(secondary_image_native); | ||
| if ((secondary_image.Width() != static_cast<unsigned int>(camera_width)) or |
There was a problem hiding this comment.
It doesn't make sense to upscale. Only makes sense to downscale. Zone coordinates are percentages now. Motion detect results scan be scaled to match either image.
Also, we should still honour y-image for motion detection.
There was a problem hiding this comment.
by that I mean we can use the y-image on the secondary stream for even LESS cpu.
| // full-res primary. When active, the primary is only decoded on demand (for | ||
| // live viewers), so DECODING_ALWAYS is downgraded to viewer-driven decoding. | ||
| secondary_analysis = (analysis_source == ANALYSIS_SECONDARY) and !second_path.empty(); | ||
| effective_decoding = decoding; |
There was a problem hiding this comment.
What if we are encoding in videostore? This isn't enough. We should honour the setting. If the user wants decoding=ondemand, they can set that. We will just always decode the secondary.
|
THanks for taking this on, we have been slowing working towards it but you have jumped right in, which should force us to move faster. That being said, the reason it was taking so long is it is a bit more complicated than this initial effort addresses. I have concerns about doing the open and capture outside of the Camera object. Not opposed to it being it's own thread, but I think it should be injecting into the packet queue. I'm not convinced that this implementation will stay synced between the streams. We need to sync and link the two streams, likely in wallclock units instead of pts/dts. And now I will sic the AI on it. |
There was a problem hiding this comment.
Pull request overview
Wires the existing AnalysisSource=Secondary setting to actually analyze a monitor’s configured SecondPath substream, reducing CPU use by avoiding full-resolution primary-stream decoding when no live viewers are present.
Changes:
- Adds
SecondStreamThreadto decode the secondary/substream in a sidecar thread and publish the latest analysis-ready frame. - Updates
Monitor::Analyse()to source motion detection input from the sidecar whenAnalysisSource=Secondary, and introduceseffective_decodingto make primary decoding viewer-driven in that mode. - Extends
FFmpeg_Inputwith anOpen(..., AVDictionary**)overload plus a “force software decode” option to support the sidecar’s RTSP/timeouts and non-HW decode needs.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/zm_second_stream_thread.h | Declares the sidecar substream decode thread and its mailbox-style image accessors. |
| src/zm_second_stream_thread.cpp | Implements substream open/reconnect/backoff, frame conversion, and mailbox publishing. |
| src/zm_monitor.h | Adds secondary-analysis state, sidecar ownership, and getMotionSourceImage() API. |
| src/zm_monitor.cpp | Routes motion detection through the secondary stream when configured; adjusts decode policy via effective_decoding; manages sidecar lifecycle. |
| src/zm_ffmpeg_input.h | Adds Open(filename, AVDictionary**) and a flag to force software decoding. |
| src/zm_ffmpeg_input.cpp | Plumbs AVDictionary options into avformat_open_input and skips HW codecs when software decode is forced. |
| src/CMakeLists.txt | Adds zm_second_stream_thread.cpp to the build. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Decodes a monitor's low-res analysis substream (SecondPath) in its own thread | ||
| // and publishes the latest frame as an analysis-ready Image, scaled/converted to | ||
| // the primary camera dimensions. Used when AnalysisSource=Secondary so that | ||
| // motion detection does not require software-decoding the full-resolution | ||
| // primary stream while nobody is watching live. |
| Image latest_image_; | ||
| bool have_image_; | ||
| uint64_t sequence_; | ||
| SystemTimePoint capture_time_; |
There was a problem hiding this comment.
Just to add some further info here: If we use a steady clock, we should start it at capture, and then base timestamps for both streams off that start point.
| Stop(); // Signal any running thread to terminate first | ||
| if (thread_.joinable()) thread_.join(); | ||
| terminate_ = false; | ||
| thread_ = std::thread(&SecondStreamThread::Run, this); | ||
| } |
| int dummy[4]; | ||
| int in_range, out_range, brightness, contrast, saturation; | ||
| sws_getColorspaceDetails(convert_context_, reinterpret_cast<int **>(&dummy), &in_range, | ||
| reinterpret_cast<int **>(&dummy), &out_range, | ||
| &brightness, &contrast, &saturation); |
| if (StringToUpper(url.substr(0, 4)) == "RTSP") { | ||
| // Bound socket reads so a dead substream is detected (and the thread can be | ||
| // joined on shutdown) instead of blocking forever in av_read_frame. The | ||
| // RTSP option was renamed stimeout -> timeout in ffmpeg 5.0; set both so the | ||
| // bound applies regardless of the linked ffmpeg version. |
| return Open(filepath, nullptr); | ||
| } | ||
|
|
||
| int FFmpeg_Input::Open(const char *filepath, AVDictionary **options) { |
There was a problem hiding this comment.
I think we could just use AVDictionary **options = nullptr in the declaration to avoid two ::Open functions.
|
Thanks for the review and feedback. I'm just flying by the seat of my pants, I'm not very familiar with zm's non-perl code. To be frank I'm avoiding the packet queue because that would add a lot of complexity and touches a lot of things I don't understand. It's probably over my head. I will make a decent attempt at addressing the other issues you've raised as time permits, it would be really nice to have this functionality in mainline. |
It's really hot outside and my NVR was melting down to zmc CPU usage. Here's a fun fix with some help from the slop machine. This is an absolutely massive improvement, my machine went from flamethrower to mostly idle.
-snip-
What
Uses SecondPath (should be configured as a lower resolution stream from the camera) for frame analysis when AnalysisSource is set to Secondary.
AnalysisSource already exists in the schema and UI but was a no-op. This wires it up.
Why
In Modect,
zmcsoftware-decodes the entire primary stream just to feed motiondetection, even with no live viewers. On a 5MP/10fps monitor that is roughly a
full CPU core (measured: ~59% libavcodec + ~13% libswscale). Most cameras
already publish a low-res substream; analysing that instead cuts idle decode
cost, while motion events still record the primary at full resolution.
How it works
SecondStreamThread(one per monitor, started only whenSecondPathisset and
AnalysisSource=Secondary) owns its ownFFmpeg_Inputon thesubstream, reconnects with exponential backoff, and publishes each decoded
frame into a mutex-guarded mailbox at the substream's native resolution. It
never enqueues packets into the packetqueue, so the single-video-stream
invariants there are untouched.
Monitor::getMotionSourceImage()feedsAnalyse(). In secondary mode itreturns the mailbox image, upscaled to camera dimensions only when a frame is
actually scored (a cheap sequence/age peek gates the copy + upscale). Zones
stay in primary coordinates and need no changes, because the analysis image is
scaled to the same
camera_width x camera_heightthe primary path alreadyproduces.
AnalysisImagesetting (YChannel vs FullColour) drives whetherthe sidecar extracts the Y plane or swscales to colour.
effective_decodingdowngradesDECODING_ALWAYSto on-demand while secondaryanalysis is active, so the primary decodes only for live viewers. Alarm/event
flow is unchanged: scores attach to the primary packet, so pre-event buffering
and recording use the primary as before.
FFmpeg_Input::set_no_hwaccel()): theD1 substream is cheap on the CPU, hardware-format frames cannot be consumed by
the mailbox, and this avoids contending with the primary over the VAAPI device.
FFmpeg_Inputgains anOpen(filename, AVDictionary**)overload so thesidecar can pass
rtsp_transportand a socket read timeout (timeoutforffmpeg >= 5.0,
stimeoutfor older). The existing single-argumentOpen()delegates to it unchanged.
Interaction with existing
SecondPathuseSecondPathis already consumed elsewhere, so reviewers should be aware:FfmpegCameraopens the substream as an audio source (mSecondInput) whenthe primary has no audio stream; go2rtc and RTSP2Web restreaming also open it.
AnalysisSource=Secondaryand no primary audio,zmcwill hold twoconnections to the substream (audio + analysis); with restreaming enabled the
camera may see more. This is additive load only — the sidecar owns a separate
FFmpeg_Inputand shares no state withmSecondInput, so the existingteardown paths are unaffected. Cameras with tight session limits are the
constraint here.
Behaviour when the substream is unavailable
no scoring, no forced state change, no alarm storm while blind.
which also keepalives the RTSP session so it does not sit in CLOSE-WAIT.
Upgrade note
Because
AnalysisSourcewas previously a no-op, any monitor already set toSecondaryin the database will change behaviour after upgrade: it will beginopening and analysing the substream. A monitor with a stale or misconfigured
SecondPathwill log reconnect errors and report "no motion data" rather thananalysing the primary. Worth a mention in release notes.
Testing
cmake --build . --target zmc), no warnings.Limitations
primary dimensions is blocky). Zone areas and pixel-count thresholds stay
valid because the analysis image is at primary resolution.
it; zones still function.