fix(product): defer list count annotations - #15447
Conversation
Signed-off-by: Emre Koca <110906681+kocaemre@users.noreply.github.com>
|
Closed accidentally due to the bugfix branch getting deleted by some faulty release automation. |
valentijnscholten
left a comment
There was a problem hiding this comment.
Thanks for digging into #15378 — the slow query is real and the location annotations are indeed involved. But I don't think this diff fixes it, and I'd like to suggest a much smaller change instead. I compiled the querysets against Django 5.2 to look at the generated SQL.
The counting side is already covered
Two mechanisms already exist:
get_page_items()indojo/utils.pycallsget_page_items_and_count(..., do_count=False), so there is no extratotal_countquery for this page.- Django masks unused annotations out of the Paginator's own
COUNTquery (Query.get_aggregation()→set_annotation_mask(aggregates)). With only thefindings_countsubquery annotated,count()compiles to literallySELECT COUNT(*) FROM dojo_product— the subquery is dropped.
So "annotations before pagination are expensive" isn't the mechanism here. A correlated Subquery annotation before pagination is nearly free.
The actual cause is narrower
The two Count("locations…", distinct=True) annotations add a LEFT JOIN and force a GROUP BY — and Django puts the correlated findings subquery into that GROUP BY, by ordinal:
GROUP BY "dojo_product"."id", ..., 3 -- 3 = the findings_count subqueryThat is the 60,168 loops in the issue: the subquery becomes a grouping key, so it is evaluated once per joined location row instead of once per product. The aggregate also makes has_existing_aggregation true, which defeats the count-query masking above, so the Paginator's count wraps the same join + group and pays it a second time.
Why this PR doesn't remove the fan-out
prefetch_for_product() already carries six correlated subqueries (active_engagement_count, closed_engagement_count, last_engagement_date, active_finding_count, active_verified_finding_count, total_reimport_count). Moving the location aggregates into that queryset reproduces the same shape in the page query, now with ~7 subqueries as grouping keys — and LIMIT 25 is applied after the grouping, so it doesn't bound the scan:
SELECT DISTINCT product.*, <7 correlated subqueries>,
COUNT(DISTINCT loc.host), COUNT(DISTINCT loc.id)
FROM dojo_product LEFT OUTER JOIN dojo_locationproductreference ...
GROUP BY product.id, product.name, 5, 4, ...
ORDER BY product.name LIMIT 25So the PR fixes the count query and leaves the dominant page query fanning out the same way. An EXPLAIN (ANALYZE, BUFFERS) on a dataset like the reporter's would confirm, but the mechanism their plan shows is unchanged by this diff.
Suggested fix
Keep aggregates off the product queryset entirely and count LocationProductReference rows per product with subqueries — build_count_subquery() in dojo/query_utils.py exists for exactly this trap (see the comment in that file), and dojo/product_type/ui/views.py already uses it this way. No join, no GROUP BY, and the counts can then live pre- or post-pagination without any of the rest of this diff:
location_refs = LocationProductReference.objects.filter(product_id=OuterRef("pk"))
prods = prods.annotate(
location_count=Coalesce(build_count_subquery(location_refs, group_field="product_id"), Value(0)),
location_host_count=Coalesce(
Subquery(
location_refs.order_by().values("product_id")
.annotate(c=Count("location__url__host", distinct=True))
.order_by("product_id").values("c")[:1],
output_field=IntegerField(),
),
Value(0),
),
)Smaller points, if you rework it:
product_list_orders_by_findings_count()hand-parses theoparameter, reimplementing what theOrderingFilterindojo/product/ui/filters.pyalready owns. It becomes unnecessary once the location counts are subqueries.findings_count=F("active_finding_count")emits the identicalCOALESCE((SELECT COUNT(...)))twice in the SELECT list, and in the sort pathfindings_countends up annotated twice (no error, since it's a model property rather than a field, but it's confusing). Converging the template on one of the two names would be cleaner.- The unconditional
.distinct()costs every request withV3_FEATURE_LOCATIONSaSELECT COUNT(*) FROM (SELECT DISTINCT ...)for pagination. The duplicate-row risk really comes from the host filter indojo/filters.py(locations__location__url__host__icontainscan match several references per product; the plainlocations__location=filter can't, thanks to theunique_location_and_productconstraint), so dedup belongs in those filter methods rather than on every request. - The tests are
SimpleTestCase+MagicMockasserting the view's internal call order. They don't touch a database, so they can't demonstrate the fix, and they'll break on any harmless refactor.assertNumQueries/CaptureQueriesContextasserting that the page query has noGROUP BY, plus a query test showing a host filter doesn't duplicate products, is what would actually pin the behaviour.
Signed-off-by: Emre Koca <110906681+kocaemre@users.noreply.github.com>
|
Thanks for the detailed SQL analysis — I reworked this in the narrower direction you suggested. Latest commit now:
Local verification: The test run still reports the existing local |
|
This pull request contains multiple critical findings where the user 'kocaemre' modified numerous sensitive codepaths without being on the allowed authors list. These unauthorized changes affect various core modules including authorization, API, and finding management.
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in
|
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/authorization/serializer_guards.py' matches configured sensitive codepath pattern 'dojo/authorization/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/authorization/url_permissions.py (drs_b784c541)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/authorization/url_permissions.py' matches configured sensitive codepath pattern 'dojo/authorization/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/context_processors.py (drs_7c66e5d4)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/context_processors.py' matches configured sensitive codepath pattern 'dojo/context_processors.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/decorators.py (drs_77e1e869)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/decorators.py' matches configured sensitive codepath pattern 'dojo/decorators.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/endpoint/models.py (drs_0beb4e1a)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/endpoint/models.py' matches configured sensitive codepath pattern 'dojo/endpoint/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/engagement/services.py (drs_ac394247)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/engagement/services.py' matches configured sensitive codepath pattern 'dojo/engagement/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/finding/deduplication.py (drs_7352cf1c)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/finding/deduplication.py' matches configured sensitive codepath pattern 'dojo/finding/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/finding/helper.py (drs_de4c043e)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/finding/helper.py' matches configured sensitive codepath pattern 'dojo/finding/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/finding/models.py (drs_5d88b872)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/finding/models.py' matches configured sensitive codepath pattern 'dojo/finding/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/finding/queries.py (drs_0aa4b4fa)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/finding/queries.py' matches configured sensitive codepath pattern 'dojo/finding/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/forms.py (drs_bf40da3a)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/forms.py' matches configured sensitive codepath pattern 'dojo/forms.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/importers/auto_create_context.py (drs_4eac17c2)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/importers/auto_create_context.py' matches configured sensitive codepath pattern 'dojo/importers/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/importers/base_location_manager.py (drs_73673dd6)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/importers/base_location_manager.py' matches configured sensitive codepath pattern 'dojo/importers/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/jira/helper.py (drs_adfdac53)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/jira/helper.py' matches configured sensitive codepath pattern 'dojo/jira/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/reports/queries.py (drs_3d116b39)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/reports/queries.py' matches configured sensitive codepath pattern 'dojo/reports/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/reports/widgets.py (drs_75715438)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/reports/widgets.py' matches configured sensitive codepath pattern 'dojo/reports/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/search/views.py (drs_827aab15)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/search/views.py' matches configured sensitive codepath pattern 'dojo/search/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/templatetags/display_tags.py (drs_0a0c6db5)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/templatetags/display_tags.py' matches configured sensitive codepath pattern 'dojo/templatetags/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/urls.py (drs_9627a590)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/urls.py' matches configured sensitive codepath pattern 'dojo/urls.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/utils.py (drs_5755467f)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/utils.py' matches configured sensitive codepath pattern 'dojo/utils.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/filters.py (drs_88e13bbb)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/filters.py' matches configured sensitive codepath pattern 'dojo/filters.py' and was modified by 'kocaemre' (commit b5fe86e) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/api_v2/prefetch/prefetcher.py (drs_f48a9315)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/api_v2/prefetch/prefetcher.py' matches configured sensitive codepath pattern 'dojo/api_v2/**/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/api_v2/serializers.py (drs_da65fd1e)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/api_v2/serializers.py' matches configured sensitive codepath pattern 'dojo/api_v2/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/api_v2/views.py (drs_1fde6f06)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/api_v2/views.py' matches configured sensitive codepath pattern 'dojo/api_v2/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/authorization/api_permissions.py (drs_b2bcc0f9)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/authorization/api_permissions.py' matches configured sensitive codepath pattern 'dojo/authorization/*.py' and was modified by 'kocaemre' (commit dfb80b6) who is not in the allowed authors list. |
We've notified @mtesauro.
Comment to provide feedback on these findings.
Report false positive: @dryrunsecurity fp [FINDING ID] [FEEDBACK]
Report low-impact: @dryrunsecurity nit [FINDING ID] [FEEDBACK]
Example: @dryrunsecurity fp drs_90eda195 This code is not user-facing
All finding details can be found in the DryRun Security Dashboard.
valentijnscholten
left a comment
There was a problem hiding this comment.
Thanks for tackling the Product-list query planning. The join→subquery conversion for location_count/location_host_count looks correct and equivalent to the old counts (Product.locations is the reverse relation to LocationProductReference, and its manager applies no default filter, so the row sets match). Two things should be addressed before merge, plus a description mismatch.
1. dojo/filters.py — the de-duplication fix is incomplete; products still duplicate on the other location filters (blocking)
.distinct() is added only to filter_endpoints_host_base (the endpoints__host path). But the Product filter also joins the to-many locations relation in two other places that have no .distinct():
location_status = MultipleChoiceFilter(field_name="locations__status", ...)(dojo/product/ui/filters.py:103)- the
endpointsfilter →filter_endpoints_base(dojo/filters.py)
So the stated goal ("location joins do not duplicate products") isn't fully met: a product with two locations whose statuses are both in the selected location_status set will appear twice, with wrong pagination totals — when filtering by location status or endpoint id without an endpoint host. Please dedupe those paths too (or apply distinct() once at the view level after filtering, which would cover all of them).
2. dojo/filters.py — .distinct() on the shared helper changes the Finding list too, out of scope and untested (blocking)
filter_endpoints_host_base is shared: it's also called by the Finding filter (dojo/finding/ui/filters.py:124), so this change alters the Finding list's endpoints__host behavior (now de-duplicates findings). That may be a reasonable fix, but it's an unmentioned change to a different list and isn't covered by the added test (only test_product_list_pagination). Please either scope the distinct() to the product filter, or explicitly confirm + add Finding-list coverage that pagination/ordering/counts don't shift unexpectedly. (Note: all-column DISTINCT won't dedupe if the finding queryset carries a to-many-join annotation at that point, so the effect there needs checking either way.)
Description mismatch (please reconcile)
The PR description doesn't match the diff: it says it "defers count annotations until after filtering and pagination," keeps findings_count before pagination "only when sorted by that field," and "moves annotations into prefetch_for_product()." None of that is in the diff — annotations are still applied before pagination (just extracted into helper functions), there's no sort-conditional gating, and nothing moves into prefetch_for_product(). Could you update the description to describe the actual change (join→subquery + a filter-level distinct())? It matters for reviewing whether this fully addresses #15378.
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
|
Conflicts have been resolved. A maintainer will review the pull request shortly. |
|
Resolved the Resolution notes:
Local verification:
I also tried the targeted Django test directly:
|
…cation-counts-clean # Conflicts: # dojo/product/ui/views.py
63ef69b to
dfb80b6
Compare
|
Addressed the two blocking review points in Changes:
Local verification:
Notes:
|
This is a focused bugfix for an open issue: fixes #15378.
Description
Defers Product list count annotations until after filtering and pagination unless the request explicitly sorts by
findings_count.With
V3_FEATURE_LOCATIONSenabled, the Product list currently adds finding and location count annotations before pagination. That can make PostgreSQL evaluate expensive correlated finding-count subqueries for intermediate rows created by location joins, even though the page only renders 25 products.This PR:
findings_countannotation before pagination only when the Product list is sorted by that field;distinct()to the filtered v3 product queryset before pagination so location joins do not duplicate products;findings_count,location_host_count, andlocation_countannotations intoprefetch_for_product().Test results
.venv/bin/python manage.py test unittests.test_product_list_pagination -v 2.venv/bin/python -m ruff check --config ruff.toml dojo/product/ui/views.py unittests/test_product_list_pagination.py.venv/bin/python -m py_compile dojo/product/ui/views.py unittests/test_product_list_pagination.py/root/.local/share/uv/python/cpython-3.13-linux-x86_64-gnu/bin/python3.13 -m py_compile dojo/product/ui/views.py unittests/test_product_list_pagination.pygit diff --check HEAD~1..HEADThe Django test run passed with the existing local warning that
components/node_modulesis missing fromSTATICFILES_DIRS.Documentation
No documentation update needed; this is a Product list query-planning/performance bugfix.
Checklist
dev.dev.bugfixbranch.