Fix PathNotFound error in msexports ETL ingestion on first run

[Michael Flanakin (flanakin)]

Summary

• The "Get Existing Parquet Files" GetMetadata activity fails with PathNotFound when the destination folder doesn't exist yet (e.g. first ingestion for a scope/dataset/month combination). Reservation recommendation exports hit this most often because their destination path adds an extra exportName segment, but any export type can hit it on a first run.
• Fixed in both places this pattern exists: msexports_ETL_ingestion (Microsoft.CostManagement/Exports/app.bicep) and the analytics ingestion ETL pipeline (Microsoft.FinOpsHubs/Analytics/app.bicep).
• Fix: add 'exists' to the GetMetadata activity's fieldList. Per ADF's GetMetadata docs, specifying exists makes the activity return exists: false instead of throwing when the path is missing — "If exists isn't specified in the field list, the Get Metadata activity fails if the object isn't found." The downstream Filter activity's items expression now checks .output.exists before reading .output.childItems, so a missing folder resolves to an empty list instead of propagating the failure.
• This is the same idiom already used elsewhere in this codebase for "path may not exist yet" scenarios: Check Schema in Exports/app.bicep, and both GetMetadata activities in IngestionQueries/app.bicep (one of which — "Get Existing Parquet Files" — already combines exists + childItems exactly like this fix does).

Root cause verification (re: issue #2088 comment)

A prior comment on the issue did solid root-cause analysis but its line numbers had drifted and its proposed fix options were speculative (custom error handling / parent-folder existence checks / wrapping in an If Condition). I re-verified against the current code:

• Microsoft.CostManagement/Exports/app.bicep: GetMetadata activity is at lines 1090-1122, Filter Out Current Exports at 1123-1147 (comment's estimate of ~1090-1122 / ~1123-1147 was accurate).
• Microsoft.FinOpsHubs/Analytics/app.bicep: identical vulnerable pattern confirmed at lines 1668-1707 (comment said ~1667-1706, essentially correct).
• The actual cleanest fix is ADF's documented, built-in behavior for this exact case (exists in fieldList), not a custom If Condition or Web-activity workaround — and this idiom was already present elsewhere in the repo, so this change makes the two vulnerable call sites consistent with the rest of the codebase rather than introducing a new pattern.

Test plan

• bicep build src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/app.bicep --stdout — builds cleanly
• bicep build src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/app.bicep --stdout — builds cleanly
• Verified compiled ARM JSON includes "exists" in both fieldList arrays
• pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Lint" — 3418/3418 passed
• Manual end-to-end validation against a real deployment with no prior export data (not performed in this environment)

Fixes #2088

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Roland Krummenacher (RolandKrummenacher)]

Unguarded .output.exists breaks the Completed dependency tolerance.

Filter Out Current Exports depends on Get Existing Parquet Files with dependencyConditions: ['Completed'] (not Succeeded) - it is deliberately designed to run even when the GetMetadata activity fails. That is exactly why the original expression used only contains(output, 'childItems'): a failed activity returns an output object without that property, so the expression degraded to json('[]') and the pipeline continued. (Both the Completed condition and the contains guard were introduced together in #1512.)

Adding exists to the fieldList stops the PathNotFound failure, but the activity can still fail for other reasons - transient ADLS Gen2 5xx/throttling, an RBAC error on the ingestion container, or the 12h timeout. In those cases output has no exists property either, so and(activity('Get Existing Parquet Files').output.exists, ...) now fails evaluation ("the expression cannot be evaluated because property 'exists' doesn't exist"), the Filter activity fails, For Each Old File (Succeeded dependency) is skipped, and msexports_ETL_ingestion fails - where it previously degraded gracefully to an empty file list.

The output.exists term is also redundant: when exists is false, GetMetadata returns no childItems, so the pre-existing contains(output, 'childItems') check alone already yields json('[]'). The simplest fix is to keep only the fieldList addition and leave this expression unchanged (or, if you want the explicit check, guard it with contains(activity('Get Existing Parquet Files').output, 'exists') first).

Note the precedent cited in the PR description (IngestionQueries/app.bicep:363) is safe only because that Filter uses a Succeeded dependency, so it never evaluates against a failed-activity output.

Suggested change

	value: '@if(and(activity(\'Get Existing Parquet Files\').output.exists, contains(activity(\'Get Existing Parquet Files\').output, \'childItems\')), activity(\'Get Existing Parquet Files\').output.childItems, json(\'[]\'))'
	value: '@if(contains(activity(\'Get Existing Parquet Files\').output, \'childItems\'), activity(\'Get Existing Parquet Files\').output.childItems, json(\'[]\'))'

[Roland Krummenacher (RolandKrummenacher)]

This changes which error the user gets, but the pipeline still fails on a missing folder.

When ingestion_ExecuteETL runs for a folderPath whose ingestion folder does not exist yet, Get Existing Parquet Files now succeeds with exists: false, Filter Out Folders yields an empty array, and then If No Files (below, @equals(length(activity('Filter Out Folders').output.Value), 0)) fires the Files Not Found Fail activity with errorCode IngestionFilesNotFound.

So the cascade into ingestion_ExecuteETL described in #2088 is not actually resolved here - PathNotFound is just replaced by IngestionFilesNotFound, whose message ("Please confirm the folder path is the full path, including the "ingestion" container and not starting with or ending with a slash") points the operator at a path-formatting problem that is not the real cause.

If a first-run / no-data folder should be a no-op, If No Files needs to distinguish "folder absent" (exists == false) from "folder present but empty / path malformed". If failing loudly is still the intent here, it would be worth saying so in the PR description, since the current text implies this hunk fixes the failure.