Best practice for handling "no data yet" scenarios vs. transient failures in flows?

[jwaes]

Hi all

I have a flow that fetches data from an external API that typically becomes available around noon — but in reality, it's often delayed. I’m trying to find the cleanest way to handle this gracefully in Prefect.

Here’s what I want to achieve:

• If the task fails due to a transient issue (e.g. connection errors, timeouts), I want it to retry a few times using standard task-level retries.
• If the API responds with a "no data available yet" message (e.g. an XML payload or specific error indicating that the data isn’t ready), I consider that a different kind of failure — not a task error, but a signal that the flow should retry later (e.g. with backoff).

My question:
What’s the best way to model this pattern?

• Should I raise a custom exception and trigger a flow-level retry using .with_options(retries=..., retry_delay_seconds=...)?
• Or would it be better to fail the flow and use Prefect Automations to reschedule or retry it?
• Is there another pattern you’d recommend for this “data not ready yet, try later” use case?

Appreciate any guidance or real-world patterns others are using for this!

Thanks

[zzstoatzz]

hi @jwaes

i would say there are 2 different ways to go about this, which is: "wait on current infra" or "schedule on new infra"

and which you choose depends on:

• how long might you have to wait?
• how expensive is it to spin up new flow run infra?

personally, if the wait is not extreme, I'd likely prefer to wait on the current infrastructure (ie handle retries in the flow and via task level retry settings). That way I can see the full span of relevant events in one flow run.

is that helpful?