[Dag Processor] Parsing is skipped when there are too many callbacks

[trina242]

Hi all,

I recently encountered an issue (after upgrade to 3.x) where users were complaining about their DAGs not being updated. I'm running Airflow on a big scale, with 75K task instances daily and DAGs generated dynamically.

It's been a hell to debug - no errors, no parsing timeouts, everything working perfectly locally with the same DAG setup. The only lead was a random 0 DAGs result on file parsing and no "Last Run At" displayed, when "Duration" was there all right. Finally I managed to find the only line in the code where this situation occurrs and it's the part where callbacks are executed and "actual" DAG parsing is skipped. In my case, with a lot of callbacks, it sometimes causes even an hour of delay after changing something in the DAG definition.

The fix seems easy - remove return None statement when callbacks are executed and let processor parse the DAG anyway. Indeed it worked for me when I made a dirty patch on a production environment. But I understand that I might not realize the possible consequences of such move, which is why I'm starting a discussion here before opening a PR.

To be honest, I was kind of surprised to learn that DAG processor is responsible for executing callbacks. What do you think, is it the right place to do so?

I'll appreciate all the comments 🙏

[cookesan]

You found the relevant branch. In the current processor code, a parse request with callback requests builds the DagBag, executes the callbacks, and then returns None instead of serializing DAGs.

I would be cautious about changing only the return None. It may make your environment fresher, but it also changes the meaning of a callback request from "load the DAG file to execute callbacks" into "also publish a parse result", which could affect processor load and stale-DAG behavior.

The underlying issue sounds more like callback work competing with normal parse freshness. A cleaner PR would probably separate or prioritize those paths: for example, avoid letting a callback backlog consume parse slots needed for DAG updates, or make callback execution budgeted/observable. The useful evidence for a PR would be processor counts, callback queue size, parse interval, how many callback requests arrive per minute, and a minimal reproduction showing a normal DAG edit delayed behind callback-heavy files.

As an operational workaround, I would reduce callback volume or increase processor capacity before carrying a production patch that always serializes after callbacks.