[Idea] LLM-powered smart retry operator for Airflow 3.x

[ertancelik]

Problem

Airflow's current retry mechanism is static — it waits the same fixed interval
and retries blindly regardless of the error type. This causes:

• Rate limit errors being retried too fast (same error again)
• Auth errors being retried pointlessly (will never succeed)
• Network errors not being retried fast enough

Proposed Solution

An operator that uses a local LLM (via Ollama) to analyze the error log and
make an intelligent retry decision:

• Should we retry at all? (auth errors → no, fail fast)
• How long should we wait? (rate limits → 60s, network → 0s)
• What type of error is this? (rate_limit / network / auth / data_schema)

Implementation

I built a working prototype as a standalone provider package:
👉 https://github.com/ertancelik/airflow-provider-smart-retry

Works with Airflow 3.x and Ollama (local LLM, no API key needed).

Example

from smart_retry.operator import LLMSmartRetryOperator

smart_task = LLMSmartRetryOperator(
    task_id="my_task",
    task_callable=my_function,
    ollama_base_url="http://localhost:11434",
    model="llama3.1:8b",
    max_retries=3,
)

Would love to hear community feedback — is this something worth contributing
to the core or as an official provider?

[rahrajlat]

This is a really interesting idea — the limitations of static retries in Airflow are definitely real, and your examples make a lot of sense.

I’ve been exploring something similar from a broader AIOps perspective here:
https://github.com/rahrajlat/airflow-ai-ops

In my case, I’ve also been trying a self-learning approach, where the system improves its error classification over time based on past logs and outcomes, instead of relying only on fixed rules. The idea is to combine historical patterns with LLM-based reasoning, so decisions become more accurate as the system sees more failures.

There’s definitely overlap in using logs + LLMs to guide decisions like retries or alerts. I’ve been looking at it not just for retries, but also for things like smarter alerting, failure categorisation, and suggesting next actions.

Personally, this feels like a great fit as a provider package rather than core Airflow, since it keeps things optional and flexible, and avoids adding complexity to the core retry mechanism.

[potiuk]

Actually @kaxil just opened discussion about more comprehensive solution using our newly added common.ai providers and AIP about it https://lists.apache.org/thread/1n1k7jkm9f47qn5c7jf3xcgb1xhrrqn4