[airflow] rule for deprecated task_concurrency parameter (AIR303) #14616
Conversation
|
So many airflow rules :) Are you working/coordinating with @uranusjr ? I just want to ensure we have "one" authority that has a big picture in mind for all tne new airflow rules. |
MichaReiser
added
rule
|
|
Yes and a whole lot more coming. |
abhishekbhakat
changed the title
abhishekbhakat
force-pushed
the
deprecated-task-concurrency
branch
from
9e3bb47 to
d478d29
Compare
|
Are there other parameters that have been deprecated? I wonder if we should club them under a single rule similar to the NumPy 2.0 deprecations where one rule would look at the imports and the other would look at function parameters. |
|
There are a lot of breaking changes in Airflow 3 coming from Airflow 2. |
abhishekbhakat
force-pushed
the
deprecated-task-concurrency
branch
from
d478d29 to
46ba0e9
Compare
- Add new rule to detect deprecated task_concurrency parameter - Update documentation and tests - Add snapshot tests
abhishekbhakat
force-pushed
the
deprecated-task-concurrency
branch
from
46ba0e9 to
9a60fb9
Compare
|
Oh wow, that's a very long list. I didn't expect there to be that many changes. I think it would be great to write up a short proposal on what rules you plan on adding and how you want to group the deprecations. We want to avoid having 50ish airflow deprecation rules. For example: Can we have one rule for:
We can incrementally add to those rules. I think a good place to discuss the new rules is #14626 |
| # Using deprecated task_concurrency parameter | ||
| task1 = PythonOperator( | ||
| task_id="task1", | ||
| task_concurrency=2, # This should trigger AIR303 |
There was a problem hiding this comment.
@abhishekbhakat We should raise a deprecation error in Airflow 2.2+ too: apache/airflow#17708 (that was when we had original deprecation)
In Airflow 3, it is "removed" entirely.
|
What's the status of this rule? Is it still needed according to the new categorization? |
|
I'll put this as a draft PR. We are working on the consolidated rules. We'll revisit this after that is ready. |
Summary
Airflow 3.0 renames the
task_concurrencyparameter tomax_active_tis_per_dagacross all operators. The old parameter will no longer have any effect in 3.0. This rule detects usage of the deprecated parameter to help users migrate their DAGs to the new concurrency control pattern.Ref: #14626
Test Plan
A test fixture has been included for the rule.