Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Airflow 2 to 3 auto migration rules - ruff #44556

Open
Tracked by #41641
Lee-W opened this issue Dec 2, 2024 · 3 comments
Open
Tracked by #41641

Airflow 2 to 3 auto migration rules - ruff #44556

Lee-W opened this issue Dec 2, 2024 · 3 comments
Assignees
@Lee-W @sunank200 @prabhusneha
Labels
area:dependencies Issues related to dependencies problems area:dev-tools area:upgrade Facilitating migration to a newer version of Airflow kind:feature Feature Requests
Milestone
Airflow 3.0.0

Comments

@Lee-W
Copy link
Member

Lee-W commented Dec 2, 2024 •
edited
Loading

Description

Ruff

AIR302

context key (@sunank200)

args (and their subclasses in airflow) (@Lee-W)

attribute (@Lee-W)

resource key (not sure whether we can do it through ruff, probably not) (@Lee-W)

AIR303: moved to provider (@prabhusneha)

AIR303 (TODO) Some of the paths are not correct, but can be easily fixed by reading the origin PR

package

module

class

constant / variable

AIR310: models related changes (AIP-72) not going to do it

Rules to blocked by AIP-72

In review rules

AIR302

method (re matching)

  • airflow.secrets.base_secrets.BaseSecretsBackend.get_conn_uri → airflow.secrets.base_secrets.BaseSecretsBackend.get_conn_value
  • airflow.secrets.base_secrets.BaseSecretsBackend.get_connections → airflow.secrets.base_secrets.BaseSecretsBackend.get_connection
  • airflow.hooks.base.BaseHook.get_connections → use get_connection
  • airflow.datasets.BaseDataset.iter_datasets → airflow.sdk.definitions.asset.BaseAsset.iter_assets
  • airflow.datasets.BaseDataset.iter_dataset_aliases → airflow.sdk.definitions.asset.BaseAsset.iter_asset_aliases

property (@Lee-W)

class attrubite (@Lee-W)

Merged rules

AIR302

args

names

method call

AIR303

Use case/motivation

No response

Related issues

#41641

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@Lee-W Lee-W added area:dev-tools kind:feature Feature Requests labels Dec 2, 2024
@Lee-W Lee-W self-assigned this Dec 2, 2024
@dosubot dosubot bot added area:dependencies Issues related to dependencies problems area:upgrade Facilitating migration to a newer version of Airflow labels Dec 2, 2024
This was referenced Dec 2, 2024
Open
Open
Closed
Merged
@Lee-W
Copy link
Member Author

Lee-W commented Dec 3, 2024 •
edited
Loading

Current progress

Updated: 12/20

This was referenced Dec 4, 2024
Merged
Merged
MichaReiser pushed a commit to astral-sh/ruff that referenced this issue Dec 6, 2024
@Lee-W
[airflow]: extend removed names (AIR302) (#14804)
39623f8 
## Summary

Airflow 3.0 removes various deprecated functions, members, modules, and
other values. They have been deprecated in 2.x, but the removal causes
incompatibilities that we want to detect. This PR deprecates the
following names.
The full list of rules we will extend
apache/airflow#44556


#### package
* `airflow.contrib.*` 

#### module
* `airflow.operators.subdag.*` 

#### class
* `airflow.sensors.external_task.ExternalTaskSensorLink` →
`airflow.sensors.external_task.ExternalDagLin`
* `airflow.operators.bash_operator.BashOperator` →
`airflow.operators.bash.BashOperator`
* `airflow.operators.branch_operator.BaseBranchOperator` →
`airflow.operators.branch.BaseBranchOperator`
* `airflow.operators.dummy.EmptyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.dummy.DummyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.dummy_operator.EmptyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.dummy_operator.DummyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.email_operator.EmailOperator` →
`airflow.operators.email.EmailOperator`
* `airflow.sensors.base_sensor_operator.BaseSensorOperator` →
`airflow.sensors.base.BaseSensorOperator`
* `airflow.sensors.date_time_sensor.DateTimeSensor` →
`airflow.sensors.date_time.DateTimeSensor`
* `airflow.sensors.external_task_sensor.ExternalTaskMarker` →
`airflow.sensors.external_task.ExternalTaskMarker`
* `airflow.sensors.external_task_sensor.ExternalTaskSensor` →
`airflow.sensors.external_task.ExternalTaskSensor`
* `airflow.sensors.external_task_sensor.ExternalTaskSensorLink` →
`airflow.sensors.external_task.ExternalTaskSensorLink`
* `airflow.sensors.time_delta_sensor.TimeDeltaSensor` →
`airflow.sensors.time_delta.TimeDeltaSensor`

#### function
* `airflow.utils.decorators.apply_defaults`
* `airflow.www.utils.get_sensitive_variables_fields` →
`airflow.utils.log.secrets_masker.get_sensitive_variables_fields`
* `airflow.www.utils.should_hide_value_for_key` →
`airflow.utils.log.secrets_masker.should_hide_value_for_key`
* `airflow.configuration.get` → `airflow.configuration.conf.get` 
* `airflow.configuration.getboolean` →
`airflow.configuration.conf.getboolean`
* `airflow.configuration.getfloat` →
`airflow.configuration.conf.getfloat`
* `airflow.configuration.getint` → `airflow.configuration.conf.getint` 
* `airflow.configuration.has_option` →
`airflow.configuration.conf.has_option`
* `airflow.configuration.remove_option` →
`airflow.configuration.conf.remove_option`
* `airflow.configuration.as_dict` → `airflow.configuration.conf.as_dict`
* `airflow.configuration.set` → `airflow.configuration.conf.set` 
* `airflow.secrets.local_filesystem.load_connections` →
`airflow.secrets.local_filesystem.load_connections_dict`
* `airflow.secrets.local_filesystem.get_connection` →
`airflow.secrets.local_filesystem.load_connections_dict`
* `airflow.utils.helpers.chain` → `airflow.models.baseoperator.chain` 
* `airflow.utils.helpers.cross_downstream` →
`airflow.models.baseoperator.cross_downstream`

#### attribute
* in `airflow.utils.trigger_rule.TriggerRule`
    * `DUMMY` 
    * `NONE_FAILED_OR_SKIPPED` 

#### constant / variable
* `airflow.PY\d\d`
MichaReiser pushed a commit to astral-sh/ruff that referenced this issue Dec 6, 2024
@Lee-W
[airflow]: extend removed args (AIR302) (#14765)
3ea14d7 
## Summary

Airflow 3.0 removes various deprecated functions, members, modules, and
other values. They have been deprecated in 2.x, but the removal causes
incompatibilities that we want to detect. This PR deprecates the
following names.

* in `DAG`
    * `sla_miss_callback` was removed
* in `airflow.operators.trigger_dagrun.TriggerDagRunOperator`
    * `execution_date` was removed
* in `airflow.operators.weekday.DayOfWeekSensor`,
`airflow.operators.datetime.BranchDateTimeOperator` and
`airflow.operators.weekday.BranchDayOfWeekOperator`
* `use_task_execution_day` was removed in favor of
`use_task_logical_date`

The full list of rules we will extend
apache/airflow#44556

## Test Plan

<!-- How was it tested? -->
A test fixture is included in the PR.
@Lee-W Lee-W mentioned this issue Dec 13, 2024
Merged
@prabhusneha
Copy link
Contributor

I'd like to work on AIR303 tasks

@Lee-W
Copy link
Member Author

Lee-W commented Dec 16, 2024

I'd like to work on AIR303 tasks

That would be really nice! Thanks for helping out! I tried to put some guideline in the first AIR303 PR I created astral-sh/ruff#14764. Please also tag me when you create the PR. I think the ruff team need our help for confirming the rule part (they'll check the rust part 👍 )

@uranusjr uranusjr added this to the Airflow 3.0.0 milestone Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Lee-W Lee-W

@sunank200 sunank200

@prabhusneha prabhusneha

Labels
area:dependencies Issues related to dependencies problems area:dev-tools area:upgrade Facilitating migration to a newer version of Airflow kind:feature Feature Requests
Projects
None yet
Milestone
Airflow 3.0.0
Development

No branches or pull requests
4 participants
@uranusjr @Lee-W @sunank200 @prabhusneha