Status of testing Providers that were prepared on September 21, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 4.0.0rc1

• Provider Airbyte: breaking change, update provider to use Airbyte API Python SDK (#41122): @marcosmarxm
• Fix wrong casing in airbyte hook. (#42170): @potiuk
  Linked issues:
  • Linked Issue #42154: @jscheffl
• Pin airbyte-api to 0.51.0 (#42154) (#42155): @jscheffl
  Linked issues:
  • Linked Issue #42154: @jscheffl

Provider alibaba: 2.9.1rc1

• log handler deprecated filename_template argument removal (#41552): @dirrao

Provider amazon: 8.29.0rc1

• Adding support for volume configurations in ECSRunTaskOperator (#42087): @srehman420
• Openlineage s3 to redshift operator integration (#41575): @Artuz37
• ECS Exec: Drop params that aren't compatible with EC2 (#42228): @o-nikolas
  Linked issues:
  • Linked Issue #41824: @AdamStrLSEC
• Fix GlueDataBrewStartJobOperator template fields (#42073): @ferruzzi
• Validate aws service exceptions in waiters (#41941): @gopidesupavan
  Linked issues:
  • Linked Issue #41918: @sarkch
• Fix treatment of "#" in S3Hook.parse_s3_url() (#41796): @GlenboLake
• fix: remove part of openlineage extraction from S3ToRedshiftOperator (#41631): @kacpermuda
• filename template arg in providers file task handlers backward compitability support (#41633): @dirrao
  Linked issues:
  • Linked Issue #41552: @dirrao
• fix: custom query should have precedence over default query in RedshiftToS3Operator (#41634): @kacpermuda
  Linked issues:
  • Linked Issue #37861: @okirialbert
• Actually move saml to amazon provider (mistakenly added in papermill) (#42148): @potiuk
  Linked issues:
  • Linked Issue #42137: @potiuk
• Use base aws classes in AWS Glue DataBrew Operators/Triggers (#41848): @gopidesupavan
  Linked issues:
  • Linked Issue #35278: @Taragolis
• Move register_views to auth manager interface (#41777): @vincbeck
• Limit watchtower as depenendcy as 3.3.0 breaks main. (#41612): @potiuk
• log handler deprecated filename_template argument removal (#41552): @dirrao

Provider apache.hdfs: 4.5.1rc1

• log handler deprecated filename_template argument removal (#41552): @dirrao

Provider apache.impala: 1.5.1rc1

• Make kerberos an optional and devel dependency for impala and fab (#41616): @potiuk

Provider apache.livy: 3.9.1rc1

• bugfix/livy-set-base-url (#41658): @harryshi10

Provider apache.spark: 4.11.0rc1

• Add kerberos related connection fields(principal, keytab) on SparkSubmitHook (#40757): @seyoon-lim

Provider cloudant: 4.0.0rc1

• Switch cloudant provider from cloudant library to ibmcloudant library (#41555): @topherinternational
  Linked issues:
  • Linked Issue #21004: @uranusjr

Provider cncf.kubernetes: 8.4.2rc1

• Deprecated configuration removed (#42129): @dirrao

Provider common.io: 1.4.1rc1

• Protect against None components of universal pathlib xcom backend (#41921): @potiuk
  Linked issues:
  • Linked Issue #41723: @potiuk

Provider common.sql: 1.17.0rc1

• Generalize caching of connection in DbApiHook to improve performance (#40751): @dabla
• feat: log client db messages for provider postgres (#40171): @dondaum
  Linked issues:
  • Linked Issue #40116: @hpereira98

Provider databricks: 6.10.0rc1

• [FEAT] databricks repair run with reason match and appropriate new settings (#41412): @gaurav7261
• Removed deprecated method reference airflow.www.auth.has_access when min airflow version >= 2.8.0 (#41747): @dirrao

Provider docker: 3.14.0rc1

• Added logging device and logging device options (#41416): @geraj1010
  Linked issues:
  • Linked Issue #40533: @xjchew
• Add retrieve output docker swarm operator (#41531): @rgriffier

Provider elasticsearch: 5.5.1rc1

• Fix ElasticSearch SQLClient deprecation warning (#41871): @Owen-CH-Leung
• Generalize caching of connection in DbApiHook to improve performance (#40751): @dabla
• filename template arg in providers file task handlers backward compitability support (#41633): @dirrao
  Linked issues:
  • Linked Issue #41552: @dirrao
• log handler deprecated filename_template argument removal (#41552): @dirrao

Provider fab: 1.4.0rc1

• Add FAB migration commands (#41804): @ephraimbuddy
• Separate FAB migration from Core Airflow migration (#41437): @ephraimbuddy
• Deprecated kerberos auth removed (#41693): @dirrao
• Deprecated configuration removed (#42129): @dirrao
• Move is_active user property to FAB auth manager (#42042): @vincbeck
• Move register_views to auth manager interface (#41777): @vincbeck
• Revert "Provider fab auth manager deprecated methods removed (#41720)" (#41960): @potiuk
• provider fab auth manager deprecated methods removed (#41720): @dirrao
• Make kerberos an optional and devel dependency for impala and fab (#41616): @potiuk

Provider google: 10.23.0rc1

• Add ability to create Flink Jobs in dataproc cluster (#42342): @VladaZakharova
• Add new Google Search 360 Reporting Operators (#42255): @molcay
• Add return_immediately parameter to PubSubPullSensor class (#41842): @arnaubadia
  Linked issues:
  • Linked Issue #41838: @arnaubadia
  • Linked Issue #23231: @eladkal
• Add parent_model param in UploadModelOperator (#42091): @jx2lee
  Linked issues:
  • Linked Issue #41194: @GLaDAP
• Add DataflowStartYamlJobOperator (#41576): @moiseenkov
• Add RunEvaluationOperator for Google Vertex AI Rapid Evaluation API (#41940): @CYarros10
• Add CountTokensOperator for Google Generative AI CountTokensAPI (#41908): @CYarros10
• Add Supervised Fine Tuning Train Operator, Hook, Tests, Docs (#41807): @CYarros10
• Minor fixes to ensure successful Vertex AI LLMops pipeline (#41997): @CYarros10
• Exclude partition from BigQuery table name (#42130): @moiseenkov
• [Fix #41763]: Redundant forward slash in SFTPToGCSOperator when destination_path is not specified or have default value (#41928): @Mayuresh16
  Linked issues:
  • Linked Issue #41763: @mpospis
• Fix poll_interval in GKEJobTrigger (#41712): @gopidesupavan
  Linked issues:
  • Linked Issue #41705: @bwatan
• update pattern for dataflow job id extraction (#41794): @lukas-mi
• Fix 'do_xcom_push' and 'get_logs' functionality for KubernetesJobOperator (#40814): @MaksYermak
• Mark VertexAI AutoMLText deprecation (#42251): @olegkachur-e
• Remove system test for deprecated Google analytics operators (#41946): @moiseenkov
• Unpin google-cloud-bigquery package version for Google provider (#41839): @MaksYermak
• Move away from deprecated DAG.following_schedule() method (#41773): @jscheffl
• log handler deprecated filename_template argument removal (#41552): @dirrao

Provider jdbc: 4.5.1rc1

• Generalize caching of connection in DbApiHook to improve performance (#40751): @dabla

Provider microsoft.azure: 10.5.0rc1

• Allow custom api versions in MSGraphAsyncOperator (#41331): @dabla
• Add callback to process Azure Service Bus message contents (#41601): @perry2of5
• log handler deprecated filename_template argument removal (#41552): @dirrao

Provider microsoft.mssql: 3.9.1rc1

• Generalize caching of connection in DbApiHook to improve performance (#40751): @dabla

Provider mongo: 4.2.1rc1

• Adjust typing in Mongo hook to prevent mypy errors (#42354): @jscheffl
• Improve Mongo connection validation and UI (#41717): @topherinternational
  Linked issues:
  • Linked Issue #41371: @topherinternational

Provider mysql: 5.7.1rc1

• Generalize caching of connection in DbApiHook to improve performance (#40751): @dabla

Provider odbc: 4.7.1rc1

• Generalize caching of connection in DbApiHook to improve performance (#40751): @dabla

Provider openai: 1.4.0rc1

• feat(providers/openai): support batch api in hook/operator/trigger (#41554): @josix
  Linked issues:
  • Linked Issue #41336: @josix

Provider openlineage: 1.12.0rc1

• [feat: notify about potential serialization failures when sending DagRun, don't serialize unnecessary params, guard listener for exceptions (feat: notify about potential serialization failures when sending DagRun, don't serialize unnecessary params, guard listener for exceptions #41690)](Fix a few typos in CHANGELOG of v1-10-test #4169
• fix: revert behavior of flattening lists in OpenLineage's InfoJsonEncodable (#41786): @mobuchowski
• chore: bump OL provider dependencies versions (#42059): @kacpermuda
• move to dag_run.logical_date from execution date in OpenLineage provider (#41889): @mobuchowski
• Unify DAG schedule args and change default to None (#41453): @uranusjr

Provider papermill: 3.8.1rc1

• Actually move saml to amazon provider (mistakenly added in papermill) (#42148): @potiuk
  Linked issues:
  • Linked Issue #42137: @potiuk
• Make SAML a required dependency of Amazon provider (#42137): @potiuk

Provider postgres: 5.13.0rc1

• feat: log client db messages for provider postgres (#40171): @dondaum
  Linked issues:
  • Linked Issue #40116: @hpereira98
• Generalize caching of connection in DbApiHook to improve performance (#40751): @dabla

Provider snowflake: 5.7.1rc1

• Update snowflake naming for account names and locators. (#41775): @JDarDagran

All users involved in the PRs:
@gaurav7261 @ephraimbuddy @jscheffl @harryshi10 @dabla @moiseenkov @marcosmarxm @arnaubadia @o-nikolas @Artuz37 @geraj1010 @perry2of5 @josix @kacpermuda @dondaum @Mayuresh16 @olegkachur-e @ferruzzi @mob

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[jscheffl]

Signed-off my fixes I contributed. They were mainly related to failing main, if pytests are fine then all is good.

[Owen-CH-Leung]

Confirmed that the deprecation warning is gone with version 5.5.1rc1. Thanks

[dondaum]

Tested #40171. Worked as expected.

[seyoon-lim]

I have tested apache.spark: 4.11.0rc1(#40757) and it works as intended. Thanks

[josix]

Tested openai: 1.4.0rc1 #41336, it works as expeted, thanks!

[topherinternational]

Checked #41555 and #41717, they are good to go.

[moiseenkov]

#41576, #42130, #41946 are good

[molcay]

#42255 works as expected

[VladaZakharova]

#42342 looks good

[GlenboLake]

#41796 looks good

[JDarDagran]

#41775 verified working

[gopidesupavan]

Verified #41918 working fine. Thank you for release.
For #41848 have environment issues to test, Just seen vincent message system tests are working fine on the latest provider package.

[ephraimbuddy]

Tested #41804, and #41437. No interactions with Airflow 2

[gopidesupavan]

Tested #41712 with system test and no issues, working fine.

[harryshi10]

#41658 has been tested and works as expected.

[dirrao]

#42129, #41552, #41633, #41747, #41720
I have validated basic things and they looks good to me. However, I couldn't able to test all functionality of auth, configuration and logging changes.

[eladkal]

Thank you everyone. Providers are released.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.

[geraj1010]

I tested #41416 before the PR was closed and the logging worked as expected. Apologies for the late response, my GitHub account was just recently restored.