Add support for serverless cluster in Databricks operator in databricks provider

[HariGS-DB]

Description

The current Databricks operator to run jobs (DatabricksNotebookOperator, DatabricksTaskOperator and DatabricksWorkflowTaskGroupOperator) doesn't support serverless. All the above operators expect either an existing_cluster_id or job_cluster_key, whereas neither should be passed for serverless workflow. The work required is :

• add a named parameter to support environments which are used for serverless
• add a named_parameter called is_serverless as boolean. This flag will make existing_cluster_id and job_cluster_key optional and submit a serverless job
• Update doc to reflect the new addition

Use case/motivation

This change will allow users to submit a Databricks serverless job for DatabricksNotebookOperator, DatabricksTaskOperator and DatabricksWorkflowTaskGroupOperator which is currently supported for only DatabricksSubmitRunOperator

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

feel free