Azure Databricks Adding Support for GA API Version #21895

mvvsubbu · 2022-12-14T19:44:40Z

ARM API Information (Control Plane)

MSFT employees can try out our new experience at OpenAPI Hub - one location for using our validation tools and finding your workflow.

Azure 1st Party Service can try out the Shift Left experience to initiate API design review from ADO code repo. If you are interested, may request engineering support by filling in with the form https://aka.ms/ShiftLeftSupportForm.

Changelog

Add a changelog entry for this PR by answering the following questions:

What's the purpose of the update?
- new service onboarding
- new API version
- update existing version for new feature
- update existing version to fix swagger quality issue in s360
- Other, please clarify
When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month. 01/31/2023
When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month. 02/01/2023
By default, Azure SDKs of all languages (.NET/Python/Java/JavaScript for both management-plane SDK and data-plane SDK, Go for management-plane SDK only ) MUST be refreshed with/after swagger of new version is published. If you prefer NOT to refresh any specific SDK language upon swagger updates in the current PR, please leave details with justification here.
Our team hasn't released any SDK yet, this is something our team has in backlogs to decide.

Contribution checklist (MS Employees Only):

I commit to follow the Breaking Change Policy of "no breaking changes"
I have reviewed the documentation for the workflow.
Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.

ARM API Review Checklist

Applicability: ⚠️

If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

Change to data plane APIs

Adding new properties

All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

Check this box if any of the following apply to the PR so that the label "ARMReview" and "WaitForARMFeedback" will be added by bot to kick off ARM API Review. Missing to check this box in the following scenario may result in delays to the ARM manifest review and deployment.
- Adding a new service
- Adding new API(s)
- Adding a new API version
  - To review changes efficiently, ensure you copy the existing version into the new directory structure for first commit and then push new changes, including version updates, in separate commits. You can use OpenAPIHub to initialize the PR for adding a new version. For more details refer to the wiki.
Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.
If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist

If you have any breaking changes as defined in the Breaking Change Policy, request approval from the Breaking Change Review Board.

Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Additional details on the process and office hours are on the Breaking Change Wiki.

NOTE: To update API(s) in public preview for over 1 year (refer to Retirement of Previews)

Please follow the link to find more details on PR review process.

openapi-workflow-bot · 2022-12-14T19:44:43Z

Hi, @mvvsubbu Thanks for your PR. I am workflow bot for review process. Here are some small tips.

Please ensure to do self-check against checklists in first PR comment.

PR assignee is the person auto-assigned and responsible for your current PR reviewing and merging.

For specs comparison cross API versions, Use API Specs Comparison Report Generator

If there is CI failure(s), to fix CI error(s) is mandatory for PR merging; or you need to provide justification in PR comment for explanation. How to fix?

Any feedback about review process or workflow bot, pls contact swagger and tools team. [email protected]

openapi-pipeline-app · 2022-12-14T19:44:45Z

Swagger Validation Report

️️✔️BreakingChange succeeded [Detail] [Expand]

There are no breaking changes.

️️✔️Breaking Change(Cross-Version) succeeded [Detail] [Expand]

There are no breaking changes.

compared swaggers (via Oad v0.10.4)]	new version	base version
databricks.json	2023-02-01(91ead59)	2018-04-01(main)
databricks.json	2023-02-01(91ead59)	2022-04-01-preview(main)
vnetpeering.json	2023-02-01(91ead59)	2018-04-01(main)
vnetpeering.json	2023-02-01(91ead59)	2022-04-01-preview(main)

️️✔️CredScan succeeded [Detail] [Expand]

There is no credential detected.

️⚠️LintDiff: 0 Warnings warning [Detail]

compared tags (via openapi-validator v2.0.0)	new version	base version
package-2023-02-01	package-2023-02-01(91ead59)	default(main)

The following errors/warnings exist before current PR submission:

Only 30 items are listed, please refer to log for more details.

Rule	Message
`❌` ResourceNameRestriction	The resource name parameter 'workspaceName' should be defined with a 'pattern' restriction. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L37
`❌` DeleteOperationAsyncResponseValidation	An async DELETE operation must set long running operation options 'x-ms-long-running-operation-options' Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L84
`❌` LroLocationHeader	A 202 response should include an Location response header. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L113
`❌` CreateOperationAsyncResponseValidation	An async PUT operation must set long running operation options 'x-ms-long-running-operation-options' Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L128
`❌` PatchSkuProperty	The patch operation body parameter schema should contains property 'sku'. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L209
`❌` LroLocationHeader	A 202 response should include an Location response header. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L244
`❌` OperationsApiResponseSchema	The response schema of operations API '/providers/Microsoft.Databricks/operations' does not match the ARM specification. Please standardize the schema. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L357
`❌` ResourceNameRestriction	The resource name parameter 'workspaceName' should be defined with a 'pattern' restriction. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L373
`❌` ResourceNameRestriction	The resource name parameter 'workspaceName' should be defined with a 'pattern' restriction. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L419
`❌` ResourceNameRestriction	The resource name parameter 'workspaceName' should be defined with a 'pattern' restriction. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L465
`❌` ResourceNameRestriction	The resource name parameter 'workspaceName' should be defined with a 'pattern' restriction. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L511
`❌` ResourceNameRestriction	The resource name parameter 'privateEndpointConnectionName' should be defined with a 'pattern' restriction. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L511
`❌` CreateOperationAsyncResponseValidation	An async PUT operation must set long running operation options 'x-ms-long-running-operation-options' Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L556
`❌` CreateOperationAsyncResponseValidation	An async PUT operation must return 201. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L596
`❌` CreateOperationAsyncResponseValidation	Only 201 is the supported response code for PUT async response. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L603
`❌` LroLocationHeader	A 202 response should include an Location response header. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L603
`❌` DeleteOperationAsyncResponseValidation	An async DELETE operation must set long running operation options 'x-ms-long-running-operation-options' Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L617
`❌` LroLocationHeader	A 202 response should include an Location response header. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L649
`❌` ResourceNameRestriction	The resource name parameter 'workspaceName' should be defined with a 'pattern' restriction. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L667
`❌` MissingTypeObject	The schema 'Workspace' is considered an object but without a 'type:object', please add the missing 'type:object'. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L712
`❌` MissingTypeObject	The schema 'WorkspaceProperties' is considered an object but without a 'type:object', please add the missing 'type:object'. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L740
`❌` MissingTypeObject	The schema 'TrackedResource' is considered an object but without a 'type:object', please add the missing 'type:object'. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L854
`❌` MissingTypeObject	The schema 'Resource' is considered an object but without a 'type:object', please add the missing 'type:object'. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L887
`❌` MissingTypeObject	The schema 'Sku' is considered an object but without a 'type:object', please add the missing 'type:object'. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L908
`❌` MissingTypeObject	The schema 'ManagedIdentityConfiguration' is considered an object but without a 'type:object', please add the missing 'type:object'. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L961
`❌` GuidUsage	Usage of Guid is not recommended. If GUIDs are absolutely required in your service, please get sign off from the Azure API review board. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L966
`❌` GuidUsage	Usage of Guid is not recommended. If GUIDs are absolutely required in your service, please get sign off from the Azure API review board. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L972
`❌` MissingTypeObject	The schema 'WorkspaceCustomBooleanParameter' is considered an object but without a 'type:object', please add the missing 'type:object'. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L997
`❌` MissingTypeObject	The schema 'WorkspaceCustomObjectParameter' is considered an object but without a 'type:object', please add the missing 'type:object'. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L1013
`❌` MissingTypeObject	The schema 'WorkspaceCustomStringParameter' is considered an object but without a 'type:object', please add the missing 'type:object'. Location: Microsoft.Databricks/stable/2023-02-01/databricks.json#L1029

️⚠️Avocado: 1 Warnings warning [Detail]

Rule	Message
⚠️ MULTIPLE_API_VERSION	The default tag contains multiple API versions swaggers. readme: specification/databricks/resource-manager/readme.md tag: specification/databricks/resource-manager/readme.md#tag-package-2023-02-01

️️✔️ApiReadinessCheck succeeded [Detail] [Expand]

️⚠️~[Staging] ServiceAPIReadinessTest: 0 Warnings warning [Detail]

API Test is not triggered due to precheck failure. Check pipeline log for details.

️️✔️~[Staging] SwaggerAPIView succeeded [Detail] [Expand]

️️✔️~[Staging] CadlAPIView succeeded [Detail] [Expand]

️️✔️ModelValidation succeeded [Detail] [Expand]

Validation passes for ModelValidation.

️️✔️SemanticValidation succeeded [Detail] [Expand]

Validation passes for SemanticValidation.

️️✔️PoliCheck succeeded [Detail] [Expand]

Validation passed for PoliCheck.

️️✔️PrettierCheck succeeded [Detail] [Expand]

Validation passes for PrettierCheck.

️️✔️SpellCheck succeeded [Detail] [Expand]

Validation passes for SpellCheck.

️️✔️Lint(RPaaS) succeeded [Detail] [Expand]

Validation passes for Lint(RPaaS).

️️✔️CadlValidation succeeded [Detail] [Expand]

Validation passes for CadlValidation.

️️✔️PR Summary succeeded [Detail] [Expand]

Validation passes for Summary.

_{Posted by Swagger Pipeline | How to fix these errors?}

openapi-pipeline-app · 2022-12-14T19:44:45Z

Swagger Generation Artifacts

️️✔️ApiDocPreview succeeded [Detail] [Expand]

 Please click here to preview with your @microsoft account.

️❌SDK Breaking Change Tracking failed [Detail]

Breaking Changes Tracking

❌azure-sdk-for-python-track2 - track2_azure-mgmt-databricks - Approved - 1.1.0b2

+	Model PrivateLinkServiceConnectionState no longer has parameter action_required

️⚠️ azure-sdk-for-python-track2 warning [Detail]

⚠️Warning [Logs]Release - Generate from e0a8a42. SDK Automation 14.0.0

command	sh scripts/automation_init.sh ../azure-sdk-for-python_tmp/initInput.json ../azure-sdk-for-python_tmp/initOutput.json
cmderr	[automation_init.sh] WARNING: Skipping azure-nspkg as it is not installed.
command	sh scripts/automation_generate.sh ../azure-sdk-for-python_tmp/generateInput.json ../azure-sdk-for-python_tmp/generateOutput.json
cmderr	[automation_generate.sh]
cmderr	[automation_generate.sh] npm notice New major version of npm available! 8.19.3 -> 9.4.0
cmderr	[automation_generate.sh] npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.4.0>
cmderr	[automation_generate.sh] npm notice Run `npm install -g [email protected]` to update!
cmderr	[automation_generate.sh] npm notice

️✔️track2_azure-mgmt-databricks [View full logs] [Release SDK Changes] Breaking Change Detected

info	[Changelog] ### Features Added
info	[Changelog]
info	[Changelog]   - Model AccessConnector has a new parameter system_data
info	[Changelog]   - Model OperationDisplay has a new parameter description
info	[Changelog]   - Model PrivateEndpointConnectionProperties has a new parameter group_ids
info	[Changelog]   - Model PrivateLinkServiceConnectionState has a new parameter actions_required
info	[Changelog]
info	[Changelog] ### Breaking Changes
info	[Changelog]
info	[Changelog]   - Model PrivateLinkServiceConnectionState no longer has parameter action_required

️️✔️ azure-sdk-for-js succeeded [Detail] [Expand]

️✔️Succeeded [Logs]Release - Generate from e0a8a42. SDK Automation 14.0.0

command	sh .scripts/automation_init.sh ../azure-sdk-for-js_tmp/initInput.json ../azure-sdk-for-js_tmp/initOutput.json
warn	File azure-sdk-for-js_tmp/initOutput.json not found to read
command	sh .scripts/automation_generate.sh ../azure-sdk-for-js_tmp/generateInput.json ../azure-sdk-for-js_tmp/generateOutput.json

️✔️@azure/arm-databricks [View full logs] [Release SDK Changes]
- azure-arm-databricks-3.0.0-beta.5.tgz
```
info	[Changelog]
error	breakingChangeTracking is enabled, but version or changelogItem is not found in output.
```

️⚠️ azure-resource-manager-schemas warning [Detail]

⚠️Warning [Logs]Release - Generate from e0a8a42. Schema Automation 14.0.0

command	.sdkauto/initScript.sh ../azure-resource-manager-schemas_tmp/initInput.json ../azure-resource-manager-schemas_tmp/initOutput.json
cmderr	[initScript.sh]  WARN old lockfile
cmderr	[initScript.sh] npm WARN old lockfile The package-lock.json file was created with an old version of npm,
cmderr	[initScript.sh] npm WARN old lockfile so supplemental metadata must be fetched from the registry.
cmderr	[initScript.sh] npm WARN old lockfile
cmderr	[initScript.sh] npm WARN old lockfile This is a one-time fix-up, please be patient...
cmderr	[initScript.sh] npm WARN old lockfile
warn	File azure-resource-manager-schemas_tmp/initOutput.json not found to read
command	.sdkauto/generateScript.sh ../azure-resource-manager-schemas_tmp/generateInput.json ../azure-resource-manager-schemas_tmp/generateOutput.json

️✔️databricks [View full logs] [Release Schema Changes]

️️✔️ azure-powershell succeeded [Detail] [Expand]

️✔️Succeeded [Logs]Release - Generate from e0a8a42. SDK Automation 14.0.0

command	sh ./tools/SwaggerCI/init.sh ../azure-powershell_tmp/initInput.json ../azure-powershell_tmp/initOutput.json
command	pwsh ./tools/SwaggerCI/psci.ps1 ../azure-powershell_tmp/generateInput.json ../azure-powershell_tmp/generateOutput.json

️✔️Az.Databricks [View full logs] [Release SDK Changes]
- Az.Databricks.0.1.0.nupkg

_{Posted by Swagger Pipeline | How to fix these errors?}

openapi-pipeline-app · 2022-12-14T19:44:46Z

Generated ApiView

Language	Package Name	ApiView Link
JavaScript	@azure/arm-databricks	https://apiview.dev/Assemblies/Review/9dc7fd451c984adfb9fb6fe4022d641a

openapi-workflow-bot · 2022-12-14T19:53:36Z

Hi, @mvvsubbu, For review efficiency consideration, when creating a new api version, it is required to place API specs of the base version in the first commit, and push new version updates into successive commits. You can use OpenAPIHub to initialize the PR for adding a new version. For more details refer to the wiki. Or you could onboard API spec pipeline

openapi-workflow-bot · 2022-12-14T19:54:44Z

Hi, @mvvsubbu your PR are labelled with WaitForARMFeedback. A notification email will be sent out shortly afterwards to notify ARM review board([email protected]).

mvvsubbu · 2022-12-14T20:10:04Z

API readiness check failed because our service is not yet deployed. We are targetting to merge this PR after service is deployed by 01/31/2023.

msaranga

Approved.

ghost · 2023-01-22T16:02:59Z

Hi, @mvvsubbu. Your PR has no update for 14 days and it is marked as stale PR. If no further update for over 14 days, the bot will close the PR. If you want to refresh the PR, please remove no-recent-activity label.

mvvsubbu · 2023-02-01T01:23:00Z

@xiaoxuqi-ms can you merge this PR, our service changes has been deployed.

xiaoxuqi-ms · 2023-02-01T02:19:35Z

/azp run

azure-pipelines · 2023-02-01T02:19:44Z

Azure Pipelines successfully started running 1 pipeline(s).

xiaoxuqi-ms · 2023-02-01T02:33:56Z

SDK breaking change is added after this pr is approved and brought by the last pipeline run which is to check the API readiness

xiaoxuqi-ms · 2023-02-01T02:34:33Z

@msyyc , could you help to take a look at the python breaking change? thanks

mvvsubbu added 3 commits December 14, 2022 11:26

Base Commit for new API Version

670d50d

Updating the API Version to 2023-02-01

5602799

Updated the read me to include GA API version

91ead59

mvvsubbu requested review from msaranga and sabose1 December 14, 2022 19:50

azure-sdk mentioned this pull request Dec 14, 2022

[AutoPR databricks] Azure Databricks Adding Support for GA API Version azure-sdk/azure-resource-manager-schemas#1673

Closed

AzureRestAPISpecReview added CI-MissingBaseCommit new-api-version resource-manager labels Dec 14, 2022

openapi-workflow-bot bot added ARMReview WaitForARMFeedback <valid label in PR review process> add this label when ARM review is required labels Dec 14, 2022

mvvsubbu removed the CI-MissingBaseCommit label Dec 14, 2022

mvvsubbu marked this pull request as ready for review December 14, 2022 20:06

openapi-assignment-bot bot assigned xiaoxuqi-ms Dec 14, 2022

mvvsubbu added the DoNotMerge <valid label in PR review process> use to hold merge after approval label Dec 14, 2022

mvvsubbu changed the title ~~Azure Databricks Adding Support for GA API Version~~ [DON'T MERGE] Azure Databricks Adding Support for GA API Version Dec 16, 2022

raosuhas added the ARMSignedOff <valid label in PR review process>add this label when ARM approve updates after review label Dec 19, 2022

openapi-workflow-bot bot removed the WaitForARMFeedback <valid label in PR review process> add this label when ARM review is required label Dec 19, 2022

msaranga approved these changes Jan 4, 2023

View reviewed changes

ghost added the no-recent-activity label Jan 22, 2023

mvvsubbu removed the no-recent-activity label Jan 23, 2023

mvvsubbu changed the title ~~[DON'T MERGE] Azure Databricks Adding Support for GA API Version~~ Azure Databricks Adding Support for GA API Version Jan 31, 2023

mvvsubbu removed the DoNotMerge <valid label in PR review process> use to hold merge after approval label Jan 31, 2023

AzureRestAPISpecReview added CI-MissingBaseCommit ReadyForApiTest <valid label in PR review process>add this label when swagger and service APIs are ready for test labels Feb 1, 2023

openapi-pipeline-app bot added the CI-BreakingChange-Python-Track2 label Feb 1, 2023

msyyc added the Approved-SdkBreakingChange-Python label Feb 1, 2023

xiaoxuqi-ms merged commit e0a8a42 into main Feb 1, 2023

xiaoxuqi-ms deleted the users/subraman/databricks_ga_api branch February 1, 2023 02:48

Azure Databricks Adding Support for GA API Version #21895

Azure Databricks Adding Support for GA API Version #21895

Uh oh!

Conversation

mvvsubbu commented Dec 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ARM API Information (Control Plane)

Changelog

Contribution checklist (MS Employees Only):

ARM API Review Checklist

Breaking Change Review Checklist

Uh oh!

openapi-workflow-bot bot commented Dec 14, 2022

Uh oh!

openapi-pipeline-app bot commented Dec 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Swagger Validation Report

Uh oh!

openapi-pipeline-app bot commented Dec 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Swagger Generation Artifacts

Breaking Changes Tracking

Uh oh!

openapi-pipeline-app bot commented Dec 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Generated ApiView

Uh oh!

openapi-workflow-bot bot commented Dec 14, 2022

Uh oh!

openapi-workflow-bot bot commented Dec 14, 2022

Uh oh!

mvvsubbu commented Dec 14, 2022

Uh oh!

msaranga left a comment

Choose a reason for hiding this comment

Uh oh!

ghost commented Jan 22, 2023

Uh oh!

mvvsubbu commented Feb 1, 2023

Uh oh!

xiaoxuqi-ms commented Feb 1, 2023

Uh oh!

azure-pipelines bot commented Feb 1, 2023

Uh oh!

xiaoxuqi-ms commented Feb 1, 2023

Uh oh!

xiaoxuqi-ms commented Feb 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mvvsubbu commented Dec 14, 2022 •

edited

Loading

openapi-pipeline-app bot commented Dec 14, 2022 •

edited

Loading

openapi-pipeline-app bot commented Dec 14, 2022 •

edited

Loading

openapi-pipeline-app bot commented Dec 14, 2022 •

edited

Loading