add apitestErrorCode doc #14749
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| <!-- | ||
| Copyright (c) 2021 Microsoft Corporation | ||
|
|
||
| This software is released under the MIT License. | ||
| https://opensource.org/licenses/MIT | ||
| --> | ||
|
|
||
| # API testing error code | ||
|
|
||
| OAV api testing define some rules and check whether actual service response match with example. Example is very important for downstream SDK test code generation, code samples and docs. The goal to detect example quality issues and check service behaviors consistent with example. | ||
|
|
||
| ## Rule descriptions | ||
|
|
||
| #### INCORRECT_PROVISIONING_STATE | ||
|
|
||
| If service return 200 status code,it means the operation is succeed and finished. So the provisioning state should be one of the terminal states ["succeeded", "failed", "canceled", "ready", "created", "deleted"]. | ||
|
|
||
| The provisioning state is very important for downstream terraform or cli to manage resource status. For more details about provisioning state, please refer to this [ARM RPC provisioning state](https://github.com/Azure/azure-resource-manager-rpc/blob/master/v1.0/Addendum.md#provisioningstate-property) | ||
|
|
||
| #### RESPONSE_MISSING_VALUE | ||
|
|
||
| **Error message**: The response value is missing. Path: {}. Expected: {}. Actual: undefined | ||
|
|
||
| The example has defined response value, but actually the server doesn't return that value. | ||
|
|
||
| Example: | ||
|
|
||
| ```diff | ||
| { | ||
| "properties":{ | ||
| "targetType":"blobNfs", | ||
| "junctions":[ | ||
| { | ||
| "namespacePath":"/blobnfs" | ||
| } | ||
| ], | ||
| "blobNfs":{ | ||
| "target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs", | ||
| - "usageModel":"WRITE_WORKLOAD_15" | ||
|
There was a problem hiding this comment. suggest to show what is actual response and what correct should be in 2 example There was a problem hiding this comment. Sure. Will be covered in next PR. |
||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| #### RESPONSE_ADDITIONAL_VALUE | ||
|
|
||
| **Error message** Return additional response value. Path: {}. Expected: undefined. Actual: {} | ||
|
|
||
| The example doesn't define the response value, but service actually return this value. | ||
|
|
||
| Example: | ||
|
|
||
| ```diff | ||
| { | ||
| "properties":{ | ||
| "targetType":"blobNfs", | ||
| "junctions":[ | ||
| { | ||
| "namespacePath":"/blobnfs" | ||
| } | ||
| ], | ||
| "blobNfs":{ | ||
| "target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs", | ||
| "usageModel":"WRITE_WORKLOAD_15" | ||
| + "enableFeature": true | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| #### RESPONSE_INCONSISTENT_VALUE | ||
|
|
||
| **Error message** The actual response value is different from example. Path: {}. Expected: {}. Actual: {} | ||
|
|
||
| The service returned value is different from example value. | ||
|
|
||
| Example: | ||
|
|
||
| ```diff | ||
| { | ||
| "properties":{ | ||
| "targetType":"blobNfs", | ||
| "junctions":[ | ||
| { | ||
| "namespacePath":"/blobnfs" | ||
| } | ||
| ], | ||
| "blobNfs":{ | ||
| "target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs", | ||
| - "usageModel":"WRITE_WORKLOAD_15" | ||
| + "usageModel":"WORK_LOAD_14" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| #### ROUNDTRIP_INCONSISTENT_PROPERTY | ||
|
|
||
| **Error message** The property's value in the response is different from what was set in the request. Path: {}. Request: {}. Response: {} | ||
|
|
||
| Example: The sku in request parameters is `default`, but actual return is `standard`. | ||
|
|
||
| ```diff | ||
| { | ||
| "parameters":{ | ||
| "properties":{ | ||
| "name":"myService", | ||
| "SKU":"default" | ||
| } | ||
| }, | ||
| "responses":{ | ||
| "200":{ | ||
| "properties":{ | ||
| "name":"myService", | ||
| + "SKU":"standard" | ||
| - "SKU":"default" | ||
|
|
||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| ``` | ||
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| # Test scenario integrate with armTemplate | ||
|
|
||
| ## Background | ||
|
|
||
| In some cases, we need to do some more complex operations before creating a resource. For example, | ||
|
|
||
| - Create Azure SignalR service with a global unique name. Using ARMTemplate to generate a random unique string. | ||
| - Create VM with a storage account. Using ARMTemplate to provision storage account and passing the storage account resourceId as VM creation parameter. | ||
|
|
||
| ## Examples | ||
|
|
||
| Here is an example about `generate unique resource name for signalR service` | ||
|
|
||
| #### Generate unique resource name | ||
|
|
||
| We use `armTemplate output` to overwrite `resourceName` variable and following `createResource` step will use this variable. Below is generate unique name armTemplate. This armTemplate output `resourceName` variables, so test scenario following step will using the output variable. | ||
|
|
||
| ```json | ||
| { | ||
| "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", | ||
| "contentVersion": "1.0.0.0", | ||
| "parameters": { | ||
| "resourceNamePrefix": { | ||
| "type": "string", | ||
| "defaultValue": "signalr-" | ||
| } | ||
| }, | ||
| "variables": { | ||
| "resourceName": "[concat(parameters('resourceNamePrefix'), uniqueString(resourceGroup().id))]" | ||
| }, | ||
| "resources": [], | ||
| "outputs": { | ||
| "resourceName": { | ||
| "type": "string", | ||
| "value": "[variables('resourceName')]" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| After we have this armTemplate, we could define current test scenario file. We defined `resourceName` variable globally. `./generate_unique_string.json` is armTemplate. | ||
|
|
||
| `SignalR_CreateOrUpdate.json` | ||
|
|
||
| ```json | ||
| { | ||
| "parameters": { | ||
| "parameters": { | ||
| "tag": { | ||
| "key1": "tag1" | ||
| }, | ||
| "properties":{ | ||
| ... | ||
| } | ||
| }, | ||
| "api-version": "2020-07-01-preview", | ||
| "subscriptionId": "00000000-0000-0000-0000-000000000000", | ||
| "resourceGroupName": "myResourceGroup", | ||
| "resourceName": "mySignalRService123xx" | ||
| }, | ||
| "responses": { | ||
| "200": { | ||
| "body": { | ||
| ... | ||
| } | ||
| }, | ||
| "201": { | ||
| "body": { | ||
| ... | ||
| } | ||
| }, | ||
| "202": {} | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| > NOTE: the example file `../examples/SignalR_CreateOrUpdate` has the same parameter name `resourceName`. So it will be automatically overwrite in runner. | ||
|
|
||
| ```yaml | ||
| scope: ResourceGroup | ||
| variables: | ||
| resourceName: "" | ||
| testScenarios: | ||
| - description: Microsoft.SignalRService/signalR CRUD | ||
| steps: | ||
| - step: Generate_Unique_string | ||
| armTemplateDeployment: ./generate_unique_string.json | ||
| - step: SignalR_checknameAvailability | ||
| exampleFile: ../examples/SignalR_CheckNameAvailability.json | ||
| - step: SignalR_CreateOrUpdate | ||
| exampleFile: ../examples/SignalR_CreateOrUpdate.json | ||
| - step: SignalR_ListKey | ||
| exampleFile: ../examples/SignalR_ListKeys.json | ||
| - step: SignalR_Delete | ||
| exampleFile: ../examples/SignalR_Delete.json | ||
| ``` | ||
|
|
||
| **Result**: | ||
|
|
||
|  | ||
|
|
||
| ## Reference | ||
|
|
||
| - [ARMTemplate deployment script](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-script-template) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| { | ||
| "parameters": { | ||
| "api-version": "2017-04-01" | ||
| }, | ||
| "responses": { | ||
| "200": { | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| scope: ResourceGroup | ||
| testScenarios: | ||
| - description: Microsoft.SignalRService/SignalR CreateOrUpdate | ||
| steps: | ||
| - step: CreateOrUpdate | ||
| exampleFile: ../examples/CreateOrUpdate.json | ||
| - step: Delete | ||
| exampleFile: ../examples/Delete.json |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| { | ||
| "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", | ||
| "contentVersion": "1.0.0.0", | ||
| "parameters": { | ||
| "resourceNamePrefix": { | ||
| "type": "string", | ||
| "defaultValue": "signalr-" | ||
| } | ||
| }, | ||
| "variables": { | ||
| "resourceName": "[concat(parameters('resourceNamePrefix'), uniqueString(resourceGroup().id))]" | ||
| }, | ||
| "resources": [ | ||
| ], | ||
| "outputs": { | ||
| "resourceName": { | ||
| "type": "string", | ||
| "value": "[variables('resourceName')]" | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| scope: ResourceGroup | ||
| testScenarios: | ||
| - description: operationsList | ||
| steps: | ||
| - step: operationsList | ||
| exampleFile: ../examples/Operations_List.json |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#", | ||
| "contentVersion": "1.0.0.0", | ||
| "parameters": { | ||
| "configStoreName": { | ||
| "value": "ruowantest1111" | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,17 @@ | ||
| scope: ResourceGroup | ||
| variables: | ||
| resourceName: "" | ||
| resourceGroupPrefix: "apiTest-" | ||
| testScenarios: | ||
| - description: Microsoft.SignalRService/signalR SignalR_CreateOrUpdate | ||
| steps: | ||
| - step: Generate_Unique_string | ||
| armTemplateDeployment: ./generate_unique_string.json | ||
| - step: SignalR_checknameAvailability | ||
| exampleFile: ../examples/SignalR_CheckNameAvailability.json | ||
| - step: SignalR_CreateOrUpdate | ||
| exampleFile: ../examples/SignalR_CreateOrUpdate.json | ||
| - step: SignalR_ListKey | ||
| exampleFile: ../examples/SignalR_ListKeys.json | ||
| - step: SignalR_Delete | ||
| exampleFile: ../examples/SignalR_Delete.json |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| { | ||
| "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", | ||
| "contentVersion": "1.0.0.0", | ||
| "parameters": { | ||
| "configStoreName": { | ||
| "type": "string", | ||
| "metadata": { | ||
| "description": "Specifies the name of the app configuration store." | ||
| } | ||
| }, | ||
| "location": { | ||
| "type": "string", | ||
| "defaultValue": "[resourceGroup().location]", | ||
| "metadata": { | ||
| "description": "Specifies the Azure location where the app configuration store should be created." | ||
| } | ||
| }, | ||
| "skuName": { | ||
| "type": "string", | ||
| "defaultValue": "standard", | ||
| "metadata": { | ||
| "description": "Specifies the SKU of the app configuration store." | ||
| } | ||
| } | ||
| }, | ||
| "resources": [ | ||
| { | ||
| "type": "Microsoft.AppConfiguration/configurationStores", | ||
| "apiVersion": "2019-10-01", | ||
| "name": "[parameters('configStoreName')]", | ||
| "location": "[parameters('location')]", | ||
| "sku": { | ||
| "name": "[parameters('skuName')]" | ||
| } | ||
| } | ||
| ] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| scope: ResourceGroup | ||
| testScenarios: | ||
| - description: operationsList | ||
| steps: | ||
| - step: operationsList | ||
| exampleFile: ../examples/Operations_List.json |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a great starting point we have these error codes, can we add a section focus on each rule to describe how to fix? It's good to be more specific to guide service team to fix them, and it seems missing x-ms-secret or readonly are common issues spotting from the pilot
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. Maybe I need to write a doc about validation logic. It will be covered in next PR.