Skip to content

Commit 45bf753

Browse files
ruowanruowan
authored
add apitestErrorCode doc (#14749)
* add apitestErrorCode doc * add armRPC doc * add roundTripInconsistentProperty doc * small fix * update * update signalR armTemplate * update yaml * new file * add testScenario with armTemplate * update doc * update readme.md * update doc * update NOTE section * add signalRCreateOrUpdate example file * udpate doc * add generate test scenario section * update doc * add serviceFacbric test scenario file * update managedClusters.yaml * update generateABasicTestScenario.md * update features doc * add run api test gif
1 parent 682c968 commit 45bf753

File tree

22 files changed

+430
-4
lines changed

22 files changed

+430
-4
lines changed

‎documentation/test-scenario/APITestErrorCode.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
<!--
2+
Copyright (c) 2021 Microsoft Corporation
3+
4+
This software is released under the MIT License.
5+
https://opensource.org/licenses/MIT
6+
-->
7+
8+
# API testing error code
9+
10+
OAV api testing define some rules and check whether actual service response match with example. Example is very important for downstream SDK test code generation, code samples and docs. The goal to detect example quality issues and check service behaviors consistent with example.
11+
12+
## Rule descriptions
13+
14+
#### INCORRECT_PROVISIONING_STATE
15+
16+
If service return 200 status code,it means the operation is succeed and finished. So the provisioning state should be one of the terminal states ["succeeded", "failed", "canceled", "ready", "created", "deleted"].
17+
18+
The provisioning state is very important for downstream terraform or cli to manage resource status. For more details about provisioning state, please refer to this [ARM RPC provisioning state](https://github.com/Azure/azure-resource-manager-rpc/blob/master/v1.0/Addendum.md#provisioningstate-property)
19+
20+
#### RESPONSE_MISSING_VALUE
21+
22+
**Error message**: The response value is missing. Path: {}. Expected: {}. Actual: undefined
23+
24+
The example has defined response value, but actually the server doesn't return that value.
25+
26+
Example:
27+
28+
```diff
29+
{
30+
"properties":{
31+
"targetType":"blobNfs",
32+
"junctions":[
33+
{
34+
"namespacePath":"/blobnfs"
35+
}
36+
],
37+
"blobNfs":{
38+
"target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs",
39+
- "usageModel":"WRITE_WORKLOAD_15"
40+
}
41+
}
42+
}
43+
```
44+
45+
#### RESPONSE_ADDITIONAL_VALUE
46+
47+
**Error message** Return additional response value. Path: {}. Expected: undefined. Actual: {}
48+
49+
The example doesn't define the response value, but service actually return this value.
50+
51+
Example:
52+
53+
```diff
54+
{
55+
"properties":{
56+
"targetType":"blobNfs",
57+
"junctions":[
58+
{
59+
"namespacePath":"/blobnfs"
60+
}
61+
],
62+
"blobNfs":{
63+
"target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs",
64+
"usageModel":"WRITE_WORKLOAD_15"
65+
+ "enableFeature": true
66+
}
67+
}
68+
}
69+
```
70+
71+
#### RESPONSE_INCONSISTENT_VALUE
72+
73+
**Error message** The actual response value is different from example. Path: {}. Expected: {}. Actual: {}
74+
75+
The service returned value is different from example value.
76+
77+
Example:
78+
79+
```diff
80+
{
81+
"properties":{
82+
"targetType":"blobNfs",
83+
"junctions":[
84+
{
85+
"namespacePath":"/blobnfs"
86+
}
87+
],
88+
"blobNfs":{
89+
"target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs",
90+
- "usageModel":"WRITE_WORKLOAD_15"
91+
+ "usageModel":"WORK_LOAD_14"
92+
}
93+
}
94+
}
95+
```
96+
97+
#### ROUNDTRIP_INCONSISTENT_PROPERTY
98+
99+
**Error message** The property's value in the response is different from what was set in the request. Path: {}. Request: {}. Response: {}
100+
101+
Example: The sku in request parameters is `default`, but actual return is `standard`.
102+
103+
```diff
104+
{
105+
"parameters":{
106+
"properties":{
107+
"name":"myService",
108+
"SKU":"default"
109+
}
110+
},
111+
"responses":{
112+
"200":{
113+
"properties":{
114+
"name":"myService",
115+
+ "SKU":"standard"
116+
- "SKU":"default"
117+
118+
}
119+
}
120+
}
121+
}
122+
123+
```

‎documentation/test-scenario/armTemplate.png‎

172 KB
Loading

‎documentation/test-scenario/genTestScenario.gif‎

65.7 KB
Loading

‎documentation/test-scenario/generateABasicTestScenario.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Generate a basic test scenario file
2+
3+
## Prerequisite
4+
5+
We use `oav` tools to generate basic test scenario. `oav` analyze swagger file and use swagger example as test scenario steps.
6+
7+
## Introduction
8+
9+
`oav` support rule based test scenario file generation. We use this command to generate test scenario file.
10+
11+
`oav generate-static-test-scenario --readme <readme> --tag <tag> --rules <generated-rules>`
12+
13+
- readme: swagger readme file.
14+
- tag: which tag to generate. oav will analyze swagger file under the tag and generate test scenario.
15+
16+
Example:
17+
18+
![](./genTestScenario.gif)
19+
20+
This command will load and analyze swagger and generate a basic test scenario file.
21+
22+
Result: the output contains two files
23+
24+
- test-scenarios/signalR.yaml: The test scenario file.
25+
- readme.test.md: The entry for SDK test generation
26+
27+
The generated test scenario file: The generated test scenario file contains two steps. Create signalR and delete it. It's a basic test scenario and developer can add more step based on the basic test scenario file.
28+
29+
```
30+
scope: ResourceGroup
31+
testScenarios:
32+
- description: Microsoft.SignalRService/signalR SignalR_CreateOrUpdate
33+
steps:
34+
- step: SignalR_CreateOrUpdate
35+
exampleFile: ../examples/SignalR_CreateOrUpdate.json
36+
- step: SignalR_Delete
37+
exampleFile: ../examples/SignalR_Delete.json
38+
```
39+
40+
## Reference
41+
42+
- [oav](https://github.com/Azure/oav/tree/develop)

‎documentation/test-scenario/readme.md‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,24 @@
11
# Test Scenario Documentation
22

3+
## Features
4+
5+
`oav` support run test scenario file.
6+
7+
- Very easy to use and run.
8+
- Support postman collection format. Debug easily.
9+
- Request response validation. `oav` implement a powerful validation algorithm and help developer to detect service issue in the early phase.
10+
- Validation result report. After each run test scenario, developer will get a validation report which contains detect issue in api test.
11+
- Integrate everywhere. Easily integrate with azure-pipeline, cloud-test.
12+
13+
#### Demo gif
14+
15+
![](./runApiTest.gif)
16+
317
## Quick start
418

5-
[QuickStart](./QuickStart.md)
19+
- [Example: Write and run your first test scenario file](./QuickStart.md)
20+
- [Example: Generate a basic test scenario file](./generateABasicTestScenario.md)
21+
- [Example: use armTemplate to generate unique resourceName](./testScenarioWithARMTemplate.md)
622

723
## References
824

‎documentation/test-scenario/runApiTest.gif‎

2.42 MB
Loading

‎documentation/test-scenario/testScenarioWithARMTemplate.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# Test scenario integrate with armTemplate
2+
3+
## Background
4+
5+
In some cases, we need to do some more complex operations before creating a resource. For example,
6+
7+
- Create Azure SignalR service with a global unique name. Using ARMTemplate to generate a random unique string.
8+
- Create VM with a storage account. Using ARMTemplate to provision storage account and passing the storage account resourceId as VM creation parameter.
9+
10+
## Examples
11+
12+
Here is an example about `generate unique resource name for signalR service`
13+
14+
#### Generate unique resource name
15+
16+
We use `armTemplate output` to overwrite `resourceName` variable and following `createResource` step will use this variable. Below is generate unique name armTemplate. This armTemplate output `resourceName` variables, so test scenario following step will using the output variable.
17+
18+
```json
19+
{
20+
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
21+
"contentVersion": "1.0.0.0",
22+
"parameters": {
23+
"resourceNamePrefix": {
24+
"type": "string",
25+
"defaultValue": "signalr-"
26+
}
27+
},
28+
"variables": {
29+
"resourceName": "[concat(parameters('resourceNamePrefix'), uniqueString(resourceGroup().id))]"
30+
},
31+
"resources": [],
32+
"outputs": {
33+
"resourceName": {
34+
"type": "string",
35+
"value": "[variables('resourceName')]"
36+
}
37+
}
38+
}
39+
```
40+
41+
After we have this armTemplate, we could define current test scenario file. We defined `resourceName` variable globally. `./generate_unique_string.json` is armTemplate.
42+
43+
`SignalR_CreateOrUpdate.json`
44+
45+
```json
46+
{
47+
"parameters": {
48+
"parameters": {
49+
"tag": {
50+
"key1": "tag1"
51+
},
52+
"properties":{
53+
...
54+
}
55+
},
56+
"api-version": "2020-07-01-preview",
57+
"subscriptionId": "00000000-0000-0000-0000-000000000000",
58+
"resourceGroupName": "myResourceGroup",
59+
"resourceName": "mySignalRService123xx"
60+
},
61+
"responses": {
62+
"200": {
63+
"body": {
64+
...
65+
}
66+
},
67+
"201": {
68+
"body": {
69+
...
70+
}
71+
},
72+
"202": {}
73+
}
74+
}
75+
```
76+
77+
> NOTE: the example file `../examples/SignalR_CreateOrUpdate` has the same parameter name `resourceName`. So it will be automatically overwrite in runner.
78+
79+
```yaml
80+
scope: ResourceGroup
81+
variables:
82+
resourceName: ""
83+
testScenarios:
84+
- description: Microsoft.SignalRService/signalR CRUD
85+
steps:
86+
- step: Generate_Unique_string
87+
armTemplateDeployment: ./generate_unique_string.json
88+
- step: SignalR_checknameAvailability
89+
exampleFile: ../examples/SignalR_CheckNameAvailability.json
90+
- step: SignalR_CreateOrUpdate
91+
exampleFile: ../examples/SignalR_CreateOrUpdate.json
92+
- step: SignalR_ListKey
93+
exampleFile: ../examples/SignalR_ListKeys.json
94+
- step: SignalR_Delete
95+
exampleFile: ../examples/SignalR_Delete.json
96+
```
97+
98+
**Result**:
99+
100+
![](./armTemplate.png)
101+
102+
## Reference
103+
104+
- [ARMTemplate deployment script](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-script-template)

‎specification/servicebus/resource-manager/Microsoft.ServiceBus/preview/2021-01-01-preview/examples/SBOperations_List.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"parameters": {
3-
"api-version": "2021-01-01-preview"
3+
"api-version": "2017-04-01"
44
},
55
"responses": {
66
"200": {

‎specification/servicebus/resource-manager/Microsoft.ServiceBus/preview/2021-01-01-preview/operations.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
"info": {
44
"title": "ServiceBusManagementClient",
55
"description": "Azure Service Bus client",
6-
"version": "2021-01-01-preview"
6+
"version": "2017-04-01"
77
},
88
"host": "management.azure.com",
99
"schemes": [

‎specification/servicefabric/resource-manager/Microsoft.ServiceFabric/preview/2020-01-01-preview/test-scenarios/managedClusters.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
scope: ResourceGroup
2+
testScenarios:
3+
- description: >-
4+
Microsoft.ServiceFabric/managedClusters Put a cluster with minimum
5+
parameters
6+
steps:
7+
- step: Put a cluster with minimum parameters
8+
exampleFile: ../examples/ManagedClusterPutOperation_example_min.json
9+
- step: Delete a cluster
10+
exampleFile: ../examples/ManagedClusterDeleteOperation_example.json
11+
- step: Put a cluster with maximum parameters
12+
exampleFile: ../examples/ManagedClusterPutOperation_example_max.json
13+
- step: Delete a cluster
14+
exampleFile: ../examples/ManagedClusterDeleteOperation_example.json

0 commit comments

Comments
 (0)