Skip to content

Commit a9a34f0

Browse files
loosealelooseale
authored
Merge pull request #2049 from aml-org/publish-5.6.2-master
Publish 5.6.2
2 parents cd063dc + 6ef5cbc commit a9a34f0

23 files changed

+364
-60
lines changed

‎adrs/0014-avro-support.md

+10-10
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Accepted
1010

1111
## Context
1212

13-
Async 2.x supports AVRO Schemas, and we currently don't.
13+
Async 2.x supports AVRO Schemas, and we currently don't.
1414
We want to add support of AVRO Schemas:
1515
- inside Async APIs
1616
- as a standalone document
@@ -54,33 +54,33 @@ We've also added 3 AVRO-specific fields to the `AnyShape` Model via the `AvroFie
5454

5555
### Where we support and DON'T support AVRO Schemas
5656
We Support AVRO Schemas (inline or inside a `$ref`):
57-
- as a standalone document or file
58-
- we encourage users to use the `.avsc` file type to indicate that's an avro file, for better suggestions and so on in the platform)
57+
- as a standalone document or file
58+
- we encourage users to use the `.avsc` file type to indicate that's an avro file
5959
- must use the specific `AvroConfiguration`
6060
- inside a message payload in an AsyncAPI
6161
- the key `schemaFormat` MUST be declared and specify it's an AVRO payload
62-
- we only support avro schema version 1.9.0
62+
- **we only support avro schema version 1.9.0**
6363
- the avro specific document `AvroSchemaDocument` can only be parsed with the specific `AvroConfiguration`
6464

6565
We don't support AVRO Schemas:
6666
- inside components --> schemas in an AsyncAPI
6767
- because we can't determine if it's an AVRO Schema or any other schema
6868

6969
### AVRO Validation
70-
We'll use the Apache official libraries for JVM and JS.
70+
We'll use the Apache official libraries for JVM and JS, in the version 1.11.3, due Java8 binary compatibility requirements.
7171

7272
## Consequences / Constraints
7373

74-
The validation plugins differ in interfaces and implementations, and each has some constraints:
74+
The validation libraries differ in interfaces and implementations, and each has some constraints:
7575

76-
### JVM avro validation plugin
76+
### JVM avro validation library
7777
- validation per se is not supported, we try to parse an avro schema and throw parsing results if there are any
7878
- this means it's difficult to have location of where the error is thrown, we may give an approximate location from our end post-validation
7979
- when a validation is thrown, the rest of the file is not being searched for more validations
8080
- this is particularly important in large avro schemas, where many errors can be found but only one is shown
8181

82-
### Both JVM & JS validation plugins
82+
### Both JVM & JS validation libraries
8383
- `"default"` values are not being validated when the type is `bytes`, `map`, or `array`
8484
- the validator treats as invalid an empty array as the default value for arrays (`"default": []`) even though the [Avro Schema Specification](https://avro.apache.org/docs/1.12.0/specification) has some examples with it
85-
- if an avro record has a field that is a union that includes the root record itself (recursive reference) we fail to validate it correctly because we treat that shape as an unresolved/undefined shape
86-
- in the future we'll try to ignore the cases that we are now failing and/or show a warning instead
85+
- avro schemas of type union are not valid at the root level of the schema, but they are accepted as field types in a record or items in an array. For this reason, it is not possible to generate a `PayloadValidator` for an avro union shape.
86+
- the avro libraries are very restrictives with the allowed characters in naming definition (names of records for example). They only allow letters, numbers and `_` chars. We are not modifying this behavior.

‎amf-api-contract/shared/src/main/scala/amf/apicontract/internal/spec/avro/parser/domain/AvroArrayShapeParser.scala

+3-1
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,9 @@ case class AvroArrayShapeParser(map: YMap)(implicit ctx: AvroSchemaContext)
1111
override def setMembers(anyShape: AnyShape): Unit =
1212
shape.setWithoutId(ArrayShapeModel.Items, anyShape, Annotations.inferred())
1313

14-
override def parseMembers(e: YMapEntry): AnyShape = AvroTextParser(e.value).parse()
14+
override def parseMembers(e: YMapEntry): AnyShape = {
15+
AvroInlineTypeParser(e.value).parse()
16+
}
1517

1618
override def parseSpecificFields(): Unit = {}
1719
}

‎amf-api-contract/shared/src/main/scala/amf/apicontract/internal/spec/avro/parser/domain/AvroTextParser.scala ‎amf-api-contract/shared/src/main/scala/amf/apicontract/internal/spec/avro/parser/domain/AvroInlineTypeParser.scala

+6-3
Original file line numberDiff line numberDiff line change
@@ -2,17 +2,20 @@ package amf.apicontract.internal.spec.avro.parser.domain
22

33
import amf.apicontract.internal.spec.avro.parser.context.AvroSchemaContext
44
import amf.shapes.client.scala.model.domain.AnyShape
5-
import org.yaml.model.{YMap, YNode, YScalar}
5+
import org.yaml.model.{YMap, YNode, YScalar, YSequence}
66

7-
case class AvroTextParser(node: YNode)(implicit ctx: AvroSchemaContext) extends AvroKeyExtractor {
7+
case class AvroInlineTypeParser(node: YNode)(implicit ctx: AvroSchemaContext) extends AvroKeyExtractor {
88
def parse(): AnyShape = {
99
node.value match {
1010
case scalar: YScalar =>
1111
val avroType = scalar.text
1212
val parsedShape = AvroTextTypeParser(scalar.text, None).parse()
1313
annotatedAvroShape(parsedShape, avroType, node)
1414
case map: YMap => // todo: putting an empty AnyShape when the union type is incorrect is kinda weird behavior
15-
new AvroShapeParser(map).parse().getOrElse(AnyShape())
15+
new AvroShapeParser(map).parse().getOrElse(AnyShape(map))
16+
case seq: YSequence => // The only valid case here for a seq is a Union
17+
AvroInlineUnionShapeParser(seq).parse()
18+
case _ => AnyShape(node)
1619
}
1720
}
1821
}

‎amf-api-contract/shared/src/main/scala/amf/apicontract/internal/spec/avro/parser/domain/AvroUnionShapeParser.scala

+22-4
Original file line numberDiff line numberDiff line change
@@ -3,16 +3,34 @@ package amf.apicontract.internal.spec.avro.parser.domain
33
import amf.apicontract.internal.spec.avro.parser.context.AvroSchemaContext
44
import amf.core.client.scala.model.domain.AmfArray
55
import amf.core.internal.parser.domain.Annotations
6-
import amf.shapes.client.scala.model.domain.UnionShape
6+
import amf.shapes.client.scala.model.domain.{AnyShape, UnionShape}
77
import amf.shapes.internal.domain.metamodel.UnionShapeModel
8-
import org.yaml.model.{YMap, YNode}
8+
import org.yaml.model.{YMap, YNode, YSequence, YValue}
99

1010
case class AvroUnionShapeParser(members: Seq[YNode], node: YNode)(implicit ctx: AvroSchemaContext)
11-
extends AvroComplexShapeParser(node.as[YMap]) {
11+
extends AvroComplexShapeParser(node.as[YMap])
12+
with AvroUnionLikeShapeParser {
1213
override val shape: UnionShape = UnionShape(node).withSynthesizeName("union")
1314

1415
override def parseSpecificFields(): Unit = {
15-
val parsedMembers = members.map(node => AvroTextParser(node).parse())
16+
val parsedMembers = parseMembers(members)
1617
shape.setWithoutId(UnionShapeModel.AnyOf, AmfArray(parsedMembers, Annotations(node)), Annotations(node))
1718
}
1819
}
20+
21+
case class AvroInlineUnionShapeParser(seq: YSequence)(implicit ctx: AvroSchemaContext)
22+
extends AvroUnionLikeShapeParser {
23+
24+
def parse(): AnyShape = {
25+
val shape: UnionShape = UnionShape(seq).withSynthesizeName("union")
26+
val parsedMembers = parseMembers(seq.nodes)
27+
shape.setWithoutId(UnionShapeModel.AnyOf, AmfArray(parsedMembers, Annotations(seq)), Annotations(seq))
28+
}
29+
30+
}
31+
32+
trait AvroUnionLikeShapeParser {
33+
def parseMembers(members: Seq[YNode])(implicit ctx: AvroSchemaContext): Seq[AnyShape] = {
34+
members.map(node => AvroInlineTypeParser(node).parse())
35+
}
36+
}

‎amf-apicontract.versions

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
amf.apicontract=5.6.1
1+
amf.apicontract=5.6.2
22
amf.aml=6.6.0
33
amf.model=3.9.0
44
antlr4Version=0.7.25

‎amf-cli/shared/src/test/resources/avro/reports/union-nullable-record-invalid.report.js

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
ModelId: file://amf-cli/shared/src/test/resources/avro/schemas/union-nullable-record-invalid.json
2+
Profile:
3+
Conforms: false
4+
Number of results: 1
5+
6+
Level: Violation
7+
8+
- Constraint: http://a.ml/vocabularies/amf/parser#invalid-avro-schema
9+
Message: Internal error during Avro validation: Error: invalid "null": 1234
10+
Severity: Violation
11+
Target: file://amf-cli/shared/src/test/resources/avro/schemas/union-nullable-record-invalid.json#/shape/unionNullableRecord
12+
Property:
13+
Range:
14+
Location: file://amf-cli/shared/src/test/resources/avro/schemas/union-nullable-record-invalid.json

‎amf-cli/shared/src/test/resources/avro/reports/union-nullable-record-invalid.report.jvm

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
ModelId: file://amf-cli/shared/src/test/resources/avro/schemas/union-nullable-record-invalid.json
2+
Profile:
3+
Conforms: false
4+
Number of results: 1
5+
6+
Level: Violation
7+
8+
- Constraint: http://a.ml/vocabularies/amf/parser#invalid-avro-schema
9+
Message: invalid schema type: Invalid default for field nullableUnion: 1234 not a ["null","string"]
10+
Severity: Violation
11+
Target: file://amf-cli/shared/src/test/resources/avro/schemas/union-nullable-record-invalid.json#/shape/unionNullableRecord
12+
Property:
13+
Range:
14+
Location: file://amf-cli/shared/src/test/resources/avro/schemas/union-nullable-record-invalid.json

‎amf-cli/shared/src/test/resources/avro/reports/union-root-invalid.report.js

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
ModelId: file://amf-cli/shared/src/test/resources/avro/schemas/union-root-invalid.json
2+
Profile:
3+
Conforms: false
4+
Number of results: 1
5+
6+
Level: Violation
7+
8+
- Constraint: http://a.ml/vocabularies/amf/parser#invalid-avro-schema
9+
Message: Internal error during Avro validation: Error: unknown type: ["string","int"]
10+
Severity: Violation
11+
Target: file://amf-cli/shared/src/test/resources/avro/schemas/union-root-invalid.json#/union/simpleUnion
12+
Property:
13+
Range:
14+
Location: file://amf-cli/shared/src/test/resources/avro/schemas/union-root-invalid.json

‎amf-cli/shared/src/test/resources/avro/reports/union-root-invalid.report.jvm

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
ModelId: file://amf-cli/shared/src/test/resources/avro/schemas/union-root-invalid.json
2+
Profile:
3+
Conforms: false
4+
Number of results: 1
5+
6+
Level: Violation
7+
8+
- Constraint: http://a.ml/vocabularies/amf/parser#invalid-avro-schema
9+
Message: No type: {"name":"simpleUnion","type":["string","int"],"default":"something"}
10+
Severity: Violation
11+
Target: file://amf-cli/shared/src/test/resources/avro/schemas/union-root-invalid.json#/union/simpleUnion
12+
Property:
13+
Range:
14+
Location: file://amf-cli/shared/src/test/resources/avro/schemas/union-root-invalid.json

‎amf-cli/shared/src/test/resources/avro/reports/union-simple-array-invalid.report.js

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
ModelId: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-array-invalid.json
2+
Profile: Avro
3+
Conforms: false
4+
Number of results: 1
5+
6+
Level: Violation
7+
8+
- Constraint: http://a.ml/vocabularies/amf/validation#example-validation-error
9+
Message: '132' is not a valid value (of type '"string"') for '0.string'
10+
Severity: Violation
11+
Target: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-array-invalid.json#/array/default-array/array_1
12+
Property: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-array-invalid.json#/array/default-array/array_1
13+
Range: [(4,13)-(4,49)]
14+
Location: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-array-invalid.json

‎amf-cli/shared/src/test/resources/avro/reports/union-simple-array-invalid.report.jvm

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
ModelId: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-array-invalid.json
2+
Profile: Avro
3+
Conforms: false
4+
Number of results: 1
5+
6+
Level: Violation
7+
8+
- Constraint: http://a.ml/vocabularies/amf/validation#schema-exception
9+
Message: invalid schema type: Expected string. Got VALUE_NUMBER_INT
10+
Severity: Violation
11+
Target: null
12+
Property: Expected string. Got VALUE_NUMBER_INT
13+
Range: [(1,0)-(5,1)]
14+
Location: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-array-invalid.json

‎amf-cli/shared/src/test/resources/avro/reports/union-simple-record-invalid.report.js

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
ModelId: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-record-invalid.json
2+
Profile:
3+
Conforms: false
4+
Number of results: 1
5+
6+
Level: Violation
7+
8+
- Constraint: http://a.ml/vocabularies/amf/parser#invalid-avro-schema
9+
Message: Internal error during Avro validation: Error: invalid "string": true
10+
Severity: Violation
11+
Target: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-record-invalid.json#/shape/unionSimpleRecord
12+
Property:
13+
Range:
14+
Location: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-record-invalid.json

‎amf-cli/shared/src/test/resources/avro/reports/union-simple-record-invalid.report.jvm

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
ModelId: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-record-invalid.json
2+
Profile:
3+
Conforms: false
4+
Number of results: 1
5+
6+
Level: Violation
7+
8+
- Constraint: http://a.ml/vocabularies/amf/parser#invalid-avro-schema
9+
Message: invalid schema type: Invalid default for field simpleUnion: true not a ["string","int"]
10+
Severity: Violation
11+
Target: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-record-invalid.json#/shape/unionSimpleRecord
12+
Property:
13+
Range:
14+
Location: file://amf-cli/shared/src/test/resources/avro/schemas/union-simple-record-invalid.json

‎amf-cli/shared/src/test/resources/avro/schemas/union-nullable-record-invalid.json

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"type": "record",
3+
"name": "unionNullableRecord",
4+
"namespace": "root",
5+
"fields": [
6+
{
7+
"name": "nullableUnion",
8+
"type": [ "null", "string"],
9+
"default": 1234
10+
}
11+
]
12+
}

‎amf-cli/shared/src/test/resources/avro/schemas/union-nullable-record-valid.json

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"type": "record",
3+
"name": "unionNullableRecord",
4+
"namespace": "root",
5+
"fields": [
6+
{
7+
"name": "nullableUnion",
8+
"type": [ "null", "string"],
9+
"default": null
10+
}
11+
]
12+
}

‎amf-cli/shared/src/test/resources/avro/schemas/union-root-invalid.json

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"name": "simpleUnion",
3+
"type": [ "string", "int"],
4+
"default": "something"
5+
}

‎amf-cli/shared/src/test/resources/avro/schemas/union-simple-array-invalid.json

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"type": "array",
3+
"items": [ "string", "int"],
4+
"default": [{"string": 132}, {"string": "b"}]
5+
}

‎amf-cli/shared/src/test/resources/avro/schemas/union-simple-array-valid.json

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"type": "array",
3+
"items": [ "string", "int"],
4+
"default": [{"string": "a"}, {"string": "b"}]
5+
}

‎amf-cli/shared/src/test/resources/avro/schemas/union-simple-record-invalid.json

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"type": "record",
3+
"name": "unionSimpleRecord",
4+
"namespace": "root",
5+
"fields": [
6+
{
7+
"name": "simpleUnion",
8+
"type": [ "string", "int"],
9+
"default": true
10+
}
11+
]
12+
}

‎amf-cli/shared/src/test/resources/avro/schemas/union-simple-record-valid.json

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"type": "record",
3+
"name": "unionSimpleRecord",
4+
"namespace": "root",
5+
"fields": [
6+
{
7+
"name": "simpleUnion",
8+
"type": [ "string", "int"],
9+
"default": "something"
10+
}
11+
]
12+
}

‎amf-cli/shared/src/test/scala/amf/avro/AvroSchemaValidationTest.scala

+57
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,63 @@ class AvroSchemaValidationTest extends MultiPlatformReportGenTest {
165165
)
166166
}
167167

168+
test("validate simple union in record - valid") {
169+
validate(
170+
"union-simple-record-valid.json",
171+
None,
172+
configOverride = Some(config)
173+
)
174+
}
175+
176+
test("validate simple union in record - invalid") {
177+
validate(
178+
"union-simple-record-invalid.json",
179+
Some("union-simple-record-invalid.report"),
180+
configOverride = Some(config)
181+
)
182+
}
183+
184+
test("validate nullable union in record - valid") {
185+
validate(
186+
"union-nullable-record-valid.json",
187+
None,
188+
configOverride = Some(config)
189+
)
190+
}
191+
192+
test("validate nullable union in record - invalid") {
193+
validate(
194+
"union-nullable-record-invalid.json",
195+
Some("union-nullable-record-invalid.report"),
196+
configOverride = Some(config)
197+
)
198+
}
199+
200+
test("validate simple union in array - valid") {
201+
validate(
202+
"union-simple-array-valid.json",
203+
None,
204+
configOverride = Some(config)
205+
)
206+
}
207+
208+
test("validate simple union in array - invalid") {
209+
validate(
210+
"union-simple-array-invalid.json",
211+
Some("union-simple-array-invalid.report"),
212+
configOverride = Some(config)
213+
)
214+
}
215+
216+
test("validate union at root level is invalid") {
217+
validate(
218+
"union-root-invalid.json",
219+
Some("union-root-invalid.report"),
220+
configOverride = Some(config)
221+
)
222+
}
223+
224+
168225
// TODO We need to see how implement this in with AVRO 1.11.3
169226

170227
// if (platform.name == "jvm") { // We were able only to change this behavior in JVM validator. JS one is still strict (only letter, numbers and '_')

0 commit comments

Comments
 (0)