prohibit the use of fragments in $id and $schema

[karenetheridge]

We permit empty fragments in $id, and say nothing at all about fragments in $schema. IMO we should prohibit all kinds of fragments in both keywords' URIs.

cross-reference: OAI/OpenAPI-Specification#2433 (comment)

[jdesrosiers]

I'd rather go the other direction and make $id and $schema consistent with $ref. The absolute-URI portion determines the document fetched from the schema-store and the fragment portion determines which part of the document to start evaluation when processing the schema.

For $schema, this would mean that all of the following would identify the same dialect, but might resolve to different meta-schemas.

• https://json-schema.org/draft/future
• https://json-schema.org/draft/future#
• https://json-schema.org/draft/future#strict
• https://json-schema.org/draft/future#uber-strict

For $id, this would mean that embedded schema evaluation would need to take into account the fragment to determine where to begin evaluation.

{
  "type": "object",
  "properties": {
    "foo": {
      "$id": "/schema/bundle#/$defs/foo",
      "$defs": {
        "string": { "type": "string" },
        "foo": { "$ref": "#/defs/string" }
      }
    }
  }
}

In this example, /properties/foo resolves to { "type": "string" }. It might seem strange when we look at it embedded like this, but this is exactly what is happening when we resolve a $ref. We begin validation at the location identified by the fragment, but the whole document has to be in scope in order to resolve references.

If we change that example slightly, I think it's a little easier to see how the $id with fragment is the same as $ref with fragment.

{
  "type": "object",
  "properties": {
    "foo": { "$ref": "/schema/bundle#/$defs/foo" }
  },
  "$defs": {
    "": {
      "$id": "/schema/bundle",
      "$defs": {
        "string": { "type": "string" },
        "foo": { "$ref": "#/defs/string" }
      }
    }
  }
}

It's certainly more intuitive written like this, but I don't think aesthetics are good enough reason to forbid using fragments in $id.

[notEthan]

allowing pointer fragments in $id was an oversight to begin with. disallowing it resolved a confusing/undefined situation where a schema could identify itself as being at one pointer, while actually being at a different pointer in the resource or document.

@jdesrosiers I'm not sure why $id should allow a fragment, even in your proposal - which I understand is not the same as declaring the location of the schema itself as schemas used to sometimes do. what you suggest looks like a confusing alternative to $ref, and I think your example schema has an equivalent using $ref:

{
  "type": "object",
  "properties": {
    "foo": {
      "$id": "/schema/bundle",
      "$ref": "#/$defs/foo",
      "$defs": {
        "string": { "type": "string" },
        "foo": { "$ref": "#/defs/string" }
      }
    }
  }
}

I see no upside to putting a fragment in $id, only confusion.

whether $id should allow an empty fragment, I think it should be disallowed at some point. it has remained for historical reasons, I think, but every new draft breaks compatibility of more major things than that anyway.

as for a $schema with a fragment, I don't know that it matters. if the $schema URI does not point to a metaschema the implementation is able to use, fragment or no, the implementation will presumably error attempting to resolve that.

it seems faintly plausible to me that a metaschema could be identified by $schema using a fragment (my implementation does support metaschemas which are not at the root of their document) - but it's hard to imagine the value of such a thing.

[MikeRalphson]

it seems faintly plausible to me that a metaschema could be identified by $schema using a fragment (my implementation does support metaschemas which are not at the root of their document) - but it's hard to imagine the value of such a thing.

This was exactly the use-case we were discussing over at OAS. We already have a meta-schema for OAS 3.0 (and 1.2 and 2.0) but it contains embedded definitions of the OAS schemaObject (which in these versions is a superset-subset of proper JSON Schema). It would be useful for users who want to continue to use OAS 3.0-compatible schemaObjects to be able to reference the schemaObject definition within the existing metaschema using $schema or our jsonSchemaDialect property, without the oai/TSC having to extract that schemaObject definition to its own resource and maintain it separately.

[notEthan]

@MikeRalphson that is interesting and worth thinking about in this context, although I think that embedded metaschema will almost certainly have an $id, and so not need to use any fragment to refer to it with $schema. if the document it is embedded in has another id, it might be possible to address that metaschema using that id with a fragment, but it would probably be strongly recommended against (which is why I say above I don't imagine such a construct probably has much value). I'm making a number of suppositions here without knowledge of the openapi work, though.

[MikeRalphson]

@notEthan That's useful - thanks. The use of a separate $id might be work for us, though I think some JSON Schema-related tools do still make assumptions that URIs are network-reachable URLs.

[jdesrosiers]

@notEthan

what you suggest looks like a confusing alternative to $ref

In 2019-09, $id was redefined as an embedded reference. They already are alternatives. This proposal doesn't introduce that property.

disallowing [fragments in $ids] resolved a confusing/undefined situation where a schema could identify itself as being at one pointer, while actually being at a different pointer in the resource or document.

I know the situation you're referring to. It is confusing, but it has never been undefined. That has always been an unambiguously invalid use of $id. Before 2019-09, it wasn't well defined what a URI with a fragment meant in an $id. In 2019-09, when $id was redefined to be an embedded reference, fragments were no longer undefined. It should be expected that they work the same as references. But, it was decided to disallow fragments anyway.

I think your example schema has an equivalent using $ref

Yep, that's another equivalent example. Allowing $id to have a fragment doesn't allow you to do anything that you can't already do. I only suggest it in order to simplify the conceptual model.

I see no upside to putting a fragment in $id, only confusion.

It makes for a simpler conceptual model. Fewer concepts with fewer exceptions means a simpler spec and implementations that are easier to write, have better performance, and are less bug prone. That's the main benefit. I agree that that benefit is marginal, but I'd rather not have exceptions to rules without a good reason, and don't see a good reason.

whether $id should allow an empty fragment, I think it should be disallowed at some point. it has remained for historical reasons

The same URI can be represented by multiple strings. It's the normalized URI that's important. For example, https://json-schema.org/draft/2020-12/schema and https://json-schema.org/draft/../draft/2020-12/schema are the same URI. I'll have to check the RFC to be 100% sure, but I'm pretty sure a schema with an empty hash normalizes to the hash being removed. If I'm right about that, I see no good reason to forbid an empty fragment. It doesn't hurt anything and it represents the same URI.

it's hard to imagine the value of such a thing

I agree, but I don't want to do extra work to forbid something that isn't causing any harm just in case someone comes up with a good use for it.

[karenetheridge]

I want to be clear that I opened this issue to discuss these constructs:

• { "$id": "https://example.com#" } NOT { "$id": "https://example.com#/foo" } -- the latter is already disallowed
• { "$schema": "https://example.com#" } or { "$schema": "https://example.com#/foo" } but note that the $schema keyword MUST already have an adjacent $id, so the uri in the $schema keyword is using a non-canonical URI when there is already a fragmentless URI available that would work just as well

$id with a json pointer or plain-name fragment makes no sense because this isn't a canonical URI. it's moving the goalposts to bring those possibilities back into play when there is no reason to allow them.

[jdesrosiers]

First of all, sorry for hijacking this issue a bit, but I thought it was necessary to give an explanation for why I'd prefer to go a different direction. I'm happy to drop this if there's not support, but ...

it's moving the goalposts to bring those possibilities back into play when there is no reason to allow them.

this statement makes me think I haven't presented the proposal clearly. I'm not proposing bringing anything back. Every problem we have previously fixed will stay fixed. My proposal introduces something that never existed before. Originally, fragments in $ids were allowed but they had no semantics. Then fragments were forbidden. Now I propose fragments be allowed, but this time with defined semantics and those semantics would align with $ref.

$id with a json pointer or plain-name fragment makes no sense because this isn't a canonical URI.

I'm not following your point here, but URIs with fragments can be canonical.

Example from the spec ...

Consider the following schema document that contains another schema resource embedded within it:

{
  "$id": "https://example.com/foo",
  "items": {
    "$id": "https://example.com/bar",
    "additionalProperties": { }
  }
}

The URI "https://example.com/foo#/items/additionalProperties" points to the schema of the "additionalProperties" keyword in the embedded resource. The canonical URI of that schema, however, is "https://example.com/bar#/additionalProperties".

[karenetheridge]

I think modifications to the $schema and $id keywords would be best discussed in a new issue (and separate issues, as they are two very different things).

• allowing $schema to handle fragments (either plain-name or json pointer) could possibly have a use (it is similar to $ref in that evaluation first resolves the URI, and then does something with the referenced document); however this is currently prohibited by the spec (which mandates that $schema must appear at a resource root, i.e. an adjacent $id must be present, and I think there were good reasons for making that requirement (edit: I mixed up the rules for where a $schema keyword can be located, and the content of its URI itself)
• I did not follow your (Jason's) proposal for fragments in $id at all, either in how it would work or the motivation behind the proposed change, sorry

[jdesrosiers]

I did not follow your (Jason's) proposal for fragments in $id at all

I'll drop it then. I might bring it up again in the future if I come up with a better way to explain it.

allowing $schema to handle fragments [...] is currently prohibited by the spec (which mandates that $schema must appear at a resource root, i.e. an adjacent $id must be present

What I had in mind doesn't alter the resource root, only the evaluation starting point (just like $ref). No matter where evaluation starts, the JSON Schema engine can always find the schema declaration at #/$schema.

{
  "$id": "https://example.com/my-schema",
  "$schema": "https://json-schema.org/draft/future/schema#strict"
}

{
  "$id": "https://json-schema.org/draft/future/schema",
  "$schema": "https://json-schema.org/draft/future/schema",

  ... standard schema ...

  "$defs": {
    "strict": {
      "$anchor": "strict"
      ... strict schema ...
    }
  }
}

Nothing has to change about where $schema is allowed. The fragment just says, start evaluating the schema at #strict rather than at #. It still finds the $id and $schema in the same place it always has.

[jdesrosiers]

I'll have to check the RFC to be 100% sure, but I'm pretty sure a schema with an empty hash normalizes to the hash being removed. If I'm right about that, I see no good reason to forbid an empty fragment. It doesn't hurt anything and it represents the same URI.

I looked this up and I was wrong. Normalization does not remove an empty fragment.

The fragment component is not subject to any scheme-based normalization; thus, two URIs that differ only by the suffix "#" are considered different regardless of the scheme.

• https://tools.ietf.org/html/rfc3986#section-6.2.3

[karenetheridge]

I revised my earlier comment - I mixed up the rules for where a $schema keyword can be, and the content of its URI itself, sorry! I think the spec is silent at the moment about whether a $schema URI can have a fragment.. so perhaps it is already possible today to have a schema containing e.g. "$schema": "https://example.com/metaschema.json#/properties/$defs/my_schema" ?

[handrews]

I would be thrilled to see the empty fragment banned from both $id and $schema. Both should be non-fragment URIs. The empty fragment in $id was for historical compatibility, and in $schema to be able to match $ids again for historical compatibility.

@jdesrosiers the reason fragments do not have scheme-based normalization is that the fragment syntax and semantics are entirely defined by the media type. The URI scheme has nothing to do with fragments, nor does the RFC 3986 normalization process. For JSON Schema, an empty fragment and no fragment are the same, because we defined them to be that way. This is why the spec for $id says and MUST resolve to an absolute-URI because at some point we had a discussion about this and "resolve to" was the language people were most comfortable with. It is specifically separate from normalization which is in the previous sentence.

Re: non-empty fragments in $schema – people get confused enough with meta-schemas as it is. I really thought we had language for $schema to be an absolute URI (with scheme, without fragment) or equivalent to one (empty fragment, because of our media-type-specific fragment semantics). Please don't encourage people to do this! EDIT: I still really don't like fragments in $schema but I see OAS has a use for it. idk why people don't like having multiple files as I split things up every chance I get, but there's always this lumper vs splitter thing that shows up in many contexts so 🤷

I would also really hate to see fragments returned to $id. I don't plan to do more here than drop in occasionally, so I know my comments don't carry much weight anymore and that's fine- I am thrilled to see work continue and don't want to disrupt that! But removing fragments from $id and making it sensible is something I consider one of my more significant achievements. EDIT: In terms of impact, I mean — y'all don't need to keep features solely because of my pride in them! It took years of incremental changes to get the point where we could split off fragments into $anchor and people were comfortable with that. $id was one of the reasons that OpenAPI would not support full compatibility (I don't remember if it was specifically the fragment part, but there were a lot of things wrong with what was then id when OAS 3.0 went out). The changes we made were necessary to get JSON Schema out in the world more broadly.

[handrews]

disallowing [fragments in $ids] resolved a confusing/undefined situation where a schema could identify itself as being at one pointer, while actually being at a different pointer in the resource or document.

I know the situation you're referring to. It is confusing, but it has never been undefined. That has always been an unambiguously invalid use of $id.

It was definitely possible to do, and it was not clearly defined. I know because I did a very, very detailed implementation of id (pre-$!) and $ref as defined in draft-05 (which does not exist! 😉 ) and could not figure out how to handle several cases including that one. Should you allow both? What happens if someone does a same-document reference under an id with a JSON Pointer fragment where the same-document reference is also a JSON Pointer fragment, but one that would be impossible to co-exist with the one in the id? But that if resolved against the id base URI does point to a valid schema?

Once you started dealing with not only the behavior of the fragment in id but the base URI behavior and how that interacts with fragments in $ref it got really, really messy. It's the combination that really exposed the deeper problems.

[jdesrosiers]

disallowing [fragments in $ids] resolved a confusing/undefined situation where a schema could identify itself as being at one pointer, while actually being at a different pointer in the resource or document.

I know the situation you're referring to. It is confusing, but it has never been undefined. That has always been an unambiguously invalid use of $id.

It was definitely possible to do, and it was not clearly defined.

Ever since draft-05, it's been defined what characters are allowed in id fragments and those characters do not include /. Therefore, JSON Pointers are not allowed in ids. That's why I say using JSON Pointers in $id fragments is an unambiguously invalid use of $id. I thought that was part of draft-04 as well, but I checked and it I was wrong. It was added in draft-05.