Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@

**Support user cases**

- Add `Options::emit_anchor_events` (default `false`) to emit `Event::Anchor(id, name)`
immediately after the anchored scalar or collection-start event. Its span covers the complete
`&name` token, and its ID matches the node and resolved aliases without relying on allocation
order. Anchor events are presentation metadata (`is_node()` is `false`); existing node payloads
and default event streams are unchanged. Together with alias-token spans and `Span::tag_span()`,
this exposes original property spellings without separate anchor and tag scans.
`Event::anchor_id()` also recognizes nonzero IDs on `Event::Anchor`; filter with `is_node()`
when counting anchored nodes to avoid counting each definition twice.
- Add `Span::tag_span()` for the complete raw explicit-tag token range on parser node events
(we only had tag start but no end).
- Deprecate `Span::with_tag_start`, retaining it for compatibility. Use `Span::tag_span()` on
Expand All @@ -25,6 +33,9 @@

**Fixed**

- Track anchor-property presence independently of its numeric ID, including across comments.
With `set_anchor_offset(0)`, duplicate anchor properties now return a scan error instead of
panicking when anchor events and comments are enabled, and empty anchored nodes parse correctly.
- Fix a panic when a parser is used again after `Parser::load` or `Parser::try_load` returned a
scan error. The error could interrupt the transition into an indentless sequence entry, as with
the input `":\n- !%"`. A later `load` also no longer re-emits `StreamEnd` after an error raised
Expand Down
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,11 @@ aliases (see [`Event::UnresolvedAlias`](https://docs.rs/granit-parser/latest/gra
and continuing with the next document after a document error using
[`Parser::proceed_next_document`](https://docs.rs/granit-parser/latest/granit_parser/struct.Parser.html#method.proceed_next_document).

### Anchor events
Anchors were traditionally seen as opaque numeric IDs.
The new opt-in [`Event::Anchor`](https://docs.rs/granit-parser/latest/granit_parser/enum.Event.html#variant.Anchor) event
provides the anchor name if required for advanced processing. The companion `Span` of this event provides anchor location.

### Unsupported features
YAML 1.1 line breaks. YAML 1.1 defines NEL (U+0085), LS (U+2028) and PS (U+2029) as line breaks, this parser
uses `\n`/`\r` only line-breaking that as expected in YAML 1.2.
Expand Down
11 changes: 8 additions & 3 deletions examples/event_walker.rs
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ impl Walker {
(role, depth)
}

/// A node event gets a context; structural and comment events get `None`.
/// A node event gets a context; structural and metadata events get `None`.
fn observe(&mut self, event: &Event<'_>) -> Result<Option<(Role, usize)>, &'static str> {
match event {
Event::Scalar(..) | Event::Alias(_) | Event::UnresolvedAlias(_) => {
Expand Down Expand Up @@ -107,6 +107,8 @@ impl Walker {
Ok(None)
}
Event::StreamStart | Event::StreamEnd => Ok(None),
// Anchor metadata describes the preceding node and consumes no slot.
Event::Anchor(..) => Ok(None),
#[cfg(feature = "comments")]
Event::Comment(..) => {
// Comments are presentation metadata, so they never advance a
Expand Down Expand Up @@ -139,7 +141,7 @@ fn node_label<'event>(event: &'event Event<'_>) -> &'event str {
}
}

/// Print the input, then each node as it arrives, checking its context against a teaching table.
/// Print the input, nodes, and anchor metadata, checking node contexts against a teaching table.
/// The tables are fixtures, not a collected event stream or a document tree.
fn walk(
title: &str,
Expand All @@ -157,6 +159,9 @@ fn walk(
// separate makes the same Walker usable with another parser input type.
let (event, _span) = next?;
let Some((role, depth)) = walker.observe(&event)? else {
if let Event::Anchor(id, name) = &event {
println!(" anchor metadata: &{name} (ID {id})");
}
continue;
};

Expand Down Expand Up @@ -234,7 +239,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
walk(
"Aliases are nodes, including keys and self-references",
"original: &key name\n*key : value\nloop: &loop [*loop]\n",
Options::default(),
granit_parser::options! { emit_anchor_events: true },
&[
("block mapping", Root, 0),
("original", MappingKey, 1),
Expand Down
19 changes: 18 additions & 1 deletion fuzz/fuzz_targets/common.rs
Original file line number Diff line number Diff line change
Expand Up @@ -247,8 +247,15 @@ fn trace<T>(mut iterator: impl Iterator<Item = Result<T, ScanError>>) -> Trace<T

fn validate_string_parser_trace(input: &str, trace: &Trace<(Event<'_>, Span)>) {
let byte_offsets = byte_offsets(input);
for (_, span) in &trace.items {
for (event, span) in &trace.items {
validate_string_span(input, &byte_offsets, *span);
if let Event::Anchor(_, name) = event {
assert_eq!(
span.slice(input).and_then(|token| token.strip_prefix('&')),
Some(name.as_ref()),
"anchor metadata span must cover its complete source token"
);
}
}
if let Some(error) = &trace.error {
validate_string_marker(input, &byte_offsets, *error.marker());
Expand Down Expand Up @@ -408,6 +415,16 @@ fn validate_parser_structure(events: &[(Event<'_>, Span)], complete: bool) {
"alias refers to an anchor not yet defined in this document"
);
}
Event::Anchor(anchor, _) => {
assert!(document_open, "anchor metadata outside a document");
let (node, _) = &events[index - 1];
assert!(
node.is_node(),
"anchor metadata must immediately follow its node"
);
assert_eq!(node.anchor_id(), Some(*anchor));
assert!(anchors.contains(anchor));
}
_ => {}
}
}
Expand Down
9 changes: 6 additions & 3 deletions fuzz/fuzz_targets/options.rs
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,12 @@ pub fn check_input(data: &[u8]) {
return;
};

// The high flag bit selects strict indentation for every raw oracle, including comment
// suppression. Lower flags and the five-byte header keep their existing meanings.
let mut options = granit_parser::options! { strict_indentation: flags & 0x80 != 0 };
// The two high flag bits select strict indentation and anchor metadata for every raw oracle,
// including comment suppression. Lower flags and the five-byte header retain their meanings.
let mut options = granit_parser::options! {
strict_indentation: flags & 0x80 != 0,
emit_anchor_events: flags & 0x40 != 0,
};
if mode % 3 == 2 {
check_comment_suppression_with_options(input, options);
return;
Expand Down
44 changes: 44 additions & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,50 @@
//! excluding the line break. With [`Parser::new_from_str`], [`Span::slice`] returns that source
//! comment text.
//!
//! # Anchor and tag source metadata
//!
//! Set [`Options::emit_anchor_events`] to receive [`Event::Anchor`] with the assigned ID and
//! original name, without `&`. Its companion span covers the complete `&name` token. This event
//! immediately follows the anchored scalar or collection start, before any collection children.
//! It is presentation metadata, so [`Event::is_node`] returns `false`. Its source range precedes
//! the node content even though the event follows the node. Emission defaults to `false`, leaving
//! existing event streams unchanged.
//!
//! Resolved [`Event::Alias`] events carry the target ID and a complete `*name` token span. An
//! unresolved alias can instead be exposed through [`Options::emit_unresolved_aliases`]. Explicit
//! tags remain on node events; [`Span::tag_span`] preserves their complete raw token, including
//! verbatim notation and percent escapes that normalized tag helpers may not reproduce.
//!
//! ```rust
//! use std::collections::BTreeMap;
//! use granit_parser::{Event, Parser};
//!
//! let yaml = "message: &msg !note hello\ncopy: *msg\nexample: \"&msg is just text\"\n";
//! let options = granit_parser::options! { emit_anchor_events: true };
//! let mut anchors = BTreeMap::new();
//! for result in Parser::new_from_str_with_options(yaml, options) {
//! let (event, span) = result?;
//! if let Some(tag_span) = span.tag_span() {
//! assert_eq!(tag_span.slice(yaml), Some("!note"));
//! }
//! match event {
//! Event::Anchor(id, name) => {
//! assert_eq!(span.slice(yaml), Some("&msg"));
//! anchors.insert(id, (name, span));
//! }
//! Event::Alias(id) => {
//! let (name, definition_span) = &anchors[&id];
//! assert_eq!(name, "msg");
//! assert_eq!(definition_span.slice(yaml), Some("&msg"));
//! assert_eq!(span.slice(yaml), Some("*msg"));
//! }
//! _ => {}
//! }
//! }
//! assert_eq!(anchors.len(), 1); // The quoted text does not define an anchor.
//! # Ok::<(), granit_parser::ScanError>(())
//! ```
//!
//! # Limits
//!
//! [`Options::strict_indentation`] enables strict YAML indentation for flow collections. It
Expand Down
16 changes: 16 additions & 0 deletions src/options.rs
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,21 @@ pub struct Options {
/// # Ok::<(), granit_parser::ScanError>(())
/// ```
pub emit_unresolved_aliases: bool,
/// Emit anchor definition metadata immediately after its node event.
///
/// Defaults to `false`, preserving the existing event stream. When enabled, each anchored
/// scalar or collection start is immediately followed by [`crate::Event::Anchor`] carrying
/// the assigned ID and original name (without `&`). Its companion span covers the complete
/// `&name` token. Collection anchors are emitted before their children, including aliases
/// referring to the collection itself.
///
/// The node event comes first so the parser never emits anchor metadata for a node that
/// failed before its event could be emitted, including when recovering to the next document.
///
/// Anchor events are presentation metadata, not additional YAML nodes. Their spans refer
/// back to properties preceding the node, so event spans need not be in source order.
/// This option is independent of comment emission and has no effect on a standalone scanner.
pub emit_anchor_events: bool,
/// Whether scanners emit comment tokens and parsers emit comment events.
///
/// The default is `true` with the `comments` Cargo feature and `false` without it.
Expand Down Expand Up @@ -138,6 +153,7 @@ impl Default for Options {
strict_indentation: false,
disallow_directives: false,
emit_unresolved_aliases: false,
emit_anchor_events: false,
emit_comments: cfg!(feature = "comments"),
#[cfg(feature = "comments")]
max_buffered_comment_events: 96,
Expand Down
Loading
Loading