Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,12 @@
- Deprecate `Span::with_tag_start`, retaining it for compatibility. Use `Span::tag_span()` on
parser-emitted spans to obtain complete tag-token ranges.
- Add `Parser::proceed_next_document()` for explicit document-boundary recovery after a syntax
error or skipping the remainder of a document. Parsing resumes at the next document with original
source positions, or emits `StreamEnd`. Abandoned collections and document-scoped state are
discarded without synthetic closing events. Source failures remain terminal; ordinary parsing
remains fail-fast unless recovery is requested.
error or skipping the remainder of a document. It returns `Result<bool, ScanError>`: `true` when
another document remains, without guaranteeing its validity, and `false` at clean EOF. Pending
comments and an unread `StreamEnd` remain available with original source positions. Comment
lookahead respects the configured buffer limit and can return `TooManyComments`. Abandoned
collections and document-scoped state are discarded without synthetic closing events. Source
failures remain terminal; ordinary parsing remains fail-fast unless recovery is requested.
- Clarify in documentation and emphasize in tests that `Parser::new_from_str` supplies byte offsets for every event span.
- Add boolean option `emit_unresolved_aliases` (default `false`). When enabled, unknown anchors
produce `Event::UnresolvedAlias(name)` with the alias token's span and structural parsing
Expand Down
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,12 @@ Document starts are emitted as [`Event::DocumentStart(explicit, version)`](https

For editors processing multiple documents, `Parser::proceed_next_document()` resumes after a
syntax error at the next separated `---` or valid `...` at column zero. Call it after recording the
error, then continue reading events normally: the parser reports the next document or `StreamEnd`
with original source positions. It also skips the current document during successful parsing.
error, then continue reading events normally with original source positions. Its
`Result<bool, ScanError>` is `Ok(true)` when another document remains (including an empty or malformed
document), and `Ok(false)` at clean EOF. Pending comments and an unread `StreamEnd` remain available
through normal event reads. Looking ahead past comments respects `Options::max_buffered_comment_events`
and can return `ErrorKind::TooManyComments` when comment emission is enabled.
It also skips the current document during successful parsing.
Consumers must abandon or mark their unfinished tree nodes; recovery emits no synthetic closing
events or error nodes. Anchors remain scoped to their document, and source failures remain terminal.
This recovers later documents, not structure within the broken document.
Expand Down
4 changes: 3 additions & 1 deletion examples/tooling_diagnostics.rs
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,9 @@ fn main() -> Result<(), ScanError> {
Err(error) => {
report_error(&error);
diagnostics += 1;
parser.proceed_next_document()?;
if !parser.proceed_next_document()? {
break;
}
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ impl core::error::Error for InputIoError {
#[derive(Clone, PartialEq, Debug, Eq, Hash)]
#[non_exhaustive]
pub enum ErrorKind {
/// Too many consecutive comments were buffered before a collection entry.
/// Too many comments were buffered while looking ahead to a collection entry or document.
#[cfg(feature = "comments")]
TooManyComments,
/// Reading from the input source failed.
Expand Down Expand Up @@ -363,7 +363,7 @@ impl fmt::Display for ErrorKind {
match self {
#[cfg(feature = "comments")]
Self::TooManyComments => {
f.write_str("too many consecutive comments before resolving collection entry")
f.write_str("too many buffered comments while looking ahead")
}
Self::InputIo { error } => write!(f, "input I/O error: {error}"),
Self::InputDecoding { message } => {
Expand Down
2 changes: 1 addition & 1 deletion src/options.rs
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ pub struct Options {
)]
pub emit_comments: bool,
/// Maximum number of consecutive comment events buffered while resolving an ambiguous
/// collection entry.
/// collection entry or looking for the next document during recovery.
///
/// The default is 96. A value of zero rejects the first comment that would need buffering.
/// This option is available only with the `comments` Cargo feature.
Expand Down
92 changes: 89 additions & 3 deletions src/parser.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1174,6 +1174,12 @@ impl<'input, T: BorrowedInput<'input>> Parser<'input, T> {
/// skips the first document. Immediately after [`Event::DocumentEnd`] it leaves the next
/// document ready to read. Repeated calls skip successive documents.
///
/// Returns `true` when another document remains, including an empty or malformed document,
/// and `false` at clean EOF. This looks past inter-document comments and end markers without
/// consuming the next document's events. Comments inspected by this lookahead are buffered
/// for subsequent reads; with comment emission enabled, the configured comment-buffer limit
/// applies. Syntax errors encountered in the next document remain available to normal parsing.
///
/// Read subsequent events using the usual parser methods. They start with the next
/// [`Event::DocumentStart`] (possibly preceded by comments), or [`Event::StreamEnd`] if no
/// document remains. An already consumed stream end stays exhausted. This method does not
Expand Down Expand Up @@ -1206,7 +1212,7 @@ impl<'input, T: BorrowedInput<'input>> Parser<'input, T> {
/// let error = parser.by_ref().find_map(Result::err).unwrap();
/// assert!(parser.next().is_none());
/// // Record the diagnostic and abandon any open nodes in the consumer's partial tree.
/// parser.proceed_next_document()?;
/// assert!(parser.proceed_next_document()?);
/// assert!(matches!(parser.next().unwrap()?.0, Event::DocumentStart(true, _)));
/// let rest = parser.collect::<Result<Vec<_>, _>>()?;
/// assert!(rest.iter().any(|(event, _)| event.scalar().is_some_and(|(v, _)| v == "healthy")));
Expand All @@ -1217,12 +1223,40 @@ impl<'input, T: BorrowedInput<'input>> Parser<'input, T> {
/// # Errors
/// Returns a source failure (including I/O, decoding, or cancellation errors supplied by the
/// input). Source failures cannot be recovered and leave the parser exhausted.
pub fn proceed_next_document(&mut self) -> Result<(), ScanError> {
/// With comment emission enabled, also returns `ErrorKind::TooManyComments` if looking ahead
/// would exceed the comment-buffer limit; parsing then stays exhausted until recovery is
/// requested again.
pub fn proceed_next_document(&mut self) -> Result<bool, ScanError> {
// A consumed StreamEnd stays exhausted. A terminal error still needs recovery below.
if self.stream_end_emitted && !self.failed {
return Ok(());
return Ok(false);
}

self.skip_document()?;
let result = self.next_document_exists();
match result {
Ok(false) => {
// Stay between documents so repeated calls preserve unread trailing comments
// as well as StreamEnd instead of consuming them in the skip loop.
self.at_document_end = true;
}
Err(_) => {
self.current = None;
self.current_error = None;
#[cfg(feature = "comments")]
{
self.queued_events.clear();
self.deferred_error = None;
}
self.failed = true;
self.stream_end_emitted = true;
}
Ok(true) => {}
}
result
}

fn skip_document(&mut self) -> Result<(), ScanError> {
loop {
if !self.failed {
if self.at_document_end {
Expand Down Expand Up @@ -1269,6 +1303,58 @@ impl<'input, T: BorrowedInput<'input>> Parser<'input, T> {
}
}

/// Probe only the next document's prefix, retaining comments and any syntax error for readers.
fn next_document_exists(&mut self) -> Result<bool, ScanError> {
if matches!(self.current, Some((Event::StreamEnd, _))) {
return Ok(false);
}

#[cfg(feature = "comments")]
if self.has_pending_document_directives() {
// A header paused at a comment already establishes document presence. Leave EOF
// or an invalid end marker for normal parsing, but retain terminal source failures.
self.scanner.check_source_error()?;
return Ok(true);
}

loop {
let token = match self.peek_token() {
Ok(token) => token,
Err(error) => {
// Source failures can carry any ErrorKind. Consult the scanner's source
// latch rather than classifying the error by its kind.
self.scanner.check_source_error()?;
#[cfg(feature = "comments")]
{
self.deferred_error = Some(error);
}
#[cfg(not(feature = "comments"))]
{
self.current_error = Some(error);
self.failed = true;
}
return Ok(true);
}
};
match token.1 {
QueuedTokenType::StreamEnd => return Ok(false),
QueuedTokenType::DocumentEnd => self.skip(),
#[cfg(feature = "comments")]
QueuedTokenType::Comment(_) => {
let mark = token.0.start;
if self.queued_events.len() >= self.max_buffered_comment_events {
return Err(ScanError::from_kind(mark, ErrorKind::TooManyComments));
}
let comment = self
.next_comment_event()?
.expect("comment token disappeared after peek");
self.queued_events.push_back(comment);
}
_ => return Ok(true),
}
}
}

fn recover_document(&mut self) -> Result<(), ScanError> {
let result = self.scanner.proceed_next_document(self.token.take());
self.current = None;
Expand Down
2 changes: 1 addition & 1 deletion src/scanner.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1790,7 +1790,7 @@ impl<'input, T: BorrowedInput<'input>> Scanner<'input, T> {
}

/// Source failures cannot be recovered by discarding YAML syntax.
fn check_source_error(&mut self) -> ScanResult {
pub(crate) fn check_source_error(&mut self) -> ScanResult {
if let Some(kind) = self.input.take_source_error() {
self.source_error
.get_or_insert_with(|| ScanError::from_kind(self.mark, kind));
Expand Down
Loading
Loading