Action 344 - Lookahead function

[mbeckerle]

This extension needs a write up as a proposed extension to DFDL.

Internal Daffodil design note:
https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+DFDLX+lookAhead

This feature, as implemented today in Daffodil, is in heavy use. There are quite a few formats that need this.

This feature is for "fixed length" data formats - it makes no sense for use in more general data formats.

[mbeckerle]

This is the writeup of the feature from the ASF Daffodil web site:

dfdlx:lookAhead(offset, bitSize)

: Read bitSize bits, where the first bit is located at an offset (in bits)
from the current location. The result is a xs:nonNegativeInteger. Restrictions:

• offset >=0
• bitSize >= 1
• distance + bitSize <= Implementation defined limit no less than 512 bits
• Cannot be called during unparse
• Parse Error if the offset results in attempting to look ahead past EOF
• Undefined behavior if the offset results in attempting to look past the current data limit of
  a dfdl:lengthKind="explicit" surrounding element.
• The dfdl:bitOrder and dfdl:byteOrder are determined by the current schema component
  and data location.
• DFDL property changes between the current location and the location containing
  the data being read will not be used.

Examples of dfdlx:lookAhead

The following two elements both populate element a with the value of the next 3 bits as an
unsignedInt. They are not completely equivalent because the first will consume 3 bits of the
input stream where the second will not advance the input stream.

<xs:element name="a" type="xs:unsignedInt" dfdl:length="3" dfdl:lengthUnits="bits" />

<xs:element name="a" type="xs:unsignedInt" dfdl:inputValueCalc="{ dfdlx:lookAhead(0,3) }" />

The following example demonstrates using lookAhead to branch based on a field in the future.
In this case the choice of elements a vs. b depends on the value of the tag field which is
found after fields a and b:

<xs:choice dfdl:choiceDispatchKey="{ dfdlx:lookAhead(16,8) }">
  <xs:element name="a" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="1"/>
  <xs:element name="b" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="2"/>
</xs:choice>
<xs:element name="tag" type="xs:int" dfdl:length="8" />

[mbeckerle]

These two powerpoint slides make the argument for why forward lookahead is needed, particularly in fixed-length data formats that have evolved over time:

P-forward-reference-in-fixed-length-data.pptx

The images are below just to make the content obvious without having to download and open a pptx file.

[mbeckerle]

Experience document for review #74.

[smhdfdl]

Looking at your slides example, is it not possible to use a discriminator of type regex to look ahead? It's fixed length data after all.

[mbeckerle]

It is possible. It's just miserably slow. This sort of format wants the schema to have no speculation going on whatsoever.

[mbeckerle]

Please see issue #75 which is a more powerful proposal which if realized, would eliminate the need for this simpler lookahead function.