Skip to content

Fix low cardinality of duration values in randomly generated Parquet payloads - #1081

Merged
atextor merged 2 commits into
eclipse-esmf:mainfrom
sekikn:feature/#1080-fix-low-cardinality-of-parquet-generation
Oct 30, 2026
Merged

atextor merged 2 commits into
eclipse-esmf:mainfrom
sekikn:feature/#1080-fix-low-cardinality-of-parquet-generation

Conversation

@sekikn

@sekikn sekikn commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Description

This PR fixes ParquetExampleValueGenerator to generate random values for properties of types xsd:yearMonthDuration and xsd:dayTimeDuration, instead of using a small set of predefined values.

Fixes #1080

Type of change

  • New feature (non-breaking change which adds functionality)

Checklist:

  • My code follows the style guidelines of this project

  • I have performed a self-review of my own code

  • I have commented my code, particularly in hard-to-understand areas

    • This fix is so simple and straightforward that no additional comments are needed.
  • I have made corresponding changes to the documentation

    • There is no corresponding documentation.
  • My changes generate no new warnings

  • I have added tests that prove my fix is effective or that my feature works

Additional notes:

We can verify the new behavior as follows:

$ java -jar tools/samm-cli/target/samm-cli-DEV-SNAPSHOT.jar aspect core/esmf-test-aspect-models/src/main/resources/valid/org.eclipse.esmf.test/1.0.0/AspectWithDurationTypeForRangeConstraints.ttl to parquet -o /tmp/aspect.parquet --count 100 --ignore-example-value --random-timestamp
$ python3 -c 'import polars as pl; pl.read_parquet("/tmp/aspect.parquet").sql("SELECT testPropertyWithYearMonthDuration, count(*) FROM self GROUP BY testPropertyWithYearMonthDuration").show(limit=20)'
shape: (20, 2)
┌─────────────────────────────────┬─────┐
│ testPropertyWithYearMonthDurat… ┆ len │
│ ---                             ┆ --- │
│ str                             ┆ i64 │
╞═════════════════════════════════╪═════╡
│ -P18Y4M                         ┆ 1   │
│ P89Y3M                          ┆ 1   │
│ -P26Y11M                        ┆ 1   │
│ P89Y5M                          ┆ 1   │
│ -P99Y5M                         ┆ 1   │
│ -P57Y5M                         ┆ 1   │
│ P54Y3M                          ┆ 1   │
│ -P52Y1M                         ┆ 1   │
│ -P31Y9M                         ┆ 1   │
│ -P35Y6M                         ┆ 1   │
│ P65Y4M                          ┆ 1   │
│ P47Y9M                          ┆ 1   │
│ P94Y10M                         ┆ 1   │
│ P88Y4M                          ┆ 2   │
│ P82Y4M                          ┆ 1   │
│ -P63Y8M                         ┆ 1   │
│ P19Y3M                          ┆ 1   │
│ P25Y5M                          ┆ 1   │
│ -P7Y8M                          ┆ 1   │
│ -P23Y7M                         ┆ 1   │
└─────────────────────────────────┴─────┘

@atextor atextor left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sekikn, absolutely reasonable fix. Thank you!

@atextor
atextor merged commit 3f64105 into eclipse-esmf:main Oct 30, 2026
5 checks passed
@sekikn
sekikn deleted the feature/#1080-fix-low-cardinality-of-parquet-generation branch October 30, 2026 05:44
@github-actions

Copy link
Copy Markdown

Release v2.16.2 addresses this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Task] Fix low cardinality of duration values in randomly generated Parquet payloads

2 participants