[pull] master from apache:master - #650
Merged
Merged
Conversation
Bumps [transformers](https://github.com/huggingface/transformers) from 4.38.0 to 5.10.1. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v4.38.0...v5.10.1) --- updated-dependencies: - dependency-name: transformers dependency-version: 5.10.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.294.0 to 0.295.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.294.0...v0.295.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.295.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Upgrade GCP Libraries BOM to 26.87.0 - Upgrades com.google.cloud:libraries-bom to 26.87.0 - Updates synchronized versions: gax (2.84.0), grpc (1.82.4), netty (4.1.133.Final), protobuf (4.33.6), google-api-services-bigquery (v2-rev20260731-2.0.0), google-cloud-datastore (3.4.0) - Updates dep_urls_java.yaml license entry - Aligns Bigtable change streams cell mutation timestamps to milliseconds to comply with Bigtable timestamp granularity requirements * chore: Resolve spotless complaint * test: Align BigtableChangeStreamIT mutation timestamps to milliseconds
* Accept integral JSON values that a double represents exactly
doubleValueExtractor's validator compared asLong() against a round-trip
through asInt():
&& jsonNode.asLong() == (long) (double) jsonNode.asInt()
asInt() truncates anything outside int range, so the two sides can never
agree for a larger integral literal and the value is rejected -- even
when a double holds it exactly. Epoch millis is the everyday case:
{"f": 1609459200000} against a DOUBLE field fails today.
Swapping asInt() for asLong() looks like the fix and is not. (double)
Long.MAX_VALUE rounds up to 2^63, and narrowing 2^63 back to long
saturates at Long.MAX_VALUE rather than overflowing, so the round-trip
appears to succeed and the extractor stores 9223372036854775808 -- an
over-rejection traded for silent corruption.
Comparing through BigDecimal is exact, and it is what the decimal branch
directly below already does.
Tests: four supported cases (epoch millis, its negative, 2^31, 2^53) and
two rejections (2^53+1 and Long.MAX_VALUE). The rejections get a method
each because testUnsupportedConversion uses the ExpectedException rule,
which is satisfied by the first exception to leave the test method -- a
second call in the same body never runs.
master fails the supported case; the asLong() variant fails the
Long.MAX_VALUE case; this passes 78/78.
* Reject an integral JSON value that a float only saturates back to
floatValueExtractor validates an integral literal by round-tripping it
through float and back to int:
jsonNode.asInt() == (int) (float) jsonNode.asInt()
Narrowing a float that is out of int range saturates rather than
overflowing, so the trip is not the identity check it looks like.
Measured across the interesting values:
asInt (float)asInt (int)(float) current correct
2147483647 2.14748365E9 2147483647 accept REJECT
-2147483648 -2.14748365E9 -2147483648 accept accept
16777216 1.6777216E7 16777216 accept accept
16777217 1.6777216E7 16777216 reject reject
2147483583 2.14748352E9 2147483520 reject reject
Integer.MAX_VALUE is the one value the check gets wrong: the float it
goes through is 2147483648, and narrowing that back to int saturates at
2147483647, so the equality holds and the value is accepted. The
extractor then stores 2147483648.0 -- a document saying 2147483647 reads
back as a different number, with no error.
Compared through BigDecimal instead, matching the doubleValueExtractor
branch this PR already fixes. Integer.MIN_VALUE and 2^24 stay accepted:
both are exactly representable, so this is not an across-the-board
tightening.
Two supported cases go into the existing method. The unsupported case
gets its own, for the reason noted there: the ExpectedException rule is
satisfied by the first exception to leave the method, so a later call in
the same body never runs. That is also why the existing
testUnsupportedFloatConversions cannot cover this -- its INT_STRING call
is the fourth in the body and has never executed.
Reverting only the float validator fails
testUnsupportedFloatConversionAtIntegerMaxValue and nothing else: 79
tests, 1 failed.
Bumps [github.com/apache/thrift](https://github.com/apache/thrift) from 0.23.0 to 0.24.0. - [Release notes](https://github.com/apache/thrift/releases) - [Changelog](https://github.com/apache/thrift/blob/master/CHANGES.md) - [Commits](apache/thrift@v0.23.0...v0.24.0) --- updated-dependencies: - dependency-name: github.com/apache/thrift dependency-version: 0.24.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…emas.canonical, CollectDistinctSchemas) (#39933) New, not yet wired code that turns a PCollection of file paths into the list of distinct schemas those files carry, with file counts. The pre-pass that will use it exists because manifest entries are immutable: a file registered before the table knows one of its columns never gets stats for that column, so the table schema has to be brought up to date before any file is registered, and that requires looking at every footer first. ReadFooterSchema (DoFn<String, String>) reads each Parquet footer on the BoundedAsyncTasks pool and emits the file's canonical schema as JSON. Non-Parquet paths and unknown extensions contribute nothing; a footer that cannot be read or converted is logged and counted (numFooterReadErrors) but never fails the pipeline: the per-file registration step reports such files individually later. FileSchemas.canonical sorts struct fields by name at every level and renumbers ids in deterministic order. The ids are positional and never consumed downstream: the commit side reconciles columns by name (unionByNameWith). CollectDistinctSchemas is a CombineFn over the canonical JSON strings (Map<String, Long> accumulator) producing List<KV<String, Long>> ordered most common first, ties broken by the JSON text for determinism. The most common schema goes first because the commit side uses it as the seed when the table does not exist yet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )