Skip to content

feat: basic read operation for Iceberg #944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
lonless9 merged 37 commits into from
Oct 17, 2025
Merged

feat: basic read operation for Iceberg #944

lonless9 merged 37 commits into from
Oct 17, 2025

Conversation

@lonless9
Copy link
Contributor

@lonless9 lonless9 commented Sep 30, 2025 •
edited
Loading

@github-actions
Copy link

github-actions bot commented Sep 30, 2025 •
edited
Loading

Spark 3.5.5 Test Report

Commit Information

Commit Revision Branch
After 012b03e refs/pull/944/merge
Before 12eb82e refs/heads/main

Test Summary

Suite Commit Failed Passed Skipped Warnings Time (s)
doctest-catalog After 11 14 6 4.97
Before 11 14 6 5.06
doctest-column After 33 2 5.42
Before 33 2 5.51
doctest-dataframe After 25 80 2 3 7.94
Before 25 80 2 3 8.05
doctest-functions After 84 318 7 7 15.01
Before 84 318 7 7 15.66
test-connect After 177 837 157 628 131.81
Before 177 837 157 628 136.69

Test Details

Error Counts 
          297 Total
          154 Total Unique
-------- ---- ----------------------------------------------------------------------------------------------------------
           22 DocTestFailure
           14 AssertionError: AnalysisException not raised
           14 UnsupportedOperationException: lambda function
           13 PySparkAssertionError: [DIFFERENT_PANDAS_DATAFRAME] DataFrames are not almost equal:
           10 handle add artifacts
            9 UnsupportedOperationException: hint
            6 AssertionError: False is not true
            6 UnsupportedOperationException: PlanNode::CacheTable
            6 UnsupportedOperationException: function: window
            5 UnsupportedOperationException: function: monotonically_increasing_id
            4 AssertionError: "TABLE_OR_VIEW_NOT_FOUND" does not match "view not found: v"
            4 AssertionError: Attributes of DataFrame.iloc[:, 7] (column name="8_timestamp_t") are different
            4 PySparkNotImplementedError: [NOT_IMPLEMENTED] rdd() is not implemented.
            4 UnsupportedOperationException: function: input_file_name
            4 UnsupportedOperationException: unknown aggregate function: hll_sketch_agg
            3 AnalysisException: No files found in the specified paths: file:///home/runner/work/sail/sail/.venvs/test-spark.spark-3.5.5/lib/python3.11/site-packages/pyspark/python/test_support/sql/ages_newlines.cs...
            3 AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="time") are different
            3 UnsupportedOperationException: handle analyze input files
            3 ValueError: Converting to Python dictionary is not supported when duplicate field names are present
            2 AnalysisException: Could not find config namespace "spark"
            2 AnalysisException: No table format found for: orc
            2 AnalysisException: not supported: list functions
            2 AnalysisException: two values expected: [Column(Column { relation: None, name: "#2" }), Column(Column { relation: None, name: "#3" }), Literal(Utf8("/"), None)]
            2 AssertionError
            2 AssertionError: AnalysisException not raised by <lambda>
            2 AssertionError: Lists differ: [Row([22 chars](key=1, value='1'), Row(key=10, value='10'), R[2402 chars]99')] != [Row([22 chars](key=0, value='0'), Row(key=1, value='1'), Row[4882 chars]99')]
            2 IllegalArgumentException: expected value at line 1 column 1
            2 IllegalArgumentException: invalid argument: found FUNCTION at 5:13 expected 'DATABASE', 'SCHEMA', 'TABLE', 'VIEW', 'TEMP', 'TEMPORARY', or 'FUNCTIONS'
            2 UnsupportedOperationException: Aggregate can not be used as a sliding accumulator because `retract_batch` is not implemented: avg@6jabwh68iuheayv3cstdmwtmp(#9) PARTITION BY [#8] ORDER BY [#9 ASC NULLS...
            2 UnsupportedOperationException: approx quantile
            2 UnsupportedOperationException: collect metrics
            2 UnsupportedOperationException: freq items
            2 UnsupportedOperationException: function: array_sort
            2 UnsupportedOperationException: function: format_number
            2 UnsupportedOperationException: function: from_json
            2 UnsupportedOperationException: function: schema_of_csv
            2 UnsupportedOperationException: handle analyze is local
            2 UnsupportedOperationException: handle analyze same semantics
            2 UnsupportedOperationException: pivot
            2 UnsupportedOperationException: unresolved regex
            2 UnsupportedOperationException: user defined data type should only exist in a field
            2 UnsupportedOperationException: with watermark
            2 handle artifact statuses
            2 received metadata size exceeds hard limit (19714 vs. 16384);  :status:42B content-type:60B grpc-status:45B grpc-message:9084B grpc-status-details-bin:7876B ValueError: Code in Status proto (StatusCode...
            1 AnalysisException: Error parsing timestamp from '2023-01-01' using format '%d-%m-%Y': input contains invalid characters
            1 AnalysisException: Failed to parse placeholder id: cannot parse integer from empty string
            1 AnalysisException: No files found in the specified paths: file:///home/runner/work/sail/sail/.venvs/test-spark.spark-3.5.5/lib/python3.11/site-packages/pyspark/sql/functions.py
(+1)        1 AnalysisException: No files found in the specified paths: file:///tmp/tmp0cq47z9t/
(+1)        1 AnalysisException: No files found in the specified paths: file:///tmp/tmpdswuvgr5/text-0.text, file:///tmp/tmpdswuvgr5/text-1.text, file:///tmp/tmpdswuvgr5/text-2.text
(+1)        1 AnalysisException: No files found in the specified paths: file:///tmp/tmpwuoq35of/
(+1)        1 AnalysisException: UNION queries have different number of columns: left has 3 columns whereas right has 2 columns
            1 AnalysisException: not supported: function exists
            1 AnalysisException: table already exists: tbl1
            1 AnalysisException: temporary view not found: tab2
            1 AssertionError: "2000000" does not match "raise_error expects a single UTF-8 string argument"
(+1)        1 AssertionError: "Database 'memory:6151ff57-2b61-46f2-9e6c-90b8fe036da7' dropped." does not match "No table format found for: jdbc"
(+1)        1 AssertionError: "Database 'memory:6c1b688b-1204-4bd5-a9c2-b0c2bdb7ace1' dropped." does not match "No table format found for: jdbc"
            1 AssertionError: "attribute.*missing" does not match "cannot resolve attribute: ObjectName([Identifier("b")])"
            1 AssertionError: "foobar" does not match "raise_error expects a single UTF-8 string argument"
            1 AssertionError: '+--------------------------------+-------------------[411 chars]-+\n' != '+-----------+-----------+\n|from_csv(a)|from_csv(b)|\[105 chars]-+\n'
            1 AssertionError: '+---[102 chars] c -> NULL}|\n|           {c -> de}|\n+--------------------+\n' != '+---[102 chars] c -> NULL}|\n|                NULL|\n+--------------------+\n'
            1 AssertionError: '+---[17 chars]-----+\n|                        x|\n+--------[132 chars]-+\n' != '+---[17 chars]----------+\n|update_fields(x, WithField(e))|\[167 chars]-+\n'
            1 AssertionError: '4.0.0' != '3.5.5'
            1 AssertionError: 2 != 6
            1 AssertionError: ArrayIndexOutOfBoundsException not raised
            1 AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="a") are different
            1 AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="ts") are different
            1 AssertionError: Exception not raised
            1 AssertionError: Exception not raised by <lambda>
            1 AssertionError: Lists differ: [(1, 2), (3, 4), (None, 5), (0, 0)] != [(1, 2), (3, 4), (None, 5), (None, None)]
            1 AssertionError: Lists differ: [Row([14 chars] _c1=25, _c2='I am Hyukjin\n\nI love Spark!'),[86 chars]om')] != [Row([14 chars] _c1='25', _c2='I am Hyukjin\n\nI love Spark!'[92 chars]om')]
            1 AssertionError: Lists differ: [Row(id=90, name='90'), Row(id=91, name='91'), Ro[176 chars]99')] != [Row(id=15, name='15'), Row(id=16, name='16'), Ro[176 chars]24')]
            1 AssertionError: Lists differ: [Row(key='0'), Row(key='1'), Row(key='10'), Row(ke[1435 chars]99')] != [Row(key=0), Row(key=1), Row(key=10), Row(key=11),[1235 chars]=99)]
            1 AssertionError: Lists differ: [Row(ln(id)=0.0, ln(id)=0.0, struct(id, name)=Row(id=[1232 chars]0'))] != [Row(ln(id)=4.31748811353631, ln(id)=4.31748811353631[1312 chars]4'))]
            1 AssertionError: Lists differ: [Row(name='Andy', age=30), Row(name='Justin', [34 chars]one)] != [Row(_corrupt_record=' "age":19}\n', name=None[104 chars]el')]
            1 AssertionError: Row(point='[1.0, 2.0]', pypoint='[3.0, 4.0]') != Row(point='(1.0, 2.0)', pypoint='[3.0, 4.0]')
            1 AssertionError: StorageLevel(False, True, True, False, 1) != StorageLevel(False, False, False, False, 1)
            1 AssertionError: Struc[30 chars]estampType(), True), StructField('val', IntegerType(), True)]) != Struc[30 chars]estampType(), True), StructField('val', IntegerType(), False)])
            1 AssertionError: Struc[32 chars]e(), False), StructField('b', DoubleType(), Fa[158 chars]ue)]) != Struc[32 chars]e(), True), StructField('b', DoubleType(), Tru[154 chars]ue)])
            1 AssertionError: Struc[40 chars]ue), StructField('val', ArrayType(DoubleType(), False), True)]) != Struc[40 chars]ue), StructField('val', PythonOnlyUDT(), True)])
            1 AssertionError: Struc[64 chars]Type(), True), StructField('i', StringType(), True)]), False)]) != Struc[64 chars]Type(), True), StructField('i', StringType(), True)]), True)])
            1 AssertionError: Struc[69 chars]e(), True), StructField('name', StringType(), True)]), True)]) != Struc[69 chars]e(), True), StructField('name', StringType(), True)]), False)])
            1 AssertionError: YearMonthIntervalType(0, 1) != YearMonthIntervalType(0, 0)
            1 AssertionError: [1.0, 2.0] != ExamplePoint(1.0,2.0)
            1 AssertionError: dtype('<M8[us]') != 'datetime64[ns]'
            1 AttributeError: 'DataFrame' object has no attribute '_ipython_key_completions_'
            1 AttributeError: 'DataFrame' object has no attribute '_joinAsOf'
            1 IllegalArgumentException: invalid argument: found FUNCTION at 7:15 expected 'DATABASE', 'SCHEMA', 'OR', 'TEMP', 'TEMPORARY', 'EXTERNAL', 'TABLE', 'GLOBAL', or 'VIEW'
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] foreach() is not implemented.
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] foreachPartition() is not implemented.
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] localCheckpoint() is not implemented.
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] sparkContext() is not implemented.
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] toJSON() is not implemented.
            1 PythonException:  AttributeError: 'NoneType' object has no attribute 'partitionId'
            1 PythonException:  AttributeError: 'list' object has no attribute 'x'
            1 PythonException:  AttributeError: 'list' object has no attribute 'y'
            1 SparkRuntimeException: Cast error: Cannot cast string 'abc' to value of Float64 type
            1 SparkRuntimeException: Cast error: Cannot cast to Decimal128(30, 15). Overflowing on NaN
            1 SparkRuntimeException: Invalid argument error: 83140 is too large to store in a Decimal128 of precision 4. Max is 9999
            1 SparkRuntimeException: Invalid argument error: column types must match schema types, expected Int64 but found List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: ...
            1 SparkRuntimeException: Invalid argument error: column types must match schema types, expected LargeUtf8 but found Utf8 at column index 0
            1 SparkRuntimeException: Json error: Not valid JSON: EOF while parsing a list at line 1 column 1
            1 SparkRuntimeException: Json error: Not valid JSON: expected value at line 1 column 2
            1 SparkRuntimeException: Parser error: Error parsing timestamp from '1997/02/28 10:30:00': error parsing date
            1 SparkRuntimeException: Parser error: Error while parsing value '0
            1 UnsupportedOperationException: Aggregate can not be used as a sliding accumulator because `retract_batch` is not implemented: avg@6jabwh68iuheayv3cstdmwtmp(#9) PARTITION BY [#8] ORDER BY [#9 ASC NULLS...
            1 UnsupportedOperationException: Aggregate can not be used as a sliding accumulator because `retract_batch` is not implemented: avg@6jabwh68iuheayv3cstdmwtmp(plus_one@3fz6es6jadwlqp420cnsho3cq(#9)) PART...
            1 UnsupportedOperationException: PlanNode::ClearCache
            1 UnsupportedOperationException: PlanNode::IsCached
            1 UnsupportedOperationException: PlanNode::RecoverPartitions
            1 UnsupportedOperationException: SHOW FUNCTIONS
            1 UnsupportedOperationException: Support for 'approx_distinct' for data type Float64 is not implemented
            1 UnsupportedOperationException: bucketing for writing listing table format
            1 UnsupportedOperationException: deduplicate within watermark
            1 UnsupportedOperationException: function: format_string
            1 UnsupportedOperationException: function: java_method
            1 UnsupportedOperationException: function: json_tuple
            1 UnsupportedOperationException: function: printf
            1 UnsupportedOperationException: function: reflect
            1 UnsupportedOperationException: function: regexp_extract
            1 UnsupportedOperationException: function: regexp_extract_all
            1 UnsupportedOperationException: function: regexp_substr
            1 UnsupportedOperationException: function: schema_of_json
            1 UnsupportedOperationException: function: sentences
            1 UnsupportedOperationException: function: session_window
            1 UnsupportedOperationException: function: soundex
            1 UnsupportedOperationException: function: spark_partition_id
            1 UnsupportedOperationException: function: to_char
            1 UnsupportedOperationException: function: to_csv
            1 UnsupportedOperationException: function: to_json
            1 UnsupportedOperationException: function: to_varchar
            1 UnsupportedOperationException: function: xpath
            1 UnsupportedOperationException: function: xpath_boolean
            1 UnsupportedOperationException: function: xpath_double
            1 UnsupportedOperationException: function: xpath_float
            1 UnsupportedOperationException: function: xpath_int
            1 UnsupportedOperationException: function: xpath_long
            1 UnsupportedOperationException: function: xpath_number
            1 UnsupportedOperationException: function: xpath_short
            1 UnsupportedOperationException: function: xpath_string
            1 UnsupportedOperationException: handle analyze semantic hash
            1 UnsupportedOperationException: unknown aggregate function: bitmap_construct_agg
            1 UnsupportedOperationException: unknown aggregate function: bitmap_or_agg
            1 UnsupportedOperationException: unknown aggregate function: count_min_sketch
            1 UnsupportedOperationException: unknown aggregate function: grouping_id
            1 UnsupportedOperationException: unknown aggregate function: histogram_numeric
            1 UnsupportedOperationException: unknown aggregate function: percentile
            1 UnsupportedOperationException: unknown aggregate function: try_avg
            1 UnsupportedOperationException: unknown aggregate function: try_sum
            1 UnsupportedOperationException: unknown function: distributed_sequence_id
            1 UnsupportedOperationException: unknown function: product
            1 ValueError: Code in Status proto (StatusCode.INTERNAL) doesn't match status code (StatusCode.RESOURCE_EXHAUSTED)
            1 ValueError: The column label 'id' is not unique.
            1 ValueError: The column label 'struct' is not unique.
(-1)        0 AnalysisException: No files found in the specified paths: file:///tmp/tmp3td47__e/text-0.text, file:///tmp/tmp3td47__e/text-1.text, file:///tmp/tmp3td47__e/text-2.text
(-1)        0 AnalysisException: No files found in the specified paths: file:///tmp/tmpa1062kbz/
(-1)        0 AnalysisException: No files found in the specified paths: file:///tmp/tmpprbiy6rq/
(-1)        0 AnalysisException: UNION queries have different number of columns: left has 2 columns whereas right has 3 columns
(-1)        0 AssertionError: "Database 'memory:a1cb9245-fb5f-4b48-adcc-bff6451bb911' dropped." does not match "No table format found for: jdbc"
(-1)        0 AssertionError: "Database 'memory:faa02b33-5237-40d8-96f1-fe652406051a' dropped." does not match "No table format found for: jdbc"
Passed Tests Diff

(empty)

Failed Tests 
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.cacheTable
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.clearCache
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.createTable
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.functionExists
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.getFunction
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.isCached
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.listFunctions
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.recoverPartitions
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.refreshByPath
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.refreshTable
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.uncacheTable
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame._ipython_key_completions_
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame._joinAsOf
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.checkpoint
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.coalesce
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.colRegex
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.dropDuplicatesWithinWatermark
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.explain
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.foreach
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.foreachPartition
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.hint
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.inputFiles
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.isLocal
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.isStreaming
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.localCheckpoint
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.observe
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.randomSplit
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.rdd
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.repartition
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.repartitionByRange
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.sameSemantics
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.sampleBy
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.storageLevel
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.toJSON
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.withWatermark
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrameStatFunctions.sampleBy
pyspark/sql/functions.py::pyspark.sql.functions.aggregate
pyspark/sql/functions.py::pyspark.sql.functions.approx_percentile
pyspark/sql/functions.py::pyspark.sql.functions.array_position
pyspark/sql/functions.py::pyspark.sql.functions.array_sort
pyspark/sql/functions.py::pyspark.sql.functions.array_union
pyspark/sql/functions.py::pyspark.sql.functions.arrays_zip
pyspark/sql/functions.py::pyspark.sql.functions.bitmap_construct_agg
pyspark/sql/functions.py::pyspark.sql.functions.bitmap_or_agg
pyspark/sql/functions.py::pyspark.sql.functions.broadcast
pyspark/sql/functions.py::pyspark.sql.functions.collect_set
pyspark/sql/functions.py::pyspark.sql.functions.count_min_sketch
pyspark/sql/functions.py::pyspark.sql.functions.exists
pyspark/sql/functions.py::pyspark.sql.functions.filter
pyspark/sql/functions.py::pyspark.sql.functions.first
pyspark/sql/functions.py::pyspark.sql.functions.forall
pyspark/sql/functions.py::pyspark.sql.functions.format_number
pyspark/sql/functions.py::pyspark.sql.functions.format_string
pyspark/sql/functions.py::pyspark.sql.functions.from_csv
pyspark/sql/functions.py::pyspark.sql.functions.from_json
pyspark/sql/functions.py::pyspark.sql.functions.grouping_id
pyspark/sql/functions.py::pyspark.sql.functions.histogram_numeric
pyspark/sql/functions.py::pyspark.sql.functions.hll_sketch_agg
pyspark/sql/functions.py::pyspark.sql.functions.hll_sketch_estimate
pyspark/sql/functions.py::pyspark.sql.functions.hll_union
pyspark/sql/functions.py::pyspark.sql.functions.hll_union_agg
pyspark/sql/functions.py::pyspark.sql.functions.ilike
pyspark/sql/functions.py::pyspark.sql.functions.input_file_block_length
pyspark/sql/functions.py::pyspark.sql.functions.input_file_block_start
pyspark/sql/functions.py::pyspark.sql.functions.input_file_name
pyspark/sql/functions.py::pyspark.sql.functions.java_method
pyspark/sql/functions.py::pyspark.sql.functions.json_tuple
pyspark/sql/functions.py::pyspark.sql.functions.kurtosis
pyspark/sql/functions.py::pyspark.sql.functions.like
pyspark/sql/functions.py::pyspark.sql.functions.make_timestamp
pyspark/sql/functions.py::pyspark.sql.functions.map_entries
pyspark/sql/functions.py::pyspark.sql.functions.map_filter
pyspark/sql/functions.py::pyspark.sql.functions.map_zip_with
pyspark/sql/functions.py::pyspark.sql.functions.mode
pyspark/sql/functions.py::pyspark.sql.functions.monotonically_increasing_id
pyspark/sql/functions.py::pyspark.sql.functions.percentile
pyspark/sql/functions.py::pyspark.sql.functions.percentile_approx
pyspark/sql/functions.py::pyspark.sql.functions.printf
pyspark/sql/functions.py::pyspark.sql.functions.product
pyspark/sql/functions.py::pyspark.sql.functions.rand
pyspark/sql/functions.py::pyspark.sql.functions.randn
pyspark/sql/functions.py::pyspark.sql.functions.reduce
pyspark/sql/functions.py::pyspark.sql.functions.reflect
pyspark/sql/functions.py::pyspark.sql.functions.regexp_extract
pyspark/sql/functions.py::pyspark.sql.functions.regexp_extract_all
pyspark/sql/functions.py::pyspark.sql.functions.regexp_instr
pyspark/sql/functions.py::pyspark.sql.functions.regexp_substr
pyspark/sql/functions.py::pyspark.sql.functions.regr_avgy
pyspark/sql/functions.py::pyspark.sql.functions.regr_intercept
pyspark/sql/functions.py::pyspark.sql.functions.regr_r2
pyspark/sql/functions.py::pyspark.sql.functions.regr_slope
pyspark/sql/functions.py::pyspark.sql.functions.regr_sxy
pyspark/sql/functions.py::pyspark.sql.functions.regr_syy
pyspark/sql/functions.py::pyspark.sql.functions.schema_of_csv
pyspark/sql/functions.py::pyspark.sql.functions.schema_of_json
pyspark/sql/functions.py::pyspark.sql.functions.sentences
pyspark/sql/functions.py::pyspark.sql.functions.session_window
pyspark/sql/functions.py::pyspark.sql.functions.soundex
pyspark/sql/functions.py::pyspark.sql.functions.spark_partition_id
pyspark/sql/functions.py::pyspark.sql.functions.to_char
pyspark/sql/functions.py::pyspark.sql.functions.to_csv
pyspark/sql/functions.py::pyspark.sql.functions.to_json
pyspark/sql/functions.py::pyspark.sql.functions.to_varchar
pyspark/sql/functions.py::pyspark.sql.functions.transform
pyspark/sql/functions.py::pyspark.sql.functions.transform_keys
pyspark/sql/functions.py::pyspark.sql.functions.transform_values
pyspark/sql/functions.py::pyspark.sql.functions.try_avg
pyspark/sql/functions.py::pyspark.sql.functions.try_sum
pyspark/sql/functions.py::pyspark.sql.functions.window
pyspark/sql/functions.py::pyspark.sql.functions.window_time
pyspark/sql/functions.py::pyspark.sql.functions.xpath
pyspark/sql/functions.py::pyspark.sql.functions.xpath_boolean
pyspark/sql/functions.py::pyspark.sql.functions.xpath_double
pyspark/sql/functions.py::pyspark.sql.functions.xpath_float
pyspark/sql/functions.py::pyspark.sql.functions.xpath_int
pyspark/sql/functions.py::pyspark.sql.functions.xpath_long
pyspark/sql/functions.py::pyspark.sql.functions.xpath_number
pyspark/sql/functions.py::pyspark.sql.functions.xpath_short
pyspark/sql/functions.py::pyspark.sql.functions.xpath_string
pyspark/sql/functions.py::pyspark.sql.functions.zip_with
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_add_archive
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_add_file
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_add_pyfile
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_add_zipped_package
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_basic_requests
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_cache_artifact
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_copy_from_local_to_fs
pyspark/sql/tests/connect/client/test_artifact.py::LocalClusterArtifactTests::test_add_archive
pyspark/sql/tests/connect/client/test_artifact.py::LocalClusterArtifactTests::test_add_file
pyspark/sql/tests/connect/client/test_artifact.py::LocalClusterArtifactTests::test_add_pyfile
pyspark/sql/tests/connect/client/test_artifact.py::LocalClusterArtifactTests::test_add_zipped_package
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_collect
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_collect_timestamp
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_column_regexp
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_create_global_temp_view
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_deduplicate_within_watermark_in_batch
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_describe
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_explain_string
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_extended_hint_types
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_grouped_data
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_hint
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_input_files
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_invalid_column
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_is_local
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_join_hint
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_json
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_multi_paths
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_namedargs_with_global_limit
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_numeric_aggregation
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_observe
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_orc
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_random_split
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_same_semantics
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_schema
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_semantic_hash
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_simple_read_without_schema
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_simple_udt_from_read
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_sql_with_command
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_sql_with_pos_args
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_stat_approx_quantile
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_stat_freq_items
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_stat_sample_by
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_streaming_local_relation
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_tail
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_to
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_version
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_with_local_list
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_with_local_ndarray
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_write_operations
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectSessionTests::test_error_stack_trace
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_column_accessor
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_column_arithmetic_ops
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_column_field_ops
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_columns
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_decimal
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_distributed_sequence_id
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_aggregation_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_broadcast
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_collection_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_csv_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_date_ts_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_generator_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_json_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_lambda_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_map_collection_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_math_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_nested_lambda_function
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_normal_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_string_functions_multi_args
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_string_functions_one_arg
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_time_window_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_udf
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_udtf
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_window_functions
pyspark/sql/tests/connect/test_parity_arrow.py::ArrowParityTests::test_createDataFrame_duplicate_field_names
pyspark/sql/tests/connect/test_parity_arrow.py::ArrowParityTests::test_createDataFrame_with_schema
pyspark/sql/tests/connect/test_parity_arrow.py::ArrowParityTests::test_pandas_round_trip
pyspark/sql/tests/connect/test_parity_arrow.py::ArrowParityTests::test_pandas_self_destruct
pyspark/sql/tests/connect/test_parity_arrow.py::ArrowParityTests::test_timestamp_dst
pyspark/sql/tests/connect/test_parity_arrow.py::ArrowParityTests::test_timestamp_nat
pyspark/sql/tests/connect/test_parity_arrow.py::ArrowParityTests::test_toPandas_arrow_toggle
pyspark/sql/tests/connect/test_parity_arrow.py::ArrowParityTests::test_toPandas_duplicate_field_names
pyspark/sql/tests/connect/test_parity_arrow.py::ArrowParityTests::test_toPandas_nested_timestamp
pyspark/sql/tests/connect/test_parity_arrow.py::ArrowParityTests::test_toPandas_respect_session_timezone
pyspark/sql/tests/connect/test_parity_arrow.py::ArrowParityTests::test_toPandas_timestmap_tzinfo
pyspark/sql/tests/connect/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_nondeterministic_udf_in_aggregate
pyspark/sql/tests/connect/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_in_join_condition
pyspark/sql/tests/connect/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_not_supported_in_join_condition
pyspark/sql/tests/connect/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_with_input_file_name
pyspark/sql/tests/connect/test_parity_arrow_python_udf.py::UDFParityTests::test_nondeterministic_udf_in_aggregate
pyspark/sql/tests/connect/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_in_join_condition
pyspark/sql/tests/connect/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_not_supported_in_join_condition
pyspark/sql/tests/connect/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_with_input_file_name
pyspark/sql/tests/connect/test_parity_catalog.py::CatalogParityTests::test_function_exists
pyspark/sql/tests/connect/test_parity_catalog.py::CatalogParityTests::test_get_function
pyspark/sql/tests/connect/test_parity_catalog.py::CatalogParityTests::test_list_functions
pyspark/sql/tests/connect/test_parity_catalog.py::CatalogParityTests::test_refresh_table
pyspark/sql/tests/connect/test_parity_catalog.py::CatalogParityTests::test_table_cache
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_cache_dataframe
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_cache_table
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_create_dataframe_from_pandas_with_dst
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_duplicate_field_names
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_extended_hint_types
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_freqItems
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_generic_hints
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_input_files
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_join_without_on
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_require_cross
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_to
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_to_pandas
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_checking_csv_header
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_encoding_json
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_ignore_column_of_all_nulls
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_ignorewhitespace_csv
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_jdbc
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_jdbc_format
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_linesep_json
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_linesep_text
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_multiline_csv
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_multiline_json
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_read_multiple_orc_file
pyspark/sql/tests/connect/test_parity_errors.py::ErrorsParityTests::test_array_index_out_of_bounds_exception
pyspark/sql/tests/connect/test_parity_errors.py::ErrorsParityTests::test_date_time_exception
pyspark/sql/tests/connect/test_parity_errors.py::ErrorsParityTests::test_number_format_exception
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_approxQuantile
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_assert_true
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_functions_broadcast
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_inline
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_input_file_name_udf
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_nested_higher_order_function
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_raise_error
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_sampleby
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_window_time
pyspark/sql/tests/connect/test_parity_pandas_grouped_map.py::GroupedApplyInPandasTests::test_grouped_over_window
pyspark/sql/tests/connect/test_parity_pandas_grouped_map.py::GroupedApplyInPandasTests::test_grouped_over_window_with_key
pyspark/sql/tests/connect/test_parity_pandas_grouped_map_with_state.py::GroupedApplyInPandasWithStateTests::test_apply_in_pandas_with_state_python_worker_random_failure
pyspark/sql/tests/connect/test_parity_pandas_map.py::MapInPandasParityTests::test_large_variable_types
pyspark/sql/tests/connect/test_parity_pandas_udf_grouped_agg.py::PandasUDFGroupedAggParityTests::test_invalid_args
pyspark/sql/tests/connect/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_nondeterministic_vectorized_udf_in_aggregate
pyspark/sql/tests/connect/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_scalar_iter_udf_init
pyspark/sql/tests/connect/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_vectorized_udf_check_config
pyspark/sql/tests/connect/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_vectorized_udf_invalid_length
pyspark/sql/tests/connect/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_bounded_mixed
pyspark/sql/tests/connect/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_bounded_simple
pyspark/sql/tests/connect/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_shrinking_window
pyspark/sql/tests/connect/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_sliding_window
pyspark/sql/tests/connect/test_parity_readwriter.py::ReadwriterParityTests::test_bucketed_write
pyspark/sql/tests/connect/test_parity_readwriter.py::ReadwriterParityTests::test_insert_into
pyspark/sql/tests/connect/test_parity_readwriter.py::ReadwriterParityTests::test_save_and_load
pyspark/sql/tests/connect/test_parity_readwriter.py::ReadwriterParityTests::test_save_and_load_builder
pyspark/sql/tests/connect/test_parity_readwriter.py::ReadwriterV2ParityTests::test_create_without_provider
pyspark/sql/tests/connect/test_parity_readwriter.py::ReadwriterV2ParityTests::test_table_overwrite
pyspark/sql/tests/connect/test_parity_types.py::TypesParityTests::test_cast_to_string_with_udt
pyspark/sql/tests/connect/test_parity_types.py::TypesParityTests::test_cast_to_udt_with_udt
pyspark/sql/tests/connect/test_parity_types.py::TypesParityTests::test_complex_nested_udt_in_df
pyspark/sql/tests/connect/test_parity_types.py::TypesParityTests::test_negative_decimal
pyspark/sql/tests/connect/test_parity_types.py::TypesParityTests::test_parquet_with_udt
pyspark/sql/tests/connect/test_parity_types.py::TypesParityTests::test_udf_with_udt
pyspark/sql/tests/connect/test_parity_types.py::TypesParityTests::test_udt_with_none
pyspark/sql/tests/connect/test_parity_types.py::TypesParityTests::test_yearmonth_interval_type
pyspark/sql/tests/connect/test_parity_udf.py::UDFParityTests::test_nondeterministic_udf_in_aggregate
pyspark/sql/tests/connect/test_parity_udf.py::UDFParityTests::test_udf_in_join_condition
pyspark/sql/tests/connect/test_parity_udf.py::UDFParityTests::test_udf_not_supported_in_join_condition
pyspark/sql/tests/connect/test_parity_udf.py::UDFParityTests::test_udf_with_input_file_name
pyspark/sql/tests/connect/test_parity_udtf.py::ArrowUDTFParityTests::test_udtf_arrow_sql_conf
pyspark/sql/tests/connect/test_parity_udtf.py::ArrowUDTFParityTests::test_udtf_with_table_argument_malformed_query
pyspark/sql/tests/connect/test_parity_udtf.py::ArrowUDTFParityTests::test_udtf_with_table_argument_multiple
pyspark/sql/tests/connect/test_parity_udtf.py::ArrowUDTFParityTests::test_udtf_with_table_argument_unknown_identifier
pyspark/sql/tests/connect/test_parity_udtf.py::UDTFParityTests::test_udtf_with_table_argument_malformed_query
pyspark/sql/tests/connect/test_parity_udtf.py::UDTFParityTests::test_udtf_with_table_argument_multiple
pyspark/sql/tests/connect/test_parity_udtf.py::UDTFParityTests::test_udtf_with_table_argument_unknown_identifier
pyspark/sql/tests/connect/test_utils.py::ConnectUtilsTests::test_assert_approx_equal_decimaltype_custom_rtol_pass
pyspark/sql/tests/connect/test_utils.py::ConnectUtilsTests::test_assert_equal_approx_pandas_on_spark_df
pyspark/sql/tests/connect/test_utils.py::ConnectUtilsTests::test_assert_equal_exact_pandas_on_spark_df
pyspark/sql/tests/connect/test_utils.py::ConnectUtilsTests::test_assert_equal_nested_struct_str_duplicate
pyspark/sql/tests/connect/test_utils.py::ConnectUtilsTests::test_assert_error_pandas_pyspark_df
pyspark/sql/tests/connect/test_utils.py::ConnectUtilsTests::test_assert_type_error_pandas_df

@github-actions
Copy link

github-actions bot commented Sep 30, 2025 •
edited
Loading

Spark 4.0.0 Test Report

Commit Information

Commit Revision Branch
After 012b03e refs/pull/944/merge
Before 12eb82e refs/heads/main

Test Summary

Suite Commit Failed Passed Skipped Warnings Time (s)
doctest-catalog After 11 14 2 5.80
Before 11 14 2 5.87
doctest-column After 35 6.10
Before 35 6.28
doctest-dataframe After 30 86 4 2 9.70
Before 30 86 4 2 9.74
doctest-functions After 110 332 10 7 24.40
Before 110 332 10 7 24.83
test-connect After 516 1179 227 441 270.90
Before 516 1179 227 441 274.57

Test Details

Error Counts 
          667 Total
          239 Total Unique
-------- ---- ----------------------------------------------------------------------------------------------------------
           41 IllegalArgumentException: missing argument: Python UDTF return type
           25 register data source command
           24 DocTestFailure
           23 UnsupportedOperationException: function: parse_json
           23 UnsupportedOperationException: with relations
           18 UnsupportedOperationException: handle add artifacts
           18 UnsupportedOperationException: named argument expression
           14 AssertionError: 1 != 0 : dict_keys([])
           14 UnsupportedOperationException: lambda function
           13 UnsupportedOperationException: variant data type
           12 AssertionError: AnalysisException not raised
           12 AssertionError: False is not true
           12 IllegalArgumentException: expected value at line 1 column 1
           12 PySparkAssertionError: [DIFFERENT_PANDAS_DATAFRAME] DataFrames are not almost equal:
           12 UnsupportedOperationException: unresolved table valued function
           11 IllegalArgumentException: invalid argument: expected function for lateral table factor
           10 UnsupportedOperationException: hint
           10 UnsupportedOperationException: lateral join
            9 AssertionError: 3 != 0 : []
            8 AssertionError
            6 UnsupportedOperationException: PlanNode::CacheTable
            6 UnsupportedOperationException: collect metrics
            6 UnsupportedOperationException: function: spark_partition_id
            6 UnsupportedOperationException: function: window
            6 UnsupportedOperationException: handle analyze is local
            5 AssertionError: `query_context_type` is required when QueryContext exists. QueryContext: [].
            5 UnsupportedOperationException: named function arguments
            5 UnsupportedOperationException: user defined data type should only exist in a field
            5 checkpoint command
            4 AnalysisException: temporary view not found: t2
            4 AssertionError: "TABLE_OR_VIEW_NOT_FOUND" does not match "view not found: v"
            4 AssertionError: AnalysisException not raised by <lambda>
            4 PythonException:  PySparkRuntimeError: [UDTF_EVAL_METHOD_ARGUMENTS_DO_NOT_MATCH_SIGNATURE] Failed to evaluate the user-defined table function '' because the function arguments did not match the expect...
            4 UnsupportedOperationException: approx quantile
            4 UnsupportedOperationException: function: input_file_name
            4 UnsupportedOperationException: function: monotonically_increasing_id
            4 UnsupportedOperationException: unknown aggregate function: hll_sketch_agg
            3 AnalysisException: No files found in the specified paths: file:///home/runner/work/sail/sail/.venvs/test-spark.spark-4.0.0/lib/python3.11/site-packages/pyspark/python/test_support/sql/ages_newlines.cs...
            3 AssertionError: 1 != 0
            3 IllegalArgumentException: invalid argument: extraction must be a literal
            3 UnsupportedOperationException: function: from_json
            3 UnsupportedOperationException: function: shuffle
            3 UnsupportedOperationException: handle analyze input files
            3 UnsupportedOperationException: pivot
            3 UnsupportedOperationException: transpose
            3 UnsupportedOperationException: unknown table function: IDENTIFIER
            3 ValueError: Converting to Python dictionary is not supported when duplicate field names are present
            2 AnalysisException: Failed to parse placeholder id: cannot parse integer from empty string
            2 AnalysisException: Invalid Python user-defined table function return type. Expect a struct type, but got Int32.
            2 AnalysisException: No table format found for: orc
            2 AnalysisException: ambiguous attribute: ObjectName([Identifier("id")])
            2 AnalysisException: not supported: list functions
            2 AnalysisException: temporary view not found: variant_table
            2 AnalysisException: two values expected: [Column(Column { relation: None, name: "#2" }), Column(Column { relation: None, name: "#3" }), Literal(Utf8("/"), None)]
            2 AssertionError: 3 != 0 : dict_keys([])
            2 AssertionError: unexpectedly None
            2 IllegalArgumentException: invalid argument: found FUNCTION at 5:13 expected 'DATABASE', 'SCHEMA', 'TABLE', 'VIEW', 'TEMP', 'TEMPORARY', or 'FUNCTIONS'
            2 IllegalArgumentException: invalid argument: found PARTITION at 281:290 expected '.', '[', '::', 'ESCAPE', 'IS', 'NOT', 'IN', '*', '/', '%', 'DIV', '+', '-', '||', '>>>', '>>', '<<', '&', '^', '|', '!=...
            2 IllegalArgumentException: invalid argument: found PARTITION at 295:304 expected '.', '[', '::', 'ESCAPE', 'IS', 'NOT', 'IN', '*', '/', '%', 'DIV', '+', '-', '||', '>>>', '>>', '<<', '&', '^', '|', '!=...
            2 IllegalArgumentException: invalid argument: found PARTITION at 59:68 expected '.', '[', '::', 'ESCAPE', 'IS', 'NOT', 'IN', '*', '/', '%', 'DIV', '+', '-', '||', '>>>', '>>', '<<', '&', '^', '|', '!=',...
            2 IllegalArgumentException: invalid argument: found WITH at 171:175 expected '.', '[', '::', 'ESCAPE', 'IS', 'NOT', 'IN', '*', '/', '%', 'DIV', '+', '-', '||', '>>>', '>>', '<<', '&', '^', '|', '!=', '!...
            2 IllegalArgumentException: invalid argument: found WITH at 279:283 expected '.', '[', '::', 'ESCAPE', 'IS', 'NOT', 'IN', '*', '/', '%', 'DIV', '+', '-', '||', '>>>', '>>', '<<', '&', '^', '|', '!=', '!...
            2 PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 99.50000 % )
            2 PythonException:  AssertionError: assert None is not None
            2 PythonException:  AttributeError: 'NoneType' object has no attribute 'cpus'
            2 PythonException:  KeyError: 'a'
            2 UnsupportedOperationException: Aggregate can not be used as a sliding accumulator because `retract_batch` is not implemented: avg@1lyeq07trczbg3c9ihccuvt1t(#9) PARTITION BY [#8] ORDER BY [#9 ASC NULLS...
            2 UnsupportedOperationException: Aggregate can not be used as a sliding accumulator because `retract_batch` is not implemented: mean_udf@2yjkzpz26kl3afqanhvf3n18m(#3) PARTITION BY [#2] ORDER BY [#3 ASC ...
            2 UnsupportedOperationException: CLUSTER BY for write
            2 UnsupportedOperationException: LATERAL JOIN with criteria
            2 UnsupportedOperationException: Physical plan does not support logical expression Wildcard { qualifier: None, options: WildcardOptions { ilike: None, exclude: None, except: None, replace: None, rename:...
            2 UnsupportedOperationException: freq items
            2 UnsupportedOperationException: function: format_number
            2 UnsupportedOperationException: function: from_xml
            2 UnsupportedOperationException: function: randstr
            2 UnsupportedOperationException: function: to_variant_object
            2 UnsupportedOperationException: function: try_make_interval
            2 UnsupportedOperationException: function: try_make_timestamp
            2 UnsupportedOperationException: function: try_make_timestamp_ltz
            2 UnsupportedOperationException: function: try_make_timestamp_ntz
            2 UnsupportedOperationException: function: try_parse_json
            2 UnsupportedOperationException: handle analyze same semantics
            2 UnsupportedOperationException: unknown function: distributed_sequence_id
            2 UnsupportedOperationException: unresolved regex
            2 UnsupportedOperationException: wildcard with plan ID
            2 UnsupportedOperationException: with watermark
            2 create resource profile command
            2 handle artifact statuses
            2 received metadata size exceeds hard limit (19714 vs. 16384);  :status:42B content-type:60B grpc-status:45B grpc-message:9084B grpc-status-details-bin:7876B ValueError: Code in Status proto (StatusCode...
            1 AnalysisException: Could not find config namespace "mapred"
            1 AnalysisException: Error parsing timestamp from '082017' using format '%m%Y': input is not enough for unique date and time
            1 AnalysisException: Error parsing timestamp from '2014-31-12' using format '%Y-%d-%pa': input contains invalid characters
            1 AnalysisException: Error parsing timestamp from '2023-01-01' using format '%d-%m-%Y': input contains invalid characters
            1 AnalysisException: No files found in the specified paths: file:///home/runner/work/sail/sail/.venvs/test-spark.spark-4.0.0/lib/python3.11/site-packages/pyspark/sql/functions/builtin.py
(+1)        1 AnalysisException: No files found in the specified paths: file:///tmp/test_multi_paths1m8stvtnz/text-0.text, file:///tmp/test_multi_paths1m8stvtnz/text-1.text, file:///tmp/test_multi_paths1m8stvtnz/te...
(+1)        1 AnalysisException: No files found in the specified paths: file:///tmp/tmpsp5b6cah/
(+1)        1 AnalysisException: No files found in the specified paths: file:///tmp/tmpybstx8po/
            1 AnalysisException: No table format found for: xml
            1 AnalysisException: Schema contains duplicate unqualified field name "nth_value(#5,Int32(2)) RESPECT NULLS PARTITION BY [#3] ORDER BY [#4 ASC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ...
(+1)        1 AnalysisException: UNION queries have different number of columns: left has 2 columns whereas right has 3 columns
            1 AnalysisException: ambiguous attribute: ObjectName([Identifier("b")])
            1 AnalysisException: ambiguous attribute: ObjectName([Identifier("i")])
            1 AnalysisException: cannot resolve attribute: ObjectName([Identifier("x")])
            1 AnalysisException: element_at expects List or Map type as first argument, got Null
            1 AnalysisException: not supported: function exists
            1 AnalysisException: table already exists: tbl1
            1 AnalysisException: temporary view not found: tab2
            1 AnalysisException: temporary view not found: v2
            1 AnalysisException: too big
(+1)        1 AssertionError: "Database 'memory:83a57abb-955f-4ded-bc6d-d0e9e27da9d4' dropped." does not match "No table format found for: jdbc"
(+1)        1 AssertionError: "Database 'memory:fc60311a-6e0b-4ac9-a6ea-962d1582da73' dropped." does not match "No table format found for: jdbc"
            1 AssertionError: "Invalid return type" does not match " AttributeError: 'Series' object has no attribute 'columns'
            1 AssertionError: "PARTITION_TRANSFORM_EXPRESSION_NOT_IN_PARTITIONED_BY" does not match "unknown function: years"
            1 AssertionError: "UNRESOLVED_COLUMN.WITH_SUGGESTION" does not match "cannot resolve attribute: ObjectName([Identifier("b")])"
            1 AssertionError: "foobar" does not match "raise_error expects a single UTF-8 string argument"
            1 AssertionError: "requirement failed: Cogroup keys must have same size: 2 != 1" does not match "invalid argument: child plan grouping expressions must have the same length"
            1 AssertionError: '+--------------------------------+-------------------[411 chars]-+\n' != '+-----------+-----------+\n|from_csv(a)|from_csv(b)|\[105 chars]-+\n'
            1 AssertionError: '+---[102 chars] c -> NULL}|\n|           {c -> de}|\n+--------------------+\n' != '+---[102 chars] c -> NULL}|\n|                NULL|\n+--------------------+\n'
            1 AssertionError: '+---[17 chars]-----+\n|                        x|\n+--------[132 chars]-+\n' != '+---[17 chars]----------+\n|update_fields(x, WithField(e))|\[167 chars]-+\n'
            1 AssertionError: '+---[177 chars]     1|            2|\n|        2|           3[75 chars]-+\n' != '+---[177 chars]     2|            2|\n|        2|           3[75 chars]-+\n'
            1 AssertionError: '+---[23 chars]---+-----+\n|  1|    1|\n+---+-----+\nonly showing top 1 row' != '+---[23 chars]---+-----+\n|  1|    1|\n+---+-----+\nonly showing top 1 row\n'
            1 AssertionError: 'deadbeef' is not None
            1 AssertionError: 0 not greater than 0
            1 AssertionError: 0.40248566366484795 != 0.9531453492357947 : Column<'rand(1)'>
            1 AssertionError: 2 != 6
            1 AssertionError: 6 != 0 : []
            1 AssertionError: ArrayIndexOutOfBoundsException not raised
            1 AssertionError: Exception not raised
            1 AssertionError: Exception not raised by <lambda>
            1 AssertionError: Lists differ: [(1, 2), (3, 4), (None, 5), (0, 0)] != [(1, 2), (3, 4), (None, 5), (None, None)]
            1 AssertionError: Lists differ: [Row([14 chars] _c1=25, _c2='I am Hyukjin\n\nI love Spark!'),[86 chars]om')] != [Row([14 chars] _c1='25', _c2='I am Hyukjin\n\nI love Spark!'[92 chars]om')]
            1 AssertionError: Lists differ: [Row(id=90, name='90'), Row(id=91, name='91'), Ro[176 chars]99')] != [Row(id=15, name='15'), Row(id=16, name='16'), Ro[176 chars]24')]
            1 AssertionError: Lists differ: [Row(key='0'), Row(key='1'), Row(key='10'), Row(ke[1435 chars]99')] != [Row(key=0), Row(key=1), Row(key=10), Row(key=11),[1235 chars]=99)]
            1 AssertionError: Lists differ: [Row(ln(id)=0.0, ln(id)=0.0, struct(id, name)=Row(id=[1232 chars]0'))] != [Row(ln(id)=4.31748811353631, ln(id)=4.31748811353631[1312 chars]4'))]
            1 AssertionError: Lists differ: [Row(name='Andy', age=30), Row(name='Justin', [34 chars]one)] != [Row(_corrupt_record=' "age":19}\n', name=None[104 chars]el')]
            1 AssertionError: Row(point='[1.0, 2.0]', pypoint='[3.0, 4.0]') != Row(point='(1.0, 2.0)', pypoint='[3.0, 4.0]')
            1 AssertionError: SparkConnectGrpcException not raised
            1 AssertionError: StorageLevel(False, True, True, False, 1) != StorageLevel(False, False, False, False, 1)
            1 AssertionError: Struc[30 chars]estampType(), True), StructField('val', IntegerType(), True)]) != Struc[30 chars]estampType(), True), StructField('val', IntegerType(), False)])
            1 AssertionError: Struc[32 chars]e(), False), StructField('b', DoubleType(), Fa[158 chars]ue)]) != Struc[32 chars]e(), True), StructField('b', DoubleType(), Tru[154 chars]ue)])
            1 AssertionError: Struc[40 chars]ue), StructField('val', ArrayType(DoubleType(), False), True)]) != Struc[40 chars]ue), StructField('val', PythonOnlyUDT(), True)])
            1 AssertionError: True is not false : Default URL is not secure
            1 AssertionError: YearMonthIntervalType(0, 1) != YearMonthIntervalType(0, 0)
            1 AssertionError: [1.0, 2.0] != ExamplePoint(1.0,2.0)
            1 AttributeError: 'NoneType' object has no attribute 'extract_graph'
            1 AttributeError: 'NoneType' object has no attribute 'toText'
            1 FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/sail/sail/.venvs/test-spark.spark-4.0.0/lib/python3.11/site-packages/pyspark/data/artifact-tests/junitLargeJar.jar'
(+1)        1 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpwn7xnxgz'
            1 IllegalArgumentException: data did not match any variant of untagged enum JsonDataType
            1 IllegalArgumentException: invalid argument: empty data type
            1 IllegalArgumentException: invalid argument: field not found in input schema: col1
            1 IllegalArgumentException: invalid argument: found ( at 114:115 expected ':', data type, ',', or ')'
            1 IllegalArgumentException: invalid argument: found FUNCTION at 7:15 expected 'DATABASE', 'SCHEMA', 'OR', 'TEMP', 'TEMPORARY', 'EXTERNAL', 'TABLE', 'GLOBAL', or 'VIEW'
            1 IllegalArgumentException: invalid argument: found abc at 0:3 expected something else, ';', statement, or end of input
            1 IllegalArgumentException: invalid argument: found collate at 13:20 expected string, '.', '[', '::', 'ESCAPE', 'IS', 'NOT', 'IN', '*', '/', '%', 'DIV', '+', '-', '||', '>>>', '>>', '<<', '&', '^', '|',...
            1 IllegalArgumentException: invalid argument: grouping sets with grouping expressions
            1 IllegalArgumentException: invalid argument: invalid user-defined window function type
            1 IllegalArgumentException: invalid argument: table does not exist: ObjectName([Identifier("test_table")])
(+1)        1 PySparkAssertionError: Received incorrect server side session identifier for request. Please create a new Spark Session to reconnect. (58b48f67-cbba-4087-ba0c-1fb2e619098e != d327ee86-64a1-4681-9f5a-2...
(+1)        1 PySparkAssertionError: Received incorrect server side session identifier for request. Please create a new Spark Session to reconnect. (db8b5dce-a457-46fd-abe0-ed409233b1ae != cebda777-3f47-41ee-bd4a-a...
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] rdd is not implemented.
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] toJSON() is not implemented.
            1 PythonException:  AttributeError: 'NoneType' object has no attribute 'partitionId'
            1 PythonException:  AttributeError: 'list' object has no attribute 'x'
            1 PythonException:  AttributeError: 'list' object has no attribute 'y'
            1 PythonException:  TypeError: net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for pyspark.sql.types._create_row).
            1 SparkRuntimeException: Cast error: Cannot cast string 'abc' to value of Float64 type
            1 SparkRuntimeException: Cast error: Cannot cast to Decimal128(30, 15). Overflowing on NaN
            1 SparkRuntimeException: Compute error: Cannot perform a binary operation on arrays of different length
            1 SparkRuntimeException: Invalid argument error: column types must match schema types, expected Int64 but found List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: ...
            1 SparkRuntimeException: Invalid argument error: column types must match schema types, expected LargeUtf8 but found Utf8 at column index 0
            1 SparkRuntimeException: Json error: Not valid JSON: EOF while parsing a list at line 1 column 1
            1 SparkRuntimeException: Json error: Not valid JSON: expected value at line 1 column 2
            1 SparkRuntimeException: Parser error: Error while parsing value '0
            1 UnsupportedOperationException: Aggregate can not be used as a sliding accumulator because `retract_batch` is not implemented: avg@1lyeq07trczbg3c9ihccuvt1t(#9) PARTITION BY [#8] ORDER BY [#9 ASC NULLS...
            1 UnsupportedOperationException: Aggregate can not be used as a sliding accumulator because `retract_batch` is not implemented: avg@1lyeq07trczbg3c9ihccuvt1t(plus_one@6u0mz3nm98p9nxyf30cgvjry(#9)) PARTI...
            1 UnsupportedOperationException: PlanNode::ClearCache
            1 UnsupportedOperationException: PlanNode::IsCached
            1 UnsupportedOperationException: PlanNode::RecoverPartitions
            1 UnsupportedOperationException: SHOW FUNCTIONS
            1 UnsupportedOperationException: Support for 'approx_distinct' for data type Float64 is not implemented
            1 UnsupportedOperationException: Support for 'approx_distinct' for data type Struct(name Utf8, value Int64) is not implemented
            1 UnsupportedOperationException: as of join
            1 UnsupportedOperationException: bucketing for writing listing table format
            1 UnsupportedOperationException: deduplicate within watermark
            1 UnsupportedOperationException: function: array_sort
            1 UnsupportedOperationException: function: collate
            1 UnsupportedOperationException: function: collation
            1 UnsupportedOperationException: function: format_string
            1 UnsupportedOperationException: function: java_method
            1 UnsupportedOperationException: function: json_tuple
            1 UnsupportedOperationException: function: printf
            1 UnsupportedOperationException: function: reflect
            1 UnsupportedOperationException: function: regexp_extract
            1 UnsupportedOperationException: function: regexp_extract_all
            1 UnsupportedOperationException: function: regexp_substr
            1 UnsupportedOperationException: function: schema_of_csv
            1 UnsupportedOperationException: function: schema_of_json
            1 UnsupportedOperationException: function: schema_of_xml
            1 UnsupportedOperationException: function: sentences
            1 UnsupportedOperationException: function: session_window
            1 UnsupportedOperationException: function: soundex
            1 UnsupportedOperationException: function: to_char
            1 UnsupportedOperationException: function: to_csv
            1 UnsupportedOperationException: function: to_json
            1 UnsupportedOperationException: function: to_varchar
            1 UnsupportedOperationException: function: to_xml
            1 UnsupportedOperationException: function: try_reflect
            1 UnsupportedOperationException: function: try_url_decode
            1 UnsupportedOperationException: function: uniform
            1 UnsupportedOperationException: function: xpath
            1 UnsupportedOperationException: function: xpath_boolean
            1 UnsupportedOperationException: function: xpath_double
            1 UnsupportedOperationException: function: xpath_float
            1 UnsupportedOperationException: function: xpath_int
            1 UnsupportedOperationException: function: xpath_long
            1 UnsupportedOperationException: function: xpath_number
            1 UnsupportedOperationException: function: xpath_short
            1 UnsupportedOperationException: function: xpath_string
            1 UnsupportedOperationException: handle analyze json to ddl
            1 UnsupportedOperationException: handle analyze semantic hash
            1 UnsupportedOperationException: named window function arguments
            1 UnsupportedOperationException: unknown aggregate function: bitmap_construct_agg
            1 UnsupportedOperationException: unknown aggregate function: bitmap_or_agg
            1 UnsupportedOperationException: unknown aggregate function: count_min_sketch
            1 UnsupportedOperationException: unknown aggregate function: grouping_id
            1 UnsupportedOperationException: unknown aggregate function: histogram_numeric
            1 UnsupportedOperationException: unknown aggregate function: percentile
            1 UnsupportedOperationException: unknown aggregate function: try_avg
            1 UnsupportedOperationException: unknown aggregate function: try_sum
            1 UnsupportedOperationException: unknown function: product
            1 UnsupportedOperationException: unknown function: timestampadd
            1 UnsupportedOperationException: unknown function: timestampdiff
            1 ValueError: Code in Status proto (StatusCode.INTERNAL) doesn't match status code (StatusCode.RESOURCE_EXHAUSTED)
            1 ValueError: The column label 'id' is not unique.
            1 ValueError: The column label 'struct' is not unique.
            1 failed to decode Protobuf message: WithColumns.input: Relation.rel_type: WithColumns.input: Relation.rel_type: WithColumns.input: Relation.rel_type: WithColumns.input: Relation.rel_type: WithColumns.i...
            1 handle add artifacts
            1 received metadata size exceeds hard limit (value length 25049 vs. 16384)
(-1)        0 AnalysisException: No files found in the specified paths: file:///tmp/test_multi_paths102ng3lzu/text-0.text, file:///tmp/test_multi_paths102ng3lzu/text-1.text, file:///tmp/test_multi_paths102ng3lzu/te...
(-1)        0 AnalysisException: No files found in the specified paths: file:///tmp/tmp6lxoht_g/
(-1)        0 AnalysisException: No files found in the specified paths: file:///tmp/tmpq7kllsl1/
(-1)        0 AnalysisException: UNION queries have different number of columns: left has 3 columns whereas right has 2 columns
(-1)        0 AssertionError: "Database 'memory:0c4ff706-5873-4c3d-b527-731fc355547f' dropped." does not match "No table format found for: jdbc"
(-1)        0 AssertionError: "Database 'memory:6305ad66-7e7d-4389-8976-a3febfd3e2a5' dropped." does not match "No table format found for: jdbc"
(-1)        0 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpg_6bx1_4'
(-1)        0 PySparkAssertionError: Received incorrect server side session identifier for request. Please create a new Spark Session to reconnect. (738df7b7-acb4-47e2-bb08-eda36b84043b != 88d951a1-11cd-4b36-bfe5-9...
(-1)        0 PySparkAssertionError: Received incorrect server side session identifier for request. Please create a new Spark Session to reconnect. (ad431a50-e2ca-4afb-a275-51c3aa1cfcba != 4f85286d-fe78-4fc4-b40a-1...
Passed Tests Diff

(empty)

Failed Tests 
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.cacheTable
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.clearCache
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.createTable
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.functionExists
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.getFunction
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.isCached
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.listFunctions
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.recoverPartitions
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.refreshByPath
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.refreshTable
pyspark/sql/catalog.py::pyspark.sql.catalog.Catalog.uncacheTable
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame._joinAsOf
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.approxQuantile
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.cache
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.coalesce
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.colRegex
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.dropDuplicatesWithinWatermark
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.exists
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.explain
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.groupingSets
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.hint
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.inputFiles
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.isLocal
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.isStreaming
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.lateralJoin
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.localCheckpoint
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.observe
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.persist
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.randomSplit
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.rdd
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.repartition
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.repartitionByRange
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.sameSemantics
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.sampleBy
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.scalar
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.storageLevel
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.toJSON
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.transpose
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.withWatermark
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrameStatFunctions.approxQuantile
pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrameStatFunctions.sampleBy
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.aggregate
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.approx_count_distinct
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.approx_percentile
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.array_sort
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.bitmap_construct_agg
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.bitmap_or_agg
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.broadcast
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.collation
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.corr
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.cosh
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.count_if
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.count_min_sketch
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.degrees
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.exists
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.exp
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.filter
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.first
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.forall
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.format_number
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.format_string
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.from_csv
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.from_json
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.from_xml
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.grouping_id
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.hash
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.histogram_numeric
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.hll_sketch_agg
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.hll_sketch_estimate
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.hll_union
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.hll_union_agg
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.ilike
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.inline_outer
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.input_file_block_length
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.input_file_block_start
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.input_file_name
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.is_variant_null
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.java_method
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.json_tuple
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.kurtosis
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.like
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.log2
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.map_filter
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.map_zip_with
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.mode
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.monotonically_increasing_id
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.parse_json
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.percentile
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.percentile_approx
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.printf
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.product
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.rand
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.randn
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.randstr
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.reduce
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.reflect
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.regexp_extract
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.regexp_extract_all
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.regexp_substr
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.schema_of_csv
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.schema_of_json
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.schema_of_variant
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.schema_of_variant_agg
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.schema_of_xml
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.sentences
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.session_window
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.shuffle
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.sin
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.soundex
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.spark_partition_id
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.tan
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.timestamp_add
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.timestamp_diff
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.to_char
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.to_csv
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.to_json
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.to_varchar
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.to_variant_object
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.to_xml
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.transform
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.transform_keys
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.transform_values
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_add
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_avg
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_make_interval
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_make_timestamp
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_make_timestamp_ltz
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_make_timestamp_ntz
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_parse_json
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_reflect
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_subtract
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_sum
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_url_decode
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.try_variant_get
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.udf
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.udtf
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.uniform
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.variant_get
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.window
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.window_time
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.xpath
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.xpath_boolean
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.xpath_double
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.xpath_float
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.xpath_int
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.xpath_long
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.xpath_number
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.xpath_short
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.xpath_string
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.xxhash64
pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.zip_with
pyspark/sql/tests/connect/arrow/test_parity_arrow.py::ArrowParityTests::test_createDataFrame_pandas_duplicate_field_names
pyspark/sql/tests/connect/arrow/test_parity_arrow.py::ArrowParityTests::test_pandas_self_destruct
pyspark/sql/tests/connect/arrow/test_parity_arrow.py::ArrowParityTests::test_toPandas_duplicate_field_names
pyspark/sql/tests/connect/arrow/test_parity_arrow_cogrouped_map.py::CogroupedMapInArrowParityTests::test_with_local_data
pyspark/sql/tests/connect/arrow/test_parity_arrow_map.py::ArrowMapParityTests::test_map_in_arrow_with_barrier_mode
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_chained_udfs_with_variant
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_kwargs
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_named_arguments
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_named_arguments_and_defaults
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_named_arguments_negative
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_nondeterministic_udf_in_aggregate
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_cache
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_in_join_condition
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_not_supported_in_join_condition
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_with_complex_variant_input
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_with_complex_variant_output
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_with_input_file_name
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_with_udt
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_with_variant_input
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::ArrowPythonUDFParityTests::test_udf_with_variant_output
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_chained_udfs_with_variant
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_kwargs
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_named_arguments
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_named_arguments_and_defaults
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_named_arguments_negative
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_nondeterministic_udf_in_aggregate
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_cache
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_in_join_condition
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_not_supported_in_join_condition
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_with_complex_variant_input
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_with_complex_variant_output
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_with_input_file_name
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_with_udt
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_with_variant_input
pyspark/sql/tests/connect/arrow/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_with_variant_output
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_add_archive
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_add_file
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_add_pyfile
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_add_zipped_package
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_artifacts_cannot_be_overwritten
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_cache_artifact
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_copy_from_local_to_fs
pyspark/sql/tests/connect/client/test_artifact.py::ArtifactTests::test_single_chunked_and_chunked_artifact
pyspark/sql/tests/connect/client/test_artifact_localcluster.py::LocalClusterArtifactTests::test_add_archive
pyspark/sql/tests/connect/client/test_artifact_localcluster.py::LocalClusterArtifactTests::test_add_file
pyspark/sql/tests/connect/client/test_artifact_localcluster.py::LocalClusterArtifactTests::test_add_pyfile
pyspark/sql/tests/connect/client/test_artifact_localcluster.py::LocalClusterArtifactTests::test_add_zipped_package
pyspark/sql/tests/connect/client/test_artifact_localcluster.py::LocalClusterArtifactTests::test_artifacts_cannot_be_overwritten
pyspark/sql/tests/connect/client/test_client.py::SparkConnectClientTestCase::test_properties
pyspark/sql/tests/connect/pandas/test_parity_pandas_cogrouped_map.py::CogroupedApplyInPandasTests::test_case_insensitive_grouping_column
pyspark/sql/tests/connect/pandas/test_parity_pandas_cogrouped_map.py::CogroupedApplyInPandasTests::test_different_group_key_cardinality
pyspark/sql/tests/connect/pandas/test_parity_pandas_cogrouped_map.py::CogroupedApplyInPandasTests::test_self_join
pyspark/sql/tests/connect/pandas/test_parity_pandas_grouped_map.py::GroupedApplyInPandasTests::test_case_insensitive_grouping_column
pyspark/sql/tests/connect/pandas/test_parity_pandas_grouped_map.py::GroupedApplyInPandasTests::test_grouped_over_window
pyspark/sql/tests/connect/pandas/test_parity_pandas_grouped_map.py::GroupedApplyInPandasTests::test_grouped_over_window_with_key
pyspark/sql/tests/connect/pandas/test_parity_pandas_grouped_map_with_state.py::GroupedApplyInPandasWithStateTests::test_apply_in_pandas_with_state_basic
pyspark/sql/tests/connect/pandas/test_parity_pandas_grouped_map_with_state.py::GroupedApplyInPandasWithStateTests::test_apply_in_pandas_with_state_basic_fewer_data
pyspark/sql/tests/connect/pandas/test_parity_pandas_grouped_map_with_state.py::GroupedApplyInPandasWithStateTests::test_apply_in_pandas_with_state_basic_more_data
pyspark/sql/tests/connect/pandas/test_parity_pandas_grouped_map_with_state.py::GroupedApplyInPandasWithStateTests::test_apply_in_pandas_with_state_basic_no_state
pyspark/sql/tests/connect/pandas/test_parity_pandas_grouped_map_with_state.py::GroupedApplyInPandasWithStateTests::test_apply_in_pandas_with_state_basic_no_state_no_data
pyspark/sql/tests/connect/pandas/test_parity_pandas_grouped_map_with_state.py::GroupedApplyInPandasWithStateTests::test_apply_in_pandas_with_state_basic_with_null
pyspark/sql/tests/connect/pandas/test_parity_pandas_grouped_map_with_state.py::GroupedApplyInPandasWithStateTests::test_apply_in_pandas_with_state_python_worker_random_failure
pyspark/sql/tests/connect/pandas/test_parity_pandas_map.py::MapInPandasParityTests::test_large_variable_types
pyspark/sql/tests/connect/pandas/test_parity_pandas_map.py::MapInPandasParityTests::test_map_in_pandas_with_barrier_mode
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf.py::PandasUDFParityTests::test_pandas_udf_basic_with_return_type_string
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf.py::PandasUDFParityTests::test_pandas_udf_return_type_error
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf.py::PandasUDFParityTests::test_udf_wrong_arg
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_grouped_agg.py::PandasUDFGroupedAggParityTests::test_invalid_args
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_grouped_agg.py::PandasUDFGroupedAggParityTests::test_kwargs
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_grouped_agg.py::PandasUDFGroupedAggParityTests::test_named_arguments
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_grouped_agg.py::PandasUDFGroupedAggParityTests::test_named_arguments_and_defaults
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_grouped_agg.py::PandasUDFGroupedAggParityTests::test_named_arguments_negative
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_chained_udfs_with_complex_variant
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_chained_udfs_with_variant
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_kwargs
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_named_arguments
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_named_arguments_and_defaults
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_named_arguments_negative
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_nondeterministic_vectorized_udf_in_aggregate
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_scalar_iter_udf_init
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_udafs_with_complex_variant_input
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_udafs_with_complex_variant_output
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_udafs_with_variant_input
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_udafs_with_variant_output
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_udf_with_nested_variant_input
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_udf_with_variant_input
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_udf_with_variant_nested_output
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_udf_with_variant_output
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_vectorized_udf_check_config
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_scalar.py::PandasUDFScalarParityTests::test_vectorized_udf_invalid_length
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_bounded_mixed
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_bounded_simple
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_invalid_args
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_kwargs
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_named_arguments
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_named_arguments_negative
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_shrinking_window
pyspark/sql/tests/connect/pandas/test_parity_pandas_udf_window.py::PandasUDFWindowParityTests::test_sliding_window
pyspark/sql/tests/connect/shell/test_progress.py::SparkConnectProgressHandlerE2E::test_custom_handler_works
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_column_regexp
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_create_global_temp_view
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_explain_string
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_extended_hint_types
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_hint
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_input_files
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_is_local
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_join_hint
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_observe
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_same_semantics
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_schema
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_self_join
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_semantic_hash
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_sql_with_command
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_sql_with_named_args
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_sql_with_pos_args
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_tail
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectBasicTests::test_to
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectGCTests::test_garbage_collection_checkpoint
pyspark/sql/tests/connect/test_connect_basic.py::SparkConnectGCTests::test_garbage_collection_derived_checkpoint
pyspark/sql/tests/connect/test_connect_collection.py::SparkConnectCollectionTests::test_collect
pyspark/sql/tests/connect/test_connect_collection.py::SparkConnectCollectionTests::test_collect_timestamp
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_column_accessor
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_column_arithmetic_ops
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_column_field_ops
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_column_string_ops
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_columns
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_decimal
pyspark/sql/tests/connect/test_connect_column.py::SparkConnectColumnTests::test_distributed_sequence_id
pyspark/sql/tests/connect/test_connect_creation.py::SparkConnectCreationTests::test_streaming_local_relation
pyspark/sql/tests/connect/test_connect_creation.py::SparkConnectCreationTests::test_with_local_list
pyspark/sql/tests/connect/test_connect_creation.py::SparkConnectCreationTests::test_with_local_ndarray
pyspark/sql/tests/connect/test_connect_dataframe_property.py::SparkConnectDataFramePropertyTests::test_cached_schema_cogroup_apply_in_arrow
pyspark/sql/tests/connect/test_connect_dataframe_property.py::SparkConnectDataFramePropertyTests::test_cached_schema_cogroup_apply_in_pandas
pyspark/sql/tests/connect/test_connect_dataframe_property.py::SparkConnectDataFramePropertyTests::test_cached_schema_group_apply_in_arrow
pyspark/sql/tests/connect/test_connect_dataframe_property.py::SparkConnectDataFramePropertyTests::test_cached_schema_group_apply_in_pandas
pyspark/sql/tests/connect/test_connect_dataframe_property.py::SparkConnectDataFramePropertyTests::test_cached_schema_map_in_arrow
pyspark/sql/tests/connect/test_connect_dataframe_property.py::SparkConnectDataFramePropertyTests::test_cached_schema_map_in_pandas
pyspark/sql/tests/connect/test_connect_error.py::SparkConnectErrorTests::test_deduplicate_within_watermark_in_batch
pyspark/sql/tests/connect/test_connect_error.py::SparkConnectErrorTests::test_invalid_column
pyspark/sql/tests/connect/test_connect_error.py::SparkConnectErrorTests::test_invalid_star
pyspark/sql/tests/connect/test_connect_error.py::SparkConnectErrorTests::test_recursion_handling_for_plan_logging
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_aggregation_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_broadcast
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_collection_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_csv_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_date_ts_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_json_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_lambda_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_map_collection_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_math_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_nested_lambda_function
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_non_deterministic_with_seed
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_normal_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_string_functions_multi_args
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_string_functions_one_arg
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_time_window_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_udf
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_udtf
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_window_functions
pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_xml_functions
pyspark/sql/tests/connect/test_connect_readwriter.py::SparkConnectReadWriterTests::test_json
pyspark/sql/tests/connect/test_connect_readwriter.py::SparkConnectReadWriterTests::test_multi_paths
pyspark/sql/tests/connect/test_connect_readwriter.py::SparkConnectReadWriterTests::test_orc
pyspark/sql/tests/connect/test_connect_readwriter.py::SparkConnectReadWriterTests::test_simple_read_without_schema
pyspark/sql/tests/connect/test_connect_readwriter.py::SparkConnectReadWriterTests::test_simple_udt_from_read
pyspark/sql/tests/connect/test_connect_readwriter.py::SparkConnectReadWriterTests::test_write_operations
pyspark/sql/tests/connect/test_connect_readwriter.py::SparkConnectReadWriterTests::test_xml
pyspark/sql/tests/connect/test_connect_session.py::ChannelBuilderTests::test_sensible_defaults
pyspark/sql/tests/connect/test_connect_session.py::SparkConnectSessionTests::test_authentication
pyspark/sql/tests/connect/test_connect_session.py::SparkConnectSessionTests::test_error_enrichment_jvm_stacktrace
pyspark/sql/tests/connect/test_connect_session.py::SparkConnectSessionTests::test_error_enrichment_message
pyspark/sql/tests/connect/test_connect_session.py::SparkConnectSessionTests::test_error_stack_trace
pyspark/sql/tests/connect/test_connect_session.py::SparkConnectSessionTests::test_get_message_parameters_without_enriched_error
pyspark/sql/tests/connect/test_connect_session.py::SparkConnectSessionTests::test_reset_when_server_and_client_sessionids_mismatch
pyspark/sql/tests/connect/test_connect_session.py::SparkConnectSessionTests::test_reset_when_server_session_id_mismatch
pyspark/sql/tests/connect/test_connect_stat.py::SparkConnectStatTests::test_describe
pyspark/sql/tests/connect/test_connect_stat.py::SparkConnectStatTests::test_grouped_data
pyspark/sql/tests/connect/test_connect_stat.py::SparkConnectStatTests::test_numeric_aggregation
pyspark/sql/tests/connect/test_connect_stat.py::SparkConnectStatTests::test_random_split
pyspark/sql/tests/connect/test_connect_stat.py::SparkConnectStatTests::test_stat_approx_quantile
pyspark/sql/tests/connect/test_connect_stat.py::SparkConnectStatTests::test_stat_freq_items
pyspark/sql/tests/connect/test_connect_stat.py::SparkConnectStatTests::test_stat_sample_by
pyspark/sql/tests/connect/test_df_debug.py::SparkConnectDataFrameDebug::test_df_debug_basics
pyspark/sql/tests/connect/test_df_debug.py::SparkConnectDataFrameDebug::test_query_execution_text_format
pyspark/sql/tests/connect/test_parity_catalog.py::CatalogParityTests::test_function_exists
pyspark/sql/tests/connect/test_parity_catalog.py::CatalogParityTests::test_get_function
pyspark/sql/tests/connect/test_parity_catalog.py::CatalogParityTests::test_list_functions
pyspark/sql/tests/connect/test_parity_catalog.py::CatalogParityTests::test_refresh_table
pyspark/sql/tests/connect/test_parity_catalog.py::CatalogParityTests::test_table_cache
pyspark/sql/tests/connect/test_parity_column.py::ColumnParityTests::test_access_column
pyspark/sql/tests/connect/test_parity_column.py::ColumnParityTests::test_getitem_column
pyspark/sql/tests/connect/test_parity_creation.py::DataFrameCreationParityTests::test_schema_inference_from_pandas_with_dict
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_cache_dataframe
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_cache_table
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_coalesce_hints_with_string_parameter
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_create_df_with_collation
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_dataframe_star
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_duplicate_field_names
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_extended_hint_types
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_generic_hints
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_input_files
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_join_without_on
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_lateral_column_alias
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_local_checkpoint_dataframe
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_local_checkpoint_dataframe_with_storage_level
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_metadata_column
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_pandas_api
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_repr_behaviors
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_require_cross
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_self_join_II
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_self_join_III
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_self_join_IV
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_to
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_transpose
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_transpose_with_invalid_index_columns
pyspark/sql/tests/connect/test_parity_dataframe.py::DataFrameParityTests::test_with_column_and_generator
pyspark/sql/tests/connect/test_parity_dataframe_query_context.py::DataFrameQueryContextParityTests::test_dataframe_query_context
pyspark/sql/tests/connect/test_parity_dataframe_query_context.py::DataFrameQueryContextParityTests::test_dataframe_query_context_col
pyspark/sql/tests/connect/test_parity_dataframe_query_context.py::DataFrameQueryContextParityTests::test_query_context_complex
pyspark/sql/tests/connect/test_parity_dataframe_query_context.py::DataFrameQueryContextParityTests::test_sql_query_context
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_checking_csv_header
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_encoding_json
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_ignore_column_of_all_nulls
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_ignorewhitespace_csv
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_jdbc
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_jdbc_format
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_linesep_json
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_linesep_text
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_multiline_csv
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_multiline_json
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_read_multiple_orc_file
pyspark/sql/tests/connect/test_parity_datasources.py::DataSourcesParityTests::test_xml
pyspark/sql/tests/connect/test_parity_errors.py::ErrorsParityTests::test_array_index_out_of_bounds_exception
pyspark/sql/tests/connect/test_parity_errors.py::ErrorsParityTests::test_date_time_exception
pyspark/sql/tests/connect/test_parity_errors.py::ErrorsParityTests::test_number_format_exception
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_approxQuantile
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_assert_true
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_collation
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_enum_literals
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_functions_broadcast
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_inline
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_input_file_name_udf
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_nested_higher_order_function
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_non_deterministic_with_seed
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_nth_value
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_parse_json
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_raise_error
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_randstr_uniform
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_sampleby
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_to_variant_object
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_try_make_interval
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_try_make_timestamp
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_try_make_timestamp_ltz
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_try_make_timestamp_ntz
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_try_parse_json
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_variant_expressions
pyspark/sql/tests/connect/test_parity_functions.py::FunctionsParityTests::test_window_time
pyspark/sql/tests/connect/test_parity_group.py::GroupParityTests::test_agg_func
pyspark/sql/tests/connect/test_parity_observation.py::DataFrameObservationParityTests::test_observe
pyspark/sql/tests/connect/test_parity_observation.py::DataFrameObservationParityTests::test_observe_on_commands
pyspark/sql/tests/connect/test_parity_observation.py::DataFrameObservationParityTests::test_observe_str
pyspark/sql/tests/connect/test_parity_observation.py::DataFrameObservationParityTests::test_observe_with_same_name_on_different_dataframe
pyspark/sql/tests/connect/test_parity_python_datasource.py::PythonDataSourceParityTests::test_arrow_batch_data_source
pyspark/sql/tests/connect/test_parity_python_datasource.py::PythonDataSourceParityTests::test_arrow_batch_sink
pyspark/sql/tests/connect/test_parity_python_datasource.py::PythonDataSourceParityTests::test_custom_json_data_source_abort
pyspark/sql/tests/connect/test_parity_python_datasource.py::PythonDataSourceParityTests::test_custom_json_data_source_commit
pyspark/sql/tests/connect/test_parity_python_datasource.py::PythonDataSourceParityTests::test_custom_json_data_source_read
pyspark/sql/tests/connect/test_parity_python

(truncated)

@lonless9 lonless9 added the run spark tests Trigger Spark tests on a pull request label Oct 9, 2025
@codecov
Copy link

codecov bot commented Oct 9, 2025

Codecov Report

❌ Patch coverage is 37.29563% with 1419 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/sail-iceberg/src/spec/datatypes.rs 13.56% 344 Missing ⚠️
crates/sail-iceberg/src/spec/manifest.rs 12.61% 194 Missing ⚠️
crates/sail-iceberg/src/datasource/provider.rs 69.19% 183 Missing ⚠️
crates/sail-iceberg/src/spec/manifest_list.rs 4.54% 126 Missing ⚠️
crates/sail-iceberg/src/spec/schema.rs 50.21% 114 Missing ⚠️
crates/sail-iceberg/src/spec/snapshot.rs 5.30% 107 Missing ⚠️
crates/sail-iceberg/src/spec/partition.rs 0.00% 101 Missing ⚠️
crates/sail-iceberg/src/spec/transform.rs 0.00% 100 Missing ⚠️
crates/sail-iceberg/src/table_format.rs 68.96% 36 Missing ⚠️
crates/sail-iceberg/src/spec/sort.rs 0.00% 24 Missing ⚠️
... and 5 more
Files with missing lines Coverage Δ
crates/sail-data-source/src/registry.rs 82.85% <100.00%> (-17.15%) ⬇️
crates/sail-data-source/src/formats/iceberg.rs 83.33% <83.33%> (ø)
crates/sail-iceberg/src/arrow_conversion.rs 84.00% <84.00%> (ø)
crates/sail-iceberg/src/spec/name_mapping.rs 0.00% <0.00%> (ø)
crates/sail-iceberg/src/spec/table_metadata.rs 54.90% <54.90%> (ø)
crates/sail-iceberg/src/spec/values.rs 0.00% <0.00%> (ø)
crates/sail-iceberg/src/spec/sort.rs 0.00% <0.00%> (ø)
crates/sail-iceberg/src/table_format.rs 68.96% <68.96%> (ø)
crates/sail-iceberg/src/spec/transform.rs 0.00% <0.00%> (ø)
crates/sail-iceberg/src/spec/partition.rs 0.00% <0.00%> (ø)
... and 6 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lonless9 lonless9 removed the run spark tests Trigger Spark tests on a pull request label Oct 9, 2025
@lonless9
Copy link
Contributor Author

lonless9 commented Oct 10, 2025 •
edited
Loading

Before waiting for the upstream to solve the corresponding issue, import pydantic and pin the version.
Remind for me: Remove pydantic import.

Tracking:
pydantic/pydantic#12347
apache/iceberg-python#2590

@shehabgamin shehabgamin mentioned this pull request Oct 15, 2025
Merged
@lonless9 lonless9 changed the title feat: read operation for Iceberg feat: basic read operation for Iceberg Oct 16, 2025
@lonless9 lonless9 marked this pull request as ready for review October 16, 2025 07:53
@lonless9 lonless9 requested a review from linhr October 16, 2025 07:53
@linhr linhr mentioned this pull request Oct 16, 2025
Open
48 tasks
linhr
linhr approved these changes Oct 16, 2025
View reviewed changes
Copy link
Contributor

@linhr linhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is exciting!! 🚀🚀

crates/sail-iceberg/src/spec/views/mod.rs
Comment on lines +18 to +24
pub mod view_metadata;
pub mod view_metadata_builder;
pub mod view_version;

pub use view_metadata::*;
pub use view_metadata_builder::*;
pub use view_version::*;
Copy link
Contributor

@linhr linhr Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the view is only used for the REST API, we can remove the spec entirely from the sail-iceberg crate.

(Just a comment for future work. No need to change anything for now.)

python/pysail/tests/spark/iceberg/utils.py
catalog = load_catalog(
"test_catalog",
type="sql",
uri=f"sqlite:///{tmp_path}/pyiceberg_catalog.db",
Copy link
Contributor

@linhr linhr Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this path work on Windows?

Copy link
Contributor Author

@lonless9 lonless9 Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried the Windows environment and only had escape_sql_string_literal related issues, and so far all tests have passed!

@lonless9 lonless9 merged commit 778fbfc into main Oct 17, 2025
21 checks passed
@lonless9 lonless9 deleted the iceberg-scan branch October 17, 2025 02:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@linhr linhr linhr approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lonless9 @linhr