[SEDONA-699] Fix issue with not closing parquet files.

[Imbruced]

Did you read the Contributor Guide?

• Yes, I have read the Contributor Rules and Contributor Development Guide

• No, I haven't read it.

Is this PR related to a JIRA ticket?

• Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-XXX. The PR name follows the format [SEDONA-XXX] my subject.

• No:

  • this is a documentation update. The PR name follows the format [DOCS] my subject
  • this is a CI update. The PR name follows the format [CI] my subject

What changes were proposed in this PR?

How was this patch tested?

Did this PR include necessary documentation updates?

• Yes, I am adding a new API. I am using the current SNAPSHOT version number in vX.Y.Z format.
• Yes, I have updated the documentation.
• No, this PR does not affect any public API so no need to change the documentation.

[Imbruced]

For some file size thresholds and a certain number of them (in my case, 15 files with 30 MB in size), the data frame show gets stuck. Closing parquet files seems to fix it; I am unsure if we want to add one huge file and copy it in a test to replicate it.

[Kontinuation]

Thank you for fixing this. I have just found that the way we are reading Parquet file metadata from footer is not optimal. We should simply use ParquetFileReader.readFooter(configuration, path, SKIP_ROW_GROUPS).getFileMetaData to read the file metadata, then we don't have to bother closing the readers. It also does less work by skip reading row groups, as row groups of huge files can be quite large.

I don't think we need to add heavy weight tests for this by the way, as the fix is very straightforward, and it is not easy to write simple and quick tests for it.

[Imbruced]

@Kontinuation I'll adjust to use the readFooter method. Also, I'll create a ticket.

[Imbruced]

@jiayuasu do we proceed with that? My only concern is the readFooter function is deprecated for spark3.3.