GH-49875: [Python] Fix timezone dropped when converting tz-aware Categorical to Arrow array

[AnkitAhlawat7742]

Rationale for this change

When converting a pandas.Categorical with tz-aware datetime categories to a PyArrow array, the timezone information was silently dropped from the dictionary array's value type. This is a silent data loss bug — no warning or error is raised, but the timezone metadata is lost.

What changes are included in this PR?

In python/pyarrow/array.pxi, the Categorical conversion was using values.categories.values(raw numpy array) which strips timezone metadata since numpy does not support tz-aware datetimes. Changed to values.categories (pandas Index) and added from_pandas=True so PyArrow uses the pandas conversion path, which correctly preserves timezone metadata.

Are these changes tested?

Yes. Verified manually

Are there any user-facing changes?

Yes — this is a bug fix. Users did #49875

This PR contains a "Critical Fix" — timezone information was lost silently during conversion without any warning or error.

• GitHub Issue: [Python] Timezone being dropped in dictionary array #49875

[github-actions]

⚠️ GitHub issue #49875 has been automatically assigned in GitHub to PR creator.

[AnkitAhlawat7742]

Hi @AlenkaF
Please review these changes

[AlenkaF]

I think this change makes sense but the PR is lacking a test. The reproducible example from the issue can be reused and added to https://github.com/apache/arrow/blob/main/python/pyarrow/tests/test_pandas.py mimicking existing categorical tests.

cc @jorisvandenbossche in case of any opinions.

[AnkitAhlawat7742]

I think this change makes sense but the PR is lacking a test. The reproducible example from the issue can be reused and added to https://github.com/apache/arrow/blob/main/python/pyarrow/tests/test_pandas.py mimicking existing categorical tests.

cc @jorisvandenbossche in case of any opinions.

Thank you for the review!
I have added a regression test to python/pyarrow/tests/test_pandas.py based on the reproducible example from the issue.
Please have a look.

[AlenkaF]

CI failures are related, could you have a look?

[AnkitAhlawat7742]

CI failures are related, could you have a look?

Fixed:

1. Resolved the formatting issue.
2. CI failure was caused by pandas version compatibility in the test assertion:
   str(cat.dtype.categories.dtype) == "datetime64[us, US/Eastern]"

Please retrigger the CI

[jorisvandenbossche]

Looks good!

[github-actions]

⚠️ GitHub issue #49875 has been automatically assigned in GitHub to PR creator.

[AlenkaF]

@github-actions crossbow submit pandas

[github-actions]

Revision: 0f1f595

Submitted crossbow builds: ursacomputing/crossbow @ actions-3f725ba4fc

Task	Status
test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.12-pandas-latest-numpy-1.26
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13-pandas-nightly-numpy-nightly
test-conda-python-3.13-pandas-upstream_devel-numpy-nightly

[AlenkaF]

Thanks! The failing extended builds are not connected. Will try to see if there is an issue opened already to track that.

[AlenkaF]

Opened an issue for the failing builds here: #49920

[AlenkaF]

@jorisvandenbossche mind having one more look before I merge?

[raulcd]

This looks good to me

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit ea8cef5.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.