Fix #60766:.map,.apply would convert element type for extension array #61396
+29
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…sion array. The Int32Dtype type allows representing integers with support for null values (pd.NA). However, when using .map(f) or .apply(f), the elements passed to f are converted to float64, and pd.NA is transformed into np.nan. This happens because .map() and .apply() internally use numpy, which automatically converts the data to float64, even when the original type is Int32Dtype. The fix (just remove the method to_numpy()) ensures that when using .map() or .apply(), the elements in the series retain their original type (Int32, Float64, boolean, etc.), preventing unnecessary conversions to float64 and ensuring that pd.NA remains correctly handled.
| @@ -181,10 +187,15 @@ def test_map(self, data_missing, na_action): | |||
| def test_map_na_action_ignore(self, data_missing_for_sorting): | |||
| zero = data_missing_for_sorting[2] | |||
| result = data_missing_for_sorting.map(lambda x: zero, na_action="ignore") | |||
|
|
|||
There was a problem hiding this comment.
Better to avoid this unrelated changes
jbrockmendel
reviewed
Jul 8, 2025
| return x + 1 | ||
|
|
||
| result = s.map(transform) | ||
| expected = Series([2, 3, NA, 5], dtype=result.dtype) |
There was a problem hiding this comment.
can you be explicit about the expected dtype. i.e. is it Int32?
jbrockmendel
added
the
pyarrow dtype retention
| @@ -0,0 +1,20 @@ | |||
| from pandas import ( | |||
There was a problem hiding this comment.
can you name this file just test_map.py
| Series, | ||
| isna, | ||
| ) | ||
| from pandas.testing import assert_series_equal |
There was a problem hiding this comment.
use tm.assert_series_equal
| if data_missing.dtype.kind != "b": | ||
| for i in range(len(result)): | ||
| if result[i] is pd.NA: | ||
| result[i] = "nan" |
There was a problem hiding this comment.
isnt this pretty unwanted behavior?
|
i suspect the correct thing to do for map involves the just-implemented EA._cast_pointwise_result |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
.map&.applywould convert element type for extension array. #60766doc/source/whatsnew/v3.0.0.rstfile if fixing a bug or adding a new feature.The Int32Dtype type allows representing integers with support for null values (pd.NA). However, when using .map(f) or .apply(f), the elements passed to f are converted to float64, and pd.NA is transformed into np.nan.
This happens because .map() and .apply() internally use numpy, which automatically converts the data to float64, even when the original type is Int32Dtype.
The fix (just remove the method to_numpy()) ensures that when using .map() or .apply(), the elements in the series retain their original type (Int32, Float64, boolean, etc.), preventing unnecessary conversions to float64 and ensuring that pd.NA remains correctly handled.