-
Notifications
You must be signed in to change notification settings - Fork 24
Description
Description of the issue
While working through a Hi bug related to obs_date values I found that given a single dataset, the projected obs_date values deviated up to ~60 seconds away from the expected value of the PSET midpoint as defined by the epoch + epoch_delta / 2.
I had a suspicion that the error was due to using tt2000ns for the time that was being projected. Just to check, I converted the obs_date value in the PSET prior to projecting to tt2000s and the error was reduced to about 20-seconds.
Steps to reproduce the issue
The relevant test that exhibits this inaccuracy is in test_hi_l2.py::test_genarate_hi_map
The check on the projected obs_date value is:
np.testing.assert_allclose(
np.nanmax(sky_map.data_1d["obs_date"].data) / 1e9,
pset_midpoint,
atol=60,
)
Expected behavior (What should happen)
No response
Actual behavior (What does happen)
No response
Code Snippet:
Code
Additional notes, affected areas, and suggested fixes
From Claude... suggestions for how to handle this:
- Offset approach (recommended)
Subtract a reference timestamp before accumulating, then add it back after computing the mean:Pick a reference time (e.g., first PSET's midpoint or epoch start)
reference_time = int(pset_ds["epoch"].values[0])
Store offset during processing (small values, safe for float64)
obs_date_offset = mid_time - reference_time # Will be ~1e16 or less for 3-month maps
After weighted mean calculation, add reference back
final_obs_date = obs_date_weighted_mean + reference_time
The offset values for a 3-month map would be at most ~8e15 ns, well within float64's exact integer range (2^53 ≈ 9e15).
- Scale to coarser units during accumulation
Convert to microseconds or milliseconds before accumulating, then convert back:Accumulate in microseconds (divide by 1e3)
obs_date_us = mid_time // 1000 # int64, values ~1e15
After mean, convert back
final_obs_date = obs_date_mean_us * 1000
- Incremental weighted mean
Compute the weighted mean incrementally rather than accumulating sums, avoiding large intermediate values entirely.
The offset approach is probably cleanest since it:
- Preserves nanosecond precision
- Keeps math simple (just subtract/add a constant)
- Works naturally with the existing xarray binning operations
- Differences within a 3-month period are well within float64's exact range
Metadata
Metadata
Assignees
Labels
Type
Projects
Status