Skip to content

BUG - inaccuracy in L2 map obs_date values after projection #2383

@subagonsouth

Description

@subagonsouth

Description of the issue

While working through a Hi bug related to obs_date values I found that given a single dataset, the projected obs_date values deviated up to ~60 seconds away from the expected value of the PSET midpoint as defined by the epoch + epoch_delta / 2.

I had a suspicion that the error was due to using tt2000ns for the time that was being projected. Just to check, I converted the obs_date value in the PSET prior to projecting to tt2000s and the error was reduced to about 20-seconds.

Steps to reproduce the issue

The relevant test that exhibits this inaccuracy is in test_hi_l2.py::test_genarate_hi_map

The check on the projected obs_date value is:

np.testing.assert_allclose(
        np.nanmax(sky_map.data_1d["obs_date"].data) / 1e9,
        pset_midpoint,
        atol=60,
    )

Expected behavior (What should happen)

No response

Actual behavior (What does happen)

No response

Code Snippet:

Code

Additional notes, affected areas, and suggested fixes

From Claude... suggestions for how to handle this:

  1. Offset approach (recommended)
    Subtract a reference timestamp before accumulating, then add it back after computing the mean:

Pick a reference time (e.g., first PSET's midpoint or epoch start)

reference_time = int(pset_ds["epoch"].values[0])

Store offset during processing (small values, safe for float64)

obs_date_offset = mid_time - reference_time # Will be ~1e16 or less for 3-month maps

After weighted mean calculation, add reference back

final_obs_date = obs_date_weighted_mean + reference_time

The offset values for a 3-month map would be at most ~8e15 ns, well within float64's exact integer range (2^53 ≈ 9e15).

  1. Scale to coarser units during accumulation
    Convert to microseconds or milliseconds before accumulating, then convert back:

Accumulate in microseconds (divide by 1e3)

obs_date_us = mid_time // 1000 # int64, values ~1e15

After mean, convert back

final_obs_date = obs_date_mean_us * 1000

  1. Incremental weighted mean
    Compute the weighted mean incrementally rather than accumulating sums, avoiding large intermediate values entirely.

The offset approach is probably cleanest since it:

  • Preserves nanosecond precision
  • Keeps math simple (just subtract/add a constant)
  • Works naturally with the existing xarray binning operations
  • Differences within a 3-month period are well within float64's exact range

Metadata

Metadata

Assignees

Labels

HelperIns: HiRelated to the IMAP-Hi instrumentIns: LoRelated to the IMAP-Lo instrumentIns: UltraRelated to the IMAP-Ultra instrumentMapper ToolsWork related to common mapper toolsPhase E: Parent IssuePhase E major task indicatorbugSomething isn't working

Type

Projects

Status

Todo

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions