Fix dsd threshold behaviour and increase threshold a lil - #8
Merged
Conversation
idelder
force-pushed
the
ide/fix/dsd_thresh
branch
from
August 7, 2026 14:16
2ead44e to
56d54ec
Compare
There was a problem hiding this comment.
Pull request overview
This PR corrects the DSD percentile cutoff buffering logic in the representative-period database processing pipeline to avoid incorrectly zeroing out borderline (often duplicate) DSD values due to floating-point jitter, which can otherwise lead to missing data and downstream divide-by-zero behavior.
Changes:
- Adjust the cutoff buffer so it is applied in the direction that preserves
dsd ~= dsd_1%values rather than dropping them. - Increase the floating-point buffer magnitude to
1e-10to better tolerate compounded rounding noise.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+265
to
+267
| # Get the smallest DSD above thresh to zero out actual table | ||
| thresh_dsd = df["dsd"].loc[df["run_sum"] < utils.config["dsd_threshold"]].max() | ||
| thresh_dsd += 1e-12 # Small buffer to avoid floating point issues | ||
| thresh_dsd -= 1e-10 # Small buffer to avoid floating point issues |
There was a problem hiding this comment.
Fixed the comment in commit Fix misleading comment on DSD threshold selection. The line now reads: # Get the largest DSD whose cumulative proportion is still below the threshold
Signed-off-by: Davey Elder <iandavidelder@gmail.com>
idelder
force-pushed
the
ide/fix/dsd_thresh
branch
from
August 11, 2026 21:53
11ce1d8 to
2ee7660
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The way the DSD percentile threshold works is it finds the e.g. 1st percentile value (smallest 1% of DSD values) for each demand, then sets everything in the 1st percentile to zero. This is a nice robust way to remove the smallest DSD values, for numerical conditioning of the model, while creating, at worst, a 1% error in total annual demand.
Let
dsd_1%be the 1st percentile valueBecause of floating point errors, we can't just do
keep dsd >= dsd_1%because the dsd values vary randomly by ~1E-16 for standard double-precision 64-bit floats. To deal with these, we simply add a small threshold,
δ, to the LHS likekeep dsd + δ >= dsd_1%Except that there was a bug where this threshold was added to the RHS by accident like
keep dsd >= dsd_1% + δWhich meant that we were dropping all
dsd ~= dsd_1%, which, in the case of dsd tables with many or all identical values, was much or all of the dsd table, leading to missing data and divide-by-zero errors.This change simply subtracts the threshold from the RHS instead
keep dsd >= dsd_1% - δWe also increase the threshold to 1E-10 in case some rounding errors are compounding. Realistically, the smallest plausible dsd value would be something like, "one one-thousandth of the average for ten years of hourly data" or:
1/(8760*10*1000) = ~1E-8