Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature / columnar data optional id field support in update data #771

Merged
merged 141 commits into from
Nov 5, 2024
Merged
Changes from 1 commit
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
b9f2995
branch created
Jerry-Jinfeng-Guo Oct 1, 2024
3d1bfcc
added test with no ids
Jerry-Jinfeng-Guo Oct 7, 2024
340b740
[skip ci] functionality partially in place; wip
Jerry-Jinfeng-Guo Oct 8, 2024
1edc403
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 8, 2024
8ded552
[skip ci]
Jerry-Jinfeng-Guo Oct 8, 2024
ea6ba47
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 9, 2024
575e24d
[skip ci] added logic for has_id and id_nan
Jerry-Jinfeng-Guo Oct 9, 2024
0ea07fa
[skip ci] more component level properties
Jerry-Jinfeng-Guo Oct 10, 2024
75b1583
[skip ci] minor fix on the independence criteria
Jerry-Jinfeng-Guo Oct 14, 2024
721eec4
[skip ci] added elements per scenario of base; subsequent comparison …
Jerry-Jinfeng-Guo Oct 14, 2024
464ed64
[skip ci] added checks
Jerry-Jinfeng-Guo Oct 14, 2024
d2ce49d
[skip ci] functionality accepting sequence
Jerry-Jinfeng-Guo Oct 14, 2024
3280fa8
[skip ci] functionalities for get sequence in place; next to the upda…
Jerry-Jinfeng-Guo Oct 15, 2024
faa5966
[skip ci] removing id match assert: incompatible with optional id; wh…
Jerry-Jinfeng-Guo Oct 15, 2024
88610d2
[skip ci] empty columnar data input supported
Jerry-Jinfeng-Guo Oct 15, 2024
f584870
[skip ci] made test more complete; next to merge with `main`
Jerry-Jinfeng-Guo Oct 15, 2024
9a11e6a
[skip ci] removed commented code
Jerry-Jinfeng-Guo Oct 15, 2024
892ca09
[skip ci] address some comments
Jerry-Jinfeng-Guo Oct 16, 2024
ec5bc65
[skip ci] more comments addressed
Jerry-Jinfeng-Guo Oct 16, 2024
4d63387
[skip ci] one more comment addressed
Jerry-Jinfeng-Guo Oct 16, 2024
b06d8c5
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 17, 2024
890837a
[skip ci] branch created
Jerry-Jinfeng-Guo Oct 17, 2024
4ca60bd
[skip ci] intermediate commit marking several python tests as fail
Jerry-Jinfeng-Guo Oct 18, 2024
3b0c6b6
removed `id_exists` check and tests
Jerry-Jinfeng-Guo Oct 18, 2024
74e2225
Your code has been rated at 10.00/10, but I throw an error nonetheless
Jerry-Jinfeng-Guo Oct 18, 2024
6bbd9a8
`isort` has its own opinion
Jerry-Jinfeng-Guo Oct 18, 2024
9ea0f15
Merge branch 'main' into feature/update-python-validation-to-support-…
Jerry-Jinfeng-Guo Oct 18, 2024
8fef79c
Merge branch 'main' into feature/update-python-validation-to-support-…
Jerry-Jinfeng-Guo Oct 18, 2024
56cae49
New check logic
Jerry-Jinfeng-Guo Oct 18, 2024
4880c91
explicit optional
Jerry-Jinfeng-Guo Oct 21, 2024
f2a5177
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 21, 2024
fc9b7c8
unit test added
Jerry-Jinfeng-Guo Oct 21, 2024
22b0851
ignored two python tests
Jerry-Jinfeng-Guo Oct 21, 2024
d51d93e
typo
Jerry-Jinfeng-Guo Oct 21, 2024
23f6fcc
clang-tidy
Jerry-Jinfeng-Guo Oct 21, 2024
bf01d2e
clang-tidy 2.0
Jerry-Jinfeng-Guo Oct 21, 2024
64f3483
sonar cloud
Jerry-Jinfeng-Guo Oct 21, 2024
aee0dbb
clang-tidy
Jerry-Jinfeng-Guo Oct 21, 2024
c71493a
unit test added; next to columnar input
Jerry-Jinfeng-Guo Oct 21, 2024
33e2e62
added get id by idx2d
Jerry-Jinfeng-Guo Oct 22, 2024
bee2724
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 22, 2024
e7ff7f4
assert id check
Jerry-Jinfeng-Guo Oct 22, 2024
aa9a1e0
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 22, 2024
e035c3d
update error in calculate
Jerry-Jinfeng-Guo Oct 22, 2024
8599250
not necessarily a regular error type
Jerry-Jinfeng-Guo Oct 22, 2024
b411515
polished the cpi model test to further complete the wrong id input; p…
Jerry-Jinfeng-Guo Oct 23, 2024
d0e8c1f
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 23, 2024
b783aef
Merge branch 'main' into feature/update-python-validation-to-support-…
Jerry-Jinfeng-Guo Oct 23, 2024
5ca7124
repro UB in columnar batch update with unknown ID
mgovers Oct 23, 2024
caa8e1d
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
mgovers Oct 23, 2024
356c23f
extended unit test for validate ids
Jerry-Jinfeng-Guo Oct 23, 2024
f775c2b
extended unit test for validate ids
Jerry-Jinfeng-Guo Oct 23, 2024
5e33695
lambda to functions
Jerry-Jinfeng-Guo Oct 23, 2024
4f0fce4
address linter complain
Jerry-Jinfeng-Guo Oct 24, 2024
e57b446
thou shall not pass; linter 2.0
Jerry-Jinfeng-Guo Oct 24, 2024
4761123
there is his-py and mypy
Jerry-Jinfeng-Guo Oct 24, 2024
a43a365
addressed comment
Jerry-Jinfeng-Guo Oct 25, 2024
ac8082d
format
Jerry-Jinfeng-Guo Oct 25, 2024
2c08abd
revert `ComponentData`
Jerry-Jinfeng-Guo Oct 25, 2024
0fd17d0
fix last test
nitbharambe Oct 25, 2024
65fea67
[skip ci] debug in progress
Jerry-Jinfeng-Guo Oct 25, 2024
250b09f
bug fixed; next to merge with base branch
Jerry-Jinfeng-Guo Oct 26, 2024
384342d
Merge pull request #805 from PowerGridModel/feature/ub-update-unknown-id
Jerry-Jinfeng-Guo Oct 26, 2024
cae7b9c
has AddressSanitizer error on WSL gcc 13.0; corrected python tests
Jerry-Jinfeng-Guo Oct 27, 2024
bf02077
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 27, 2024
5878ad6
native api model remove const
Jerry-Jinfeng-Guo Oct 27, 2024
8f25bce
clang-tidy
Jerry-Jinfeng-Guo Oct 27, 2024
ad11736
change to get comp size
nitbharambe Oct 28, 2024
b672c04
in progress
Jerry-Jinfeng-Guo Oct 28, 2024
55bd9ae
all fixed; up for review
Jerry-Jinfeng-Guo Oct 28, 2024
1ccd250
Merge pull request #793 from PowerGridModel/feature/update-python-val…
TonyXiang8787 Oct 28, 2024
234be96
addressed comments
Jerry-Jinfeng-Guo Oct 28, 2024
10f1a38
wrap function call with _DEBUG
Jerry-Jinfeng-Guo Oct 28, 2024
0bdde42
_DEBUG to NDEBUG
Jerry-Jinfeng-Guo Oct 28, 2024
c880fc5
!NDEBUG
Jerry-Jinfeng-Guo Oct 28, 2024
f66e723
NDEBUG is still complained
Jerry-Jinfeng-Guo Oct 28, 2024
fbbb916
comments addressed
Jerry-Jinfeng-Guo Oct 28, 2024
add533f
removed unused lines in test
Jerry-Jinfeng-Guo Oct 28, 2024
b7e5450
sonar cloud issues
Jerry-Jinfeng-Guo Oct 29, 2024
454af2c
undo error introduced from last commit
Jerry-Jinfeng-Guo Oct 29, 2024
b79a4f9
re-added python validation
Jerry-Jinfeng-Guo Oct 29, 2024
bd459ef
mypy didn't catch mismatch, but his-py did
Jerry-Jinfeng-Guo Oct 29, 2024
c17cc2d
removed static test
Jerry-Jinfeng-Guo Oct 29, 2024
04fcf6a
comments addressed
Jerry-Jinfeng-Guo Oct 29, 2024
b3ec5e8
comments addressed
Jerry-Jinfeng-Guo Oct 29, 2024
1170719
sonar cloud
Jerry-Jinfeng-Guo Oct 29, 2024
c954095
python validation
Jerry-Jinfeng-Guo Oct 29, 2024
0e3bbb8
test deleted by mistake
Jerry-Jinfeng-Guo Oct 29, 2024
20054a3
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 29, 2024
add8802
add test
nitbharambe Oct 29, 2024
20787cf
add fix
nitbharambe Oct 29, 2024
c8e01ab
Merge pull request #812 from PowerGridModel/fix/update-input-data-opt…
Jerry-Jinfeng-Guo Oct 29, 2024
97e7c8b
[skip ci]
Jerry-Jinfeng-Guo Oct 29, 2024
963cac2
address comments
Jerry-Jinfeng-Guo Oct 30, 2024
2f063cf
add row based sparse test
nitbharambe Oct 30, 2024
679e9b8
addressed comments
Jerry-Jinfeng-Guo Oct 30, 2024
b439f33
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Oct 30, 2024
6ca043f
addressed comment
Jerry-Jinfeng-Guo Oct 31, 2024
d52d6ce
constexpr comment
Jerry-Jinfeng-Guo Oct 31, 2024
e29748e
addressed commits; handed over to Martijn and Nitish for final touch
Jerry-Jinfeng-Guo Oct 31, 2024
a8ea09d
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Oct 31, 2024
76df59d
update ids checks
mgovers Nov 4, 2024
e20d96d
fix part of optional ids
mgovers Nov 4, 2024
3d5d0bb
make compile again
mgovers Nov 4, 2024
ff8f44b
most tests pass again
mgovers Nov 4, 2024
ca6960b
fix api model tests
mgovers Nov 4, 2024
4fedd67
attempt cleanup part 1
mgovers Nov 4, 2024
d2c8b03
clang-tidy
mgovers Nov 5, 2024
8ec4bbb
fix get attribute
mgovers Nov 5, 2024
fa71eda
resolve comments
mgovers Nov 5, 2024
a0a4523
remove TODO
mgovers Nov 5, 2024
344d5da
Merge pull request #818 from PowerGridModel/feature/resolve-comments-…
Jerry-Jinfeng-Guo Nov 5, 2024
f0d4baa
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
1e4145b
sonar cloud
Jerry-Jinfeng-Guo Nov 5, 2024
ed8d0a0
format
Jerry-Jinfeng-Guo Nov 5, 2024
7a37897
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
ab3a4d7
minor
mgovers Nov 5, 2024
71f3ee0
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
161161f
use std map find
mgovers Nov 5, 2024
d1b2ccf
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
ca2c1d2
fix validator tests
nitbharambe Nov 5, 2024
2ca0fdb
channeling done; test fixed
Jerry-Jinfeng-Guo Nov 5, 2024
d1f2220
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
0513ffa
Apply suggestions from code review
mgovers Nov 5, 2024
f098446
sonar-cloud
mgovers Nov 5, 2024
1faab9f
minor
mgovers Nov 5, 2024
314af8a
Apply suggestions from code review
mgovers Nov 5, 2024
1344c9f
more resolve comments
mgovers Nov 5, 2024
d53d3ea
fix unused captured variable
mgovers Nov 5, 2024
02bf930
fix variable name
mgovers Nov 5, 2024
aa63dc8
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
8762766
fix typo
mgovers Nov 5, 2024
cfb4d1a
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
mgovers Nov 5, 2024
dba5644
fix error introduced from merging
Jerry-Jinfeng-Guo Nov 5, 2024
80d25eb
Merge branch 'main' into feature/columnar-data-support-no-id-field
Jerry-Jinfeng-Guo Nov 5, 2024
0a1ea5d
fix merge error
Jerry-Jinfeng-Guo Nov 5, 2024
69b736b
Merge branch 'feature/columnar-data-support-no-id-field' into feature…
Jerry-Jinfeng-Guo Nov 5, 2024
2c76f9c
Merge pull request #813 from PowerGridModel/feature/row-update-data-s…
Jerry-Jinfeng-Guo Nov 5, 2024
a6320cf
Revert "remove optional ids tests"
mgovers Nov 5, 2024
849d293
remove pytest mark xfail
mgovers Nov 5, 2024
74f2fef
format
mgovers Nov 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[skip ci] added elements per scenario of base; subsequent comparison …
…made easier

Signed-off-by: Jerry Guo <[email protected]>
Jerry-Jinfeng-Guo committed Oct 14, 2024
commit 721eec4e5c4ef936062aeb0eece83c6a6d6d0a1e
Original file line number Diff line number Diff line change
@@ -147,15 +147,17 @@ class MainModelImpl<ExtraRetrievableTypes<ExtraRetrievableType...>, ComponentLis

struct UpdateCompProperties {
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
std::string name{""};
bool independent{false};
bool has_id{false};
bool ids_na{false};
bool uniform{false};
bool is_columnar{false};
bool ids_match{false};
Idx elements_per_scenario{0};
bool has_id{false}; // if the component has id
bool ids_all_na{false}; // if all ids are all NA
bool ids_part_na{false}; // if some ids are NA but some are not
bool uniform{false}; // if the component is uniform
bool is_columnar{false}; // if the component is columnar
figueroa1395 marked this conversation as resolved.
Show resolved Hide resolved
bool ids_match{false}; // if the ids match, i.e., independent update
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
Idx elements_ps_in_update{0}; // count of elements for this component per scenario in update
Idx elements_ps_in_base{0}; // count of elements for this component per scenario in base
};
using UpdateCompIndependence = std::vector<UpdateCompProperties>;
using ComponentCountInBase = std::pair<std::string, Idx>;

static constexpr Idx ignore_output{-1};

@@ -194,7 +196,7 @@ class MainModelImpl<ExtraRetrievableTypes<ExtraRetrievableType...>, ComponentLis

// all component count
std::map<std::string, Idx, std::less<>> all_component_count() const {
auto const get_comp_count = [this]<typename CT>() -> std::pair<std::string, Idx> {
auto const get_comp_count = [this]<typename CT>() -> ComponentCountInBase {
return make_pair(std::string{CT::name}, this->component_count<CT>());
};
auto const all_count = run_functor_with_all_types_return_array(get_comp_count);
@@ -419,7 +421,19 @@ class MainModelImpl<ExtraRetrievableTypes<ExtraRetrievableType...>, ComponentLis
// (only applicable for independent update dataset)
SequenceIdx get_sequence_idx_map(ConstDataset const& update_data) const {
auto update_components_independence = check_components_independence(update_data);
assert(std::ranges::all_of(update_components_independence, [](auto const& comp) { return !comp.has_id || comp.ids_match; }));
assert(std::ranges::all_of(update_components_independence,
[](auto const& comp) { return !comp.has_id || comp.ids_match; }));

auto const all_comp_count_in_base = this->all_component_count();
for (auto& comp : update_components_independence) {
auto it = std::find_if(all_comp_count_in_base.begin(), all_comp_count_in_base.end(),
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
[&comp](const ComponentCountInBase& pair) { return pair.first == comp.name; });
if (it != all_comp_count_in_base.end()) {
comp.elements_ps_in_base = it->second;
}
auto const aaa = comp.name;
}

return get_sequence_idx_map(update_data, 0, update_components_independence);
}

@@ -734,7 +748,6 @@ class MainModelImpl<ExtraRetrievableTypes<ExtraRetrievableType...>, ComponentLis
template <class Component> using UpdateType = typename Component::UpdateType;

static UpdateCompIndependence check_components_independence(ConstDataset const& update_data) {
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
// update_data.get_component_info("node").elements_per_scenario;
auto check_ids_na = [](auto const& all_spans) {
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
std::vector<std::vector<bool>> ids_na{};
for (const auto& span : all_spans) {
@@ -754,18 +767,22 @@ class MainModelImpl<ExtraRetrievableTypes<ExtraRetrievableType...>, ComponentLis
// length
std::vector<std::vector<bool>> const ids_na = check_ids_na(all_spans);
result.has_id = !std::ranges::all_of(ids_na, [](const std::vector<bool>& vec) { return vec.empty(); });
result.ids_na = std::ranges::all_of(ids_na, [](const std::vector<bool>& vec) {
result.ids_all_na = std::ranges::all_of(ids_na, [](const std::vector<bool>& vec) {
return std::ranges::all_of(vec, [](bool const& obj) { return obj == true; });
});
result.ids_part_na = std::ranges::any_of(ids_na, [](const std::vector<bool>& vec) {
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
return std::ranges::any_of(vec, [](bool const& obj) { return obj == true; }) &&
std::ranges::any_of(vec, [](bool const& obj) { return obj == false; });
Jerry-Jinfeng-Guo marked this conversation as resolved.
Show resolved Hide resolved
});
auto const elements_per_scenario = static_cast<Idx>(all_spans.front().size());
result.elements_per_scenario = elements_per_scenario;
result.elements_ps_in_update = elements_per_scenario;
result.uniform = std::ranges::all_of(all_spans, [elements_per_scenario](auto const& span) {
return static_cast<Idx>(span.size()) == elements_per_scenario;
});
if (!result.uniform) {
return;
}
if (result.elements_per_scenario == 0) {
if (result.elements_ps_in_update == 0) {
return;
}
// Remember the begin iterator of the first scenario, then loop over the remaining scenarios and check the
@@ -783,11 +800,17 @@ class MainModelImpl<ExtraRetrievableTypes<ExtraRetrievableType...>, ComponentLis

auto const check_each_component = [&update_data, &process_buffer_span]<typename CT>() -> UpdateCompProperties {
// get span of all the update data
auto const comp_index = update_data.find_component(CT::name, false);
auto const n_comp = update_data.n_components();
assert(n_comp == n_comp);
UpdateCompProperties result;
result.name = CT::name;
result.is_columnar = update_data.is_columnar(result.name);
if (comp_index >= 0) {
result.elements_ps_in_update = update_data.get_component_info(comp_index).elements_per_scenario;
} else {
result.elements_ps_in_update = 0;
}
if (result.is_columnar) {
process_buffer_span.template operator()<CT>(
update_data.get_columnar_buffer_span_all_scenarios<meta_data::update_getter_s, CT>(), result);
@@ -807,8 +830,9 @@ class MainModelImpl<ExtraRetrievableTypes<ExtraRetrievableType...>, ComponentLis
if (update_data.batch_size() <= 1) {
return true;
}
auto const is_component_update_independent = check_components_independence(update_data);
return std::ranges::all_of(is_component_update_independent, [](auto const& comp) { return comp.ids_match; });
auto const all_comp_update_independence = check_components_independence(update_data);
return std::ranges::all_of(all_comp_update_independence,
[](auto const& comp) { return !comp.has_id || comp.ids_match; });
}

template <calculation_type_tag calculation_type, symmetry_tag sym> auto calculate(Options const& options) {