-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Combined table functions #104
Conversation
- the function `database_create_combined_table` takes all of a traits.build database's relational tables and merges the information into a single table. (At the moment, not including database metadata). - the functions in `unpack_combined table` have two purposes: 1. Unpack info in location properties or context properties into individual columns 2. Re-create the relational table structure There are bits not yet done: - unpacking contributors - same process as for locations, but I haven't done it - for contexts, working out how to loop across the 5 types of context - actually looping these functions through all datasets and producing final outputs But for locations and contexts, it is possible to go back and forth between combined and relational structures.
Functions to unpack the 5 context columns and the location properties column from the combined table are now working. And there are tests for each of them, including confirming that the original relational table can be recreated. Still to do: loop functions to work across many datasets; write similar function for unpacking contributor information. Also, some minor changes to database_create_combined_table to better standardise syntax
2 problems fixed: - ensure latitude & longitude columns are added for datasets that don't have them; otherwise ended up with combined table with too few columns - solution for recreated location data where lat/lon are the only location properties
Continued work on unpacking functions - all the individual tibbles with data and metadata now can be cyclically turned into "combined" format and back to "relational". The test function runs on a diversity of datasets that are missing different pieces of metadata, to ensure empty rows are added as necessary. Last thing to do - loop over many datasets and recombine them.
Within the file `database_unpack_combined_table` the two functions that unpack the locations now cycle through and recombine datasets. The same solution can be applied to all other functions in the file and I'll do that after this is reviewed.
partial edits to make `join_` functions usable within `database_create_combined_table`.
…oin_location_properties working Have update 4 above functions and integrated them into `database_create_combined_table`. For `join_location_properties` still need to add option to create `json` compaction - have marked location with `TODO`
Rework remaining join functions so they all work to join individual tables to traits.build$traits or together to recreate `database_create_combined_table`. To do: - if we want to add argument to `database_create_combined_table` that allows people to select vars for all tables. (I'm still not convinced these people shouldn't go through linking all the individual functions) - for contributors, contexts, locations, add json packing option - I haven't touched unpacking except to change 1 delimitor, and these should still work with the `single_column_pretty` option (what `database_create_combined_table` defaults to) - I'll update tests once we decide this is good - These changes actually mean that the old `join_contexts` code won't be used, because the version used in `database_create_combined_table` is more flexible to adopt to the different formatting options
Realise I need to resurrect the old `join_contexts`, at least temporarily, for `as_wide_table`, because it wants a single contexts column, not 5
…ombined_table-functions
Add json encoding to contexts, locations functions
adding tests for new joining options - something wrong with JSON compacting - see line 59
Lots of tests for join_ functions and dataset_create_combined_table and everything is working Will look just a little at unpacking functions - really to have another confirmation that everything works. Not sure which we should export.
Add new lookup_ functions for context_properties, location_properties that output vectors of property values that can be used within the join_ functions
…n_pretty" formatting * have now properly separated tests between tests for join_ functions and `database_create_combined_table` and `unpacking...` functions * haven't exported any unpacking functions at this point, but suggest that functions that unpack back to "many columns" be exported so users can do that later, after other manipulations * tests all passing
Moving unpacking functions and tests to a separate branch so we can merge `database_create_combined_table` and the updated join_ functions into develop.
@@ -37,7 +37,7 @@ as_wide_table <- function(austraits){ | |||
|
|||
# The contexts table needs the contexts collapsed to one context name per site | |||
austraits %>% | |||
join_contexts(collapse_context = TRUE) -> austraits | |||
join_contexts_old(collapse = TRUE) -> austraits |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this still available? Should we remove it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The as_wide_table
function requires the old version of join_contexts
. It isn't too hard to weave the new join_context_properties
in, but then the output is different and to maintain the austraits.build API I don't think we want to mess with that right now.
Update previous join_ functions and create new
database_create_combined_table
function to replacejoin_all
.This includes:
join_locations
into two functions:join_location_coordinates
to explicitly join latitude/longitude andjoin_location_properties
to join location metadatajoin_contexts
,join_locations
andjoin_contributors
offer different output formats: many_columns, single_column_pretty, single_column_jsonvars = "all"
, which will add all columns/location properties/context properties.Join_contexts
is reworked, using the variant that was developed on traits.build fordatabase_create_combined_table
. The oldjoin_contexts
is still required foras_wide_table
(the combined table currently output via austraits.build API) and has been moved to that file.as_wide_table
maintained for now to support austraits.build API, but will be removed in the coming months.