Combined table functions #104
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- the function `database_create_combined_table` takes all of a traits.build database's relational tables and merges the information into a single table. (At the moment, not including database metadata). - the functions in `unpack_combined table` have two purposes: 1. Unpack info in location properties or context properties into individual columns 2. Re-create the relational table structure There are bits not yet done: - unpacking contributors - same process as for locations, but I haven't done it - for contexts, working out how to loop across the 5 types of context - actually looping these functions through all datasets and producing final outputs But for locations and contexts, it is possible to go back and forth between combined and relational structures.
Functions to unpack the 5 context columns and the location properties column from the combined table are now working. And there are tests for each of them, including confirming that the original relational table can be recreated. Still to do: loop functions to work across many datasets; write similar function for unpacking contributor information. Also, some minor changes to database_create_combined_table to better standardise syntax
2 problems fixed: - ensure latitude & longitude columns are added for datasets that don't have them; otherwise ended up with combined table with too few columns - solution for recreated location data where lat/lon are the only location properties
Continued work on unpacking functions - all the individual tibbles with data and metadata now can be cyclically turned into "combined" format and back to "relational". The test function runs on a diversity of datasets that are missing different pieces of metadata, to ensure empty rows are added as necessary. Last thing to do - loop over many datasets and recombine them.
Within the file `database_unpack_combined_table` the two functions that unpack the locations now cycle through and recombine datasets. The same solution can be applied to all other functions in the file and I'll do that after this is reviewed.
partial edits to make `join_` functions usable within `database_create_combined_table`.
…oin_location_properties working Have update 4 above functions and integrated them into `database_create_combined_table`. For `join_location_properties` still need to add option to create `json` compaction - have marked location with `TODO`
Rework remaining join functions so they all work to join individual tables to traits.build$traits or together to recreate `database_create_combined_table`. To do: - if we want to add argument to `database_create_combined_table` that allows people to select vars for all tables. (I'm still not convinced these people shouldn't go through linking all the individual functions) - for contributors, contexts, locations, add json packing option - I haven't touched unpacking except to change 1 delimitor, and these should still work with the `single_column_pretty` option (what `database_create_combined_table` defaults to) - I'll update tests once we decide this is good - These changes actually mean that the old `join_contexts` code won't be used, because the version used in `database_create_combined_table` is more flexible to adopt to the different formatting options
Realise I need to resurrect the old `join_contexts`, at least temporarily, for `as_wide_table`, because it wants a single contexts column, not 5
…ombined_table-functions
Add json encoding to contexts, locations functions
adding tests for new joining options - something wrong with JSON compacting - see line 59
Lots of tests for join_ functions and dataset_create_combined_table and everything is working Will look just a little at unpacking functions - really to have another confirmation that everything works. Not sure which we should export.
Add new lookup_ functions for context_properties, location_properties that output vectors of property values that can be used within the join_ functions
…n_pretty" formatting * have now properly separated tests between tests for join_ functions and `database_create_combined_table` and `unpacking...` functions * haven't exported any unpacking functions at this point, but suggest that functions that unpack back to "many columns" be exported so users can do that later, after other manipulations * tests all passing
Moving unpacking functions and tests to a separate branch so we can merge `database_create_combined_table` and the updated join_ functions into develop.
dfalster
approved these changes
Sep 26, 2024
| @@ -37,7 +37,7 @@ as_wide_table <- function(austraits){ | |||
|
|
|||
| # The contexts table needs the contexts collapsed to one context name per site | |||
| austraits %>% | |||
| join_contexts(collapse_context = TRUE) -> austraits | |||
| join_contexts_old(collapse = TRUE) -> austraits | |||
There was a problem hiding this comment.
Why is this still available? Should we remove it?
There was a problem hiding this comment.
The as_wide_table function requires the old version of join_contexts. It isn't too hard to weave the new join_context_properties in, but then the output is different and to maintain the austraits.build API I don't think we want to mess with that right now.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update previous join_ functions and create new
database_create_combined_tablefunction to replacejoin_all.This includes:
join_locationsinto two functions:join_location_coordinatesto explicitly join latitude/longitude andjoin_location_propertiesto join location metadatajoin_contexts,join_locationsandjoin_contributorsoffer different output formats: many_columns, single_column_pretty, single_column_jsonvars = "all", which will add all columns/location properties/context properties.Join_contextsis reworked, using the variant that was developed on traits.build fordatabase_create_combined_table. The oldjoin_contextsis still required foras_wide_table(the combined table currently output via austraits.build API) and has been moved to that file.as_wide_tablemaintained for now to support austraits.build API, but will be removed in the coming months.