Add function to Optionally get `site.info` if not present #3324

Sweetdevil144 · 2024-07-05T18:35:11Z

Description

This PR aims to refactor site.id and improve str_ns. The end goal is to make the the System independent of DB. Currently, I'm refactoring to create a siteID if not present already. I'll add test cases to check this util function too.
Work still in pending

Some comments from @mdietze :

Here's my slightly different interpretation :

Require the user to input a dataframe with lat/lon and optionally siteID; if siteID is not provided, >construct a unique identifier from lat/lon

replace str_ns with provided unique identifier

Provide a helper function that takes in a BETY connection and site IDs and returns lat, lon, and str_ns

Motivation and Context

May Fix a Subtask of #3307

Review Time Estimate

Immediately
Within one week
When possible

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My change requires a change to the documentation.
My name is in the list of CITATION.cff
I have updated the CHANGELOG.md.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

In do_conversion.R Signed-off-by: Abhinav Pandey <[email protected]>

Signed-off-by: Abhinav Pandey <[email protected]>

Sweetdevil144 · 2024-07-05T20:33:59Z

@meetagrawal09 can you cross check if corresponding changes in test.met.process are valid or not? :)

Signed-off-by: Abhinav Pandey <[email protected]>

modules/data.atmosphere/tests/testthat/test.met.process.R

base/db/R/get.new.site.R

base/workflow/R/do_conversions.R

infotroph · 2024-07-08T16:41:32Z

I think you're taking the wrong approach to task 2 here: Yes, "if siteID is not provided, construct a unique identifier from lat/lon", but it needs to be constructed without using the DB. As we discussed in Slack, this could be as simple as something like id_str <- paste0(lat, "_", lon).

base/db/R/get.new.site.R

Co-authored-by: Chris Black <[email protected]>

infotroph

Posting my notes from our live review -- they're a bit cryptic, but hopefully enough to remind you of the key points from the conversation.

Basically I think the entire get.new.site function is trying to handle too many cases, and instead when the db isn't available we should simply paste together lat and lon to use as a siteid.

infotroph · 2024-08-21T16:29:23Z

base/db/R/get.new.site.R

+
+        if (is.null(site$id) | is.na(site$id)) {
+            if ((!is.null(site$lat) && !is.null(site$lon)) |
+                (!is.na(site$lat) && !is.na(site$lon))


Weird little gotcha here -- the !is.null will fire when lat and lon are NA, and the !is.na will return a missing value when lat and lon are null.

You probably want && instead of |

infotroph · 2024-08-21T16:31:29Z

base/db/R/get.new.site.R

+                    lat = site$lat,
+                    lon = site$lon
+                )
+                str_ns <- paste0(new.site$lat, "-", new.site$lon)


Might want to use a string other than "-" to concatenate here, to avoid confusion between separator characters and negative lon values

Replaced current logic with a "_" sign

infotroph · 2024-08-21T16:36:56Z

base/db/R/get.new.site.R

+        }
+    }
+
+    site.info <- list(new.site = new.site, str_ns = str_ns)


Q in live review: why this structure? A: to match what's expected at met.process line 165

Detailed Explanation : Current structure is followed to match data flow of what information is later on utilised in met.process for other function calls

infotroph · 2024-08-21T16:40:10Z

base/db/R/get.new.site.R

+generate_new_siteID <- function() {
+    # Generate a random number. Assuming higher order integers to increase randomness in IDs
+    random_id <- sample(10000:99999999, 1)
+    return(as.numeric(random_id))
+}


We want IDs to be deterministic -- don't need this

Can you suggest a method to perform so? I am unable to determine one right now :) !!

One approach I suggest is as follow : Hashing to generate a unique id using lat and lon of the site. This would ensure greater precision upto 8 decimal points.

I've implemented hasing in #3423 thus:

mutate(id = digest::digest(geometry, algo = "xxhash64"))

where 'geometry' could be replaced by paste0(lat, lon) or, if we wanted to be consistent and convert it to an sf object (I'm not certain I see the point) but could do

data.frame(lat=100, lon = -90) |> sf::st_as_sf(coords = c("lon", "lat"), crs = 4326) |> digest::digest(algo="xxhash64")

base/db/R/get.new.site.R

base/workflow/R/do_conversions.R

infotroph · 2024-08-21T16:48:16Z

modules/data.atmosphere/R/met.process.R

+  site.info <- PEcAn.DB::get.new.site(site, con=con, latlon = latlon)
+
+  # extract new.site and str_ns from site.info
+  new.site <- site.info$new.site


Hypothesis to confirm before acting: No downstream object uses new.site$id, can remove it here

new.site$id is later on passed on to download.raw.met.module for further conversions in convert_input function. I guess for that instance we will have to keep it rather than duplicating the code.

modules/data.land/R/ic_process.R

modules/data.land/R/soil_process.R

Signed-off-by: Abhinav Pandey <[email protected]>

Sweetdevil144

From previous comments on #3348, I couldn't resist removing unnecessary changes. I will post a discussion regarding this PR and gather insights from other 'members & contributors' too.

Signed-off-by: Abhinav Pandey <[email protected]>

mdietze · 2025-01-16T12:29:43Z

base/db/R/get.new.site.R

+##' @title Get New Site Info
+##' @param site a dataframe with site information including id, lat, lon, and time_zone.
+##' @param con Database connection object.
+##' @param latlon Optional global latlon object which will store updated lat/lon.


I'm not following why latlon exists as an option here, as information is never extracted from it in any of the cases below within this function, and none of the calls you implemented to this function pass in anything other than NULL.

mdietze · 2025-01-16T12:31:18Z

modules/data.atmosphere/R/met.process.R

-                         lat = latlon$lat, 
-                         lon = latlon$lon)
-  str_ns <- paste0(new.site$id %/% 1e+09, "-", new.site$id %% 1e+09)
+  latlon <- NULL


Why is latlon defined to be null here? Why not just delete this line and the , latlon = latlon in the function call as you already set the default for latlon to NULL.

mdietze · 2025-01-16T12:32:51Z

modules/data.land/R/ic_process.R

-                         lon = latlon$lon)
+  latlon <- NULL
+  if(is.null(site$lat) | is.null(site$lon)) {
+    site.info <- PEcAn.DB::get.new.site(site, con=con, latlon = latlon)


Same point as earlier about latlon not needing to be defined here.

mdietze · 2025-01-16T12:36:25Z

modules/data.land/R/ic_process.R

+      id = as.numeric(site$id),
+      lat = site$lat,
+      lon = site$lon
+    )


I'm not following why this if/else exists. Doesn't get.new.site already support this else case? And if not, shouldn't it?

mdietze · 2025-01-16T12:38:35Z

modules/data.land/R/soil_process.R

@@ -30,6 +30,7 @@ soil_process <- function(settings, input, dbfiles, overwrite = FALSE,run.local=T
  # set up bety connection
  con <- PEcAn.DB::db.open(dbparms$bety)
  on.exit(PEcAn.DB::db.close(con), add = TRUE)
+
  # get site info
  latlon <- PEcAn.DB::query.site(site$id, con = con)[c("lat", "lon")]


Not following why this function only has whitespace change as it seems like the code chunk here is doing the same thing as what you were trying to support with your new function, and also would benifit from having a database-independent option.

Sweetdevil144 added 8 commits July 3, 2024 20:08

Accept site.data as function input

cb41971

In do_conversion.R Signed-off-by: Abhinav Pandey <[email protected]>

Pass site.data via args

8b82f2b

Signed-off-by: Abhinav Pandey <[email protected]>

Read Command Line arguments

8c998fd

Signed-off-by: Abhinav Pandey <[email protected]>

Update str_ns declaration

7a5a848

Signed-off-by: Abhinav Pandey <[email protected]>

Update site$lon

74365fd

Signed-off-by: Abhinav Pandey <[email protected]>

Update if-else block for site.id

a0ae689

Signed-off-by: Abhinav Pandey <[email protected]>

Add helper function to get new.site

18dd086

Signed-off-by: Abhinav Pandey <[email protected]>

Roxygenise the documentations

7a69113

Signed-off-by: Abhinav Pandey <[email protected]>

github-actions bot added Modules Base labels Jul 5, 2024

Sweetdevil144 added 3 commits July 6, 2024 00:15

Update Function Name

604634b

Signed-off-by: Abhinav Pandey <[email protected]>

Export dependency to DB rather than utils

0f154e5

Signed-off-by: Abhinav Pandey <[email protected]>

Update test case

fae37a8

Signed-off-by: Abhinav Pandey <[email protected]>

github-actions bot added the Tests label Jul 5, 2024

Update 'exampes'

cb42033

Signed-off-by: Abhinav Pandey <[email protected]>

Sweetdevil144 added 10 commits July 6, 2024 16:27

Return null when site$id is not present

5ee450a

Signed-off-by: Abhinav Pandey <[email protected]>

Remove NULL calling

e37b93b

Signed-off-by: Abhinav Pandey <[email protected]>

Fixing 'no global binding err'

144ce9c

Signed-off-by: Abhinav Pandey <[email protected]>

update global variable handling

54ce644

Signed-off-by: Abhinav Pandey <[email protected]>

Remove global binding changes

6d76455

Signed-off-by: Abhinav Pandey <[email protected]>

Update global variables

ca0b50e

Signed-off-by: Abhinav Pandey <[email protected]>

Ipdate all instances of latlon

8082194

Signed-off-by: Abhinav Pandey <[email protected]>

Roxygenise documents

65cdea7

Signed-off-by: Abhinav Pandey <[email protected]>

Usage Updates added

acb8507

Signed-off-by: Abhinav Pandey <[email protected]>

Doc Updates added

59f451a

Signed-off-by: Abhinav Pandey <[email protected]>

meetagrawal09 reviewed Jul 8, 2024

View reviewed changes

modules/data.atmosphere/tests/testthat/test.met.process.R Outdated Show resolved Hide resolved

infotroph reviewed Jul 8, 2024

View reviewed changes

base/db/R/get.new.site.R Show resolved Hide resolved

infotroph reviewed Jul 8, 2024

View reviewed changes

base/workflow/R/do_conversions.R Outdated Show resolved Hide resolved

Sweetdevil144 and others added 5 commits August 6, 2024 00:23

Merge branch 'develop' into siteID-refactor

92b9b06

Merge branch 'develop' into siteID-refactor

e76124d

Merge branch 'develop' into siteID-refactor

dff7ab8

Merge branch 'develop' into siteID-refactor

903efc9

Merge branch 'develop' into siteID-refactor

14950aa

infotroph reviewed Aug 21, 2024

View reviewed changes

base/db/R/get.new.site.R Outdated Show resolved Hide resolved

Update base/db/R/get.new.site.R

d392847

Co-authored-by: Chris Black <[email protected]>

infotroph requested changes Aug 21, 2024

View reviewed changes

remove browndog support from benchmark

0ac5b39

Signed-off-by: Abhinav Pandey <[email protected]>

github-actions bot added the Dockerfile label Aug 28, 2024

Sweetdevil144 added 4 commits August 28, 2024 22:37

Merge branch 'develop' into siteID-refactor

2639e88

Merge branch 'develop' into siteID-refactor

308380e

Refactor changes from review

9edc268

Signed-off-by: Abhinav Pandey <[email protected]>

Merge branch 'develop' into siteID-refactor

a7c4b7d

infotroph mentioned this pull request Sep 3, 2024

Remove Browndog #3348

Merged

13 tasks

Sweetdevil144 added 3 commits September 5, 2024 12:06

Implement temporary hashing to generate UID

59b293e

Signed-off-by: Abhinav Pandey <[email protected]>

Merge branch 'develop' into siteID-refactor

b685c58

Remove unrequired changes which were accidently pushed to this branch

65b976d

Signed-off-by: Abhinav Pandey <[email protected]>

Sweetdevil144 commented Sep 5, 2024

View reviewed changes

Sweetdevil144 and others added 9 commits September 5, 2024 13:50

Refactor Dependency chart

99e4345

Signed-off-by: Abhinav Pandey <[email protected]>

Using openssl for hashing lat/lon

a4025a8

Signed-off-by: Abhinav Pandey <[email protected]>

Merge branch 'develop' into siteID-refactor

f6390eb

Merge branch 'develop' into siteID-refactor

5135e82

Merge branch 'develop' into siteID-refactor

c411a6e

Merge branch 'develop' into siteID-refactor

0009c6e

Merge branch 'develop' into siteID-refactor

9039c23

Merge branch 'PecanProject:develop' into siteID-refactor

71e293c

Merge branch 'develop' into siteID-refactor

3c537f6

mdietze requested changes Jan 16, 2025

View reviewed changes

infotroph mentioned this pull request Feb 1, 2025

only abbreviate siteid if numeric and over a billion #3428

Merged

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add function to Optionally get `site.info` if not present #3324

Add function to Optionally get `site.info` if not present #3324

Sweetdevil144 commented Jul 5, 2024 •

edited

Loading

Sweetdevil144 commented Jul 5, 2024

infotroph commented Jul 8, 2024

infotroph left a comment •

edited

Loading

infotroph Aug 21, 2024

infotroph Aug 21, 2024

infotroph Aug 21, 2024

Sweetdevil144 Sep 1, 2024

infotroph Aug 21, 2024

Sweetdevil144 Sep 1, 2024

infotroph Aug 21, 2024

Sweetdevil144 Aug 31, 2024

Sweetdevil144 Sep 1, 2024

dlebauer Feb 4, 2025

infotroph Aug 21, 2024

Sweetdevil144 Aug 31, 2024

Sweetdevil144 left a comment

mdietze Jan 16, 2025

mdietze Jan 16, 2025

mdietze Jan 16, 2025

mdietze Jan 16, 2025

mdietze Jan 16, 2025

Add function to Optionally get site.info if not present #3324

Are you sure you want to change the base?

Add function to Optionally get site.info if not present #3324

Conversation

Sweetdevil144 commented Jul 5, 2024 • edited Loading

Description

Motivation and Context

Review Time Estimate

Types of changes

Checklist:

Sweetdevil144 commented Jul 5, 2024

infotroph commented Jul 8, 2024

infotroph left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sweetdevil144 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add function to Optionally get `site.info` if not present #3324

Add function to Optionally get `site.info` if not present #3324

Sweetdevil144 commented Jul 5, 2024 •

edited

Loading

infotroph left a comment •

edited

Loading