Freshdesk: #220943 & #222192 SA Museum data load: 18/12/2024

[timhicks-ala]

From https://support.ehelp.edu.au/a/tickets/220943:

"There are two files, one for our tissues data and one for everything else. They are both intended as complete uploads to replace the appropriate existing data."

sam ala 2024-12-18.zip

Additional record removal from: https://support.ehelp.edu.au/a/tickets/222192

This record needs to be removed:

https://biocache.ala.org.au/occurrences/7a20dc1a-30ce-457c-b03c-18db2b56f8c7

[timhicks-ala]

Note that they have requested R71850 be removed from the dataset: https://support.ehelp.edu.au/a/tickets/222192

[rosemaryjoconnor]

07/01/2024

Code: https://github.com/AtlasOfLivingAustralia/databox/blob/master/data-resources/dr346-SAMA/sourceCode/dr346-SAMA-update.py

Data has additional backslashes and double-quote characters in the Locality field that cause issues with loading the csv files to dataframe. Also some spurious newlines within cells.
Code removes all of the above before processing.

• Removed record with catalogNumber: R71850
• Clean and merge datasets.
• Test in databox
• Load production
• Index check

Record counts

• Current Prod: 565,382
• New: 570,362