Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OECD Refresh and Automation #1125

Open
wants to merge 24 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
fa8afef
Created test and download scripts for autoscript
HarishC727 Nov 25, 2024
7475700
fixed lint
HarishC727 Nov 25, 2024
1fd7e28
added files for autorefresh
HarishC727 Dec 5, 2024
880e9ed
added try catch blocks
HarishC727 Dec 5, 2024
c4e6a86
Merge branch 'master' into oecd_autoref
HarishC727 Dec 5, 2024
ebdaca2
fixed try catch
HarishC727 Dec 5, 2024
41fd60b
resolved internal comments
HarishC727 Dec 9, 2024
4db62a1
Merge branch 'datacommonsorg:master' into oecd_autoref
HarishC727 Dec 9, 2024
0988a19
Merge branch 'master' into oecd_autoref
HarishC727 Dec 10, 2024
8bb9b0d
fixed process script for population
HarishC727 Dec 9, 2024
e0d15b7
Merge branch 'master' into oecd_autoref
HarishC727 Dec 10, 2024
af62434
Merge branch 'master' into oecd_autoref
HarishC727 Dec 17, 2024
c7bce4d
Merge branch 'master' into oecd_autoref
HarishC727 Dec 18, 2024
74a5034
Merge branch 'master' into oecd_autoref
HarishC727 Dec 26, 2024
906db77
Merge branch 'datacommonsorg:master' into oecd_autoref
HarishC727 Jan 7, 2025
c859a41
corrected places and fixed scripts
HarishC727 Jan 7, 2025
ce2dd32
fixed lint and test
HarishC727 Jan 7, 2025
bfa9cbf
reverted to PR comments
HarishC727 Jan 8, 2025
577e3c4
reverted to PR comments
HarishC727 Jan 8, 2025
392b0dc
Merge branch 'master' into oecd_autoref
HarishC727 Jan 9, 2025
ab4fe9a
Merge branch 'master' into oecd_autoref
HarishC727 Jan 10, 2025
316978e
Merge branch 'master' into oecd_autoref
HarishC727 Jan 10, 2025
0cd47d1
modified manifest.json
HarishC727 Jan 10, 2025
1e26470
fixed place bugs
HarishC727 Jan 19, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion scripts/oecd/regional_demography/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,16 +72,32 @@ In each subdirectory, the StatVarObservation template MCF files are saved as `OE

### Script

The script file `preprocess_csv.py` in each subdirectory is used to generate cleaned CSV and template MCF files.
The script file `preprocess_csv.py` in each subdirectory is used to download, process and generate cleaned CSV and template MCF files.

## Generating Artifacts

To generate the cleaned csv and template MCF files, run

- To download the input file of the particular file run-
```bash
python3 preprocess_csv.py --mode=download
```

- To process the input file of the particular file run-
```bash
python3 preprocess_csv.py --mode=process
```

- To download and process the input file of the particular file run-
```bash
python3 preprocess_csv.py
```

- To run the test scripts
```bash
python3 -m unittest process_csv_test.py
```

For population, following the Raw Data section, first copy its raw data to population direction.
Then run the same preprocess_csv command, it will generate artifacts as well as cleaned CSV.

Expand Down
Loading
Loading