Update read me

.gitignore

@@ -4,3 +4,4 @@
 
 .vscode
 dataset-generation-scripts/output/*
+dataset-generation-scripts/.env

dataset-generation-scripts/.env.example

@@ -1,5 +1,5 @@
 POSTGRES_DB_USERNAME=db_user_name
 POSTGRES_DB_PSWD=db_user_pswd
 POSTGRES_DB_HOST=localhost
-POSTGRES_DB_PORT=3306
+POSTGRES_DB_PORT=15432
 POSTGRES_TMP_DB_NAME=vn_provinces_tmp

dataset-generation-scripts/README.md

@@ -1,64 +1,128 @@
-> ⚠️ **Heads-up!**  
-This section is only useful for the repository maintainers. Feel free to ignore this section if you are just looking for the Vietnamese Database Dataset.
+> ⚠️ **Heads-up!**
+> This section is for maintainers who need to regenerate the dataset. If you're looking to use the dataset, see the [root README](../README.md).
 
-# Vietnames Provinces Database Dataset Automation Scripts
+# Vietnamese Provinces Database Dataset Automation Scripts
 
-Vietnamese Government will gradually issue new decree to make change to the administrative units. The change is vary: promoting ward to a higher administrative unit, merging ward, etc...  
-Therefore, it's challenging to keep the dataset up-to-date. Before the development of this automation scripts, I have to manually read the decree change and compare it with the new xls sheet (which is kind of dumb and counter-productive way 🙈)  
-Then it finally come to the development of this autogenerated patch script to make this repository maintainers's life easy.
+The Vietnamese Government periodically issues decrees that change administrative units — merging wards, promoting them to higher units, etc. These automation scripts ingest the latest administrative data, enrich it with GIS geometry, and generate import scripts across multiple database formats.
 
-## How does it work
-[![image.png](https://i.postimg.cc/BnY9f29C/image.png)](https://postimg.cc/dhyS8k07)
+## How it works
 
-The scripts are divided into two main phases
+[![architecture diagram](https://i.postimg.cc/BnY9f29C/image.png)](https://postimg.cc/dhyS8k07)
 
-- **Dumper** (`internal/dumper/`): Reads administrative data from the DVHCVN SOAP source (or manual seeds), performs data transformation and inserts the records into a temporary Postgres database
-- **Dataset Writer** (`internal/dataset_writer/`): Reads from the persisted temporary Postgres database and generates import scripts for multiple databases (PostgreSQL/MySQL, SQL Server, Oracle), plus JSON, MongoDB, and Redis exports
+The scripts operate in two phases:
+
+- **Dumper** (`internal/dumper/`): Reads administrative data from the DVHCVN SOAP source, transforms the records, and inserts them into a temporary Postgres database
+- **Dataset Writer** (`internal/dataset_writer/`): Reads from the temporary Postgres database and generates import scripts for multiple databases (PostgreSQL/MySQL, SQL Server, Oracle), plus JSON, MongoDB, and Redis exports
 
 See [CLAUDE.md](CLAUDE.md) for detailed subsystem context.
-## How to run
-### Prerequisite
-You must have these thing installed on your machine
-- Postgresql
-- Go
-### Setting thing up
-#### Postgres temporary database 
-The easiest way is to use the Docker Postgres/PostGIS container:
-
-```shell
+
+## Prerequisites
+
+- **Go 1.24+** (matches `go.mod`)
+- **Docker** (for the temporary Postgres/PostGIS database)
+
+All required data files (including GeoJSON geometry) are committed in the repository — no extra downloads or extraction needed.
+
+## Setup
+
+### 1. Clone the repository
+
+```bash
+git clone git@github.com:ThangLeQuoc/vietnamese-provinces-database.git
+cd vietnamese-provinces-database
+```
+
+### 2. Start the temporary Postgres database
+
+```bash
+cd dataset-generation-scripts
 docker compose -f docker/docker-compose.yaml up -d
 ```
 
-This starts a Postgres/PostGIS container on port `15432` with database `vn_provinces_tmp`.
+This starts a Postgres/PostGIS container named `vn_provinces_postgres_container` on port `15432` with database `vn_provinces_tmp`.
+
+### 3. Configure the `.env` file
+
+```bash
+cp .env.example .env
+```
+
+The default values in `.env.example` match the Docker container — no edits needed for the standard setup. If you customized the Docker port or credentials, update `.env` accordingly.
 
-Database credentials are configured via a `.env` file (see `.env.example` for the template). The connection is built programmatically in `internal/database/postgres_connector.go` from these environment variables:
+## Run
+
+```bash
+go run main.go
+```
+
+Results land in the `output/` directory. By default, the script generates:
+
+- SQL import scripts for PostgreSQL/MySQL, SQL Server, Oracle
+- JSON, MongoDB, and Redis exports
+- GIS SQL scripts and GeoJSON files
+
+**Skipping GIS**: The `INCLUDE_GIS` constant in `main.go` defaults to `true`. Set it to `false` for a faster, admin-only run that skips GIS data fetching and geometry output — no internet connection required.
+
+## Output structure
+
+After a successful run, the `output/` directory contains:
 
 ```
-POSTGRES_DB_USERNAME=postgres
-POSTGRES_DB_PSWD=root
-POSTGRES_DB_HOST=localhost
-POSTGRES_DB_PORT=15432
-POSTGRES_TMP_DB_NAME=vn_provinces_tmp
+output/
+├── postgresql_mysql_generated_ImportData_vn_units_*.sql   # PostgreSQL & MySQL import
+├── mssql_generated_ImportData_vn_units_*.sql               # SQL Server import
+├── oracle_generated_ImportData_vn_units_*.sql              # Oracle import
+├── json/
+│   ├── full_json_generated_data_vn_units_*.json            # Full dataset (provinces + wards + districts)
+│   ├── simplified_json_generated_data_vn_units_*.json      # Simplified names
+│   └── vn_only_simplified_json_generated_data_vn_units_*.json  # Vietnamese-only simplified
+├── mongodb/
+│   ├── administrative_regions_*.json
+│   ├── administrative_units_*.json
+│   └── mongo_data_vn_unit_*.json                           # Full MongoDB import
+├── redis/
+│   └── redis_vn_provinces_dataset_*.redis                   # Redis commands
+└── gis/                                                     # (only if INCLUDE_GIS=true)
+    ├── *_ImportData_gis_*.sql                               # GIS SQL imports per engine
+    ├── *_ImportData_gis_*.sql.zip                           # Compressed versions
+    ├── vn_provinces_wards_geojson_*.zip                     # Combined GeoJSON archive
+    └── geojson/                                             # Per-province GeoJSON
+        ├── README.md
+        ├── 01_ha_noi/
+        │   ├── 01_ha_noi.geojson                            # Province boundary
+        │   └── wards/                                       # Per-ward boundaries
+        │       ├── 00004_ba_dinh.geojson
+        │       └── ...
+        ├── 04_cao_bang/
+        └── ...
 ```
 
-Copy `.env.example` to `.env` and adjust the values for your local setup.
+## Verify success
+
+Check the expected record counts in the temporary database:
 
-#### Git LFS
-⚠️ This project uses [Git LFS](https://git-lfs.github.com/) for large GIS dataset files.  
-Make sure to run:
-```shell
-git lfs install
-git lfs pull
-git lfs fetch --all
+```bash
+docker exec vn_provinces_postgres_container psql -U postgres -d vn_provinces_tmp \
+  -c "SELECT COUNT(*) FROM provinces_tmp; SELECT COUNT(*) FROM wards_tmp;"
 ```
-before running the project.
 
-#### Executing
-At the root of the `dataset-generation-scripts` folder, run
-```shell
-go run main.go
+Expected: 34 provinces, 3,321 wards _(counts may change with new government decrees)_.
+
+## Tests
+
+```bash
+# From dataset-generation-scripts/
+go test -v ./...
 ```
-And check the result in the `output` folder. The final result includes SQL import scripts for multiple database engines (PostgreSQL/MySQL, SQL Server, Oracle), plus JSON, MongoDB, and Redis exports. If `INCLUDE_GIS` is enabled in `main.go`, GIS SQL and GeoJSON files are also generated.
 
-# Run testcase
-> go test -v ./...
+**Important**: Docker must be running — most tests connect to the temporary Postgres database. Packages that require the database include `internal/sapnhap_bando/...` and `internal/dumper/...`. Pure unit tests (e.g., `internal/common/viet/`) will pass without Docker.
+
+## Troubleshooting
+
+| Symptom | Likely cause | Fix |
+|---------|-------------|-----|
+| `connect: connection refused` on port 15432 | Docker not running | `docker compose -f docker/docker-compose.yaml up -d` |
+| Port 15432 already in use | Another Postgres instance on that port | Change the host port in `docker/docker-compose.yaml` and update `POSTGRES_DB_PORT` in `.env` |
+| GIS pipeline errors or missing geometry | GeoJSON files missing from `resources/gis/geojson_11Mar2026/` | Ensure the repository was cloned correctly; the GeoJSON files are committed in Git |
+| `package ... is not in GOROOT` | Go module dependencies not downloaded | Run `go mod download` from `dataset-generation-scripts/` |
+| Tests fail with database connection errors | Database container not started | Verify with `docker ps`, check `.env` values match the Docker config |