Clean up dataset generation script code

AGENTS.md

@@ -53,48 +53,63 @@ vietnamese-provinces-database/
 │   ├── main.go                        # Entry point
 │   ├── go.mod / go.sum               # Go dependencies (Bun, postgres driver)
 │   ├── .env                          # Database credentials
-│   ├── cmd/
-│   │   └── compare-gis/              # CLI tool for GIS comparison workflows
 │   ├── docker/
 │   │   └── docker-compose.yaml       # Local Postgres/PostGIS container
 │   ├── internal/
-│   │   ├── common/                   # Shared helpers (e.g., Vietnamese text handling)
-│   │   ├── dvhcvn_data_downloader/   # Direct DVHCVN administrative data ingestion
-│   │   ├── sapnhap_bando/           # Geographic data service (formerly API, now file-based)
+│   │   ├── common/                   # Shared helpers
+│   │   │   └── viet/                 # Vietnamese text handling (tone marks, normalization)
+│   │   ├── dvhcvn_data_downloader/   # Direct DVHCVN administrative data ingestion (SOAP parser)
+│   │   ├── sapnhap_bando/           # Geographic data service (HTTP API to sapnhap.bando.com.vn)
 │   │   │   ├── fetcher/             # Loads JSON metadata & GeoJSON files
-│   │   │   ├── service/             # Business logic
-│   │   │   ├── repository/          # Database operations
-│   │   │   ├── model/               # Domain models
-│   │   │   └── dto/                 # Data transfer objects
+│   │   │   ├── service/             # Business logic (GIS fetch, backfill, metadata)
+│   │   │   ├── repository/          # Database operations (sapnhap, gis, geojson_objects)
+│   │   │   ├── model/               # Domain models (sapnhap, geojson)
+│   │   │   ├── dto/                 # Data transfer objects (geojson_file, gis_server, sapnhap_api)
+│   │   │   └── util/                # Name normalization utilities
 │   │   ├── dumper/                  # Reads admin data, persists to DB
+│   │   │   ├── config/              # Constant mappings
+│   │   │   ├── helper/              # Dumper helper functions
+│   │   │   ├── model/               # Dumper domain models
+│   │   │   ├── repository/          # Seed data repository
+│   │   │   └── service/             # DVHCVN SOAP seed dumper, corrector, manual seed dumper
 │   │   ├── dataset_writer/          # Generates SQL/JSON/NoSQL output
-│   │   ├── vn_provinces_tmp/        # Core VN provinces data layer
+│   │   │   └── dataset_file_writer/ # Per-format file writers (postgres/mysql, mssql, oracle, json, mongodb, redis, geojson)
+│   │   │       ├── dto/             # Output DTOs (geojson, json, mongo)
+│   │   │       └── helper/          # DTO mappers
+│   │   ├── vn_provinces_tmp/        # Core VN provinces data layer (provinces_tmp, wards_tmp tables)
+│   │   │   ├── model/               # Bun ORM models (Province, Ward, AdministrativeUnit, AdministrativeRegion)
+│   │   │   └── repository/          # Repository queries
 │   │   ├── gis/                     # GIS models and shared GIS logic
-│   │   ├── gis_comparator/          # GIS data validation
-│   │   ├── spatial_gis_comparator/  # Advanced spatial analysis
-│   │   ├── geojson_fetcher/         # GeoJSON handling
-│   │   ├── testutil/                # Test fixtures/helpers
-│   │   └── database/                # Postgres connection pool
+│   │   └── database/                # Postgres connection pool + bootstrap/SQL script execution
 │   ├── resources/
-│   │   └── gis/
-│   │       ├── bando_gisserver/     # ← Provinces/wards JSON metadata
-│   │       ├── exported/            # Exported GIS intermediate artifacts
-│   │       ├── geojson_11Mar2026/   # ← GeoJSON geometry (from deprecated API)
-│   │       └── sapnhapbando_geojson/ # Auxiliary GIS GeoJSON resources
-│   ├── resources/manual_seeds/       # Manual fallback seed data
-│   ├── sapnhap-bando-crawler/        # Historical/auxiliary crawler tooling
+│   │   ├── db_table_init.sql         # Core table schema (provinces_tmp, wards_tmp, etc.)
+│   │   ├── db_region_administrative_unit.sql  # Region & administrative unit seed data
+│   │   ├── fresh_cleanup.sql         # DB cleanup script (run before each generation)
+│   │   ├── gis/
+│   │   │   ├── geojson_11Mar2026/    # ← GeoJSON geometry (from deprecated API)
+│   │   │   ├── sapnhapbando_geojson/ # Auxiliary GIS GeoJSON resources (3,355 files)
+│   │   │   ├── sapnhap_bando_tables.sql          # GIS table schema
+│   │   │   ├── sapnhapbando_init_geo_json_objects_tbl.sql  # Geo objects table init
+│   │   │   └── sapnhapbando_geo_objects.sql       # Geo objects seed data
+│   │   ├── manual_seeds/             # Manual fallback seed data (provinces_seed.sql + wards/)
+│   │   └── rules/                    # Vietnamese text convention rules
+│   │       └── vn_tone_mark_convention.md
+│   ├── bando_co_dvch.sql             # Raw SAPNhap administrative unit SQL dump (497KB)
+│   ├── sapnhap-bando-crawler/        # Historical/auxiliary crawler tooling (Python scripts)
 │   ├── memory/
 │   │   ├── MEMORY.md                # Memory index
 │   │   └── feedback_*.md            # User preferences & learnings
-│   ├── output/                       # Generated artifacts
-│   └── patch/                        # Historical decree patches
-├── development/                       # Feature documentation
+│   ├── output/                       # Generated artifacts (gitignored, staging area)
+│   └── tmp/                          # Temporary working directory
+├── development/                       # Feature documentation & planning artifacts
 │   ├── adapt_the_removal_of_sapnhap_api.md  # Context: API → file-based migration
-│   └── [phase-plans].md              # Plans for new features
-├── docs/                              # User-facing GIS and dataset documentation
+│   ├── cleanup_old_reference/         # Completed cleanup plans (e.g., remove_bando_gisserver_references.md)
+│   └── include_geojson_export/        # GeoJSON export feature planning
+├── docs/
+│   └── gis/                          # User-facing GIS documentation (gis_readme.md, gis_readme_vi.md, gis_example_query.md)
 ├── json/, mysql/, postgresql/, oracle/, sqlserver/, mongodb/, redis/
 │   └── Generated dataset exports in various formats
-└── .github/workflows/                # CI/CD pipelines
+└── .github/workflows/                # CI/CD pipelines (test-go.yml runs Go tests with PostGIS)
 ```
 
 ---
@@ -137,10 +152,9 @@ Current generation flow:
 
 Geographic data migration context (March 2026):
 - **Before**: GIS metadata fetched from SAPNhap API (`/pcotinh`, `/ptracuu`)
-- **Now**: GIS metadata and geometry load from local files:
-  - `./resources/gis/bando_gisserver/provinces.json`
-  - `./resources/gis/bando_gisserver/wards.json`
+- **After (March 2026)**: GIS metadata and geometry load from local files:
   - `./resources/gis/geojson_11Mar2026/*.geojson`
+- **After (July 2026)**: `bando_gisserver/` local JSON files removed (dead code cleanup). Live GIS uses HTTP API (`pread_json`, `p.co_dvhc_id`) + An Giang manual patch.
 - **Key IDs**: `mahc` (province)→`sapnhap_province_matinh` (GIS), `maxa` (ward)→`sapnhap_ward_maxa` (GIS)
 - **Status**: All 3,355 records verified with 100% GIS ID match rate
 
@@ -154,7 +168,6 @@ Geographic data migration context (March 2026):
 |------|----------|-------|
 | Check province/ward counts | Query | `docker exec psql` or `/db-query` |
 | Verify data integrity (duplicates, orphans) | Query | SQL verification scripts |
-| Compare old vs new GIS data | Query + analysis | `gis_comparator`, `spatial_gis_comparator` |
 | Generate new SQL dumps | Execute script | `go run main.go` |
 | Migrate new government decree | Query → Plan → Execute | Read decree, find affected records, generate patch |
 
@@ -199,10 +212,27 @@ Typical contents in `dataset-generation-scripts/output/` include:
 ## Testing & Validation
 
 - Unit tests in each `internal/*/` package (run with `go test -v ./...`)
-- Data validation via `gis_comparator` for geometry matching
 - Historical patch verification in `patch/` directory
 - GIS server ID matching (100% validation before release)
 
+### CI/CD Pipeline (`.github/workflows/test-go.yml`)
+
+The CI pipeline runs on pull requests to `main`/`master` and on manual dispatch:
+
+1. **Service setup**: Spins up `postgis/postgis:15-3.3` as a service container on port `15432`
+2. **Schema initialization**: Runs SQL scripts in order:
+   - `resources/db_table_init.sql` — core tables
+   - `resources/db_region_administrative_unit.sql` — region/unit seed data
+   - `resources/gis/sapnhap_bando_tables.sql` — GIS tables
+   - `resources/gis/sapnhapbando_init_geo_json_objects_tbl.sql` — geo objects table
+3. **Environment variables** (required for tests):
+   - `POSTGRES_DB_HOST=localhost`, `POSTGRES_DB_PORT=15432`
+   - `POSTGRES_DB_USERNAME=postgres`, `POSTGRES_DB_PSWD=root`
+   - `POSTGRES_TMP_DB_NAME=vn_provinces_tmp`
+4. **Test execution**: `go test -v ./...` from `dataset-generation-scripts/`
+
+> **Note**: `first-workflow.yml` is a trivial hello-world stub, not functional CI.
+
 ---
 
 ## Subsystem Deep Dive
@@ -221,8 +251,7 @@ For detailed code-level context on Go services, database models, and implementat
 ## Memory & Feedback
 
 This project maintains persistent learnings:
-- [Database Skill Preference](dataset-generation-scripts/memory/feedback_database_skill.md) — Use reusable skills for recurring tasks
-- [Automatic Skill Invocation](dataset-generation-scripts/memory/feedback_auto_db_query.md) — Skills should auto-trigger based on context
+- [Automatic Skill Invocation](dataset-generation-scripts/memory/feedback_auto_db_query.md) — Skills should auto-trigger based on context, not require manual `/skill-name` invocation
 
 ---
 
@@ -232,9 +261,12 @@ This project maintains persistent learnings:
 |----------|---------|
 | [README.md](README.md) | User guide — dataset installation & usage |
 | [CLAUDE.md](dataset-generation-scripts/CLAUDE.md) | Code agent guide — detailed subsystem context |
+| [dataset-generation-scripts/README.md](dataset-generation-scripts/README.md) | Maintainer guide — how to run generation scripts |
+| [docs/gis/](docs/gis/) | User-facing GIS documentation (readme, example queries) |
 | [development/](development/) | Feature documentation & planning artifacts |
 | [patch/](patch/) | Historical decree patches & changelog |
 | [resources/gis/](dataset-generation-scripts/resources/gis/) | GeoJSON & metadata source files |
+| [resources/rules/](dataset-generation-scripts/resources/rules/) | Vietnamese text convention rules |
 
 ---
 

dataset-generation-scripts/CLAUDE.md

@@ -70,9 +70,12 @@ The upstream data site `sapnhap.bando.com.vn` deprecated their API endpoints:
 - `/ptracuu` (wards lookup)
 
 **Solution:** Migrated to local JSON and GeoJSON files:
-- `./resources/gis/bando_gisserver/provinces.json` - Province metadata
-- `./resources/gis/bando_gisserver/wards.json` - Ward metadata
-- `./resources/gis/geojson_11Mar2026/` - GeoJSON geometry files
+- `./resources/gis/geojson_11Mar2026/` - GeoJSON geometry files (used for An Giang manual patch)
+
+> **July 2026 Update:** The `bando_gisserver/` local JSON metadata files (provinces.json, wards.json) were removed.
+> The code path that loaded them was dead — never invoked from `main.go`.
+> The live GIS pipeline uses HTTP API calls to `sapnhap.bando.com.vn` directly.
+> See: `development/cleanup_old_reference/remove_bando_gisserver_references.md`
 
 **Implementation:**
 - Replaced API calls with file-based loading
@@ -87,19 +90,47 @@ The upstream data site `sapnhap.bando.com.vn` deprecated their API endpoints:
 ```
 dataset-generation-scripts/
 ├── internal/
-│   ├── sapnhap_bando/
-│   │   ├── fetcher/      # Data fetching from local files
-│   │   ├── service/      # Business logic
-│   │   ├── repository/   # Database operations
-│   │   ├── model/        # Database models
-│   │   └── dto/          # Data transfer objects
-│   ├── vn_provinces_tmp/ # VN provinces data layer
-│   └── database/         # Database connection
+│   ├── common/
+│   │   └── viet/              # Vietnamese text handling (tone marks, normalization)
+│   ├── dvhcvn_data_downloader/ # DVHCVN SOAP data ingestion (parser, model, downloader)
+│   ├── sapnhap_bando/         # Geographic data service
+│   │   ├── fetcher/           # Data fetching from HTTP API & local files
+│   │   ├── service/           # Business logic (GIS fetch, backfill, metadata)
+│   │   ├── repository/        # Database operations (sapnhap, gis, geojson_objects)
+│   │   ├── model/             # Database models (sapnhap, geojson)
+│   │   ├── dto/               # Data transfer objects (geojson_file, gis_server, sapnhap_api)
+│   │   └── util/              # Name normalization utilities
+│   ├── dumper/                # Admin data ingestion to DB
+│   │   ├── config/            # Constant mappings
+│   │   ├── helper/            # Helper functions
+│   │   ├── model/             # Domain models
+│   │   ├── repository/        # Seed data repository
+│   │   └── service/           # DVHCVN SOAP seed dumper, corrector, manual seed dumper
+│   ├── dataset_writer/        # Multi-format dataset generation
+│   │   └── dataset_file_writer/ # Per-format writers (postgres/mysql, mssql, oracle, json, mongodb, redis, geojson)
+│   │       ├── dto/           # Output DTOs
+│   │       └── helper/        # DTO mappers
+│   ├── vn_provinces_tmp/      # Core VN provinces data layer (model + repository)
+│   ├── gis/                   # GIS models
+│   ├── testutil/              # Test fixtures/helpers
+│   └── database/              # Postgres connection + bootstrap/SQL execution
 ├── resources/
-│   └── gis/
-│       ├── bando_gisserver/  # JSON metadata files
-│       └── geojson_11Mar2026/ # GeoJSON geometry files
-└── main.go
+│   ├── db_table_init.sql      # Core table schema
+│   ├── db_region_administrative_unit.sql
+│   ├── fresh_cleanup.sql
+│   ├── gis/
+│   │   ├── geojson_11Mar2026/ # GeoJSON geometry files (from deprecated API)
+│   │   ├── sapnhapbando_geojson/ # Auxiliary GIS GeoJSON resources (3,355 files)
+│   │   ├── sapnhap_bando_tables.sql
+│   │   ├── sapnhapbando_init_geo_json_objects_tbl.sql
+│   │   └── sapnhapbando_geo_objects.sql
+│   ├── manual_seeds/          # Manual fallback seed data
+│   └── rules/                 # Vietnamese text convention rules
+├── docker/
+│   └── docker-compose.yaml    # Postgres/PostGIS container (port 15432)
+├── memory/                    # AI agent memory & feedback
+├── output/                    # Generated artifacts (gitignored)
+└── main.go                    # Entry point
 ```
 
 ## Development Workflow
@@ -127,12 +158,14 @@ dataset-generation-scripts/
 
 | Package | Purpose | Key Files |
 |---------|---------|-----------|
-| `sapnhap_bando/` | Geographic data service | `fetcher/` loads JSON, `service/` orchestrates, `repository/` queries |
-| `vn_provinces_tmp/` | Core VN administrative data layer | Manages `provinces_tmp`, `wards_tmp` tables |
-| `dumper/` | Reads admin data, persists to DB | Integration point with GSO data sources |
-| `dataset_writer/` | Generates SQL/JSON/NoSQL exports | Multi-database output formatting |
-| `gis_comparator/` | Validates GIS data consistency | Ensures GIS IDs match across sources |
-| `database/` | Postgres connection pool | Manages `bun.DB` lifecycle |
+| `common/viet/` | Vietnamese text processing | Tone mark removal, Y-normalization, Unicode handling |
+| `dvhcvn_data_downloader/` | Direct DVHCVN admin data ingestion | SOAP parser (`dvhcvn_parser.go`), data downloader, models |
+| `sapnhap_bando/` | Geographic data service (HTTP API) | `fetcher/` loads data, `service/` orchestrates GIS fetch + backfill, `repository/` queries DB, `util/` normalizes names |
+| `dumper/` | Reads admin data, persists to DB | `service/` has DVHCVN SOAP dumper + corrector, manual seed dumper; `config/` has constant mappings; `repository/` for seed data |
+| `dataset_writer/` | Generates SQL/JSON/NoSQL exports | `dataset_file_writer/` has per-format writers (postgres/mysql, mssql, oracle, json, mongodb, redis, geojson) + `dto/` + `helper/` |
+| `vn_provinces_tmp/` | Core VN administrative data layer | `model/` has Bun ORM models (Province, Ward, AdministrativeUnit, AdministrativeRegion); `repository/` for queries |
+| `gis/` | GIS shared models | Geometry data structures |
+| `database/` | Postgres connection + DB bootstrap | `postgres_connector.go` (.env-based DSN), `vn_province_db_service.go` (schema bootstrap, SQL script execution) |
 | `testutil/` | Test helpers | Database seeding, fixtures |
 
 ---
@@ -153,13 +186,13 @@ dataset-generation-scripts/
 
 | File | Purpose |
 |------|---------|
-| `.env` | Database credentials (`DSN` for Postgres connection) |
-| `docker-compose.yml` | Docker services (if exists; see [../AGENTS.md](../AGENTS.md)) |
-| `go.mod` | Go module definition & dependencies |
-| `resources/gis/bando_gisserver/` | JSON metadata for provinces & wards |
+| `.env` | Database credentials — 5 env vars: `POSTGRES_DB_USERNAME`, `POSTGRES_DB_PSWD`, `POSTGRES_DB_HOST`, `POSTGRES_DB_PORT`, `POSTGRES_TMP_DB_NAME`. The DSN is constructed programmatically in `internal/database/postgres_connector.go` |
+| `.env.example` | Template for `.env` (copy to `.env` and fill in credentials) |
+| `docker/docker-compose.yaml` | Docker Postgres/PostGIS service (port 15432→5432) |
+| `go.mod` | Go module definition & dependencies (Go 1.24.0) |
 | `resources/gis/geojson_11Mar2026/` | GeoJSON geometry files from deprecated API |
-- `main.go` - Entry point for dataset generation
-- `development/` - Development documentation
+| `resources/gis/sapnhapbando_geojson/` | 3,355 auxiliary GIS GeoJSON resource files |
+| `main.go` | Entry point for dataset generation |
 
 ## GIS Server ID Matching