165: Include dataset for ElasticSearch - #177
Open
thanglequoc wants to merge 32 commits into
Open
Conversation
- Spec: docs/superpowers/specs/2026-07-18-elasticsearch-dataset-export-design.md - Plan: docs/superpowers/plans/2026-07-18-elasticsearch-dataset-export.md - Branch: 165_IncludeDataForElasticSearch Design covers: - Two ES indices: provinces (no GIS) and provinces-gis (with GIS) - Denormalized province documents with embedded nested wards - SearchKeywords array (code, codeName, tone-stripped name, nameEn) - Embedded AdministrativeUnit object - Bulk API-compatible NDJSON output - Static mapping files for both indices - Meta field (no underscore prefix - ES reserves _-prefixed fields)
- Add Elasticsearch DTOs (ElasticsearchProvinceDocument, ElasticsearchWardDocument,
ElasticsearchAdministrativeUnit, ElasticsearchGIS, ElasticsearchMeta)
- Add GenerateSearchKeywords() with Viet tone mark removal and deduplication
- Add ConvertToElasticsearchProvinceModel() mapper with embedded wards and AdminUnit
- Create ElasticsearchDatasetFileWriter with WriteToFile() for non-GIS index
and WriteElasticsearchGISDataToFile() for provinces-gis GIS index
- Generate Bulk API-compatible NDJSON (index action + document lines)
- Generate static index mappings (provinces.json, provinces-gis.json)
- Generate dataset README with curl import examples and query samples
- Add unit tests for writer, NDJSON format, parseBBox, and SearchKeywords
- Wire ES writer into ReadAndGenerateSQLDatasets() and GenerateGISSQLDatasets()
- Create top-level elasticsearch/README.md with user documentation
Output: elasticsearch/{provinces.ndjson, provinces-gis.ndjson,
- Add Elasticsearch DTOs (ElasticsearchProvinceDocument, Elasxisting)
- Include post geometry correction with Postgis ST_CollectionExtract(ST_MakeValid(ST_GeomFromText(geom_wkt, 4326)), 3) - Update test - Update main script, add post geometry fix to the main execution path - Data change verified with pre-post GIS data comparison, report provided in development/corrupted_gis_ward_data/gis_comparison_report_2026-07-24.md
…ql/, sqlserver/, oracle/) matching root layout
- Update GIS data with automated valid Geometry patch with Postgis
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ST_IsValid), so we need to correct the GIS data