Current efforts to measure the progress of the United Nations Sustainable Development Goals (SDGs) are hampered by a dearth of data. For example, poverty — one of seventeen SDGs — is not measured often because it is both spatially sparse and infrequently collected in Sub-Saharan Africa due to the high cost of surveys. Same for Syria and Iraq, due to the war. One possible way to measure these indicators is making use of the open texts available in Web. In this project, we are trying to use the Wikipedia articles attached to specific locations to predict the povert level. We demonstrate that modern NLP techniques can be used to predict community-level asset Wikipedia articles.
The 2030 Agenda for Sustainable Development, adopted by all United Nations Member States in 2015, provides a shared blueprint for peace and prosperity for people and the planet. At its heart are the 17 Sustainable Development Goals (SDGs), which are an urgent call by all countries - developed and developing - to work together toward ending poverty, improving health care and education, reducing inequality and protecting our oceans, forests and other natural resources (The 17 Goals | Sustainable Development. United Nations).
Wikipedia is a large scale knowledge-repository where human from all the world collaborate to continuously add and update knowledge in the articles. The English version of Wikipedia (AKA The English Wikipedia) contains currently 6,622,837 articles with total of 1.14 e9 edites, with average of 19.72 edits per page (Statistics. In Wikipedia accessed on 25/02/2023)
The task can be defined as predicting the poverty level of a country location (represented by a latitude and longitude pair) using geolocated Wikipedia articles from a suitable neighborhood. To accomplish that, we retrieve Wikipedia articles linked to each contry using Wikimedia API, then use NLP methods to build a regresson model to predict the continuos poverty precentage. Wikimedia API sensitivity to location is one major challenge, which makes it difficult to retrieve a sufficient number of articles linked to a country given the country geolocation. To solve that, we first get the all the cities in the country, then for each city, the city location is retrieved which is used then to retrieve the Wikipedia articles throught the Wikimedia API.
An additional challenge is posed by the by the type and the length of the articles. Some articles are short (less than 200 words), were generated automatically from structued knwoledge bases or don't really have usefull information related to the SDGs. These articles affected the model performance. To filter out these articles, we built a semantic similarity model to find the similarity between each Wikipedia article and set of base articles for each SDG. Then, filter those articles with similarity score less than a threshoold. After that, we built a regression model from the filrted articles.
The last challenge is the lack of baseline data. To build a regression model you need to have labeled data, which is the poverty level per country in our case. These data is not officially available for all countries in the last couple of years in the Unitied Nations reosouces. To overcome this issue, we relied on other sources to get the povery level data, such as Statistica and Unicef.
The work method can be summarized in the following steps:
- Retrieve the list of countries from a text file
- Retrieve the cities in each country using the CountriesNow API API
- Retrieve the geolocation for each city using the PositionStack API
- Use the locations retrieved in step 3 to get all the linked Wikipedia articles titles using the Wikimedia Geosearch API method
- Retrieve the article parsed content (raw text) using the article title retrieved in step 4 through the Wikimedia action API method
- Retrieve the SDGs base documentns listed in the sdgs_titles.json file to use as baseline to filter out unsimilar articles
- Create BERT-based Sentence Embeddings for the SDGs and the Wikiepdia-Geolocated documents
- Compute the cosine similarity between each SDG document and the Wikiepdia-Geolocated documents
- For each SDG, get those Wikiepdia-Geolocated documents with cosine similarity score greater than a measured threshold
- Feed the filtered documents into a BERT-based regression model
We ran the Wikipedia crawler with the countries specified above to reireive their geo-located Wikipedia articles. Table 1 below shows number of retrieved articles per country.
| Country | Number of Articles |
|---|---|
| Jordan | 392 |
| Lebanon | 819 |
| Turkey | 10272 |
| Syria | 1237 |
| Egypt | 1289 |
| Iraq | 569 |
| Saudi Arabia | 384 |
| Yemen | 788 |
| Cyprus | 817 |
| Qatar | 247 |
| Oman | 166 |
| Iran | 11889 |
| United Arab Emirates | 478 |
| Kuwait | 167 |
| Bahrain | 193 |
| Morocco | 1023 |
| Tunisia | 576 |
| Libya | 188 |
| Sudan | 185 |
| Mauritania | 85 |
Then we devided the data into training and test sets with (0.7, 0.3) ratio, respectivaly and trained our Wikipedia Embedding (WE) model using a Root Mean Squared Error (Rmse) loss function, 10 epochs, 0.1 dropout, and batch size of 16. The country list contains the following 21 MENA countries: Egypt, Morocco, Tunisia, Libya, Sudan, Mauritania, Jordan, Lebanon, Turkey, Syria, Egypt, Iraq, Saudi Arabia, Yemen, Cyprus, Qatar, Oman, Iran, United Arab Emirates, Kuwait and Bahrain. The model was trained on two V100 GPUs each with 24 GB memory.
We ran the expirement with different similarity lower score thresholds and recordeed the Rmse value for each. figure 1 shows the recorded observed Rmse per epoch for each test.
The lowest Rmse value is obtained at the 8th epoch when similarity threshold equals to 0.2. This doesn't fit well with our initial expectations where Rmse value should be improved when filtering out the less similar articles. However, when having a deep look at the filtered articles, we found that the number of remaining articles is significantly dropped when the lower similarity threshold is greater than 0.3, as shown in figure 2. This means the dataset becomes insufficient to build a robust model.
In this project we verified the use of geolocated Wikipedia articles for socioeconomic applications. We did this by obtaining vector representations of articles to create sentence embeds for SDG-like geolocated Wikipedia articles. We then combined these latent embeds with survey data and evaluated models to predict the poverty level. Using this framework, we found that Wikipedia articles are informative about socioeconomic indicators. We have tested this model against the first SDG indicator, ie end poverty in all its forms everywhere, and we have achieved reasonable results as measured by the Rmse. The geolocated Wikipedia article dataset finds application not only in poverty analysis, but also in more general socio-economic forecasts, such as educational and health-related outcomes. We hope that this approach will accelerate progress towards the UN SDGs by improving the way we estimate missing socio-economic indicators, particularly in developing countries, with the aim of improving responses from regional governments and international aid organizations.
To run the code:
- Update the countries.txt file with the countries list
- Run wikipedia_crawler.ipynb to retrieve the Wikipedia articles geographically linked to the countries. A new Article directory will be created in the same workspace directory along with country_articles.json file that contains the retrieved articles and their paths
- Run sdgs_classifie.ipynb to build the model