From a1285ffe21d798de8661f7421764dd2c30cfe8e3 Mon Sep 17 00:00:00 2001 From: James Green Date: Tue, 5 Dec 2023 15:03:32 +0000 Subject: [PATCH] added the ability to run docker --- .gitignore | 2 +- README.md | 72 +++++++------------------------- {test_data => data_src}/.gitkeep | 0 Dockerfile => docker/Dockerfile | 13 ++++-- docker/entrypoint.sh | 7 ++++ makefile | 47 +++++++++++++++++++-- src/main.py | 2 +- 7 files changed, 76 insertions(+), 67 deletions(-) rename {test_data => data_src}/.gitkeep (100%) rename Dockerfile => docker/Dockerfile (50%) create mode 100644 docker/entrypoint.sh diff --git a/.gitignore b/.gitignore index 3aa3450..460222a 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,6 @@ env/ *.sha256 terraform.tfstate -test_data/**.csv +data_src/**.csv ./idea/* **/__pycache__/ diff --git a/README.md b/README.md index 967ab38..88f8305 100644 --- a/README.md +++ b/README.md @@ -1,64 +1,20 @@ -# Ministry of Justice Template Repository +# Juniper mist integration script -[![repo standards badge](https://img.shields.io/endpoint?labelColor=231f20&color=005ea5&style=for-the-badge&label=MoJ%20Compliant&url=https%3A%2F%2Foperations-engineering-reports.cloud-platform.service.justice.gov.uk%2Fapi%2Fv1%2Fcompliant_public_repositories%2Fendpoint%2Ftemplate-repository&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACgAAAAoCAYAAACM/rhtAAAABmJLR0QA/wD/AP+gvaeTAAAHJElEQVRYhe2YeYyW1RWHnzuMCzCIglBQlhSV2gICKlHiUhVBEAsxGqmVxCUUIV1i61YxadEoal1SWttUaKJNWrQUsRRc6tLGNlCXWGyoUkCJ4uCCSCOiwlTm6R/nfPjyMeDY8lfjSSZz3/fee87vnnPu75z3g8/kM2mfqMPVH6mf35t6G/ZgcJ/836Gdug4FjgO67UFn70+FDmjcw9xZaiegWX29lLLmE3QV4Glg8x7WbFfHlFIebS/ANj2oDgX+CXwA9AMubmPNvuqX1SnqKGAT0BFoVE9UL1RH7nSCUjYAL6rntBdg2Q3AgcAo4HDgXeBAoC+wrZQyWS3AWcDSUsomtSswEtgXaAGWlVI2q32BI0spj9XpPww4EVic88vaC7iq5Hz1BvVf6v3qe+rb6ji1p3pWrmtQG9VD1Jn5br+Knmm70T9MfUh9JaPQZu7uLsR9gEsJb3QF9gOagO7AuUTom1LpCcAkoCcwQj0VmJregzaipA4GphNe7w/MBearB7QLYCmlGdiWSm4CfplTHwBDgPHAFmB+Ah8N9AE6EGkxHLhaHU2kRhXc+cByYCqROs05NQq4oR7Lnm5xE9AL+GYC2gZ0Jmjk8VLKO+pE4HvAyYRnOwOH5N7NhMd/WKf3beApYBWwAdgHuCLn+tatbRtgJv1awhtd838LEeq30/A7wN+AwcBt+bwpD9AdOAkYVkpZXtVdSnlc7QI8BlwOXFmZ3oXkdxfidwmPrQXeA+4GuuT08QSdALxC3OYNhBe/TtzON4EziZBXD36o+q082BxgQuqvyYL6wtBY2TyEyJ2DgAXAzcC1+Xxw3RlGqiuJ6vE6QS9VGZ/7H02DDwAvELTyMDAxbfQBvggMAAYR9LR9J2cluH7AmnzuBowFFhLJ/wi7yiJgGXBLPq8A7idy9kPgvAQPcC9wERHSVcDtCfYj4E7gr8BRqWMjcXmeB+4tpbyG2kG9Sl2tPqF2Uick8B+7szyfvDhR3Z7vvq/2yqpynnqNeoY6v7LvevUU9QN1fZ3OTeppWZmeyzRoVu+rhbaHOledmoQ7LRd3SzBVeUo9Wf1DPs9X90/jX8m/e9Rn1Mnqi7nuXXW5+rK6oU7n64mjszovxyvVh9WeDcTVnl5KmQNcCMwvpbQA1xE8VZXhwDXAz4FWIkfnAlcBAwl6+SjD2wTcmPtagZnAEuA3dTp7qyNKKe8DW9UeBCeuBsbsWKVOUPvn+MRKCLeq16lXqLPVFvXb6r25dlaGdUx6cITaJ8fnpo5WI4Wuzcjcqn5Y8eI/1F+n3XvUA1N3v4ZamIEtpZRX1Y6Z/DUK2g84GrgHuDqTehpBCYend94jbnJ34DDgNGArQT9bict3Y3p1ZCnlSoLQb0sbgwjCXpY2blc7llLW1UAMI3o5CD4bmuOlwHaC6xakgZ4Z+ibgSxnOgcAI4uavI27jEII7909dL5VSrimlPKgeQ6TJCZVQjwaOLaW8BfyWbPEa1SaiTH1VfSENd85NDxHt1plA71LKRvX4BDaAKFlTgLeALtliDUqPrSV6SQCBlypgFlbmIIrCDcAl6nPAawmYhlLKFuB6IrkXAadUNj6TXlhDcCNEB/Jn4FcE0f4UWEl0NyWNvZxGTs89z6ZnatIIrCdqcCtRJmcCPwCeSN3N1Iu6T4VaFhm9n+riypouBnepLsk9p6p35fzwvDSX5eVQvaDOzjnqzTl+1KC53+XzLINHd65O6lD1DnWbepPBhQ3q2jQyW+2oDkkAtdt5udpb7W+Q/OFGA7ol1zxu1tc8zNHqXercfDfQIOZm9fR815Cpt5PnVqsr1F51wI9QnzU63xZ1o/rdPPmt6enV6sXqHPVqdXOCe1rtrg5W7zNI+m712Ir+cer4POiqfHeJSVe1Raemwnm7xD3mD1E/Z3wIjcsTdlZnqO8bFeNB9c30zgVG2euYa69QJ+9G90lG+99bfdIoo5PU4w362xHePxl1slMab6tV72KUxDvzlAMT8G0ZohXq39VX1bNzzxij9K1Qb9lhdGe931B/kR6/zCwY9YvuytCsMlj+gbr5SemhqkyuzE8xau4MP865JvWNuj0b1YuqDkgvH2GkURfakly01Cg7Cw0+qyXxkjojq9Lw+vT2AUY+DlF/otYq1Ixc35re2V7R8aTRg2KUv7+ou3x/14PsUBn3NG51S0XpG0Z9PcOPKWSS0SKNUo9Rv2Mmt/G5WpPF6pHGra7Jv410OVsdaz217AbkAPX3ubkm240belCuudT4Rp5p/DyC2lf9mfq1iq5eFe8/lu+K0YrVp0uret4nAkwlB6vzjI/1PxrlrTp/oNHbzTJI92T1qAT+BfW49MhMg6JUp7ehY5a6Tl2jjmVvitF9fxo5Yq8CaAfAkzLMnySt6uz/1k6bPx59CpCNxGfoSKA30IPoH7cQXdArwCOllFX/i53P5P9a/gNkKpsCMFRuFAAAAABJRU5ErkJggg==)](https://operations-engineering-reports.cloud-platform.service.justice.gov.uk/public-report/template-repository) +This repo has been developed by the DevOps Lan&Wi-Fi to automate site creation on juniper mist. -This template repository equips you with the default initial files required for a Ministry of Justice GitHub repository. +## Run script -## Included Files +## Development setup -The repository comes with the following preset files: +### Prerequisites +* Docker +* IDE that integrates with docker (We use IntelliJ in this example) -- LICENSE -- .gitignore -- CODEOWNERS -- dependabot.yml -- GitHub Actions example file -- Ministry of Justice Compliance Badge (Public repositories only) +### Setup +* Run `make build-dev` +* Integrate built docker container with IDE. [here](https://www.jetbrains.com/help/idea/configuring-remote-python-sdks.html#2546d02c) is the example for intelliJ +* mark src directory & test directory within the IDE. [here](https://www.jetbrains.com/help/idea/content-roots.html) -## Setup Instructions - -Once you've created your repository using this template, ensure the following steps: - -### Update README - -Edit this README.md file to document your project accurately. Take the time to create a clear, engaging, and informative README.md file. Include information like what your project does, how to install and run it, how to contribute, and any other pertinent details. - -### Update repository description - -After you've created your repository, GitHub provides a brief description field that appears on the top of your repository's main page. This is a summary that gives visitors quick insight into the project. Using this field to provide a succinct overview of your repository is highly recommended. - -This description and your README.md will be one of the first things people see when they visit your repository. It's a good place to make a strong, concise first impression. Remember, this is often visible in search results on GitHub and search engines, so it's also an opportunity to help people discover your project. - -### Grant Team Permissions - -Assign permissions to the appropriate Ministry of Justice teams. Ensure at least one team is granted Admin permissions. Whenever possible, assign permissions to teams rather than individual users. - -### Read about the GitHub repository standards - -Familiarise yourself with the Ministry of Justice GitHub Repository Standards. These standards ensure consistency, maintainability, and best practices across all our repositories. - -You can find the standards [here](https://operations-engineering.service.justice.gov.uk/documentation/services/repository-standards.html). - -Please read and understand these standards thoroughly and enable them when you feel comfortable. - -### Modify the GitHub Standards Badge - -Once you've ensured that all the [GitHub Repository Standards](https://operations-engineering.service.justice.gov.uk/documentation/services/repository-standards.html) have been applied to your repository, it's time to update the Ministry of Justice (MoJ) Compliance Badge located in the README file. - -The badge demonstrates that your repository is compliant with MoJ's standards. Please follow these [instructions](https://operations-engineering.service.justice.gov.uk/documentation/runbooks/services/add-repo-badge.html) to modify the badge URL to reflect the status of your repository correctly. - -**Please note** the badge will not function correctly if your repository is internal or private. In this case, you may remove the badge from your README. - -### Manage Outside Collaborators - -To add an Outside Collaborator to the repository, follow the guidelines detailed [here](https://github.com/ministryofjustice/github-collaborators). - -### Update CODEOWNERS - -(Optional) Modify the CODEOWNERS file to specify the teams or users authorized to approve pull requests. - -### Configure Dependabot - -Adapt the dependabot.yml file to match your project's [dependency manager](https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#package-ecosystem) and to enable [automated pull requests for package updates](https://docs.github.com/en/code-security/supply-chain-security). - -If your repository is private with no GitHub Advanced Security license, remove the .github/workflows/dependency-review.yml file. +# Notes +* To see options run `make help` +* If you update [requirements.txt](src/requirements.txt), you will need to rebuild the docker container. diff --git a/test_data/.gitkeep b/data_src/.gitkeep similarity index 100% rename from test_data/.gitkeep rename to data_src/.gitkeep diff --git a/Dockerfile b/docker/Dockerfile similarity index 50% rename from Dockerfile rename to docker/Dockerfile index 89387f0..c07a078 100644 --- a/Dockerfile +++ b/docker/Dockerfile @@ -4,14 +4,21 @@ FROM python:3.11.5 WORKDIR /app # Copy the current directory contents into the container at /app -COPY . /app +COPY ../src/requirements.txt /tmp/requirements.txt + +# Copy entrypoint +COPY docker/entrypoint.sh /entrypoint.sh +COPY src/ /app/src + +# Make entrypoint executable +RUN chmod +x /entrypoint.sh # Install any needed packages specified in requirements.txt -RUN pip install --no-cache-dir -r src/requirements.txt +RUN pip install --no-cache-dir -r ../tmp/requirements.txt # Set the src directory ENV PYTHONPATH=/app/src # Run tests when the container launches -CMD ["python", "-m", "unittest", "discover", "-s", "/app", "-t", "/app"] +ENTRYPOINT ["/entrypoint.sh"] diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh new file mode 100644 index 0000000..d4f210b --- /dev/null +++ b/docker/entrypoint.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +if [ "$RUN_UNIT_TESTS" = True ] ; then + python -m unittest discover -s /app -t /app +else + python /app/src/main.py +fi \ No newline at end of file diff --git a/makefile b/makefile index 0c018c7..c64de3a 100644 --- a/makefile +++ b/makefile @@ -1,5 +1,44 @@ -docker-build: - docker build -t juniper-mist-tests . +#!make +.DEFAULT_GOAL := help +SHELL := '/bin/bash' -docker-run: - docker run -v src:/app/src -v test:/app/tests juniper-mist-tests \ No newline at end of file +.PHONY: build +build: ## Build the docker container + docker build -t juniper-mist -f docker/Dockerfile . + +.PHONO: create-dir +make create-dir: ## Creates a directory for end user to put CSV file into + mkdir data_src; + echo "Please put csv file into data_src then run 'make-prod'"; + +.PHONY: run-prod +run-prod: ## Run the python script only mounting the host for csv-file. + docker run -v $(shell pwd)/data_src:/data_src \ + -e MIST_API_TOKEN=$$MIST_API_TOKEN \ + -e ORG_ID=$$ORG_ID \ + juniper-mist + +.PHONY: run-dev +run-dev: ## Run the python script while mounting the host. This enables using the latest local src code without needing to wait for a container build. + docker run -v $(shell pwd)/src:/app/src \ + -v $(shell pwd)/data_src:/data_src \ + -e MIST_API_TOKEN=$$MIST_API_TOKEN \ + -e ORG_ID=$$ORG_ID \ + juniper-mist + +.PHONY: tests +tests: ## Run unit tests for the python app + docker run -v $(shell pwd)/src:/app/src \ + -v $(shell pwd)/test:/app/test \ + -e RUN_UNIT_TESTS=True juniper-mist + +.PHONY: shell +shell: ## Make interactive docker container + docker run -it --entrypoint /bin/bash \ + -v $(shell pwd)/src:/app/src \ + -v $(shell pwd)/test:/app/test \ + -v $(shell pwd)/data_src:/data_src \ + -e RUN_UNIT_TESTS=True juniper-mist + +help: + @grep -h -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' \ No newline at end of file diff --git a/src/main.py b/src/main.py index 072797f..de7b708 100644 --- a/src/main.py +++ b/src/main.py @@ -41,7 +41,7 @@ def add_geocoding_to_json(data): if __name__ == '__main__': - csv_file_path = os.getcwd() + '/../test_data/sites_with_clients.csv' + csv_file_path = os.getcwd() + '/../data_src/sites_with_clients.csv' # Convert CSV to valid JSON json_data_without_geocoding = convert_csv_to_json(csv_file_path)