Docker: Persistent Data Final Review

chaynHQ · Jan 15, 2025 · 18160cf · 18160cf
1 parent 0b0c4c6
commit 18160cf
Show file tree

Hide file tree

Showing 4 changed files with 73 additions and 130 deletions.
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -8,7 +8,7 @@ services:
       dockerfile: Dockerfile
     env_file: '.env'
     environment:
-      # To connect to local psql db instead, replace DATABASE_URL with
+      # To connect to local your local psql db, replace DATABASE_URL with:
       # postgres://postgres:[email protected]:5432/bloom
       DATABASE_URL: postgres://postgres:postgres@db:5432/bloom
     ports:
@@ -29,7 +29,7 @@ services:
       POSTGRES_PASSWORD: postgres
       POSTGRES_DB: bloom
     volumes:
-      - postgres-data:/var/lib/postgresql/data
+      - postgres-data:/var/lib/postgresql/data # path for named volume
 
 volumes:
-  postgres-data:
+  postgres-data: # named volume for persistent postgres data
diff --git a/docs/database-guide.md b/docs/database-guide.md
@@ -8,65 +8,86 @@
 
 ### Summary
 
-Most open-source contributions (e.g. running Cypress integration tests from the frontend) require adding test data to your local database. To do this, download Bloom's test data dump file, connect to the database server, then populate the database with the backup data.
+Populating your database with test data is essential for a fully functional development environment, making full-stack contributions, and running Cypress integration tests from the frontend. However, it is not necessary for smaller, isolated contributions.
+
+First, download Bloom's test data dump file. Next, connect to the database server, restore your database with the dump file, then verify with a query.
 
 ### Download Test Data Dump File
 
 First, download the test data dump file [linked here from our Google Drive](https://drive.google.com/file/d/1y6KJpAlpozEg3GqhK8JRdmK-s7uzJXtB/view?usp=drive_link). Then place this dump file in the project directory.
 
 ### Connect to Server and Add Data
 
-Next, use the appropriate command based on your setup (Fully Dockerized DB, Docker with Local DB, or Running Manually) to connect to the server and add data.
+Next, connect to the database server and add test data from the dump file, using the appropriate commands based on how you are running the app - fully containerized, containerized app with local database, or manually without Docker.
 
-- Restore the database from the dump file using pg_restore:
+1. Restore the database from the dump file by running these pg_restore commands.
 
-  ```
-  # Fully Dockerized DB:
-  docker exec -i <container_name> pg_restore -U <username> -d <database_name> --clean --if-exists < /path/to/dumpfile.dump
+   **Fully Containerized App Command:**
 
-  # Docker with Local DB or Running Manually:
-  pg_restore -U postgres -d bloom --clean --if-exists /path/to/dumpfile.dump
-  ```
+   ```
+   docker exec -i <container_name> pg_restore -U <username> -d <database_name> --clean --if-exists < /path/to/dumpfile.dump
+   ```
 
-  `container_name`, `username`, and `database_name` are defined in the `docker-compose.yml` under the ‘db’ service config.
+   `container_name`, `username`, and `database_name` are defined in the `docker-compose.yml` under the ‘db’ service. Here is the same command with the default values:
 
-- Start the bloom psql database server:
+   ```
+   docker exec -i bloom-local-db pg_restore -U postgres -d bloom --clean --if-exists < /path/to/dumpfile.dump
+   ```
 
-  ```
-  # Fully Dockerized DB:
-  docker exec -it <container_name> psql -U <username> -d <database_name>
+   **Docker with Local DB or Running Manually Command:**
 
-  # Docker with Local DB or Running Manually:
-  psql -U <username> -h localhost -p 5432 -d <database_name>
-  ```
+   ```
+   pg_restore -U postgres -d bloom --clean --if-exists /path/to/dumpfile.dump
+   ```
 
-- Within the psql server, run queries to verify the restore:
+2. Next, start the bloom psql database server.
 
-  ```
-  SELECT * FROM public."user" users WHERE users."email" = '[email protected]';
-  ```
+   **Fully Containerized App Command:**
+
+   ```
+   docker exec -it <container_name> psql -U <username> -d <database_name>
+
+   # same command with default values added:
+   docker exec -it bloom-local-db psql -U postgres -d bloom
+   ```
+
+   **Docker with Local DB or Running Manually Command:**
 
-  If the user exists, the database has successfully been seeded with test data!
+   ```
+   psql -U <username> -h localhost -p 5432 -d <database_name>
+   ```
+
+3. Verify the restore by running queries in the psql server.
+
+   ```
+   SELECT \* FROM public."user" users WHERE users."email" = '[email protected]';
+   ```
+
+   If the user exists, your database has successfully been populated with test data!
 
 ### Troubleshooting
 
-- When running in Docker, ensure paths to Docker volumes have correct permissions. Windows and WSL users may need to update the `docker-compose.yml` to remove the named volume (volumes: postgres-data:) and replace the volume path with an absolute path (e.g., /mnt/c/... for non-WSL Windows paths or /home/user/... for WSL). WSL users should store volumes on the WSL side to avoid mounting onto Windows, creating new directories if needed. If issues with volumes persist, you can remove persistent data storage by removing the volumes from the 'db' service in the `docker-compose.yml` file. Use `docker-compose up -d --force-recreate` & `docker-compose up --build` to hard reset Docker containers. Delete containers with `docker rm <container_name_or_id>` and their respective volumes & images with `docker system prune`.
+- Persistent storage is configured in the `docker-compose.yml` file using [named volumes](https://docs.docker.com/engine/storage/volumes/). This maintains your data, even if you delete your container. If you have issues with accessing persistent db storage, try replacing the volume path with an absolute path, or update your firewall settings if using WSL (especially if running integration tests). If issues with volumes persist, remove the named volumes from `docker-compose.yml` and populate your database manually as needed.
+- Ensure both the 'db' and 'api' containers are running.
+- Hard reset Docker containers `docker-compose up -d --force-recreate`.
 - If you remove **`--clean`** from the restore command but encounter duplicate object errors, the existing schema may conflict with the restore. In that case, clean the specific objects manually or use **`DROP SCHEMA public CASCADE`** before restoring.
-- Verify that the dump file is valid by running: `pg_restore --list yourfile.dump` If it fails to list contents, the dump file may be corrupted or incomplete.
-- In the psql server, verify the tables and columns exist with `\dt` , `\dt public.*` , and `\d public."user";`
+- Verify that the dump file is valid by running: `pg_restore --list yourfile.dump` If it fails to list contents, the dump file may be corrupted or incomplete. Please notify our team if this happens.
+- Verify the tables and columns exist within the psql server with `\dt` , `\dt public.*` , and `\d public."user";`
 - Run a **`DROP SCHEMA`** or truncate tables before running **`pg_restore`:**
+
   ```
   DROP SCHEMA public CASCADE;
   CREATE SCHEMA public;
   ```
-- Try the following: delete the existing db, create a new db with the same name, and try the restore on this new db. The db drop may throw an error, if so run the following command first.
-
-  `SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'bloom';`
 
+- To hard reset the database in the psql server, first delete the existing db, then create a new db with the same name, and try the restore on this new db. The db drop may throw an error, if so run the following command first:
+  ```
+  SELECT pg_terminate_backend(pid) FROM      pg_stat_activity WHERE datname = 'bloom';`
+  ```
   Then drop the database using:
-
-  `DROP DATABASE bloom;`
-
+  ```
+  DROP DATABASE bloom;
+  ```
 - If the sql dump file is outdated, you can update it by running `docker compose down` then `docker compose up` again as this is configured to run migrations.
 
 ### Chayn Staff - Heroku Directions
@@ -78,11 +99,11 @@ Chayn staff with access to Heroku, you also have the option to seed the database
 3. Replace <HEROKU_APP_NAME> with the correct Heroku app name in the `seed-local-db.sh file`
 4. Run `chmod +x ./seed-local-db.sh` in your terminal to make the file executable
 
-   After the above has been confirmed, run
+After the above has been confirmed, run
 
-   ```bash
-   bash seed-local-db.sh
-   ```
+```bash
+bash seed-local-db.sh
+```
 
 ## Database Migrations
 

diff --git a/docs/local-development.md b/docs/local-development.md
@@ -2,20 +2,18 @@
 
 ## Summary
 
-**The develop branch is our source of truth.** Fork from develop, create new feature branch, then when your PR is merged, develop will automatically merge into the main branch for deployment to production.
-
 To run Bloom's backend:
 
 1. Install prerequisites
 2. Configure environment variables
 3. Install dependencies
 4. Run the app using Docker, Dev Containers, or Manually
-5. Populate the database (only required for full-stack contributions / running integration tests from the frontend)
+5. Populate the database
 
 To test the backend:
 
 - Run unit tests
-- Run e2e integration tests from the frontend for fullstack contributions \*requires populating the database with data first
+- Run e2e integration tests from the frontend for full-stack contributions
 
 ## Prerequisites
 
@@ -39,21 +37,17 @@ yarn
 
 There are 3 methods you can use to run Bloom’s backend locally:
 
-1. **Using Docker (recommended)** - run app + PostgreSQL in a container, with options to containerize both PostgreSQL server and database (PostgreSQL installation not required) OR host the database locally (PostgreSQL required).
+1. **Using Docker (recommended)** - the backend app is fully containerized, installing PostgreSQL is optional.
 2. **Visual Studio Code Dev Container (recommended for Visual Code users)** - installs all dependencies and the PostgreSQL database container automatically.
-3. **Manually (recommended for PostgreSQL users)** - manage PostgesSQL locally.
-
-_Note: you can use an application like Postman to test the apis locally_
+3. **Manually (recommended for PostgreSQL users)** - run the app with yarn and manage PostgreSQL locally.
 
 ### Run with Docker - Recommended
 
-Prequisites: Docker, PostgreSQL optional
-
-Bloom's backend is containerized and can be run solely in Docker - both the PostgreSQL and NestJS app. To run the backend locally, ensure Docker is installed - we recommend installing [Docker Desktop](https://docs.docker.com/desktop/).
+Prequisites: Docker (we recommend [Docker Desktop](https://docs.docker.com/desktop/)), PostgreSQL (optional).
 
-If hosting the PostgreSQL database locally while running the app in Docker: switch the `DATABASE_URL` in `docker-compose.yml` to use `host.docker.internal` for enabling communications between the Docker container and your local PostgreSQL db.
+Bloom's backend is fully containerized - both PostgreSQL and NestJS app. This does not require PostgreSQL to be installed locally. To connect to a local PostgreSQL database instead, modify the `DATABASE_URL` in the `docker-compose.yml` file. This will enable communications between Docker and your local database.
 
-First, make sure the Docker app is running. Then run:
+To start the Docker container run:
 
 ```bash
 docker-compose up
@@ -89,7 +83,7 @@ See [Visual Studio Code Docs: Developing Inside a Dev Container](https://code.vi
 
 Prerequisites: PostgreSQL
 
-Log into PostgreSQL and create a database called "bloom". Ensure it is running on port `5432` (or your desired port). Finally, start the PostgreSQL server on your machine.
+Log into PostgreSQL and create a database called "bloom". Ensure it is running on port `35000` (or your desired port). Finally, start the PostgreSQL server on your machine.
 
 With the psql server running, start the app:
 
@@ -149,16 +143,16 @@ See the [database-guide.md](database-guide.md) for instructions.
 
 # Git Flow and Deployment
 
-**The develop branch is our source of truth, not main.**
+**The develop branch is our source of truth, not main.** Fork from `develop`, create new feature branch, then when your PR is merged, `develop` will automatically merge into the main branch for deployment to production. Keep your branch updated by rebasing and merging feature/bug branches into `develop` as you code.
 
-Create new branches from the `develop` base branch. There is no need to run the build command before pushing changes to GitHub, simply push and create a pull request for the new branch. GitHub Actions will run build and linting tasks automatically. Rebase and merge feature/bug branches into `develop`.
-
-This will trigger an automatic deployment to the staging app by Heroku.
+Once your PR is merged to `develop`, this will trigger an automatic deployment to the staging app by Heroku.
 
 When changes have been tested in staging, merge `develop` into `main`. This will trigger an automatic deployment to the production app by Heroku.
 
-# Swagger
+# APIs
 
 Swagger automatically reflects all of the endpoints in the app, showing their urls and example request and response objects.
 
 To access Swagger simply run the project and visit http://localhost:35001/swagger
+
+For testing APIs, we recommend using tools like Postman.
diff --git a/yarn.lock b/yarn.lock
@@ -551,14 +551,6 @@
     "@firebase/util" "1.10.2"
     tslib "^2.1.0"
 
-"@firebase/[email protected]":
-  version "0.6.11"
-  resolved "https://registry.yarnpkg.com/@firebase/component/-/component-0.6.11.tgz#228a2ff5a6b0e5970b84d4dd298bf6ed0483018e"
-  integrity sha512-eQbeCgPukLgsKD0Kw5wQgsMDX5LeoI1MIrziNDjmc6XDq5ZQnuUymANQgAb2wp1tSF9zDSXyxJmIUXaKgN58Ug==
-  dependencies:
-    "@firebase/util" "1.10.2"
-    tslib "^2.1.0"
-
 "@firebase/[email protected]":
   version "0.1.3"
   resolved "https://registry.yarnpkg.com/@firebase/data-connect/-/data-connect-0.1.3.tgz#a8184a2ece8a78b24240100b91ac4721e622cd4e"
@@ -570,30 +562,6 @@
     "@firebase/util" "1.10.2"
     tslib "^2.1.0"
 
-"@firebase/[email protected]":
-  version "2.0.1"
-  resolved "https://registry.yarnpkg.com/@firebase/database-compat/-/database-compat-2.0.1.tgz#063c4bff74782337117280fbf5b73b463a9a0638"
-  integrity sha512-IsFivOjdE1GrjTeKoBU/ZMenESKDXidFDzZzHBPQ/4P20ptGdrl3oLlWrV/QJqJ9lND4IidE3z4Xr5JyfUW1vg==
-  dependencies:
-    "@firebase/component" "0.6.11"
-    "@firebase/database" "1.0.10"
-    "@firebase/database-types" "1.0.7"
-    "@firebase/logger" "0.4.4"
-    "@firebase/util" "1.10.2"
-    tslib "^2.1.0"
-
-"@firebase/database-compat@^1.0.2":
-  version "1.0.5"
-  resolved "https://registry.yarnpkg.com/@firebase/database-compat/-/database-compat-1.0.5.tgz#18c2314f169942ac315e46b68f86cbe64bafe063"
-  integrity sha512-NDSMaDjQ+TZEMDMmzJwlTL05kh1+0Y84C+kVMaOmNOzRGRM7VHi29I6YUhCetXH+/b1Wh4ZZRyp1CuWkd8s6hg==
-  dependencies:
-    "@firebase/component" "0.6.7"
-    "@firebase/database" "1.0.5"
-    "@firebase/database-types" "1.0.3"
-    "@firebase/logger" "0.4.2"
-    "@firebase/util" "1.9.6"
-    tslib "^2.1.0"
-
 "@firebase/[email protected]", "@firebase/database-compat@^2.0.0":
   version "2.0.1"
   resolved "https://registry.yarnpkg.com/@firebase/database-compat/-/database-compat-2.0.1.tgz#063c4bff74782337117280fbf5b73b463a9a0638"
@@ -627,19 +595,6 @@
     faye-websocket "0.11.4"
     tslib "^2.1.0"
 
-"@firebase/[email protected]":
-  version "1.0.5"
-  resolved "https://registry.yarnpkg.com/@firebase/database/-/database-1.0.5.tgz#09d7162b7dbc4533f17498ac6a76d5e757ab45be"
-  integrity sha512-cAfwBqMQuW6HbhwI3Cb/gDqZg7aR0OmaJ85WUxlnoYW2Tm4eR0hFl5FEijI3/gYPUiUcUPQvTkGV222VkT7KPw==
-  dependencies:
-    "@firebase/app-check-interop-types" "0.3.2"
-    "@firebase/auth-interop-types" "0.2.3"
-    "@firebase/component" "0.6.7"
-    "@firebase/logger" "0.4.2"
-    "@firebase/util" "1.9.6"
-    faye-websocket "0.11.4"
-    tslib "^2.1.0"
-
 "@firebase/[email protected]":
   version "0.3.40"
   resolved "https://registry.yarnpkg.com/@firebase/firestore-compat/-/firestore-compat-0.3.40.tgz#3f4cfc2d25d2f25d9925cdf5903c0b49bfdaeebc"
@@ -740,13 +695,6 @@
     "@firebase/util" "1.10.2"
     tslib "^2.1.0"
 
-"@firebase/[email protected]":
-  version "0.4.4"
-  resolved "https://registry.yarnpkg.com/@firebase/logger/-/logger-0.4.4.tgz#29e8379d20fd1149349a195ee6deee4573a86f48"
-  integrity sha512-mH0PEh1zoXGnaR8gD1DeGeNZtWFKbnz9hDO91dIml3iou1gpOnLqXQ2dJfB71dj6dpmUjcQ6phY3ZZJbjErr9g==
-  dependencies:
-    tslib "^2.1.0"
-
 "@firebase/[email protected]":
   version "0.2.3"
   resolved "https://registry.yarnpkg.com/@firebase/messaging-interop-types/-/messaging-interop-types-0.2.3.tgz#e647c9cd1beecfe6a6e82018a6eec37555e4da3e"
@@ -840,9 +788,6 @@
   version "0.13.4"
   resolved "https://registry.yarnpkg.com/@firebase/storage/-/storage-0.13.4.tgz#83a3b638ffb8dbb8cb4f58d25c0961dfebc7cd9d"
   integrity sha512-b1KaTTRiMupFurIhpGIbReaWev0k5O3ouTHkAPcEssT+FvU3q/1JwzvkX4+ZdB60Fc43Mbp8qQ1gWfT0Z2FP9Q==
-  dependencies:
-    "@firebase/component" "0.6.11"
-    "@firebase/util" "1.10.2"
   dependencies:
     "@firebase/component" "0.6.11"
     "@firebase/util" "1.10.2"
@@ -855,13 +800,6 @@
   dependencies:
     tslib "^2.1.0"
 
-"@firebase/[email protected]":
-  version "1.10.2"
-  resolved "https://registry.yarnpkg.com/@firebase/util/-/util-1.10.2.tgz#4dbb565cfbdf51b4fb2081c5093dba3037d49a35"
-  integrity sha512-qnSHIoE9FK+HYnNhTI8q14evyqbc/vHRivfB4TgCIUOl4tosmKSQlp7ltymOlMP4xVIJTg5wrkfcZ60X4nUf7Q==
-  dependencies:
-    tslib "^2.1.0"
-
 "@firebase/[email protected]":
   version "1.0.2"
   resolved "https://registry.yarnpkg.com/@firebase/vertexai/-/vertexai-1.0.2.tgz#e9104361e88531d9f6f7c9f25ea64287f0a061b9"
@@ -7137,11 +7075,6 @@ [email protected]:
   dependencies:
     "@lukeed/csprng" "^1.0.0"
 
-undici-types@~6.19.8:
-  version "6.19.8"
-  resolved "https://registry.yarnpkg.com/undici-types/-/undici-types-6.19.8.tgz#35111c9d1437ab83a7cdc0abae2f26d88eda0a02"
-  integrity sha512-ve2KP6f/JnbPBFyobGHuerC9g1FYGn/F8n1LWTwNxCEzd6IfqTwUQcNXgEtmmQ6DlRrC1hrSrBnCZPokRrDHjw==
-
 undici-types@~6.20.0:
   version "6.20.0"
   resolved "https://registry.yarnpkg.com/undici-types/-/undici-types-6.20.0.tgz#8171bf22c1f588d1554d55bf204bc624af388433"
@@ -7214,12 +7147,7 @@ [email protected], uuid@^9.0.0, uuid@^9.0.1:
   resolved "https://registry.yarnpkg.com/uuid/-/uuid-9.0.1.tgz#e188d4c8853cc722220392c424cd637f32293f30"
   integrity sha512-b+1eJOlsR9K8HJpow9Ok3fiWOWSIcIzXodvv0rQjVoOVNpWMpxf1wZNpt4y9h10odCNrqnYp1OBzRktckBe3sA==
 
-uuid@^11.0.2:
-  version "11.0.3"
-  resolved "https://registry.yarnpkg.com/uuid/-/uuid-11.0.3.tgz#248451cac9d1a4a4128033e765d137e2b2c49a3d"
-  integrity sha512-d0z310fCWv5dJwnX1Y/MncBAqGMKEzlBb1AOf7z9K8ALnd0utBX/msg/fA0+sbyN1ihbMsLhrBlnl1ak7Wa0rg==
-
-uuid@^11.0.3:
+uuid@^11.0.2, uuid@^11.0.3:
   version "11.0.3"
   resolved "https://registry.yarnpkg.com/uuid/-/uuid-11.0.3.tgz#248451cac9d1a4a4128033e765d137e2b2c49a3d"
   integrity sha512-d0z310fCWv5dJwnX1Y/MncBAqGMKEzlBb1AOf7z9K8ALnd0utBX/msg/fA0+sbyN1ihbMsLhrBlnl1ak7Wa0rg==