Skip to content

74 Document using s3 vs filesystem locally #553

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,58 @@ If you have an inquiry or suggestion for the user and financial institutions man
- [Poetry](https://python-poetry.org/) is used as the package management tool. Once installed, just running `poetry install` in the root of the project should install all the dependencies needed by the app.
- [Docker](https://www.docker.com/) is used for local development where ancillary services will run.

---
### Pre-requisites
[SBL Project Repo](https://github.com/cfpb/sbl-project) contains the `docker-compose.yml` to run the ancillary services. See [LOCAL_DEV_COMPOSE.md](https://github.com/cfpb/sbl-project/blob/main/LOCAL_DEV_COMPOSE.md) for more information.
- Not all services need to run, this module `sbl-filing-api` is part of the docker compose file, which doesn't need to be ran in docker for local development.
- Issuing `docker compose up -d pg keycloak user-fi` would start the necessary services (regtech-user-fi-management, postgres, and keycloak)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A link to the sbl-project LOCAL_DEV_COMPOSE.md would be good here. We also use profiles so the backend profile pretty much does everything

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the link to the repo. Do you normally run the filing-api in docker locally during dev? I wasn't sure if that should be recommended or not since it is not necessary, although it does work as long as you have the project in a sibling folder to the sbl-project repo due to the build path in the compose file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah normally don't run this outside of docker/k8s. Just thought it was a good idea to add the link since that read me has all the details for running pg, keycloak, also.

```bash
$ cd ~/Projects/sbl-project
$ docker compose up -d pg keycloak user-fi
[+] Running 3/3
⠿ Network sbl-project_default Created 0.2s
⠿ Container sbl-project-pg-1 Started 2.6s
⠿ Container sbl-project-keycloak-1 Started 13.4s
⠿ Container sbl-project-user-fi-1 Started 17.3s
```

----
## File Upload

This application allows users to upload small business loan register files. It can be configured to store them in
either a local filesystem, or an AWS S3 bucket.

### Local Configuration

#### File System
By default, this application is configured to upload files into a directory named `upload` in the project root when
running locally within the [.env.local](src/.env.local) file. The properties involved are:
```properties
FS_UPLOAD_CONFIG__PROTOCOL="file"
FS_UPLOAD_CONFIG__ROOT="../upload"
```

#### S3
If you wish to use an Amazon S3 bucket for testing locally:

1. Create an S3 bucket in AWS. For the purposes of this README, assume it is named `sbl-filing-api`.
2. Ensure you have access to an AWS user with IAM permissions to read and write to the bucket.
3. Ensure you have the [AWS CLI](https://aws.amazon.com/cli/) installed.
4. Ensure you have ran `aws configure` to configure the AWS CLI with credentials necessary to access the given bucket.
5. Update your environment with the following environment variables (use the bucket name created above in
`FS_UPLOAD_CONFIG__ROOT` if it is different):
```bash
export FS_UPLOAD_CONFIG__PROTOCOL="s3"
export FS_UPLOAD_CONFIG__ROOT="sbl-filing-api"
```
This can be made permanent by updating e.g. your `$HOME/.zshrc` file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if you were thinking this goes hand in hand with the docker compose up above. If so, in sbl-project there is a filing.local.envthat is used by the container to load in env vars needed by the filing-api

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually here I was more thinking about running on the host, not in a container. Do you think it would be worth discussing all that here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's worth mentioning just because normally we run in docker (though you can run it in poetry by itself... just rarely done). So a link to that env just as an FYI isn't bad

6. Optionally you may update [.env.local](src/.env.local) instead if you prefer.
7. Restart the application if it is already running.

**Important:** This will make API calls and store files in S3, which may result in charges depending on usage.

If you choose to update `.env.local`, please do not commit that change.

----
## Open source licensing info
1. [TERMS](TERMS.md)
Expand Down