Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accent #1543

Merged
merged 6 commits into from
May 30, 2024
Merged

Accent #1543

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 51 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,33 +14,35 @@
- 4.1. [Heroku](#Heroku)
- 4.2. [Abraxas](#Abraxas)
- 4.3. [Docker (anywhere)](#Dockeranywhere)
- 5. [E2E tests](#E2Etests)
- 6. [GraphQL performance tests](#GraphQLperformancetests)
- 6.1. [Automation](#Automation)
- 6.2. [How to add or modify the tests](#Howtoaddormodifythetests)
- 7. [Load tests](#Loadtests)
- 7.1. [Automation](#Automation-1)
- 7.2. [Local setup](#Localsetup)
- 7.3. [Running the tests locally](#Runningthetestslocally)
- 7.4. [Recording the tests using Playwright](#RecordingthetestsusingPlaywright)
- 8. [Authentication](#Authentication)
- 8.1. [Locally](#Locally)
- 5. [Developing GitHub Actions](#DevelopingGitHubActions)
- 6. [E2E tests](#E2Etests)
- 7. [GraphQL performance tests](#GraphQLperformancetests)
- 7.1. [Automation](#Automation)
- 7.2. [How to add or modify the tests](#Howtoaddormodifythetests)
- 8. [Load tests](#Loadtests)
- 8.1. [Automation](#Automation-1)
- 8.2. [Local setup](#Localsetup)
- 8.3. [Running the tests locally](#Runningthetestslocally)
- 8.4. [Recording the tests using Playwright](#RecordingthetestsusingPlaywright)
- 9. [Authentication](#Authentication)
- 9.1. [Locally](#Locally)
- 10. [Translations](#Translations)

<!-- vscode-markdown-toc-config
numbering=true
autoSave=true
/vscode-markdown-toc-config -->
<!-- /vscode-markdown-toc -->

## Documentation
## 1. <a name='Documentation'></a>Documentation

A public documentation is available at https://visualize.admin.ch/docs/.

## Development Environment
## 2. <a name='DevelopmentEnvironment'></a>Development Environment

To start the development environment, you need a Docker runtime, e.g. [Docker Desktop](https://www.docker.com/products/docker-desktop) and [Nix](https://nixos.org).

### Setting up the dev environment
### 2.1. <a name='Settingupthedevenvironment'></a>Setting up the dev environment
ptbrowne marked this conversation as resolved.
Show resolved Hide resolved

1. Make sure that Docker is running
2. Start the Postgres database with `docker-compose up`
Expand All @@ -50,7 +52,7 @@ To start the development environment, you need a Docker runtime, e.g. [Docker De
yarn setup:dev
```

### Dev server
### 2.2. <a name='Devserver'></a>Dev server

Once the application's set up, you can start the development server with

Expand All @@ -68,15 +70,15 @@ yarn dev:ssl # If you are working with the login process

To run the application with debugging enabled through VSCode, make sure the dev server is running and the click the "Run and Debug" button in the sidebar (CMD-SHIFT-D). Then select the "Launch Chrome" configuration. This will open a new Chrome window with the dev tools open. You can now set breakpoints in the code and they will be hit.

### Postgres database
### 2.3. <a name='Postgresdatabase'></a>Postgres database

If the database server is not running, run:

```sh
docker-compose up
```

### Building the Embed script `/dist/embed.js`
### 2.4. <a name='BuildingtheEmbedscriptdistembed.js'></a>Building the Embed script `/dist/embed.js`

Currently, the embed script is not automatically built when the dev server starts.

Expand All @@ -88,7 +90,7 @@ yarn dev:rollup

> Currently, this only bundles and initializes [iframe-resizer](https://github.com/davidjbradshaw/iframe-resizer) but could be used to render charts without iframes (using [custom elements](https://developers.google.com/web/fundamentals/web-components/customelements) or render to a generic DOM element) in the future.

#### Database migrations
#### 2.4.1. <a name='Databasemigrations'></a>Database migrations

Database migrations are run automatically when a production build of the app starts. In _development_, you'll have to run them manually:

Expand All @@ -112,7 +114,7 @@ After merging the branch, you can delete the environment variables scoped to the

[visualization-tool-postgres-dev](https://vercel.com/ixt/visualization-tool/stores/postgres/store_dV3rog1asOXO3BfC/data)

## Versioning
## 3. <a name='Versioning'></a>Versioning

New versions of `package.json` are built on GitLab CI into a separate image that will be deployed to the integration env.

Expand All @@ -122,9 +124,9 @@ yarn version

This will prompt for a new version. The `postversion` script will automatically try to push the created version tag to the origin repository.

## Deployment
## 4. <a name='Deployment'></a>Deployment

### Heroku
### 4.1. <a name='Heroku'></a>Heroku

If a Heroku app is set up (as Git remote `heroku`), deploy with

Expand All @@ -136,7 +138,7 @@ Build instructions are defined in `heroku.yml`.

For details, see https://devcenter.heroku.com/articles/build-docker-images-heroku-yml

### Abraxas
### 4.2. <a name='Abraxas'></a>Abraxas

With your Abraxas credentials ...

Expand All @@ -157,7 +159,7 @@ Useful commands to use:
- `sudo /usr/local/bin/docker-compose pull web` -> Pull latest web image manually (should not be needed much)
- etc. (remember to use `sudo` for all Docker commands)

### Docker (anywhere)
### 4.3. <a name='Dockeranywhere'></a>Docker (anywhere)

To pull the latest image from the GitLab registry, run:

Expand Down Expand Up @@ -186,45 +188,45 @@ services:
- "5432:5432"
```

## Developing GitHub Actions
## 5. <a name='DevelopingGitHubActions'></a>Developing GitHub Actions

Several checks are run on different types of events that happen within the repository, including E2E or GraphQL performance tests. In order to be able to build and test the actions locally, we use [act](https://github.com/nektos/act), with example mocked event payloads defined in [this folder](https://github.com/visualize-admin/visualization-tool/tree/main/act).

After [installing](https://nektosact.com/installation/index.html) the library, you can run a given action like e.g. `act deployment_status -W ".github/workflows/performance-tests-pr.yml" -e act/deployment_status.json`, modifying the event payload or adding a new one as needed.

## E2E tests
## 6. <a name='E2Etests'></a>E2E tests

Playwright is run on every successful deployment of a branch. Screenshots are made with Percy and sent directly to their cloud service for diffing.

A special [test page](http://localhost:3000/en/__test) shows all the charts that are screenshotted.
Those charts configurations are kept in the repository.

## GraphQL performance tests
## 7. <a name='GraphQLperformancetests'></a>GraphQL performance tests

The project uses a combination of [k6](https://k6.io) and [Grafana](https://grafana.com) with [Prometheus](https://k6.io/docs/results-output/real-time/prometheus-remote-write/) for GraphQL performance testing.

### Automation
### 7.1. <a name='Automation'></a>Automation

To ensure constant monitoring of the performance of selected GraphQL queries, the performance tests are run [once an hour](https://github.com/visualize-admin/visualization-tool/blob/main/.github/workflows/performance-tests.yml) against each environment of the application. The results are then sent to the governmental Grafana dashboards.

### How to add or modify the tests
### 7.2. <a name='Howtoaddormodifythetests'></a>How to add or modify the tests

To add or modify the performance tests, go to the [k6/performance-tests](https://github.com/visualize-admin/visualization-tool/tree/main/k6/performance-tests) folder. The GitHub Action is generated by running the `yarn run github:codegen` command; be sure to run it after modifying the [generate-github-action.js](`https://github.com/visualize-admin/visualization-tool/blob/main/k6/performance-tests/generate-github-action.js`) file.

## Load tests
## 8. <a name='Loadtests'></a>Load tests

The project uses [k6](https://k6.io) for load testing.

### Automation
### 8.1. <a name='Automation-1'></a>Automation

There is a [dedicated GitHub Action](https://github.com/visualize-admin/visualization-tool/actions/workflows/manual-load-test.yml) that runs the API load tests against selected environment.
You can investigate the results by going to Actions section in GitHub and checking the summary results. They are also visible in the cloud (k6.io), if you enable the cloud option.

### Local setup
### 8.2. <a name='Localsetup'></a>Local setup

In order to run the tests locally, follow the [documentation](https://k6.io/docs/get-started/installation/) to install `k6` on your machine.

### Running the tests locally
### 8.3. <a name='Runningthetestslocally'></a>Running the tests locally

You might want to run the script locally, for example to be able to bypass the cloud limitations of k6 (e.g. max number of VUs bigger than 50). To run a given load test, simply run

Expand All @@ -237,7 +239,7 @@ by directly modifying the `options` object inside a given script and running `ya

For the GraphQL tests, you'll also need to pass `--env ENV=(test|int|prod)` flag to point to the proper environment and `--env ENABLE_GQL_SERVER_SIDE_CACHE=(true | false)` to control whether to use GQL server-side caching.

### Recording the tests using Playwright
### 8.4. <a name='RecordingthetestsusingPlaywright'></a>Recording the tests using Playwright

While some tests are written manually, other tests come from HAR recordings that span a broad set of actions.
In order to record a HAR file and convert it into k6 script, use the `testAndSaveHar` inside `e2e/har-utils.ts` file.
Expand All @@ -251,13 +253,27 @@ After the HAR file has been recorded, use [har-to-k6](https://k6.io/docs/test-au

> ⚠️ You might want to remove requests to Google afterwards manually, to remove false-positives of blocked requests.

## Authentication
## 9. <a name='Authentication'></a>Authentication

Authentication is provided by the Swiss federal government's eIAM through ADFS.
We use Next-auth to integrate our application with it, through a [custom Provider](app/auth-providers/adfs.ts).

### Locally
### 9.1. <a name='Locally'></a>Locally

You can use the ref eIAM. ADFS environment variables should be configured in your
`.env.local` file. You'll find those secret variables in our shared 1Password in
the "Visualize.admin .env.local" entry.

## 10. <a name='Translations'></a>Translations

Translations are managed via [Accent](https://accent.interactivethings.io). Right now, you need to
manually pull and push the translations. The process goes:

1. Edit components, add `<Trans />` and `t()`
2. Run `yarn locales:extract` to write the `en/messages.po`
3. Run `yarn locales:push --dry-run` to review what will be pushed to Accent
4. Run `yarn locales:push` to push the new translations to Accent
5. Edit the new messages in Accent web UI `yarn locales:browse`
6. Run `yarn locales:pull` to get the messages.po for all translated languages

In the future, we might want to integrate further Accent so that it opens pull requests.
11 changes: 11 additions & 0 deletions accent.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"apiUrl": "https://accent.interactivethings.io",
"files": [
{
"namePattern": "file",
"format": "gettext",
"source": "app/locales/en/messages.po",
"target": "app/locales/%slug%/%document_path%.po"
}
]
}
31 changes: 2 additions & 29 deletions app/locales/de/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ msgstr ""
"PO-Revision-Date: \n"
"Last-Translator: \n"
"Language-Team: \n"
"Plural-Forms: \n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"

#: app/configurator/components/chart-configurator.tsx
msgid "Add filter"
Expand Down Expand Up @@ -61,10 +61,6 @@ msgstr "Alle Datensätze durchsuchen"
msgid "browse.dataset.create-visualization"
msgstr "Visualisierung von diesem Datensatz erstellen"

#: app/browser/select-dataset-step.tsx
#~ msgid "browse.datasets.all-datasets"
#~ msgstr "Alle Datensätze"

#: app/browser/select-dataset-step.tsx
msgid "browse.datasets.description"
msgstr "Erkunden Sie die vom LINDAS Linked Data Service bereitgestellten Datensätze, indem Sie entweder nach Kategorien oder Organisationen filtern oder direkt nach bestimmten Stichworten suchen. Klicken Sie auf einen Datensatz, um detailliertere Informationen zu erhalten und Ihre eigenen Visualisierungen zu erstellen."
Expand Down Expand Up @@ -221,14 +217,11 @@ msgstr "Symbole"
msgid "chart.published.toggle.mobile.view"
msgstr "Kompakte Ansicht"

#: app/login/components/profile-tables.tsx
#~ msgid "chart.rename"
#~ msgstr "Umbenennen"

#: app/charts/table/table.tsx
msgid "chart.table.number.of.lines"
msgstr "Anzahl Zeilen"

#: app/components/chart-selection-tabs.tsx
msgid "chat-preview.delete.confirmation"
msgstr "Sind Sie sicher, dass Sie dieses Diagramm löschen möchten?"

Expand Down Expand Up @@ -1335,10 +1328,6 @@ msgstr "Filter ausblenden"
msgid "interactive.data.filters.show"
msgstr "Filter anzeigen"

#: app/login/components/profile-tables.tsx
#~ msgid "login.chart.actions.publish"
#~ msgstr "Veröffentlichen"

#: app/login/components/profile-tables.tsx
msgid "login.chart.actions.unpublish"
msgstr "Veröffentlichung aufheben"
Expand Down Expand Up @@ -1388,10 +1377,6 @@ msgstr "Sind Sie sicher, dass Sie diese Aktion durchführen wollen?"
msgid "login.profile.chart.delete.warning"
msgstr "Denken Sie daran, dass das Entfernen dieser Visualisierung sich auf alle Stellen auswirkt, an denen sie möglicherweise bereits eingebettet ist!"

#: app/login/components/profile-tables.tsx
#~ msgid "login.profile.chart.rename-dialog.title"
#~ msgstr ""

#: app/login/components/profile-content-tabs.tsx
msgid "login.profile.home"
msgstr "Empfang"
Expand All @@ -1416,10 +1401,6 @@ msgstr "Aktionen"
msgid "login.profile.my-visualizations.chart-name"
msgstr "Name"

#: app/login/components/profile-tables.tsx
#~ msgid "login.profile.my-visualizations.chart-published-date"
#~ msgstr "Veröffentlicht"

#: app/login/components/profile-tables.tsx
msgid "login.profile.my-visualizations.chart-type"
msgstr "Typ"
Expand All @@ -1436,14 +1417,6 @@ msgstr "Datensatz"
msgid "login.profile.my-visualizations.multiple-datasets"
msgstr ""

#: app/login/components/profile-tables.tsx
#~ msgid "login.rename-dialog.cancel"
#~ msgstr "Abbrechen"

#: app/login/components/profile-tables.tsx
#~ msgid "login.rename-dialog.save"
#~ msgstr "Speichern"

#: app/components/header.tsx
#: app/components/header.tsx
msgid "logo.swiss.confederation"
Expand Down
Loading
Loading