Deployment smoke tests

[ironicbadger]

We need some tests writing which will check that the hugo build was successful and the site deploys correctly. This will be used as part of a PR workflow before enabling merges to main once complete. For example if the tests fail for any reason, merging will be blocked until they pass.

I envisage this taking the form of checking the existence of critical files including configuration files and certain show pages. Then the site will be built (as part of an action most likely), spun up (again as part of the action) and then the smoketests will be performed to ensure the thing is running as expected.

[elreydetoda]

I imagine that we could have a simple hugo command (like how we do for deployment) run to try and just build the site. Which can test and give fast feedback on if it'll build at all. (Validating the config for hugo works)

---

Then, I think validating key shows or other web things (i.e. jb-live loads the iframe, media files can be accessed, host/guest pictures are loading, etc.) will go hand in hand with your ideas in matrix about deployments.

Although, personally I'm kind of more partial to #257 mainly because it doesn't require any type of deployment on JB infra. At least maybe for an initially opened PR, which will help with security because it won't be touching any JB infra until later (maybe after merging into a develop branch). While also give fast feedback to PRs authors as well.

So it could go:

PR opened against develop --> attempted deployment to netlfiy (which validates hugo config or else it won't deploy at all)

• --> if tests don't pass then person knows right away they need to fix something
• --> tests do pass, then can be merged into develop by JB team member
  • --> tests run again against .dev.jb.com to make sure everything is still working, and is now deployed (as a test run) on JB infra (like how it'll be in prod)
    • --> JB team member deploys to main/prod & tests run again, so it's getting a last check after the merge to make sure nothing on the main deployed website was messed up from the PR & if it was you can quickly revert it

I think this'll give a nice incremental rollout over JB infrastructure while also allowing for fast feedback for PR authors, and not compromising the security of JB infra by quickly deploying potentially malicious code from an unknown person who opens PRs against the repo.

---

For actually doing the test I've actually run across a really nice tool for web-ui testing tool recently that's a lot easier to use than something like selenium but has a similar in function: https://playwright.dev/

They also have a python port and it uses pytest: https://playwright.dev/python/docs/intro

Since we're already using python for the scraper, and python is big in the linux community we can leverage the port to take advantage of everyone's familiarity already with python.

It's a good testing framework which even lets you do video recordings and screenshots while it goes through, which lets you get a quick high level review of the site for extremely important sections. This is a pretty good/cool YT video about playwright:

---

BTW, @kd2flz's comment in this thread is something we still need to consider as well. Although, if we run out of them it's not that big of a deal IMO, because it's just an extra nice-to-have thing for quick feedback.

[elreydetoda]

Based on this issue happening, we need to validate the expected page isn't set to draft. When someone is doing a PR, they should include tests in the CI for ensuring this.

[elreydetoda]

Just going to start making a list of different tests we'll eventually need to implement, so I don't forget 🙃

• Episode layout (for each show) i.e. how there were two logos here: Revamp Episode Layout #400 (comment)
• Test all pages formatting
  • 404 page: Add 404 page #427
  • Live page
    • When "live": "JB is live!" indicator - on page load, call the script to highlight the #livebutton red if jupiter.tube is live. #378 (using playwright's intercept feature to mock going live, mentioned here)
      • example event:

        {"total":1,"data":[{"id":50,"uuid":"55b2e9d1-ccf3-4914-9f3f-c5d8a59bd4c1","shortUUID":"bzMpxgG2rDKn9C99GSab92","url":"https://jupiter.tube/videos/watch/55b2e9d1-ccf3-4914-9f3f-c5d8a59bd4c1","name":"LINUX Unplugged 471 - We Broke Our Server","category":{"id":15,"label":"Science & Technology"},"licence":{"id":5,"label":"Attribution - Non Commercial - Share Alike"},"language":{"id":"en","label":"English"},"privacy":{"id":1,"label":"Public"},"nsfw":false,"description":"We broke our server, so lets see if we can fix it live before the show is over!","isLocal":true,"duration":0,"views":0,"viewers":19,"likes":0,"dislikes":0,"thumbnailPath":"/static/thumbnails/b610e5f5-bbb5-4ee1-ac24-421c50f0daf5.jpg","previewPath":"/lazy-static/previews/fcd582d8-68c5-4f00-8a4a-0a8ffde1a4a5.jpg","embedPath":"/videos/embed/55b2e9d1-ccf3-4914-9f3f-c5d8a59bd4c1","createdAt":"2022-08-14T16:26:27.938Z","updatedAt":"2022-08-14T18:53:15.544Z","publishedAt":"2022-08-14T18:53:15.543Z","originallyPublishedAt":null,"isLive":true,"account":{"id":3,"displayName":"JBLive Stream","name":"jblive","url":"https://jupiter.tube/accounts/jblive","host":"jupiter.tube","avatars":[{"width":48,"path":"/lazy-static/avatars/e4d17e7f-a144-4a4a-b5d8-b297b06b727f.png","createdAt":"2022-06-07T23:43:56.565Z","updatedAt":"2022-06-07T23:43:56.565Z"},{"width":120,"path":"/lazy-static/avatars/1340f3e9-d0a7-4bc4-bcbd-a8c865eaf1b8.png","createdAt":"2022-05-30T20:36:19.005Z","updatedAt":"2022-05-30T20:36:19.005Z"}],"avatar":{"width":48,"path":"/lazy-static/avatars/e4d17e7f-a144-4a4a-b5d8-b297b06b727f.png","createdAt":"2022-06-07T23:43:56.565Z","updatedAt":"2022-06-07T23:43:56.565Z"}},"channel":{"id":2,"name":"live","displayName":"live","url":"https://jupiter.tube/video-channels/live","host":"jupiter.tube","avatars":[{"width":48,"path":"/lazy-static/avatars/8acfd2a2-ab4e-48aa-990c-3156a2765d2e.png","createdAt":"2022-06-07T23:43:56.607Z","updatedAt":"2022-06-07T23:43:56.607Z"},{"width":120,"path":"/lazy-static/avatars/e9bcfdb7-90a2-479c-b1da-1ab0e5fc9442.png","createdAt":"2022-05-30T20:38:33.632Z","updatedAt":"2022-05-30T20:38:33.632Z"}],"avatar":{"width":48,"path":"/lazy-static/avatars/8acfd2a2-ab4e-48aa-990c-3156a2765d2e.png","createdAt":"2022-06-07T23:43:56.607Z","updatedAt":"2022-06-07T23:43:56.607Z"}}}]}
        

    • when not live
• Test different devices emulations (mentioned here)
• make sure all external links open a new tab
• navigate different user journeys
• check accessibility (i.e. Accessibility check #414)
• other functionality?

More testing optimizations: https://podverse.fm/episode/1Bc04qHgk

[elreydetoda]

@CGBassPlayer, we probably need to check to make sure image paths are right. I just messed up on validating that for #506 (here), but I don't know exactly how to do that 😅

Also, posting it here because #475's convo thread is getting long and this is the original issue it spawned from.

[CGBassPlayer]

Not sure how I missed this comment! I will add image validation to the list and see if I can figure something out