Skip to content

Latest commit

 

History

History
111 lines (73 loc) · 5 KB

CONTRIBUTING.md

File metadata and controls

111 lines (73 loc) · 5 KB
Raw

youtube2zim devel

setup

See README for details about how to install with hatch virtualenv.

contributions

  • Open issues, bug reports and send PRs on github.
  • Make sure it's py3.6+ compatible.
  • Use black code formatting.

notes

In order to support all platform, we default to webm video format. mp4 one (h264), is not available in Webview on most platform due to patent restrictions.

On the other hand, webm is not supported in Safari (macOS, iOS).

We thus use ogv.js, to play webm videos on browsers that don't support it. Using video.js, we default to native playback if supported.

ogv.js is an emscripten-based JS decoder for webm and thus dynamically loads differents parts at run-time on platforms that needs them. It has two consequences:

  • file:// scheme doesn't work as the binary .wasm files are sent naively as text/html instead of application/oct-stream. If you want to use the HTML generated folder instead of ZIM, serve it through a web server and configure the Content-Type response.
  • ZIM places JS files under the /-/ namespace and binary ones under the /I/ one. Dynamically loading of JS and WASM files within ogv.js requires us to tweak it to introduce some ZIM-specific logic. See fix_ogvjs_dist.py.

Because the pagination system is implemented in JS, we also need to generate links there. For that to work both in static HTML and in-ZIM, we detect it using a <link id="favicon"> in HTML files. This link needs to be present and parsed before loading the help zim_prefix.js script.

i18n

youtube2zim has very minimal non-content text but still uses gettext through babel to internationalize.

To add a new locale (fr in this example, use only ISO-639-1):

  1. init for your locale: pybabel init -d locale -l fr
  2. make sure the POT is up to date pybabel extract -o youtube2zim/locale/messages.pot youtube2zim
  3. update your locale's catalog pybabel update -d youtube2zim/locale/ -l fr -i youtube2zim/locale/messages.pot
  4. translate the PO file (poedit is your friend)
  5. compile updated translation pybabel compile -d youtube2zim/locale -l fr

releasing

  • Update your dependencies: pip install -U setuptools wheel twine
  • Make sure CHANGELOG.md is up-to-date
  • Bump version on youtube2zim/VERSION
  • Build packages python ./setup.py sdist bdist_wheel
  • Upload to PyPI twine upload dist/youtube2zim-2.0.0*.
  • Commit your CHANGELOG.md + version bump changes
  • Tag version on git git tag -a v2.0.0

developing the ZIM UI in Vue.JS

Sometimes you need to alter something in the ZIM UI in Vue.JS but for this to work, you need assets which are generated by the scraper (e.g. channel.json, ...).

To simplify this, it is possible to:

  • run the scraper (with original code base or your modified one)
  • extract assets from generated files and place them in a directory where ZIM UI will find them
  • iterate on ZIM UI code

To achieve this, first build the Docker image based on current code base.

docker build -t local-youtube2zim .

Scrape a channel (here we use the openZIM_testing channel, but you could use any other one of interest for your UI developments).

docker run --rm -it -v "$PWD/output":/output local-youtube2zim youtube2zim --api-key <YOUR-API-KEY> --id "UC8elThf5TGMpQfQc_VE917Q" --name "openZIM_testing" --zim-file "openZIM_testing"

Extract interesting ZIM content and move it to public folder.

find zimui/public/ -mindepth 1 ! -name ".gitignore" -delete
docker run -it --rm -v $(pwd)/output:/data ghcr.io/openzim/zim-tools:latest zimdump dump --dir=/data/openZIM_testing /data/openZIM_testing.zim
sudo chown -R $(id -u -n):$(id -g -n) output/openZIM_testing
mv output/openZIM_testing/* zimui/public/
rm -rf output/openZIM_testing

Start ZIM UI locally.

cd zimui
yarn dev

Do not forget to cleanup public folder before building the docker image again, otherwise all assets will be pushed to the ZIM.

testing with cypress

Cypress is used for end-to-end testing of the ZIM UI. It allows you to write tests that simulate user interactions with the application to ensure everything works as expected.

To run the tests, you need to start the ZIM UI locally and then run the tests.

cd zimui
yarn dev
yarn test:e2e

On Linux, you might need to install additional dependencies, see Linux Prerequisites in the Cypress documentation.

running integration tests checking ZIM content

We have a bunch of integration tests checking ZIM content. Once you have a the test ZIM from openZIM channel (see instructions above for Vue.JS ZIM UI), you can run the tests locally as well:

ZIM_FILE_PATH="output/openZIM_testing.zim" pytest scraper/tests-integration/integration.py