Development

This may need revision, ping me if you need clarification as I wrote it quickly.

Prereqs

• To work on Lute v3, you'll need at least Python 3.8 and pip. You'll probably want to use some kind of virtual environments; I use venv and so will write that out here.
• Note that GitHub CI tests Python versions 3.8 through 3.11, as we can't be sure what version of Python users have, so stay away from newer language features.
• Install MeCab according to your platform as described on the User Installation Page. This is required for passing unit tests irrespective of whether you are developing features relevant to the Japanese language.

Setup and verify your dev environment

1. Clone

# ... usual cloning steps here

git checkout develop    # The main dev branch
git submodule init
git submodule update

Lute uses git submodules for the language definitions (https://github.com/LuteOrg/lute-language-defs)

2. Virtual env and dependencies

python3.8 -m venv .venv     # Your python version may vary
source .venv/bin/activate
python --version            # sanity check only

# Install dev and prod requirements
pip install flit
flit install --only-deps --deps develop

pre-commit install         # pre-commit hooks (recommended)

3. Install acceptance testing tools

Lute uses python splinter and playwright, so you'll need to install ChromeDriver and Playwright browsers

MacOS:

brew install --cask chromedriver
xattr -d com.apple.quarantine /opt/homebrew/bin/chromedriver
playwright install

4. config.yml

Copy lute/config/config.yml.example to lute/config/config.yml. Set the following parameters:

• ENV: prod for production or dev for development
• DBNAME: filename for the database (must be prefixed with test_ to run tests)
• DATAPATH: the full path to the folder used to store data (such as /Users/xyz/Documents/lute_data)
• BACKUP_PATH: the full path to the folder used to store backups (such as /Users/xyz/Documents/lute_backup)

If you're going to work on Lute, you'll need to run unit tests. The unit tests are destructive, in that they wipe and reset the configured database. To guard against mistakes:

• ENV must be dev
• DATAPATH must be set
• The DBNAME in your config.yml must start with test_

This ensures that you won't accidentally run the tests against your real Lute data. I work with this by having two completely separate environments: one for dev work, and one for real Lute usage. My prod data (actual data) stays in the latter.

5. Start Lute to verify

source .venv/bin/activate      # if necessary
python -m lute.main
# Open web browser to http://localhost:5000
# ... work work work ...
# When done, Ctl-C then
deactivate

6. Run tests locally to verify

Shut down your dev instance of Lute if it's running, and then run

inv full

This runs all tests and lints everything. It should complete without errors, as lute master and develop branch are always kept passing in CI.

Development

You may/may not find the overview docs of Lute's architecture useful ... let me know.

Commit hooks

Pre-commit hooks are installed with the pre-commit install step, and are run on every commit. I find this useful, as it stops me from having to go back and clean up, but YMMV. You can skip a step, e.g.: SKIP=pylint git commit -m "Some non-lint-compliant commit."

Testing

Testing is done with pytest and pytest-bdd. Run them as usual:

pytest
pytest path/to/file.py
pytest -k filter_string
pytest -m somemark
pytest -s
pytest -vv

Code changes and commit messages

The usual notes here. Some of these guidelines are really done by "feel", so chat with devs on Discord or in PRs as needed.

• Try to keep your PR minimal, only change what you mean to change. If your PR introduces a ton of reformatting, moving around, etc, it might be rejected for sanity's sake.
• Make nice commit messages (see https://cbea.ms/git-commit/ for notes). You don't need to go overboard if the code is self-explanatory, but a brief summary is often good.
• if you make code changes, don't forget to lint your submission (inv lint), and keep an eye on GitHub CI failures for your branch.

inv or invoke for tasks

Lute3 uses Invoke to run tasks. Tasks are in tasks.py. See inv --list for commands.

Some useful tasks:

task	desc
inv start	start the app on a development Flask server in dev/debug mode
inv lint	lint
inv accept	start a running instance of the app server if needed, and run acceptance tests
inv playwright	start a running instance of the app server if needed, and run playwright tests

Database changes

Database migrations are managed automatically when Lute starts up. See the DB schema README for notes about how the schema changes are managed.

DB migrations are stored in /lute/db/schema/migrations. To create a script, run inv db.newscript <somesuffix>, and edit the file to create a Sqlite-compliant change script. See the existing scripts for examples.

When doing releases, the baseline database gets recreated as described in Releases -- if you add a migration, you don't need to recreate the baseline.

Dependencies: pip and flit

Dependencies are managed with the pyproject.toml file, and installations of the dependencies are done using flit:

flit install --only-deps --deps develop

IMPORTANT: Make sure that your dependencies don't introduce other unexpected dependencies (e.g. that aren't under MIT licensing, that need compilers or build tools, etc), or other huge libraries!

TODOs

Todos are in the code as comments, e.g. # TODO [<group name>:] detail, <!-- TODO ... -->. inv todos collects all of these in a simple report.

The "group name" is arbitrary: it's used to group things together, which is handy when several locations need to be changed as a single unit of work.

Docker

Notes for building and running a Docker container are at ../docker/README.com.

Developing parser plugins

As of v3.4.0, Lute supports a simple "language parser plugin" capability. Set up your dev environment as explained on this page, and then head over to Developing language parser plugins.

Misc dev notes

Finding mecab.so for Docker

This is much tougher than it needs to be ...

To find the correct path, first build and run the container, then connect to it, and find libmecab.so.2 like this:

$ docker exec -it lute_v3-lute-1 bash
root@cid/# which mecab
/usr/bin/mecab
root@cid:/# ldd /usr/bin/mecab
   ...
   libmecab.so.2 => /lib/aarch64-linux-gnu/libmecab.so.2 (0x0000ffff9b540000)

Different platform architectures have this in different locations. :-/

datatables

Datatables is used for the term and book listings. See lute/static/vendor/datatables/README.md for notes about getting and updating it.

read-only db during tests

It appears that killing acceptance tests mid-run results in a zombie (?) python process that keeps a handle on the db, causing it to get locked in read-only mode.

I couldn't find a better way to kill this process than do a full machine restart. Sledgehammer approach that works.

Acceptance tests suddenly failing

Warning during run of tests with inv accept --exitfail:

WARNING  selenium.webdriver.common.selenium_manager:selenium_manager.py:139 The chromedriver version (118.0.5993.70) detected in PATH at /opt/homebrew/bin/chromedriver might not be compatible with the detected chrome version (119.0.6045.105); currently, chromedriver 119.0.6045.105 is recommended for chrome 119.*, so it is advised to delete the driver in PATH and retry

The fix (on a Mac):

brew upgrade chromedriver
/opt/homebrew/bin/chromedriver --version

The above will show a message: "“chromedriver” can’t be opened because Apple cannot check it for malicious software." Click "Show in Finder", then in Finder, click "Open" and say "OK" when it can't be verified. Yes, this is a security risk.