Create a reindexing daemon.

[msm-code]

Feature Category

• Correctness
• User Interface / User Experience
• Performance
• Other (please explain)

Describe the problem

Right now the user have to create their own cron job, to keep database and samples folder in sync

Describe the solution you'd like

Create a new service, that will automatically watch /mnt/samples directory and reindex samples every night (for example).

This script should take over the functionality of utils/reindex_local.py script and reindex all samples first, and later it should compact datasets as long as it's possible. Useful code snippet:

#!/usr/bin/env python

import logging
from lib.ursadb import UrsaDb
import time


def main() -> None:
    logging.basicConfig(level=logging.INFO)
    ursa = UrsaDb('tcp://localhost:9281')
    stage = 0
    last_datasets = None
    while True:
        datasets = set(ursa.execute_command("topology;")["result"]["datasets"].keys())
        if last_datasets:
            removed = list(last_datasets - datasets)
            created = list(datasets - last_datasets)
            logging.info("%s => %s", removed, created)
        logging.info("Stage %s: %s datasets left.", stage, len(datasets))
        if last_datasets and datasets == last_datasets:
            logging.info("Finally, a fixed point! Returning...")
            return

        start = time.time()
        ursa.execute_command("compact all;")
        end = time.time()
        logging.info("Compacting took %s seconds...", (end-start))
        stage += 1
        last_datasets = datasets


if __name__ == "__main__":
    main()

Add a way to manually trigger it easily using docker-compose (probably something like docker-compose run reindex --now)

[msm-code]

Not implemented currently. Reindexing was made friendly enough that it's not needed as much as it used to.