Refactoring: towards Fink v4

[JulienPeloton]

IMPORTANT: Please create an issue first before opening a Pull Request.
Linked to issue(s): Closes #947

What changes were proposed in this pull request?

This PR completely reorganises the core of Fink to better handle the Rubin integration.

Notable changes:

• reorganisation of the paths, to have /ztf/ and /rubin/
• new argument -s <survey> to all binaries
• renaming of db operation scripts
• addition of kafka_security_protocol in the arguments
• Removal of the sims producer --> now the CI uses ztf (or rubin later)
• update of the k8s yaml to account for new parameters
• update of the configuration files to take former CLI parameters (tns folders, etc.)
• Simplification of the configuration files --> only 2 now (fink.conf.dev & fink.conf.prod). Resources are declared within binaries (CLI)

Nightly streaming

Configuration file is unique for all jobs, and can be found at ${FINK_HOME}/conf. The nightly operations consist of 3 jobs, wrapped in ${FINK_HOME}/scheduler/launch_stream.sh.

The polling operation is launched at 00:01 on night N, where it starts polling data (which is not yet produced). Then when data finally arrives, the enrichment operation starts, followed by the distribution operation. All jobs stop at 20:00.

You can also manually launch part of the processing:

# It is 14:00 Paris. Poll only for 1 hour
./launch_stream.sh --poll_only -stop_at "15:00 today"

# Extract science until the end of the night
./launch_stream.sh --enrich_only

# Distribute until 5pm
./launch_stream.sh --distribute_only -stop_at "17:00 today"

# 8:00 AM. Reprocess entirely another night in a couple of hours
./launch_stream.sh -night 20241231 -stop_at "10:00 today"

Database operations

This is now parametrised by /home/julien.peloton/fink-broker/scheduler/ztf/launch_db.sh.

This job contains 4 steps:

fink_db -s ztf --merge
fink_db -s ztf --main_table
fink_db -s ztf --index_tables
fink_db -s ztf --clean_night

If the merge went wrong, just manually clean the files:

fink_db -s ztf --clean_merge

And restart DB operations.

New schedule

We now start on night N.

# Paris time @ VD

# Fink real-time
01 00 * * * /home/julien.peloton/fink-broker/scheduler/ztf/launch_stream.sh

# Database service
05 20 * * * /home/julien.peloton/fink-broker/scheduler/ztf/launch_db.sh

# SSOFT - once a month
0 0 1 * * /home/julien.peloton/fink-broker/scheduler/ztf/launch_ssoft.sh
0 12 1 * * /home/julien.peloton/fink-broker/scheduler/ztf/launch_sso_resolver.sh

# Operation reports four times a day
0 07 * * * /home/julien.peloton/fink-broker/scheduler/ztf/check_status.sh --telegram
0 12 * * * /home/julien.peloton/fink-broker/scheduler/ztf/check_status.sh --telegram
0 17 * * * /home/julien.peloton/fink-broker/scheduler/ztf/check_status.sh --telegram
0 22 * * * /home/julien.peloton/fink-broker/scheduler/ztf/check_status.sh --telegram

How was this patch tested?

CI & local container

[codecov]

Codecov Report

Attention: Patch coverage is 94.44444% with 4 lines in your changes missing coverage. Please review.

Project coverage is 83%. Comparing base (33e0971) to head (7a11986).

Files with missing lines	Patch %	Lines
fink_broker/ztf/mm_utils.py	66%	4 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##           master   #948   +/-   ##
=====================================
  Coverage      83%    83%           
=====================================
  Files          21     21           
  Lines        1346   1393   +47     
=====================================
+ Hits         1129   1170   +41     
- Misses        217    223    +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sonarqubecloud]

Quality Gate passed

Issues
38 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud