Skip to content

Files layout and configuration

Jump to bottom
O2eg edited this page Apr 17, 2022 · 18 revisions

Directories

Directories descriptions:

  • actiontracker - packet (package) deployment tracking module (create and manage of dbc_packets, dbc_steps, dbc_actions and dbc_locks tables)
  • conf - directory with configuration files db_converter.conf and db_converter_test.conf
  • dbccore - threads management and packet execution logic
  • matterhook - module for interaction with mattermost
  • packets - contains packages whose names are specified in the parameter --packet-name
  • psc - auxiliary modules for working with logs and database
  • tests - contains unit tests
  • log - deployment logs. A separate file will be created for each package and database.

 

 

The configuration file db_converter.conf can contain connection strings for all servers. This approach allows the management of migrations from a single host centrally. There is also a separate configuration file for tests db_converter_test.conf.

Config parameters

db_converter.conf consists of the following sections:

  • [databases] - list of databases with connection strings. Databases names (aliases) must be unique. If the password contains special characters, then use URL encoding.
  • [main] - section contains the main parameters of the db_converter
    • application_name is the name of the application connected to a database. This value is automatically postfixed with the name of the running packet.
    • lock_observer_sleep_interval - is a time interval in seconds after which the lock observer checks the status of active connections from db_converter and other active connections to the database
    • execute_sql - enable execution of SQL queries on the database. If false, queries are not executed on the database
    • conn_exception_sleep_interval - is a time interval in seconds. After this interval, the connection to the database is re-established and the query is repeated if the connection was closed from the database side (by a user, by timeout, or by lock_observer)
    • cancel_wait_tx_timeout - cancel a step's transaction if it has been waiting for another query to release the lock longer than the specified time (valid values correspond to the interval format, for example: '3 minutes', '60 seconds')
    • cancel_blocker_tx_timeout - cancel the active transaction of a step if another query waits for its completion longer than the specified time (valid values correspond to the cancel_wait_tx_timeout parameter)
    • detailed_traceback - log detailed traceback when an exception occurs. By default True
    • db_name_all_confirmation - prompt for confirmation of deployment. By default True
    • schema_location - specify a schema for placing dbc_* tables. By default public
  • [log] - logging settings
    • log_level - logging level. Possible values: Debug, Info, Error
    • log_sql - if set to 1, then all SQL queries are logged
    • file_maxbytes - is a maximum size of the log file in megabytes, upon reaching which the current one is renamed and a new one is created, in which messages continue to be written
    • file_backupcount - is a maximum number of .log files, upon reaching which the rotation is performed
  • [mattermost_hooks] - settings for interaction with the mattermost messenger
    • url - hostname or IP with a port of the messenger server, for example, https://mattermost.company.org:1443
    • chat_keys - list of chats and keys separated by a comma, for example dbc-alert/xaqxcj3axjb,dbc-dba/8ajz1z6y8
  • [slack_hooks] - settings for interaction with the slack messenger
    • url - hostname or IP with a port of the messenger server, for example, https://hooks.slack.com/services
    • chat_keys - list of chats and keys separated by a comma, for example dbc-alert=TKS/B022YC/lHCTK,dbc-dba=TKS/B022YC/s33Ju2
  • [postgresql] - session variables. This section is optional. May contain the following parameters:
Parameter Default value
deadlock_timeout 100ms
statement_timeout 1h
vacuum_cost_limit 3000
work_mem 200MB
maintenance_work_mem 1GB
timezone UTC

For boolean parameters (such as execute_sql, log_sql) values can be specified from the list: 0, 1, false, true, False, True.

Packets naming convention

For a more convenient perception of the purpose of the packets (its name in the .log files and in application_name), it is recommended to adhere to the following naming format:

Format Description
dba_task_name Administration utilities
test_some_test Test packages
alert_notification_name Notifications
maint_task_name Database maintenance
YEARMONTH[DAY optional]_TASK_NUMBER_short_descr Typical migration

All packets with dba_, alert_, and test_ prefixes are automatically used in unit tests. About how to write a unit test for a packet, see Unit test examples section.

Clone this wiki locally