Skip to content

Home

Jump to bottom
Will Haese-Hill edited this page Mar 28, 2024 · 4 revisions

Welcome to the annot-web wiki! Here, you can find instructions for setting up a Companion web server.

Web server

The production web server is a VM with 32 CPUs and 64 GB RAM. 2 TB of storage are mounted for web app data.

  • Server name: companion.gla.ac.uk
  • access via SSH is possible

The web interface is a [http://rubyonrails.org/ Ruby on Rails] application served by nginx and passenger. See https://github.com/sii-companion/annot-web/wiki/Manually-installing-dependencies#setup-nginx-with-passenger for configuration example.

There are also a dev server companion-dev.mvls.gla.ac.uk with the same resources, and a build server companion-build.mvls.gla.ac.uk with 8 CPUs, 8 GB RAM and 200 GB storage. Both of these can also be accessed via SSH. Companion dependencies have been installed manually on each of these servers in the same way (see https://github.com/sii-companion/annot-web/wiki/Manually-installing-dependencies). Sidekiq is used for running jobs, and is controlled by an always running system service at /etc/systemd/system/sidekiq.service`.

Access credentials

For each server, there is a companion user set up with sudo privileges. All Companion dependencies should be accessible from this user account, and sidekiq jobs will be initiated with this user. Please ask Scott or Thomas for the password.

Setup

The components are installed on the servers in the following places:

  • Companion: ~companion/annot-nf
  • Companion web app: /www/companion/annot-web

These are just git checkouts of the remote repositories https://github.com/sii-companion/companion and https://github.com/sii-companion/annot-web, respectively.

Configuration of the server is mainly done in a number of configuration files. Due to the site specific nature of those and the fact that they can contain passwords and other sensitive information some of them are not part of the Git repositories and need to be added manually.

companion.yml

In /www/companion/annot-web/config/companion.yml the web application itself is configured, for 'production', 'development' and 'test' environments. Currently, for each server, only the 'production' one is relevant, but they could be configured independently of each other. The following fields are important:

  • nextflowpath : Directory containing the nextflow binary
  • nextflowscript : Full path to Companion's annot.nf
  • locationconfig : Full path to a Companion location definition, e.g. /home/companion/annot-nf/loc_web.config
  • extrinsicconfig: Full path to Companion's extrinsic.cfg, e.g. /home/companion/annot-nf/data/augustus/extrinsic.cfg
  • referencedirs : List of key-value pairs containing a reference set name and the directory in which its definition is located. Example: Kinetoplastida: "/www/companion/ref_kinetoplastids"
  • weightfiles : List of key-value pairs containing a reference set name and the full path to its Companion weight file. Needed, for example, to distinguish kinetoplastids from everything else. Example: Kinetoplastida: "/home/companion/annot-nf/data/weight/weight_kinetoplastid.lua" . Use weight_multiexon_generic.lua for everything else, it's more generic as it doesn't do PTUs.
  • maxintronlengths : List of key-value pairs containing a reference set name and the maximum intron length allowed for this group. Set to super small to disallow introns, e.g. Kinetoplastida: 1 or Plasmodium: 20000.
  • fix_polycistrons : List of key-value pairs containing a reference set name and a Boolean value indicating whether these species need PTU fixing. Example: Kinetoplastida: "true", Plasmodium: "false"
  • rootdir : Full path to Companion installation, e.g. /home/companion/annot-nf/
  • workdir : Full path to Nextflow work directory, e.g. /www/companion/work.
  • tmpdir : Temporary directory to use, e.g. /data/tmp
  • example_job_id : Job ID identifying an example run to be displayed as a live example in the web page's 'Example results' tab. If missing, no live example will be displayed.
  • example_species : Example to be given for job submission form. Example: prefix: "CD36", name: "Candida dubliniensis"
  • keep_work_directories : Boolean value indicating whether to keep all work directories (for debugging). Normally they are deleted after successful runs.
  • min_work_space : Minimum amount of space (in MB) to be required to be free in the work directory. If there is less space, Companion will no longer accept new jobs until more space is freed.

mailer.rb

The file /www/companion/annot-web/config/initializers/mailer.rb contains the configuration to access a mail server to send out notifications about user jobs. Here's an example:

Rails.application.configure do
  config.action_mailer.default_url_options = {
    :host => 'companion.gla.ac.uk',
    :port => 80
  }

  config.action_mailer.delivery_method = :sendmail
  config.action_mailer.perform_deliveries = true
  config.action_mailer.raise_delivery_errors = true
end

sidekiq.yml

The YAML file /www/companion/annot-web/config/sidekiq.yml sets concurrency, i.e. the number of simultaneously running jobs, for Sidekiq. Example:

:concurrency: 1
staging:
  :concurrency: 1
production:
  :concurrency: 1
:queues:
  - default
  - [annot, 2]

would make sure only one Sidekiq job is run at a time.

loc_web.config

This file /home/companion/annot-nf/loc_web.config contains the Companion site configuration for running jobs directly on the same local system that the web server runs on. Example:

env {
    PATH = "${baseDir}/RATT:${baseDir}/ABACAS2:${HOME}/ORTHOMCLV1.4:${HOME}/genometools/bin:${HOME}/augustus/bin:${HOME}/last-581/src/:${HOME}/bin:${HOME}/augustus/scripts:$PATH"
    RATT_HOME = "${baseDir}/RATT"
    RATT_CONFIG = "${baseDir}/RATT/RATT.config_euk_NoPseudo_SpliceSite"
    GT_RETAINIDS = "yes"
    PERL5LIB = "${HOME}/ORTHOMCLV1.4/:${baseDir}/RATT/:${baseDir}/ABACAS2/:$PERL5LIB"
    PFAM = "${HOME}/Pfam-A.hmm"
    AUGUSTUS_CONFIG_PATH = "${baseDir}/data/augustus"
    FILTER_SHORT_PARTIALS_RULE = "${baseDir}/data/filters/filter_short_partials.lua"
    PFAM2GO = "${baseDir}/data/pfam2go/pfam2go.txt"
}

params.GO_OBO = "${HOME}/go.obo"
params.NCRNA_MODELS = "${baseDir}/data/cm/kinetoplastid_rnas.cm"
params.CIRCOS_CONFIG_FILE = "${baseDir}/data/circos/circos.debian.conf"
params.CIRCOS_BIN_CONFIG_FILE = "${baseDir}/data/circos/circos.bin.debian.conf"
params.SPECFILE = "${baseDir}/data/speck/output_check.lua"
params.AUGUSTUS_EXTRINSIC_CFG = "${baseDir}/data/augustus/extrinsic.cfg"

executor {
    name = 'local'
    queueSize = 4
    pollInterval = '3sec'
}

which assumes that most dependencies are installed in the user's home directory (see PATH setting). ${baseDir} here refers to the Companion installation's root directory (such as ~companion/annot-nf).

~/.bashrc

The user running the web server needs to make sure that GenomeTools' gt executable is in their $PATH.

Role email addresses

The email address [email protected] is used to send notifications to users and receive detailed error reports in case jobs fail (in this case users only get a short notice without technical details). It can be accessed via Outlook. Thomas Otto (tdo) has got the access credentials for this account.

Starting/stopping the service

Run sudo systemctl restart sidekiq nginx to ensure all front/backend components of the server are refreshed.

Temporarily closing the service

The service queue can be closed such that it no longer accepts new submissions by creating a file called CLOSED in the web application's root directory (i.e. /www/companion/annot-web). This will change the Companion user interface to display a short notice in a different colour and disable all buttons/URLs related to job submission. Finished results can still be browsed and execution of existing jobs is ''not'' suspended, to allow them to finish (typically one would use this feature before planned downtime). To re-open the queue, simply delete the CLOSED file.

Dealing with stuck jobs

It is rare but possible that a Companion process starts a UNIX process (for example exonerate) that does not finish as it should but remains running, keeping that Companion process and hence the Companion run from finishing while new jobs pile up in the queue. In this case it should suffice to just briefly stop and restart the web service (see the corresponding section here in the wiki) to re-initialize the run, triggering a retry (of the whole job).

An attempt at mitigating this problem is currently implemented by setting the maximal run time for a single exonerate job to 2 hours, and BRAKER process to 3 days.

Using the 'Delete' function in the admin panel only dequeues jobs that haven't started yet or deletes jobs that have finished. Currently running jobs are not controllable for now.

Deploying updates from GitHub

Simply change into the directories with Companion or the web app as the companion user and git pull from your personal feature branch. 'production' and 'build' servers should be restricted to only tagged releases, by running

git fetch
git checkout tags/vX.X.X -b vX.X.X

Restart the web service to pick up the changes. See https://github.com/sii-companion/annot-web/wiki/Checking-out-Companion-releases for more info.

Importing reference databases

Let's assume we already have directories with reference data for given sets of organisms, such as a VEuPathDB subproject (TriTrypDB, ...) or custom-built set. For more info on how to create these directories, see https://github.com/sii-companion/companion/wiki/Preparing-reference-data-sets. We will also assume that these have already been registered in the web app's config/companion.yml file in the referencedirs section:

production:
  ...
  referencedirs:
    Amoeba:     "/data/references/ref_amoebadb"
    Cryptosporidium: "/data/references/ref_cryptodb"
    Fungi:      "/data/references/ref_fungidb"
    Microsporidia: "/data/references/ref_microsporidiadb"
    Piroplasma: "/data/references/ref_piroplasmadb"
    Plasmodium: "/data/references/ref_plasmodb"
    Toxoplasma: "/data/references/ref_toxodb"
    Kinetoplastida: "/data/references/ref_tritrypdb"
    Vectors: "/data/references/ref_vectorbase"
    Hosts: "/data/references/ref_hostdb"
...

The directories will each contain a Ref_* subdirectory for a given species family, which each ''need'' to contain the _all subdirectory containing all-vs-all orthology data for gene cluster analysis. The guide https://github.com/sii-companion/companion/wiki/Preparing-reference-data-sets also describes how to create these.

Follow the guide https://github.com/sii-companion/annot-web/wiki/Updating-the-reference-database#update-ruby-on-rails-database-on-server to update all reference models in the web server database. This will load the contents of all the genomes defined in each directory's references.json into the database and make them available as references to be selected in the web interface.

Note that you need to have rbenv shims configured (rbenv init) for the bundler call to work (see https://github.com/sii-companion/annot-web/wiki/Updating-the-reference-database#update-ruby-on-rails-database-on-server). The corresponding rake task is located in lib/tasks/load_references.rake.

Creating web admin accounts

To inspect and maintain (e.g. delete/restart) jobs on the web server, it is possible to log in to an admin panel using the URL https://companion.gla.ac.uk/login. This will add an additional menu to the top bar of Companion, allowing one to 'Manage jobs'. To create such an admin user account, perform the following steps:

Log into the server via SSH.

In the Companion web app installation directory (e.g. /www/companion/annot-web), run the following:

RAILS_ENV=production bundle exec rake create_admin[adminuser1,[email protected],secretpassword]

where the values in the square brackets must be replaced by the desired values for username, email address and password.

Note that you need to have rbenv shims configured (rbenv init) for the bundler call to work (see https://github.com/sii-companion/annot-web/wiki/Updating-the-reference-database#update-ruby-on-rails-database-on-server). The corresponding rake task is located in lib/tasks/create_admin.rake.

Deleting old jobs from the database

To delete old jobs from the database, log into the web admin interface at https://companion.gla.ac.uk/login and for each job in the list to delete click on 'Delete' and confirm with 'Yes' on the dialog that pops up. You can also select multiple jobs with the checkbox and click "Delete all selected".

There is also a new rake task that does this:

 RAILS_ENV=production bundle exec rake expire_old

in the directory /www/companion/annot-web

It is also good to clean the cache (tmp/cache)

 RAILS_ENV=production bundle exec rake tmp:clear

A ruby gem whenever has been used to create a cron job that performs a expire_old run on a weekly basis. whenever simply takes more "human-readable" form of scheduling, which is configured in config/schedule.rb, and converts it to crontab form. Run

bundle exec whenever --update-crontab

in the directory /www/companion/annot-web in order to create the cron job that conforms to the latest code in schedule.rb.

Restarting jobs that have failed

Sometimes you might diagnose a process failure that results in job failure and want to restart the job from where it left off (i.e. prevent all the previously successful processes from having to run again).

Open the rails shell with rails c production in /www/companion/annot-web directory. Let us assume the failed job is in the Sidekiq DeadSet. You can then iterate through this set and restart the job with given job_id (see https://github.com/mperham/sidekiq/wiki/API#dead=). For example

ds = Sidekiq::DeadSet.new
ds.select do |job|
  job.jid == "2ade5abea417d56e5b491671"
end.map(&:retry)

This should add the job to the Sidekiq RetrySet, which will get picked up, and use the CACHED nextflow processes (assuming -resume flag is set in HardWorker nextflow command).

Nextflow traces

In each job's result dir there is a trace.txt file (i.e. /www/companion/annot-web/public/jobs/<job-id>/trace.txt). This provides a useful tabular display of each process and its status, workdir, duration, cpu/mem requirements, etc. This is produced whether or not a job completed successfully.

Clone this wiki locally