Initial working

Author: almahmoud

README.md

@@ -0,0 +1,88 @@
+# Bioconductor Archive Sync
+
+This Ansible playbook automates the process of syncing a Bioconductor release to the Open Storage Network (OSN) archive. It handles the full workflow of:
+
+1. Retrieving the specified Bioconductor version from the master Bioconductor server
+2. Creating the necessary directory structure locally
+3. Transferring the retrieved data to OSN for archival storage
+
+## Prerequisites
+
+### On Your Local Machine (Running Ansible)
+
+- Ansible installed (`pip install ansible`)
+- SSH access to the target server
+- SSH key for connecting to the target server
+- Knowledge of the target server's IP address and SSH user credentials
+
+### On The Target Machine (Running the Sync)
+
+- SSH key `~/.ssh/rsync.pem` for connecting to the Bioconductor master server
+- Rclone configuration file at `~/.rclone.conf` with an [osn] remote defined
+
+Note: The playbook will automatically check for these prerequisites on the target machine. If rclone is not installed on the target machine, the playbook will automatically install it.
+
+## How to Use
+
+The simplest way to run this playbook is by using the provided `run.sh` script.
+
+### Using run.sh
+
+The run.sh script simplifies execution by handling all the necessary parameters:
+
+```bash
+./run.sh <ssh_key_path> <ip_address> [ssh_user] [bioc_version]
+```
+
+#### Parameters:
+
+- **ssh_key_path**: Path to your SSH private key for connecting to the target server
+- **ip_address**: IP address of the target server where the sync will run
+- **ssh_user**: (Optional) SSH username for connecting to the target server (default: ubuntu)
+- **bioc_version**: (Optional) The Bioconductor version to sync (default: 3.21)
+
+#### Example usage:
+
+```bash
+./run.sh ~/.ssh/my_key.pem 192.168.1.100 ubuntu 3.20
+```
+
+### Best Practice: Run Multiple Times
+
+It's recommended to run the script at least twice:
+- The first run will transfer all the data, which may take significant time depending on the size of the Bioconductor release
+- Subsequent runs will be much faster and ensures that all transfers were successful
+- If the second run shows no additional files being transferred or updated, it confirms that the synchronization is complete and consistent
+
+This approach leverages rsync and rclone's internal check mechanisms - it only transfers files that have changed or are missing, making subsequent runs both a verification and a way to complete any interrupted transfers.
+
+### Best Practice: Use Screen for Persistent Sessions
+
+I'd recommended you use `screen`, especially on a VM, to ensure the process continues even if your connection to the VM is interrupted:
+
+```bash
+# Start a new screen session
+screen -S bioc-sync
+
+# Now run the script inside the screen session
+./run.sh ~/.ssh/rsync.pem 192.168.1.100 ubuntu 3.21
+
+# You can detach from the screen session with: Ctrl+A, then D
+```
+
+After starting a screen session, you can leave it unattended for a couple of hours, while the transfers happen.
+
+When returning to the session:
+
+```bash
+# If disconnected, you can reconnect to the VM and resume the session with:
+screen -r bioc-sync
+```
+
+This approach protects your sync process from:
+- Network connectivity issues between your computer and the VM
+- Local computer shutdowns or sleep mode
+- SSH session timeouts
+- Accidental terminal closing
+
+The transfer will continue running on the VM even if your connection drops, and you can easily reconnect to check progress when needed.

ansible.cfg

@@ -0,0 +1,3 @@
+[defaults]
+inventory = inventory.yml
+host_key_checking = False

inventory.yml

@@ -0,0 +1,6 @@
+all:
+  hosts:
+    bioconductor_vm:
+      ansible_host: "{{ ip_address }}"
+      ansible_user: "{{ ssh_user | default('ubuntu') }}"
+      ansible_ssh_private_key_file: "{{ ssh_key_path }}"

playbook.yml

@@ -0,0 +1,7 @@
+---
+- name: Bioconductor Sync Playbook
+  hosts: bioconductor_vm
+  vars_files:
+    - vars/main.yml
+  roles:
+    - bioconductor_sync

roles/bioconductor_sync/defaults/main.yml

@@ -0,0 +1,4 @@
+---
+# Defaults
+bioc_version: "3.20"
+target_dir: "archive-move"

roles/bioconductor_sync/tasks/main.yml

@@ -0,0 +1,101 @@
+---
+# Tasks for Bioconductor sync
+
+- name: Check for rsync.pem SSH key
+  ansible.builtin.stat:
+    path: ~/.ssh/rsync.pem
+  register: rsync_key
+
+- name: Fail if SSH key is missing
+  ansible.builtin.fail:
+    msg: "Required SSH key not found at ~/.ssh/rsync.pem. Please add the key before continuing."
+  when: not rsync_key.stat.exists
+
+- name: Check SSH key permissions
+  ansible.builtin.file:
+    path: ~/.ssh/rsync.pem
+    mode: '0600'
+  when: rsync_key.stat.exists
+
+- name: Check for rclone installation
+  ansible.builtin.command: which rclone
+  register: rclone_check
+  ignore_errors: true
+  changed_when: false
+
+- name: Install rclone if not present
+  block:
+    - name: Download rclone install script
+      ansible.builtin.get_url:
+        url: https://rclone.org/install.sh
+        dest: /tmp/rclone_install.sh
+        mode: '0755'
+      
+    - name: Run rclone installer
+      ansible.builtin.command: /tmp/rclone_install.sh
+      become: yes
+  when: rclone_check.rc != 0
+
+- name: Check for rclone config
+  ansible.builtin.stat:
+    path: ~/.rclone.conf
+  register: rclone_config
+
+- name: Fail if rclone config is missing
+  ansible.builtin.fail:
+    msg: "Rclone configuration not found at ~/.rclone.conf. Please set up rclone configuration first."
+  when: not rclone_config.stat.exists
+
+- name: Check if OSN remote exists in rclone config
+  ansible.builtin.shell: grep -q "\[osn\]" ~/.rclone.conf
+  register: osn_remote_check
+  ignore_errors: true
+  changed_when: false
+
+- name: Fail if OSN remote is not configured
+  ansible.builtin.fail:
+    msg: "The [osn] remote is not configured in your rclone.conf. Please configure the OSN remote first."
+  when: osn_remote_check.rc != 0
+
+- name: Create target directory
+  ansible.builtin.file:
+    path: "{{ target_dir }}/{{ bioc_version }}"
+    state: directory
+    mode: '0755'
+
+- name: Create file list
+  ansible.builtin.template:
+    src: file_list.txt.j2
+    dest: "{{ target_dir }}/file_list.txt"
+
+- name: Create rsync script
+  ansible.builtin.copy:
+    dest: "{{ target_dir }}/rsync_copy.sh"
+    mode: '0755'
+    content: |
+      #!/bin/bash
+      cd {{ target_dir }}
+      cat file_list.txt | xargs -i bash -c 'mkdir -p {}'
+      cat file_list.txt | xargs -i bash -c "rsync -e 'ssh -i ~/.ssh/rsync.pem' -zrtlv --delete [email protected]:{} {}"
+  register: rsync_script
+
+- name: Execute rsync copy
+  ansible.builtin.shell: "{{ target_dir }}/rsync_copy.sh"
+  register: rsync_result
+  args:
+    executable: /bin/bash
+
+- name: Create rclone copy script
+  ansible.builtin.copy:
+    dest: "{{ target_dir }}/rclone_copy.sh"
+    mode: '0755'
+    content: |
+      #!/bin/bash
+      cd {{ target_dir }}
+      cat file_list.txt | xargs -i rclone copyto -vvv ./{} osn:/bir190004-bucket01/archive.bioconductor.org/packages/{}
+
+- name: Execute rclone copy
+  ansible.builtin.shell: "{{ target_dir }}/rclone_copy.sh"
+  register: rclone_result
+  args:
+    executable: /bin/bash

roles/bioconductor_sync/templates/file_list.txt.j2

@@ -0,0 +1,13 @@
+{{ bioc_version }}/books/bin
+{{ bioc_version }}/bioc/bin
+{{ bioc_version }}/data/annotation/bin
+{{ bioc_version }}/data/experiment/bin
+{{ bioc_version }}/workflows/bin
+{{ bioc_version }}/books/src
+{{ bioc_version }}/bioc/src
+{{ bioc_version }}/data/annotation/src
+{{ bioc_version }}/data/experiment/src
+{{ bioc_version }}/workflows/src
+{{ bioc_version }}/BiocViews
+{{ bioc_version }}/BiocViews.html
+{{ bioc_version }}/index.html

run.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+# Check if ansible is installed
+if ! command -v ansible-playbook &> /dev/null; then
+    echo "Error: ansible-playbook command not found. Please install Ansible first."
+    echo "You can install it using: pip install ansible"
+    exit 1
+fi
+
+if [ "$#" -lt 2 ]; then
+    echo "Usage: $0 <ssh_key_path> <ip_address> [ssh_user] [bioc_version]"
+    echo ""
+    echo "Parameters:"
+    echo "  ssh_key_path  : Path to SSH private key for connection"
+    echo "  ip_address    : IP address of the target server"
+    echo "  ssh_user      : SSH username (default: ubuntu)"
+    echo "  bioc_version  : Bioconductor version to sync (e.g. 3.21)"
+    exit 1
+fi
+
+SSH_KEY_PATH=$1
+IP_ADDRESS=$2
+SSH_USER=${3:-ubuntu}
+BIOC_VERSION=${4:-"3.21"}
+
+echo "Starting Bioconductor sync for version $BIOC_VERSION..."
+ansible-playbook -v playbook.yml -e "ssh_key_path=$SSH_KEY_PATH ip_address=$IP_ADDRESS ssh_user=$SSH_USER bioc_version=$BIOC_VERSION"

vars/main.yml

@@ -0,0 +1,5 @@
+---
+# Variables
+ssh_user: ubuntu  # default, override as needed
+bioc_version: "3.20"  # The Bioconductor version to sync
+target_dir: "archive-move"  # The target directory for the sync