AllFiles_Archive_Compress

#!/bin/bash

# Author: Claire Bradley; claire.bradley@uq.edu.au
# Here are a few commands that I use to archive and compress files.
# This allows for much more efficient storage on the servers, and supposedly better transfer (I have not experienced this myself yet)
# The idea going forward is to run some computations in a temporary worspace (e.g.INode2), archive the results there, and transfer to UQ-RDM storage (e.g. /Volumes/XXX-QXXX), then delete from INode2.
# Ideally one copy of 'sourcedata', one copy of 'BIDSlikeraw' data and one copy of the results ('derivatives').


## ------- Taring a directory, with compression, while excluding some of its subdirectories or files
# 1. You want to place yourself in the directory above the one you want to tar, so that the whole path to the directory is not 'carried' into the archive. Consequently, upon extracting files in another location, you will end up with the directory of interest, instead of a path through all previous directories (example: extracting: data/... instead of extracting: Volumes/CBradleyQBI/TMS-EEG-Visual/data/...).
cd /Volumes/CBradleyQBI/TMS-EEG-Visual

# 2. The 'j' option indicates compression with bzip2. 'sourcedata' is excluded because it contains identifiable human data; 'derivatives' is excluded here because it does not contain analyses of interest (just a few tests at this stage)
tar -cvjf /Volumes/group_cogneuro/ClaireB/TMS-EEG-Visual/BIDSlikeraw.tar.bz2 --exclude=data/sourcedata --exclude=data/derivatives data


## ------- Listing contents of archive:
# beware, this seems to take a really long time for large archives...
cd /Volumes/group_cogneuro/ClaireB/TMS-EEG-Visual
tar -tvjf BIDSlikeraw.tar.bz2


## ------- Extracting contents of archive:
# Not sure yet how it merges with existing data; or if it replaces it rather than merging
cd /Volumes/group_cogneuro/ClaireB/TMS-EEG-Visual
tar -xvjf BIDSlikeraw.tar.bz2