Backup Everything to Cloud

Automated backup system built with Node.js that compresses specified folders and PostgreSQL databases into tar.gz archives and uploads them to cloud storage providers (Google Drive or AWS S3), with automatic cleanup of old backups.

Key Features

✅ Compress multiple folders into tar.gz archives for backup
✅ PostgreSQL database dump backup (pg_dump -Fc)
✅ Multi-uploader support: Google Drive, AWS S3 (rclone or AWS SDK)
✅ Selective upload: Choose which storage provider(s) to use
✅ Personal Gmail account support (OAuth 2.0)
✅ Retention policy (automatically delete backups older than N days)
✅ 3 retries on upload failure (exponential backoff)
✅ Keep local backup on upload failure
✅ PM2 cron scheduling support

Prerequisites

Node.js >= 18.0.0
PostgreSQL client tools (pg_dump, psql)
Uploader-specific requirements:
- Google Drive: rclone
- S3 (rclone): rclone
- S3 (SDK): AWS SDK (auto-installed)

Installation

# 1. Clone repository
git clone <repository-url>
cd backup-everything-to-cloud

# 2. Install dependencies
npm install

# 3. Install rclone (for Google Drive or S3 rclone method)
# macOS
brew install rclone

# Linux
curl https://rclone.org/install.sh | sudo bash

# 4. Copy configuration files
cp .backup.example .backup
cp .config.example .config

Configuration

1. Uploader Setup

Option A: Google Drive (rclone)

rclone config

Setup process:

n (new remote)
name: Enter gdrive
Storage: Select drive (Google Drive)
Client ID/Secret: Press Enter (use defaults)
Scope: Select 1 (full access)
Root folder: Press Enter (default)
Service Account: n
Auto config: y → Browser opens → Google login → Grant permissions
Team Drive: n
y (confirm)
q (quit)

Option B: AWS S3 (rclone)

rclone config

Setup process:

n (new remote)
name: Enter s3
Storage: Select s3 (Amazon S3)
Provider: Select AWS
Credentials: 1 (Enter AWS credentials) or 2 (use environment variables)
Access Key ID: Enter your key
Secret Access Key: Enter your secret
Region: Enter us-east-1 or your desired region
y (confirm)
q (quit)

Or use environment variables (.env file):

AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key

Option C: AWS S3 (AWS SDK)

AWS SDK can be used without rclone configuration.

Authentication methods (in priority order):

Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
~/.aws/credentials file
IAM Role (when running on EC2/ECS)

AWS CLI setup (for option 2):

aws configure
# Access Key ID: [enter]
# Secret Access Key: [enter]
# Region: us-east-1
# Output format: json

2. .backup File Configuration

Add folder paths and PostgreSQL connection strings, one per line.

# Folder paths (absolute paths)
/home/user/important-data
/var/www/project1

# PostgreSQL connection strings
postgres://username:password@localhost:5432/mydb
postgres://user2:[email protected]:5432/production_db

3. .config File Configuration

{
  "retention_days": 7,
  "schedule": "0 2 * * *",
  "uploaders": [
    {
      "type": "gdrive",
      "enabled": true,
      "folder_path": "backups"
    },
    {
      "type": "s3-sdk",
      "enabled": true,
      "bucket": "my-backup-bucket",
      "prefix": "backups/",
      "region": "us-east-1",
      "storage_class": "STANDARD_IA"
    }
  ]
}

Uploader Types

Type	Description	Pros	Cons
`gdrive`	Google Drive (rclone)	Personal Gmail support, 15GB free	Requires rclone setup
`s3-rclone`	AWS S3 (rclone)	Consistent interface	Requires rclone setup
`s3-sdk`	AWS S3 (AWS SDK)	No rclone needed, native	AWS dependency

Uploader Configuration Fields

Common fields:

type: Uploader type (required)
enabled: Enable/disable (default: true)

gdrive specific:

remote_name: rclone remote name (default: gdrive)
folder_path: Google Drive folder path (e.g., backups)

s3-rclone specific:

remote_name: rclone remote name (default: s3)
bucket: S3 bucket name (required)
prefix: S3 object key prefix (e.g., backups/)
region: AWS region (default: us-east-1)
storage_class: Storage class (default: STANDARD)

s3-sdk specific:

bucket: S3 bucket name (required)
prefix: S3 object key prefix (e.g., backups/)
region: AWS region (default: us-east-1)
storage_class: Storage class (default: STANDARD)

S3 Storage Class Guide

Class	Use Case	Cost	Retrieval Speed
`STANDARD`	Frequently accessed data	High	Immediate
`STANDARD_IA`	<1x/month access	Medium	Immediate
`GLACIER`	Archive (1-2x/year)	Low	Minutes to hours
`INTELLIGENT_TIERING`	Auto-optimization	Auto	Immediate

Recommended for backups: STANDARD_IA (cost-effective, fast recovery)

Usage

Manual Execution

npm start

Automated with PM2 (Recommended)

# Install PM2 (globally)
npm install -g pm2

# Register with PM2 and start cron schedule
npm run pm2:start

# Check status
npm run pm2:status

# View logs
npm run pm2:logs

# Restart PM2
npm run pm2:restart

# Stop PM2
npm run pm2:stop

# Remove from PM2
npm run pm2:delete

Usage Examples

Example 1: Google Drive Only

{
  "retention_days": 7,
  "schedule": "0 2 * * *",
  "uploaders": [
    {
      "type": "gdrive",
      "enabled": true,
      "folder_path": "backups"
    }
  ]
}

Example 2: S3 Only (AWS SDK)

{
  "retention_days": 30,
  "schedule": "0 3 * * *",
  "uploaders": [
    {
      "type": "s3-sdk",
      "enabled": true,
      "bucket": "my-company-backups",
      "prefix": "daily/",
      "region": "ap-northeast-2",
      "storage_class": "STANDARD_IA"
    }
  ]
}

Example 3: Google Drive + S3 Simultaneous Upload

{
  "retention_days": 7,
  "schedule": "0 2 * * *",
  "uploaders": [
    {
      "type": "gdrive",
      "enabled": true,
      "folder_path": "backups"
    },
    {
      "type": "s3-sdk",
      "enabled": true,
      "bucket": "my-backup-bucket",
      "prefix": "backups/",
      "region": "us-east-1",
      "storage_class": "GLACIER"
    }
  ]
}

Example 4: Multiple S3 (rclone + SDK)

{
  "retention_days": 14,
  "schedule": "0 1 * * *",
  "uploaders": [
    {
      "type": "s3-rclone",
      "enabled": true,
      "bucket": "primary-backups",
      "prefix": "main/",
      "region": "us-east-1",
      "storage_class": "STANDARD_IA"
    },
    {
      "type": "s3-sdk",
      "enabled": true,
      "bucket": "secondary-backups",
      "prefix": "redundant/",
      "region": "eu-west-1",
      "storage_class": "GLACIER"
    }
  ]
}

Project Structure

.
├── src/
│   ├── backup.js           # Main backup orchestration
│   ├── config.js           # Configuration management
│   ├── uploaders/          # Uploader modules
│   │   ├── base.js         # Common interface
│   │   ├── factory.js      # Uploader factory
│   │   ├── gdrive.js       # Google Drive uploader
│   │   ├── s3-rclone.js    # S3 rclone uploader
│   │   └── s3-sdk.js       # S3 SDK uploader
│   ├── postgres.js         # PostgreSQL backup module
│   ├── compress.js         # Compression utilities
│   └── utils.js            # Common utility functions
├── .backup                 # Backup targets definition (gitignored)
├── .config                 # App configuration (gitignored)
├── package.json            # NPM package definition
└── README.md               # This file

How It Works

Load Configuration: Read .backup and .config files
Initialize Uploaders: Initialize all enabled uploaders and test connections
Folder Backup: Compress each folder into tar.gz
Database Backup: Create PostgreSQL dumps with pg_dump, then compress
Upload: Upload backup files to all enabled uploaders
Cleanup Old Backups: Delete backups older than retention_days from each storage
Local Cleanup: Delete local backup files after successful upload

Troubleshooting

rclone remote not configured

# Check rclone remotes
rclone listremotes

# If 'gdrive:' or 's3:' is missing, configure with rclone config
rclone config

Google Drive Connection Test

# Check Google Drive capacity
rclone about gdrive:

# Check backup folder
rclone lsjson gdrive:backups

S3 Connection Test

# rclone method
rclone lsd s3:my-bucket

# AWS CLI method
aws s3 ls s3://my-bucket/backups/

PostgreSQL Version Mismatch

If pg_dump and PostgreSQL server versions differ, --no-sync option is automatically added.

AWS Credentials Error

Check environment variables:

echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY

Check AWS CLI credentials:

cat ~/.aws/credentials

Check IAM Role (EC2):

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/

License

MIT

Contributing

Issues and PRs are welcome!

Name		Name	Last commit message	Last commit date
Latest commit History 12 Commits
.claude		.claude
src		src
.backup.example		.backup.example
.config.example		.config.example
.env.example		.env.example
.gitignore		.gitignore
ANALYSIS_REPORT.md		ANALYSIS_REPORT.md
CODE_COMMENTS_SUMMARY.md		CODE_COMMENTS_SUMMARY.md
README.ko.md		README.ko.md
README.md		README.md
check-drive-type.js		check-drive-type.js
package-lock.json		package-lock.json
package.json		package.json
test-gdrive.js		test-gdrive.js
yarn.lock		yarn.lock

GSK-KR/backup-everything-to-cloud

Folders and files

Latest commit

History

Repository files navigation