Restructured DR backup section for clarity and thoroughness

src/current/_includes/v25.4/backups/backup-storage-collision.md

@@ -1 +1 @@
-You will encounter an error if you run multiple [backup collections]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#backup-collections) to the same storage URI. Each collection's URI must be unique.
+You will encounter an error if you run multiple [backup collections]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#backup-collections) to the same storage URI. Backup collections can contain multiple full and incremental backups, but each collection's URI must be unique. If you are using backup schedules, each schedule must have a unique URI.

src/current/v25.4/cluster-virtualization-overview.md

@@ -66,7 +66,7 @@ When cluster virtualization is enabled, [backup]({% link {{ page.version.version
 - If your deployment contains system-level customizations, you can take a separate backup of the system virtual cluster to capture them.
 - A backup of a virtual cluster can be restored as a virtual cluster in any CockroachDB cluster with cluster virtualization enabled.
 
-For more details about backing up and restoring a cluster with cluster virtualization enabled, refer to [Work with virtual clusters]({% link {{ page.version.version }}/work-with-virtual-clusters.md %}#disaster-recovery).
+For more details about backing up and restoring a cluster with cluster virtualization enabled, refer to [Work with virtual clusters]({% link {{ page.version.version }}/work-with-virtual-clusters.md %}#backup-and-restore).
 
 {{site.data.alerts.callout_success}}
 For details about configuring and using PCR for disaster recovery, refer to [Physical Cluster Replication Overview]({% link {{ page.version.version }}/physical-cluster-replication-overview.md %}).

src/current/v25.4/work-with-virtual-clusters.md

@@ -114,45 +114,182 @@ sql_txn_commit_count{tenant="demo"} 0
 
 When connected to a virtual cluster from the DB Console, metrics which measure SQL and related activity show data scoped to the virtual cluster. All other metrics are collected system-wide and display the same data on all virtual clusters including the system virtual cluster.
 
-## Disaster recovery
+## Backup and restore
 
-When cluster virtualization is enabled, [backup]({% link {{ page.version.version }}/backup.md %}) and [restore]({% link {{ page.version.version }}/restore.md %}) commands are scoped to the virtual cluster by default.
+Cockroach Labs recommends that you regularly [back up]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#full-backups) your data. When using virtual clusters, perform backups on the _application virtual cluster (app VC)_. Only the app VC's data and settings are included in these backups, and data and settings for other virtual clusters or for the _system virtual cluster (system VC)_ are omitted. 
 
-### Back up a virtual cluster
+### Create a backup schedule for your app VC
 
-To back up a virtual cluster:
+Cockroach Labs recommends using [backup schedules]({% link {{ page.version.version }}/create-schedule-for-backup.md %}) to automate full and incremental backups of your data.
 
-1. [Connect to the virtual cluster](#connect-to-a-virtual-cluster) you want to back up as a user with the `admin` role on the virtual cluster.
-1. [Back up the cluster]({% link {{ page.version.version }}/backup.md %}). Only the virtual cluster's data and settings are included in the backup, and data and settings for other virtual clusters or for the system virtual cluster is omitted.
+Use the following process to create a schedule for a cluster-level backup of your app VC. In this example, the schedule takes revision history for the backup every day at midnight.
 
-For details about restoring a backup of a virtual cluster, refer to [Restore a virtual cluster](#restore-a-virtual-cluster).
+1. [Connect](#connect-to-a-virtual-cluster) to the app VC as a user with [supported privileges]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) on the app VC. In this example the user has the `BACKUP` privilege.
 
-### Back up the entire cluster
+    {% include_cached copy-clipboard.html %}
+    ~~~ shell
+    cockroach sql --url \
+    "postgresql://root@{primary node IP or hostname}:26257?options=-ccluster={app_virtual_cluster_name}&sslmode=verify-full" \
+    --certs-dir "certs"
+    ~~~
 
-To back up the entire CockroachDB cluster, including all virtual clusters and the system virtual cluster:
+1. [Create a backup schedule]({% link {{ page.version.version }}/create-schedule-for-backup.md %}#create-a-scheduled-backup-for-a-cluster):
 
-1. [Connect to the system virtual cluster](#connect-to-the-system-virtual-cluster) as a user with the `admin` role on the system virtual cluster.
-1. [Back up the cluster]({% link {{ page.version.version }}/backup.md %}), and include the `INCLUDE_ALL_SECONDARY_TENANTS` flag in the `BACKUP` command. All virtual clusters and the system virtual cluster are included in the backup.
+    {% include_cached copy-clipboard.html %}
+    ~~~ sql
+    CREATE SCHEDULE schedule_label
+    FOR BACKUP INTO 's3://test/backups/schedule_test?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=x'
+        WITH revision_history
+        RECURRING '@daily';
+    ~~~
 
-### Restore a virtual cluster
+    ~~~
+        schedule_id     |     name       |                     status                     |            first_run             | schedule |                                                                               backup_stmt
+    ---------------------+----------------+------------------------------------------------+----------------------------------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------
+      588796190000218113 | schedule_label | PAUSED: Waiting for initial backup to complete | NULL                             | @daily   | BACKUP INTO LATEST IN 's3://test/schedule-test?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=x' WITH revision_history, detached
+      588796190012702721 | schedule_label | ACTIVE                                         | 2020-09-10 16:52:17.280821+00:00 | @weekly  | BACKUP INTO 's3://test/schedule-test?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=x' WITH revision_history, detached
+    (2 rows)
+    ~~~
+
+    {% include {{ page.version.version }}/backups/backup-storage-collision.md %}
+
+For information on scheduling backups at different levels or with other options, consult [CREATE SCHEDULE FOR BACKUP]({% link {{ page.version.version }}/create-schedule-for-backup.md %}).
+
+### Back up your app VC
+
+Use the following process to take a one-off full backup of your app VC.
+
+1. [Connect](#connect-to-a-virtual-cluster) to the app VC as a user with [supported privileges]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) on the app VC. In this example the user has the `BACKUP` privilege.
+
+    {% include_cached copy-clipboard.html %}
+    ~~~ shell
+    cockroach sql --url \
+    "postgresql://root@{primary node IP or hostname}:26257?options=-ccluster={app_virtual_cluster_name}&sslmode=verify-full" \
+    --certs-dir "certs"
+    ~~~
+
+1. [Perform a full backup]({% link {{ page.version.version }}/backup.md %}#back-up-a-cluster):
+
+    {% include_cached copy-clipboard.html %}
+    ~~~ sql
+    BACKUP INTO 'external://backup_s3/app' AS OF SYSTEM TIME '-10s';
+    ~~~
+
+### Back up a database from your app VC
+
+Use the following process to take a one-off database-level backup on your app VC.
+
+1. [Connect](#connect-to-a-virtual-cluster) to the app VC as a user with [supported privileges]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) on the app VC. In this example the user has the `BACKUP` privilege.
+
+    {% include_cached copy-clipboard.html %}
+    ~~~ shell
+    cockroach sql --url \
+    "postgresql://root@{primary node IP or hostname}:26257?options=-ccluster={app_virtual_cluster_name}&sslmode=verify-full" \
+    --certs-dir "certs"
+    ~~~
+
+1. [Back up]({% link {{ page.version.version }}/backup.md %}#back-up-a-database) a single database:
+
+    {% include_cached copy-clipboard.html %}
+    ~~~ sql
+    BACKUP DATABASE bank INTO 'external://backup_s3/app/db' AS OF SYSTEM TIME '-10s';
+    ~~~
+
+    Or back up multiple databases:
+
+    {% include_cached copy-clipboard.html %}
+    ~~~ sql
+    BACKUP DATABASE bank, employees INTO 'external://backup_s3/app/db' AS OF SYSTEM TIME '-10s';
+    ~~~
+
+### Back up a table or view from your app VC
+
+Use the following procecss to take a one-off table-level or view-level backup on your app VC.
 
-You can restore a backup of a virtual cluster to:
+1. [Connect](#connect-to-a-virtual-cluster) to the app VC as a user with [supported privileges]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) on the app VC. In this example the user has the `BACKUP` privilege.
 
-- The original virtual cluster on the original CockroachDB cluster.
-- A different virtual cluster on the original CockroachDB cluster.
-- A different virtual cluster on a different CockroachDB cluster with cluster virtualization enabled.
+    {% include_cached copy-clipboard.html %}
+    ~~~ shell
+    cockroach sql --url \
+    "postgresql://root@{primary node IP or hostname}:26257?options=-ccluster={app_virtual_cluster_name}&sslmode=verify-full" \
+    --certs-dir "certs"
+    ~~~
+
+1. [Back up]({% link {{ page.version.version }}/backup.md %}#back-up-a-table-or-view) a single table or view:
+
+    {% include_cached copy-clipboard.html %}
+    ~~~ sql
+    BACKUP bank.customers INTO 'external://backup_s3/app/table' AS OF SYSTEM TIME '-10s';
+    ~~~
+
+    Or back up multiple tables:
+
+    {% include_cached copy-clipboard.html %}
+    ~~~ sql
+    BACKUP bank.customers, bank.accounts INTO 'external://backup_s3/app/table' AS OF SYSTEM TIME '-10s';
+    ~~~
+
+### Back up a schema from your app VC
+
+Use the following procecss to take a one-off schema-level backup on your app VC using a wildcard (`*`).
+
+1. [Connect](#connect-to-a-virtual-cluster) to the app VC as a user with [supported privileges]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) on the app VC. In this example the user has the `BACKUP` privilege.
+
+    {% include_cached copy-clipboard.html %}
+    ~~~ shell
+    cockroach sql --url \
+    "postgresql://root@{primary node IP or hostname}:26257?options=-ccluster={app_virtual_cluster_name}&sslmode=verify-full" \
+    --certs-dir "certs"
+    ~~~
+
+1. [Back up]({% link {{ page.version.version }}/backup.md %}#back-up-all-tables-in-a-schema) all tables in a schema:
+
+    {% include_cached copy-clipboard.html %}
+    ~~~ sql
+    BACKUP test_schema.* INTO 'external://backup_s3/app/schema' AS OF SYSTEM TIME '-10s';
+    ~~~
+
+### Back up your system VC
+
+You can also back up your system VC to preserve metadata such as users and cluster settings. Use the following process to back up your system VC.
+
+1. [Connect](#connect-to-the-system-virtual-cluster) to the system VC as a user with [supported privileges]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) on the system VC. In this example the user has the `BACKUP` privilege.
+
+    {% include_cached copy-clipboard.html %}
+    ~~~ shell
+    cockroach sql --url \
+    "postgresql://root@{primary node IP or hostname}:26257?options=-ccluster=system&sslmode=verify-full" \
+    --certs-dir "certs"
+    ~~~
+
+1. [Perform a full backup]({% link {{ page.version.version }}/backup.md %}#back-up-a-cluster):
+
+    {% include_cached copy-clipboard.html %}
+    ~~~ sql
+    BACKUP INTO 'external://backup_s3/system' AS OF SYSTEM TIME '-10s';
+    ~~~
+
+    {% include {{ page.version.version }}/backups/backup-storage-collision.md %}
+
+### Restore a virtual cluster
 
-To restore only a virtual cluster:
+If needed, you can restore backups to a new app VC with no user-created databases or tables. To restore your app VC from the latest full backup:
 
-1. [Connect to the destination virtual cluster](#connect-to-a-virtual-cluster) as a user with the `admin` role on the virtual cluster.
-1. [Restore the cluster]({% link {{ page.version.version }}/restore.md %}). Only the virtual cluster's data and settings are restored.
+1. [Connect to the destination app VC](#connect-to-a-virtual-cluster) as a user with [supported privileges]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) on the system VC. In this example the user has the `RESTORE` privilege.
 
-### Restore the entire cluster
+    {% include_cached copy-clipboard.html %}
+    ~~~ shell
+    cockroach sql --url \
+    "postgresql://root@{primary node IP or hostname}:26257?options=-ccluster={app_virtual_cluster_name}&sslmode=verify-full" \
+    --certs-dir "certs"
+    ~~~
 
-To restore the entire CockroachDB cluster, including all virtual clusters and the system virtual cluster:
+1. [Restore the cluster]({% link {{ page.version.version }}/restore.md %}):
 
-1. [Connect to the destination system virtual cluster](#connect-to-the-system-virtual-cluster) as a user with the `admin` role on the system virtual cluster.
-1. [Restore the cluster]({% link {{ page.version.version }}/restore.md %}) from a backup that included the the `INCLUDE_ALL_SECONDARY_VIRTUAL_CLUSTERS` flag. All virtual clusters and the system virtual cluster are restored.
+    {% include_cached copy-clipboard.html %}
+    ~~~ sql
+    RESTORE FROM LATEST IN 's3://bucket/path?AUTH=implicit';
+    ~~~
 
 ## Configure cluster settings