Skip to content

Commit 368ea66

Browse files
ThomasWaldmannThomasWaldmann
authored
Merge pull request borgbackup#9643 from ThomasWaldmann/blake3
add blake3, blake2b is legacy
2 parents 4e8fa92 + 9395685 commit 368ea66

12 files changed

Lines changed: 236 additions & 158 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ jobs:
145145
{"os": "ubuntu-22.04", "python-version": "3.11", "toxenv": "docs"},
146146
{"os": "ubuntu-22.04", "python-version": "3.11", "toxenv": "py311-llfuse"},
147147
{"os": "ubuntu-24.04", "python-version": "3.12", "toxenv": "py312-pyfuse3"},
148-
{"os": "ubuntu-24.04", "python-version": "3.15-dev", "toxenv": "py315-mfusepy"}
148+
{"os": "ubuntu-24.04", "python-version": "3.14", "toxenv": "py314-mfusepy"}
149149
]
150150
}' || '{
151151
"include": [
@@ -534,7 +534,7 @@ jobs:
534534
;;
535535
536536
omnios)
537-
sudo pkg install gcc14 git pkg-config python-313 gnu-make gnu-coreutils
537+
sudo pkg install gcc14 git pkg-config python-313 gnu-make gnu-coreutils rust
538538
sudo ln -sf /usr/bin/python3.13 /usr/bin/python3
539539
sudo ln -sf /usr/bin/python3.13-config /usr/bin/python3-config
540540
sudo python3 -m ensurepip

‎docs/usage/transfer.rst‎

Lines changed: 64 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -2,52 +2,88 @@
22

33
Examples
44
~~~~~~~~
5+
6+
To keep the following examples short and readable, we export the repository
7+
locations and passphrases first:
8+
9+
::
10+
11+
export BORG_REPO=ssh://borg2@borgbackup/./tests/b20
12+
export BORG_PASSPHRASE='your-borg2-repo-passphrase'
13+
export BORG_OTHER_REPO=ssh://borg2@borgbackup/./tests/b1x
14+
export BORG_OTHER_PASSPHRASE='your-borg1-repo-passphrase'
15+
516
::
617

7-
# 0. Have Borg 2.0 installed on the client AND server; have a b12 repository copy for testing.
18+
# Borg 1.x repository -> Borg 2.0 repository (hmac-sha256 -> hmac-sha256, keeping the same chunk ID algorithm)
19+
20+
# 0. Have Borg 2.0 installed on the client AND server; have a Borg 1.x repository copy for testing.
821

922
# 1. Create a new "related" repository:
10-
# Here, the existing Borg 1.2 repository used repokey-blake2 (and AES-CTR mode),
11-
# thus we use repokey-blake2-aes-ocb for the new Borg 2.0 repository.
12-
# Staying with the same chunk ID algorithm (BLAKE2) and with the same
13-
# key material (via --other-repo <oldrepo>) will make deduplication work
23+
# Here, the existing Borg 1.x repository used repokey (and AES-CTR mode),
24+
# thus we use repokey-aes-ocb for the new Borg 2.0 repository.
25+
# Staying with the same chunk ID algorithm (hmac-sha256) and with the same
26+
# key material (via BORG_OTHER_REPO) will make deduplication work
1427
# between old archives (copied with borg transfer) and future ones.
1528
# The AEAD cipher does not matter (everything must be re-encrypted and
16-
# re-authenticated anyway); you could also choose repokey-blake2-chacha20-poly1305.
17-
# In case your old Borg repository did not use BLAKE2, just remove the "-blake2".
18-
$ borg --repo ssh://borg2@borgbackup/./tests/b20 repo-create \
19-
--other-repo ssh://borg2@borgbackup/./tests/b12 -e repokey-blake2-aes-ocb
29+
# re-authenticated anyway); you could also choose repokey-chacha20-poly1305.
30+
$ borg repo-create -e repokey-aes-ocb
31+
32+
# 2. Check what and how much it would transfer:
33+
$ borg transfer --from-borg1 --dry-run
34+
35+
# 3. Transfer (copy) archives from the old repository into the new repository (takes time and space!):
36+
$ borg transfer --from-borg1
37+
38+
# 4. Check whether we have everything (same as step 2):
39+
$ borg transfer --from-borg1 --dry-run
40+
41+
::
42+
43+
# Borg 1.x repository -> Borg 2.0 repository (blake2 -> blake3, changing the chunk ID algorithm)
44+
45+
# 0. Have Borg 2.0 installed on the client AND server; have a Borg 1.x repository copy for testing.
46+
47+
# 1. Create a new "related" repository:
48+
# Here, the existing Borg 1.x repository used repokey-blake2 (and AES-CTR mode),
49+
# thus we use repokey-blake3-aes-ocb for the new Borg 2.0 repository.
50+
# We need to change from blake2 to blake3, because blake2 is not supported
51+
# for borg2 repos (blake3 is much faster). Because we change how chunk IDs are
52+
# computed, we need to re-chunk everything while doing the transfer.
53+
# The chunker parameters you provide here should be the same as you will
54+
# use for all future Borg 2.0 archives.
55+
# The AEAD cipher does not matter (everything must be re-encrypted and
56+
# re-authenticated anyway); you could also choose repokey-blake3-chacha20-poly1305.
57+
$ borg repo-create -e repokey-blake3-aes-ocb
58+
$ export CHUNKER_PARAMS="buzhash64,19,23,21,4095"
2059

2160
# 2. Check what and how much it would transfer:
22-
$ borg --repo ssh://borg2@borgbackup/./tests/b20 transfer --upgrader=From12To20 \
23-
--other-repo ssh://borg2@borgbackup/./tests/b12 --dry-run
61+
$ borg transfer --from-borg1 --chunker-params=$CHUNKER_PARAMS --dry-run
2462

2563
# 3. Transfer (copy) archives from the old repository into the new repository (takes time and space!):
26-
$ borg --repo ssh://borg2@borgbackup/./tests/b20 transfer --upgrader=From12To20 \
27-
--other-repo ssh://borg2@borgbackup/./tests/b12
64+
$ borg transfer --from-borg1 --chunker-params=$CHUNKER_PARAMS
2865

2966
# 4. Check whether we have everything (same as step 2):
30-
$ borg --repo ssh://borg2@borgbackup/./tests/b20 transfer --upgrader=From12To20 \
31-
--other-repo ssh://borg2@borgbackup/./tests/b12 --dry-run
67+
$ borg transfer --from-borg1 --chunker-params=$CHUNKER_PARAMS --dry-run
3268

33-
Keyfile considerations when upgrading from borg 1.x
69+
Keyfile considerations when upgrading from Borg 1.x
3470
++++++++++++++++++++++++++++++++++++++++++++++++++++
3571

36-
If you are using a ``keyfile`` encryption mode (not ``repokey``), borg 2
37-
may not automatically find your borg 1.x key file, because the default
72+
If you are using a ``keyfile`` encryption mode (not ``repokey``), Borg 2
73+
may not automatically find your Borg 1.x key file, because the default
3874
key file directory has changed on some platforms due to the switch to
3975
the `platformdirs <https://pypi.org/project/platformdirs/>`_ library.
4076

41-
On **Linux**, there is typically no change -- both borg 1.x and borg 2
77+
On **Linux**, there is typically no change -- both Borg 1.x and Borg 2
4278
use ``~/.config/borg/keys/``.
4379

44-
On **macOS**, borg 1.x stored key files in ``~/.config/borg/keys/``,
45-
but borg 2 defaults to ``~/Library/Application Support/borg/keys/``.
80+
On **macOS**, Borg 1.x stored key files in ``~/.config/borg/keys/``,
81+
but Borg 2 defaults to ``~/Library/Application Support/borg/keys/``.
4682

47-
On **Windows**, borg 1.x used XDG-style paths (e.g. ``~/.config/borg/keys/``),
48-
while borg 2 defaults to ``C:\Users\<user>\AppData\Roaming\borg\keys\``.
83+
On **Windows**, Borg 1.x used XDG-style paths (e.g. ``~/.config/borg/keys/``),
84+
while Borg 2 defaults to ``C:\Users\<user>\AppData\Roaming\borg\keys\``.
4985

50-
If borg 2 cannot find your key file, you have several options:
86+
If Borg 2 cannot find your key file, you have several options:
5187

5288
1. **Copy the key file** from the old location to the new one.
5389
2. **Set BORG_KEYS_DIR** to point to the old key file directory::
@@ -58,13 +94,13 @@ If borg 2 cannot find your key file, you have several options:
5894

5995
export BORG_KEY_FILE=~/.config/borg/keys/your_key_file
6096

61-
4. **Set BORG_BASE_DIR** to force borg 2 to use the same base directory
62-
as borg 1.x::
97+
4. **Set BORG_BASE_DIR** to force Borg 2 to use the same base directory
98+
as Borg 1.x::
6399

64100
export BORG_BASE_DIR=$HOME
65101

66-
This makes borg 2 use ``$HOME/.config/borg``, ``$HOME/.cache/borg``,
67-
etc., matching borg 1.x behaviour on all platforms.
102+
This makes Borg 2 use ``$HOME/.config/borg``, ``$HOME/.cache/borg``,
103+
etc., matching Borg 1.x behavior on all platforms.
68104

69105
See :ref:`env_vars` for more details on directory environment variables.
70106

‎pyproject.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ dependencies = [
4242
"xxhash>=2.0.0",
4343
"jsonargparse>=4.47.0",
4444
"PyYAML>=6.0.2", # we need to register our types with yaml, jsonargparse uses yaml for config files
45+
"blake3>=1.0.0",
4546
]
4647

4748
[project.optional-dependencies]

‎scripts/msys2-install-deps‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
#!/bin/bash
22

3-
pacman -S --needed --noconfirm git mingw-w64-ucrt-x86_64-{toolchain,pkgconf,lz4,xxhash,openssl,rclone,python-msgpack,python-argon2_cffi,python-platformdirs,python,cython,python-setuptools,python-wheel,python-build,python-pkgconfig,python-packaging,python-pip,python-paramiko}
3+
pacman -S --needed --noconfirm git mingw-w64-ucrt-x86_64-{toolchain,pkgconf,lz4,xxhash,openssl,rclone,python-msgpack,python-argon2_cffi,python-platformdirs,python,cython,python-setuptools,python-wheel,python-build,python-pkgconfig,python-packaging,python-pip,python-paramiko,rust,python-maturin}
44

55
if [ "$1" = "development" ]; then
66
pacman -S --needed --noconfirm mingw-w64-ucrt-x86_64-python-{pytest,pytest-benchmark,pytest-cov,pytest-xdist}

‎src/borg/archiver/benchmark_cmd.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -214,16 +214,20 @@ def chunkit(ch):
214214
print(f"{spec:<24} {format_file_size(size):<10} {dt:.3f}s")
215215

216216
from ..crypto.low_level import hmac_sha256, blake2b_256
217+
import blake3
217218

218219
if not args.json:
219220
print("Cryptographic hashes / MACs ====================================")
220221
else:
221222
result["hashes"] = []
222223
size = 1000000000
223-
for spec, func in [
224+
hashes_tests = [
224225
("hmac-sha256", lambda: hmac_sha256(key_256, random_10M)),
225226
("blake2b-256", lambda: blake2b_256(key_256, random_10M)),
226-
]:
227+
("blake3", lambda: blake3.blake3(random_10M, key=key_256).digest()),
228+
("blake3-mt", lambda: blake3.blake3(random_10M, key=key_256, max_threads=blake3.blake3.AUTO).digest()),
229+
]
230+
for spec, func in hashes_tests:
227231
dt = timeit(func, number=number_default)
228232
if args.json:
229233
result["hashes"].append({"algo": spec, "size": size, "time": dt})

‎src/borg/archiver/key_cmds.py‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
import os
22

33
from ..constants import * # NOQA
4-
from ..crypto.key import AESOCBRepoKey, CHPORepoKey, Blake2AESOCBRepoKey, Blake2CHPORepoKey
5-
from ..crypto.key import AESOCBKeyfileKey, CHPOKeyfileKey, Blake2AESOCBKeyfileKey, Blake2CHPOKeyfileKey
4+
from ..crypto.key import AESOCBRepoKey, CHPORepoKey, Blake3AESOCBRepoKey, Blake3CHPORepoKey
5+
from ..crypto.key import AESOCBKeyfileKey, CHPOKeyfileKey, Blake3AESOCBKeyfileKey, Blake3CHPOKeyfileKey
66
from ..crypto.keymanager import KeyManager
77
from ..helpers import PathSpec, CommandError
88
from ..helpers.argparsing import ArgumentParser
@@ -40,10 +40,10 @@ def do_key_change_location(self, args, repository, manifest, cache):
4040
key_new = AESOCBKeyfileKey(repository)
4141
elif isinstance(key, CHPORepoKey):
4242
key_new = CHPOKeyfileKey(repository)
43-
elif isinstance(key, Blake2AESOCBRepoKey):
44-
key_new = Blake2AESOCBKeyfileKey(repository)
45-
elif isinstance(key, Blake2CHPORepoKey):
46-
key_new = Blake2CHPOKeyfileKey(repository)
43+
elif isinstance(key, Blake3AESOCBRepoKey):
44+
key_new = Blake3AESOCBKeyfileKey(repository)
45+
elif isinstance(key, Blake3CHPORepoKey):
46+
key_new = Blake3CHPOKeyfileKey(repository)
4747
else:
4848
print("Change not needed or not supported.")
4949
return
@@ -52,10 +52,10 @@ def do_key_change_location(self, args, repository, manifest, cache):
5252
key_new = AESOCBRepoKey(repository)
5353
elif isinstance(key, CHPOKeyfileKey):
5454
key_new = CHPORepoKey(repository)
55-
elif isinstance(key, Blake2AESOCBKeyfileKey):
56-
key_new = Blake2AESOCBRepoKey(repository)
57-
elif isinstance(key, Blake2CHPOKeyfileKey):
58-
key_new = Blake2CHPORepoKey(repository)
55+
elif isinstance(key, Blake3AESOCBKeyfileKey):
56+
key_new = Blake3AESOCBRepoKey(repository)
57+
elif isinstance(key, Blake3CHPOKeyfileKey):
58+
key_new = Blake3CHPORepoKey(repository)
5959
else:
6060
print("Change not needed or not supported.")
6161
return

‎src/borg/archiver/transfer_cmd.py‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -135,12 +135,11 @@ def do_transfer(self, args, *, repository, manifest, cache, other_repository=Non
135135
"""archives transfer from other repository, optionally upgrade data format"""
136136
key = manifest.key
137137
other_key = other_manifest.key
138-
if not uses_same_id_hash(other_key, key):
139-
raise Error(
140-
"You must keep the same ID hash ([HMAC-]SHA256 or BLAKE2b) or deduplication will break. "
141-
"Use a related repository!"
142-
)
143-
if not uses_same_chunker_secret(other_key, key):
138+
using_same_id_hash = uses_same_id_hash(other_key, key)
139+
rechunking = args.chunker_params is not None
140+
if not using_same_id_hash and not rechunking:
141+
raise Error("You must either keep the same ID hash or use --chunker-params.")
142+
if not rechunking and not uses_same_chunker_secret(other_key, key):
144143
raise Error(
145144
"You must use the same chunker secret or deduplication will break. " "Use a related repository!"
146145
)
@@ -311,13 +310,13 @@ def build_parser_transfer(self, subparsers, common_parser, mid_common_parser):
311310
borg --repo=DST_REPO transfer --other-repo=SRC_REPO # do it!
312311
borg --repo=DST_REPO transfer --other-repo=SRC_REPO --dry-run # check! anything left?
313312
314-
Data migration / upgrade from borg 1.x
313+
Data migration / upgrade from Borg 1.x
315314
++++++++++++++++++++++++++++++++++++++
316315
317-
To migrate your borg 1.x archives into a related, new borg2 repository, usage is quite similar
316+
To migrate your Borg 1.x archives into a related, new Borg 2 repository, usage is quite similar
318317
to the above, but you need the ``--from-borg1`` option::
319318
320-
borg --repo=DST_REPO repocreate --encryption=DST_ENC --other-repo=SRC_REPO --from-borg1
319+
borg --repo=DST_REPO repo-create --encryption=DST_ENC --other-repo=SRC_REPO --from-borg1
321320
322321
# to continue using lz4 compression as you did in SRC_REPO:
323322
borg --repo=DST_REPO transfer --other-repo=SRC_REPO --from-borg1 \\
@@ -336,7 +335,7 @@ def build_parser_transfer(self, subparsers, common_parser, mid_common_parser):
336335
subparser = ArgumentParser(
337336
parents=[common_parser], description=self.do_transfer.__doc__, epilog=transfer_epilog
338337
)
339-
subparsers.add_subcommand("transfer", subparser, help="transfer of archives from another repository")
338+
subparsers.add_subcommand("transfer", subparser, help="Transfer of archives from another repository")
340339
subparser.add_argument(
341340
"-n", "--dry-run", dest="dry_run", action="store_true", help="do not change repository, just check"
342341
)

‎src/borg/constants.py‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -189,10 +189,11 @@ class KeyType:
189189
AESOCBREPO = 0x11
190190
CHPOKEYFILE = 0x20
191191
CHPOREPO = 0x21
192-
BLAKE2AESOCBKEYFILE = 0x30
193-
BLAKE2AESOCBREPO = 0x31
194-
BLAKE2CHPOKEYFILE = 0x40
195-
BLAKE2CHPOREPO = 0x41
192+
BLAKE3AESOCBKEYFILE = 0x30
193+
BLAKE3AESOCBREPO = 0x31
194+
BLAKE3CHPOKEYFILE = 0x40
195+
BLAKE3CHPOREPO = 0x41
196+
BLAKE3AUTHENTICATED = 0x50
196197

197198

198199
CACHE_TAG_NAME = "CACHEDIR.TAG"

0 commit comments

Comments
 (0)