python setup.py build_usage

docs/usage/check.rst.inc

@@ -138,15 +138,18 @@ Second, the consistency and correctness of the archive metadata is verified:
 The ``--max-duration`` option can be used to split a long-running repository check
 into multiple partial checks. After the given number of seconds the check is
 interrupted. The next partial check will continue where the previous one stopped,
-until the complete repository has been checked. Example: Assuming a full check took 7
+until the complete repository has been checked. Example: Assuming a complete check took 7
 hours, then running a daily check with --max-duration=3600 (1 hour) resulted in one
-full check per week.
-
-Attention: Partial checks can only do way less checking than a full check (only the
-CRC32 checks on segment file entries are done), and cannot be combined with the
-``--repair`` option. Partial checks may therefore be useful only with very large
-repositories where a full check took too long. Doing a full repository check aborts a
-partial check; the next partial check will restart from the beginning.
+completed check per week.
+
+Attention: A partial --repository-only check can only do way less checking than a full
+--repository-only check: only the non-cryptographic checksum checks on segment file
+entries are done, while a full --repository-only check would also do a repo index check.
+A partial check cannot be combined with the ``--repair`` option. Partial checks
+may therefore be useful only with very large repositories where a full check would take
+too long.
+Doing a full repository check aborts a partial check; the next partial check will restart
+from the beginning.
 
 The ``--verify-data`` option will perform a full integrity verification (as opposed to
 checking the CRC32 of the segment) of data, which means reading the data from the

docs/usage/compact.rst.inc

@@ -71,6 +71,7 @@ When using ``--verbose``, borg will output an estimate of the freed space.
 
 After upgrading borg (server) to 1.2+, you can use ``borg compact --cleanup-commits``
 to clean up the numerous 17byte commit-only segments that borg 1.1 did not clean up
-due to a bug. It is enough to do that once per repository.
+due to a bug. It is enough to do that once per repository. After cleaning up the
+commits, borg will also do a normal compaction.
 
 See :ref:`separate_compaction` in Additional Notes for more details.

docs/usage/create.rst.inc

@@ -37,7 +37,7 @@ borg create
     +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
     |                                                       | ``--stdin-user USER``                             | set user USER in archive for stdin data (default: 'root')                                                                                            |
     +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--stdin-group GROUP``                           | set group GROUP in archive for stdin data (default: 'wheel')                                                                                         |
+    |                                                       | ``--stdin-group GROUP``                           | set group GROUP in archive for stdin data (default: 'root')                                                                                          |
     +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
     |                                                       | ``--stdin-mode M``                                | set mode to M in archive for stdin data (default: 0660)                                                                                              |
     +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
@@ -139,7 +139,7 @@ borg create
         --no-cache-sync    experimental: do not synchronize the cache. Implies not using the files cache.
         --stdin-name NAME    use NAME in archive for stdin data (default: 'stdin')
         --stdin-user USER    set user USER in archive for stdin data (default: 'root')
-        --stdin-group GROUP    set group GROUP in archive for stdin data (default: 'wheel')
+        --stdin-group GROUP    set group GROUP in archive for stdin data (default: 'root')
         --stdin-mode M    set mode to M in archive for stdin data (default: 0660)
         --content-from-command    interpret PATH as command and store its stdout. See also section Reading from stdin below.
         --paths-from-stdin    read DELIM-separated list of paths to backup from stdin. Will not recurse into directories.

docs/usage/export-tar.rst.inc

@@ -27,6 +27,8 @@ borg export-tar
     +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+
     |                                                       | ``--list``                            | output verbose list of items (files, dirs, ...)                                                           |
     +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--tar-format FMT``                  | select tar format: BORG, PAX or GNU                                                                       |
+    +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+
     | .. class:: borg-common-opt-ref                                                                                                                                                                            |
     |                                                                                                                                                                                                           |
     | :ref:`common_options`                                                                                                                                                                                     |
@@ -63,8 +65,9 @@ borg export-tar
 
 
     optional arguments
-        --tar-filter     filter program to pipe data through
-        --list           output verbose list of items (files, dirs, ...)
+        --tar-filter         filter program to pipe data through
+        --list               output verbose list of items (files, dirs, ...)
+        --tar-format FMT     select tar format: BORG, PAX or GNU
 
 
     :ref:`common_options`
@@ -99,12 +102,18 @@ Alternatively, a ``--tar-filter`` program may be explicitly specified. It should
 read the uncompressed tar stream from stdin and write a compressed/filtered
 tar stream to stdout.
 
-The generated tarball uses the GNU tar format.
-
-export-tar is a lossy conversion:
-BSD flags, ACLs, extended attributes (xattrs), atime and ctime are not exported.
-Timestamp resolution is limited to whole seconds, not the nanosecond resolution
-otherwise supported by Borg.
+Depending on the ``-tar-format`` option, these formats are created:
+
++--------------+---------------------------+----------------------------+
+| --tar-format | Specification             | Metadata                   |
++--------------+---------------------------+----------------------------+
+| BORG         | BORG specific, like PAX   | all as supported by borg   |
++--------------+---------------------------+----------------------------+
+| PAX          | POSIX.1-2001 (pax) format | GNU + atime/ctime/mtime ns |
++--------------+---------------------------+----------------------------+
+| GNU          | GNU tar format            | mtime s, no atime/ctime,   |
+|              |                           | no ACLs/xattrs/bsdflags    |
++--------------+---------------------------+----------------------------+
 
 A ``--sparse`` option (as found in borg extract) is not supported.
 

docs/usage/help.rst.inc

@@ -249,11 +249,11 @@ and ``--remote-path`` values support these placeholders:
 
 {now}
     The current local date and time, by default in ISO-8601 format.
-    You can also supply your own `format string <https://docs.python.org/3.8/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {now:%Y-%m-%d_%H:%M:%S}
+    You can also supply your own `format string <https://docs.python.org/3.9/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {now:%Y-%m-%d_%H:%M:%S}
 
 {utcnow}
     The current UTC date and time, by default in ISO-8601 format.
-    You can also supply your own `format string <https://docs.python.org/3.8/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S}
+    You can also supply your own `format string <https://docs.python.org/3.9/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S}
 
 {user}
     The user name (or UID, if no name is available) of the user running borg.

docs/usage/import-tar.rst.inc

@@ -106,12 +106,15 @@ stdout.
 Most documentation of borg create applies. Note that this command does not
 support excluding files.
 
-import-tar is a lossy conversion:
-BSD flags, ACLs, extended attributes (xattrs), atime and ctime are not exported.
-Timestamp resolution is limited to whole seconds, not the nanosecond resolution
-otherwise supported by Borg.
-
 A ``--sparse`` option (as found in borg create) is not supported.
 
-import-tar reads POSIX.1-1988 (ustar), POSIX.1-2001 (pax), GNU tar, UNIX V7 tar
-and SunOS tar with extended attributes.
+About tar formats and metadata conservation or loss, please see ``borg export-tar``.
+
+import-tar reads these tar formats:
+
+- BORG: borg specific (PAX-based)
+- PAX: POSIX.1-2001
+- GNU: GNU tar
+- POSIX.1-1988 (ustar)
+- UNIX V7 tar
+- SunOS tar with extended attributes

docs/usage/init.rst.inc

@@ -27,6 +27,8 @@ borg init
     +-------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
     |                                                       | ``--make-parent-dirs``             | create the parent directories of the repository directory, if they are missing.                                                                                                                                                        |
     +-------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--key-algorithm``                | the algorithm we use to derive a key encryption key from your passphrase. Default: argon2                                                                                                                                              |
+    +-------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
     | .. class:: borg-common-opt-ref                                                                                                                                                                                                                                                                                                      |
     |                                                                                                                                                                                                                                                                                                                                     |
     | :ref:`common_options`                                                                                                                                                                                                                                                                                                               |
@@ -51,6 +53,7 @@ borg init
         --append-only                  create an append-only mode repository. Note that this only affects the low level structure of the repository, and running `delete` or `prune` will still be allowed. See :ref:`append_only_mode` in Additional Notes for more details.
         --storage-quota QUOTA          Set storage quota of the new repository (e.g. 5G, 1.5T). Default: no quota.
         --make-parent-dirs             create the parent directories of the repository directory, if they are missing.
+        --key-algorithm                the algorithm we use to derive a key encryption key from your passphrase. Default: argon2
 
 
     :ref:`common_options`
@@ -65,26 +68,24 @@ directory containing the deduplicated data from zero or more archives.
 Encryption mode TLDR
 ++++++++++++++++++++
 
-The encryption mode can only be configured when creating a new repository -
-you can neither configure it on a per-archive basis nor change the
-encryption mode of an existing repository.
+The encryption mode can only be configured when creating a new repository - you can
+neither configure it on a per-archive basis nor change the mode of an existing repository.
+This example will likely NOT give optimum performance on your machine (performance
+tips will come below):
 
-Use ``repokey``::
+::
 
     borg init --encryption repokey /path/to/repo
 
-Or ``repokey-blake2`` depending on which is faster on your client machines (see below)::
-
-    borg init --encryption repokey-blake2 /path/to/repo
-
 Borg will:
 
 1. Ask you to come up with a passphrase.
-2. Create a borg key (which contains 3 random secrets. See :ref:`key_files`).
-3. Encrypt the key with your passphrase.
-4. Store the encrypted borg key inside the repository directory (in the repo config).
+2. Create a borg key (which contains some random secrets. See :ref:`key_files`).
+3. Derive a "key encryption key" from your passphrase
+4. Encrypt and sign the key with the key encryption key
+5. Store the encrypted borg key inside the repository directory (in the repo config).
    This is why it is essential to use a secure passphrase.
-5. Encrypt and sign your backups to prevent anyone from reading or forging them unless they
+6. Encrypt and sign your backups to prevent anyone from reading or forging them unless they
    have the key and know the passphrase. Make sure to keep a backup of
    your key **outside** the repository - do not lock yourself out by
    "leaving your keys inside your car" (see :ref:`borg_key_export`).
@@ -92,7 +93,7 @@ Borg will:
    never sees your passphrase, your unencrypted key or your unencrypted files.
    Chunking and id generation are also based on your key to improve
    your privacy.
-6. Use the key when extracting files to decrypt them and to verify that the contents of
+7. Use the key when extracting files to decrypt them and to verify that the contents of
    the backups have not been accidentally or maliciously altered.
 
 Picking a passphrase
@@ -116,76 +117,69 @@ a different keyboard layout.
 You can change your passphrase for existing repos at any time, it won't affect
 the encryption/decryption key or other secrets.
 
-More encryption modes
-+++++++++++++++++++++
+Choosing an encryption mode
++++++++++++++++++++++++++++
 
-Only use ``--encryption none`` if you are OK with anyone who has access to
-your repository being able to read your backups and tamper with their
-contents without you noticing.
+Depending on your hardware, hashing and crypto performance may vary widely.
+The easiest way to find out about what's fastest is to run ``borg benchmark cpu``.
 
-If you want "passphrase and having-the-key" security, use ``--encryption keyfile``.
-The key will be stored in your home directory (in ``~/.config/borg/keys``).
+`repokey` modes: if you want ease-of-use and "passphrase" security is good enough -
+the key will be stored in the repository (in ``repo_dir/config``).
 
-If you do **not** want to encrypt the contents of your backups, but still
-want to detect malicious tampering use ``--encryption authenticated``.
+`keyfile` modes: if you rather want "passphrase and having-the-key" security -
+the key will be stored in your home directory (in ``~/.config/borg/keys``).
 
-If ``BLAKE2b`` is faster than ``SHA-256`` on your hardware, use ``--encryption authenticated-blake2``,
-``--encryption repokey-blake2`` or ``--encryption keyfile-blake2``. Note: for remote backups
-the hashing is done on your local machine.
+The following table is roughly sorted in order of preference, the better ones are
+in the upper part of the table, in the lower part is the old and/or unsafe(r) stuff:
 
 .. nanorst: inline-fill
 
-+----------+---------------+------------------------+--------------------------+
-| Hash/MAC | Not encrypted | Not encrypted,         | Encrypted (AEAD w/ AES)  |
-|          | no auth       | but authenticated      | and authenticated        |
-+----------+---------------+------------------------+--------------------------+
-| SHA-256  | none          | `authenticated`        | repokey                  |
-|          |               |                        | keyfile                  |
-+----------+---------------+------------------------+--------------------------+
-| BLAKE2b  | n/a           | `authenticated-blake2` | `repokey-blake2`         |
-|          |               |                        | `keyfile-blake2`         |
-+----------+---------------+------------------------+--------------------------+
++---------------------------------+---------------+---------------+------------------+-------+
+|**mode (* = keyfile or repokey)**|**ID-Hash**    |**Encryption** |**Authentication**|**V>=**|
++---------------------------------+---------------+---------------+------------------+-------+
+| ``*-blake2-chacha20-poly1305``  | BLAKE2b       | CHACHA20      | POLY1305         | 1.3   |
++---------------------------------+---------------+---------------+------------------+-------+
+| ``*-chacha20-poly1305``         | HMAC-SHA-256  | CHACHA20      | POLY1305         | 1.3   |
++---------------------------------+---------------+---------------+------------------+-------+
+| ``*-blake2-aes-ocb``            | BLAKE2b       | AES256-OCB    | AES256-OCB       | 1.3   |
++---------------------------------+---------------+---------------+------------------+-------+
+| ``*-aes-ocb``                   | HMAC-SHA-256  | AES256-OCB    | AES256-OCB       | 1.3   |
++---------------------------------+---------------+---------------+------------------+-------+
+| ``*-blake2``                    | BLAKE2b       | AES256-CTR    | BLAKE2b          | 1.1   |
++---------------------------------+---------------+---------------+------------------+-------+
+| ``*``                           | HMAC-SHA-256  | AES256-CTR    | HMAC-SHA256      | any   |
++---------------------------------+---------------+---------------+------------------+-------+
+| authenticated-blake2            | BLAKE2b       | none          | BLAKE2b          | 1.1   |
++---------------------------------+---------------+---------------+------------------+-------+
+| authenticated                   | HMAC-SHA-256  | none          | HMAC-SHA256      | 1.1   |
++---------------------------------+---------------+---------------+------------------+-------+
+| none                            | SHA-256       | none          | none             | any   |
++---------------------------------+---------------+---------------+------------------+-------+
 
 .. nanorst: inline-replace
 
-Modes `marked like this` in the above table are new in Borg 1.1 and are not
-backwards-compatible with Borg 1.0.x.
-
-On modern Intel/AMD CPUs (except very cheap ones), AES is usually
-hardware-accelerated.
-BLAKE2b is faster than SHA256 on Intel/AMD 64-bit CPUs
-(except AMD Ryzen and future CPUs with SHA extensions),
-which makes `authenticated-blake2` faster than `none` and `authenticated`.
-
-On modern ARM CPUs, NEON provides hardware acceleration for SHA256 making it faster
-than BLAKE2b-256 there. NEON accelerates AES as well.
-
-Hardware acceleration is always used automatically when available.
-
-`repokey` and `keyfile` use AES-CTR-256 for encryption and HMAC-SHA256 for
-authentication in an encrypt-then-MAC (EtM) construction. The chunk ID hash
-is HMAC-SHA256 as well (with a separate key).
-These modes are compatible with Borg 1.0.x.
-
-`repokey-blake2` and `keyfile-blake2` are also authenticated encryption modes,
-but use BLAKE2b-256 instead of HMAC-SHA256 for authentication. The chunk ID
-hash is a keyed BLAKE2b-256 hash.
-These modes are new and *not* compatible with Borg 1.0.x.
-
-`authenticated` mode uses no encryption, but authenticates repository contents
-through the same HMAC-SHA256 hash as the `repokey` and `keyfile` modes (it uses it
-as the chunk ID hash). The key is stored like `repokey`.
-This mode is new and *not* compatible with Borg 1.0.x.
-
-`authenticated-blake2` is like `authenticated`, but uses the keyed BLAKE2b-256 hash
-from the other blake2 modes.
-This mode is new and *not* compatible with Borg 1.0.x.
-
-`none` mode uses no encryption and no authentication. It uses SHA256 as chunk
-ID hash. This mode is not recommended, you should rather consider using an authenticated
-or authenticated/encrypted mode. This mode has possible denial-of-service issues
-when running ``borg create`` on contents controlled by an attacker.
-Use it only for new repositories where no encryption is wanted **and** when compatibility
-with 1.0.x is important. If compatibility with 1.0.x is not important, use
-`authenticated-blake2` or `authenticated` instead.
-This mode is compatible with Borg 1.0.x.
+`none` mode uses no encryption and no authentication. You're advised to NOT use this mode
+as it would expose you to all sorts of issues (DoS, confidentiality, tampering, ...) in
+case of malicious activity in the repository.
+
+If you do **not** want to encrypt the contents of your backups, but still want to detect
+malicious tampering use an `authenticated` mode. It's like `repokey` minus encryption.
+
+Key derivation functions
+++++++++++++++++++++++++
+
+- ``--key-algorithm argon2`` is the default and is recommended.
+  The key encryption key is derived from your passphrase via argon2-id.
+  Argon2 is considered more modern and secure than pbkdf2.
+
+- You can use ``--key-algorithm pbkdf2`` if you want to access your repo via old versions of borg.
+
+Our implementation of argon2-based key algorithm follows the cryptographic best practices:
+
+- It derives two separate keys from your passphrase: one to encrypt your key and another one
+  to sign it. ``--key-algorithm pbkdf2`` uses the same key for both.
+
+- It uses encrypt-then-mac instead of encrypt-and-mac used by ``--key-algorithm pbkdf2``
+
+Neither is inherently linked to the key derivation function, but since we were going
+to break backwards compatibility anyway we took the opportunity to fix all 3 issues at once.

docs/usage/key_change-algorithm.rst.inc

@@ -0,0 +1,76 @@
+.. IMPORTANT: this file is auto-generated from borg's built-in help, do not edit!
+
+.. _borg_key_change-algorithm:
+
+borg key change-algorithm
+-------------------------
+.. code-block:: none
+
+    borg [common options] key change-algorithm [options] [REPOSITORY] ALGORITHM
+
+.. only:: html
+
+    .. class:: borg-options-table
+
+    +-------------------------------------------------------+----------------+----------------------+
+    | **positional arguments**                                                                      |
+    +-------------------------------------------------------+----------------+----------------------+
+    |                                                       | ``REPOSITORY`` |                      |
+    +-------------------------------------------------------+----------------+----------------------+
+    |                                                       | ``ALGORITHM``  | select key algorithm |
+    +-------------------------------------------------------+----------------+----------------------+
+    | .. class:: borg-common-opt-ref                                                                |
+    |                                                                                               |
+    | :ref:`common_options`                                                                         |
+    +-------------------------------------------------------+----------------+----------------------+
+
+    .. raw:: html
+
+        <script type='text/javascript'>
+        $(document).ready(function () {
+            $('.borg-options-table colgroup').remove();
+        })
+        </script>
+
+.. only:: latex
+
+    REPOSITORY
+
+    ALGORITHM
+        select key algorithm
+
+
+    :ref:`common_options`
+        |
+
+Description
+~~~~~~~~~~~
+
+Change the algorithm we use to encrypt and authenticate the borg key.
+
+Important: In a `repokey` mode (e.g. repokey-blake2) all users share the same key.
+In this mode upgrading to `argon2` will make it impossible to access the repo for users who use an old version of borg.
+We recommend upgrading to the latest stable version.
+
+Important: In a `keyfile` mode (e.g. keyfile-blake2) each user has their own key (in ``~/.config/borg/keys``).
+In this mode this command will only change the key used by the current user.
+If you want to upgrade to `argon2` to strengthen security, you will have to upgrade each user's key individually.
+
+Your repository is encrypted and authenticated with a key that is randomly generated by ``borg init``.
+The key is encrypted and authenticated with your passphrase.
+
+We currently support two choices:
+
+1. argon2 - recommended. This algorithm is used by default when initialising a new repository.
+   The key encryption key is derived from your passphrase via argon2-id.
+   Argon2 is considered more modern and secure than pbkdf2.
+2. pbkdf2 - the legacy algorithm. Use this if you want to access your repo via old versions of borg.
+   The key encryption key is derived from your passphrase via PBKDF2-HMAC-SHA256.
+
+Examples::
+
+    # Upgrade an existing key to argon2
+    borg key change-algorithm /path/to/repo argon2
+    # Downgrade to pbkdf2 - use this if upgrading borg is not an option
+    borg key change-algorithm /path/to/repo pbkdf2
+

docs/usage/list.rst.inc

@@ -113,7 +113,7 @@ The FORMAT specifier syntax
 +++++++++++++++++++++++++++
 
 The ``--format`` option uses python's `format string syntax
-<https://docs.python.org/3.8/library/string.html#formatstrings>`_.
+<https://docs.python.org/3.9/library/string.html#formatstrings>`_.
 
 Examples:
 ::

docs/usage/upgrade.rst.inc

@@ -99,47 +99,14 @@ See
 https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability
 for details.
 
-Attic and Borg 0.xx to Borg 1.x
-+++++++++++++++++++++++++++++++
+Borg 0.xx to Borg 1.x
++++++++++++++++++++++
 
-This currently supports converting an Attic repository to Borg and also
-helps with converting Borg 0.xx to 1.0.
+This currently supports converting Borg 0.xx to 1.0.
 
 Currently, only LOCAL repositories can be upgraded (issue #465).
 
 Please note that ``borg create`` (since 1.0.0) uses bigger chunks by
-default than old borg or attic did, so the new chunks won't deduplicate
+default than old borg did, so the new chunks won't deduplicate
 with the old chunks in the upgraded repository.
-See ``--chunker-params`` option of ``borg create`` and ``borg recreate``.
-
-``borg upgrade`` will change the magic strings in the repository's
-segments to match the new Borg magic strings. The keyfiles found in
-$ATTIC_KEYS_DIR or ~/.attic/keys/ will also be converted and
-copied to $BORG_KEYS_DIR or ~/.config/borg/keys.
-
-The cache files are converted, from $ATTIC_CACHE_DIR or
-~/.cache/attic to $BORG_CACHE_DIR or ~/.cache/borg, but the
-cache layout between Borg and Attic changed, so it is possible
-the first backup after the conversion takes longer than expected
-due to the cache resync.
-
-Upgrade should be able to resume if interrupted, although it
-will still iterate over all segments. If you want to start
-from scratch, use `borg delete` over the copied repository to
-make sure the cache files are also removed::
-
-    borg delete borg
-
-Unless ``--inplace`` is specified, the upgrade process first creates a backup
-copy of the repository, in REPOSITORY.before-upgrade-DATETIME, using hardlinks.
-This requires that the repository and its parent directory reside on same
-filesystem so the hardlink copy can work.
-This takes longer than in place upgrades, but is much safer and gives
-progress information (as opposed to ``cp -al``). Once you are satisfied
-with the conversion, you can safely destroy the backup copy.
-
-WARNING: Running the upgrade in place will make the current
-copy unusable with older version, with no way of going back
-to previous versions. This can PERMANENTLY DAMAGE YOUR
-REPOSITORY!  Attic CAN NOT READ BORG REPOSITORIES, as the
-magic strings have changed. You have been warned.
+See ``--chunker-params`` option of ``borg create`` and ``borg recreate``.