Merge pull request #6463 from ThomasWaldmann/new-crypto

new AEAD crypto with session keys

docs/faq.rst

@@ -89,7 +89,7 @@ Also, you must not run borg against multiple instances of the same repo
   (which is an issue if they happen to be not the same).
   See :issue:`4272` for an example.
 - Encryption security issues if you would update repo and copy-of-repo
-  independently, due to AES counter reuse.
+  independently, due to AES counter reuse (when using legacy encryption modes).
 
 See also: :ref:`faq_corrupt_repo`
 
@@ -246,6 +246,8 @@ then use ``tar`` to perform the comparison:
 My repository is corrupt, how can I restore from an older copy of it?
 ---------------------------------------------------------------------
 
+Note: this is only required for repos using legacy encryption modes.
+
 If your repositories are encrypted and have the same ID, the recommended method
 is to delete the corrupted repository, but keep its security info, and then copy
 the working repository to the same location:
@@ -473,8 +475,11 @@ Security
 
 .. _borg_security_critique:
 
-Isn't BorgBackup's AES-CTR crypto broken?
------------------------------------------
+Isn't BorgBackup's legacy AES-CTR-based crypto broken?
+------------------------------------------------------
+
+Note: in borg 1.3 new AEAD cipher based modes with session keys were added,
+solving the issues of the legacy modes.
 
 If a nonce (counter) value is reused, AES-CTR mode crypto is broken.
 
@@ -713,6 +718,8 @@ Please disclose security issues responsibly.
 How important are the nonce files?
 ------------------------------------
 
+This only applies to repositories using legacy encryption modes.
+
 Borg uses :ref:`AES-CTR encryption <borg_security_critique>`. An
 essential part of AES-CTR is a sequential counter that must **never**
 repeat. If the same value of the counter is used twice in the same repository,
@@ -881,14 +888,14 @@ What's the expected backup performance?
 ---------------------------------------
 
 Compared to simply copying files (e.g. with ``rsync``), Borg has more work to do.
-This can make creation of the first archive slower, but saves time 
+This can make creation of the first archive slower, but saves time
 and disk space on subsequent runs. Here what Borg does when you run ``borg create``:
 
 - Borg chunks the file (using the relatively expensive buzhash algorithm)
-- It then computes the "id" of the chunk (hmac-sha256 (often slow, except 
+- It then computes the "id" of the chunk (hmac-sha256 (often slow, except
   if your CPU has sha256 acceleration) or blake2b (fast, in software))
-- Then it checks whether this chunk is already in the repo (local hashtable lookup, 
-  fast). If so, the processing of the chunk is completed here. Otherwise it needs to 
+- Then it checks whether this chunk is already in the repo (local hashtable lookup,
+  fast). If so, the processing of the chunk is completed here. Otherwise it needs to
   process the chunk:
 - Compresses (the default lz4 is super fast)
 - Encrypts (AES, usually fast if your CPU has AES acceleration as usual
@@ -896,9 +903,9 @@ and disk space on subsequent runs. Here what Borg does when you run ``borg creat
 - Authenticates ("signs") using hmac-sha256 or blake2b (see above),
 - Transmits to repo. If the repo is remote, this usually involves an SSH connection
   (does its own encryption / authentication).
-- Stores the chunk into a key/value store (the key is the chunk id, the value 
+- Stores the chunk into a key/value store (the key is the chunk id, the value
   is the data). While doing that, it computes a CRC32 of the data (repo low-level
-  checksum, used by borg check --repository) and also updates the repo index 
+  checksum, used by borg check --repository) and also updates the repo index
   (another hashtable).
 
 Subsequent backups are usually very fast if most files are unchanged and only
@@ -928,14 +935,14 @@ If you feel your Borg backup is too slow somehow, here is what you can do:
 
 - Make sure Borg has enough RAM (depends on how big your repo is / how many
   files you have)
-- Use one of the blake2 modes for --encryption except if you positively know 
+- Use one of the blake2 modes for --encryption except if you positively know
   your CPU (and openssl) accelerates sha256 (then stay with hmac-sha256).
 - Don't use any expensive compression. The default is lz4 and super fast.
   Uncompressed is often slower than lz4.
 - Just wait. You can also interrupt it and start it again as often as you like,
   it will converge against a valid "completed" state (see ``--checkpoint-interval``,
   maybe use the default, but in any case don't make it too short). It is starting
-  from the beginning each time, but it is still faster then as it does not store 
+  from the beginning each time, but it is still faster then as it does not store
   data into the repo which it already has there from last checkpoint.
 - If you don’t need additional file attributes, you can disable them with ``--noflags``,
   ``--noacls``, ``--noxattrs``. This can lead to noticable performance improvements
@@ -945,12 +952,12 @@ If you feel that Borg "freezes" on a file, it could be in the middle of processi
 large file (like ISOs or VM images). Borg < 1.2 announces file names *after* finishing
 with the file. This can lead to displaying the name of a small file, while processing the
 next (larger) file. For very big files this can lead to the progress display show some
-previous short file for a long time while it processes the big one. With Borg 1.2 this 
+previous short file for a long time while it processes the big one. With Borg 1.2 this
 was changed to announcing the filename before starting to process it.
 
 To see what files have changed and take more time processing, you can also add
-``--list --filter=AME --stats`` to your ``borg create`` call to produce more log output, 
-including a file list (with file status characters) and also some statistics at 
+``--list --filter=AME --stats`` to your ``borg create`` call to produce more log output,
+including a file list (with file status characters) and also some statistics at
 the end of the backup.
 
 Then you do the backup and look at the log output:

docs/internals/data-structures.rst

@@ -865,6 +865,31 @@ Encryption
 
 .. seealso:: The :ref:`borgcrypto` section for an in-depth review.
 
+AEAD modes
+~~~~~~~~~~
+
+Uses modern AEAD ciphers: AES-OCB or CHACHA20-POLY1305.
+For each borg invocation, a new sessionkey is derived from the borg key material
+and the 48bit IV starts from 0 again (both ciphers internally add a 32bit counter
+to our IV, so we'll just count up by 1 per chunk).
+
+The chunk layout is best seen at the bottom of this diagram:
+
+.. figure:: encryption-aead.png
+    :figwidth: 100%
+    :width: 100%
+
+No special IV/counter management is needed here due to the use of session keys.
+
+A 48 bit IV is way more than needed: If you only backed up 4kiB chunks (2^12B),
+the IV would "limit" the data encrypted in one session to 2^(12+48)B == 2.3 exabytes,
+meaning you would run against other limitations (RAM, storage, time) way before that.
+In practice, chunks are usually bigger, for big files even much bigger, giving an
+even higher limit.
+
+Legacy modes
+~~~~~~~~~~~~
+
 AES_-256 is used in CTR mode (so no need for padding). A 64 bit initialization
 vector is used, a MAC is computed on the encrypted chunk
 and both are stored in the chunk. Encryption and MAC use two different keys.
@@ -884,6 +909,9 @@ To reduce payload size, only 8 bytes of the 16 bytes nonce is saved in the
 payload, the first 8 bytes are always zeros. This does not affect security but
 limits the maximum repository capacity to only 295 exabytes (2**64 * 16 bytes).
 
+Both modes
+~~~~~~~~~~
+
 Encryption keys (and other secrets) are kept either in a key file on the client
 ('keyfile' mode) or in the repository config on the server ('repokey' mode).
 In both cases, the secrets are generated from random and then encrypted by a

docs/internals/security.rst

@@ -124,7 +124,88 @@ prompt is a set BORG_PASSPHRASE. See issue :issue:`2169` for details.
 Encryption
 ----------
 
-Encryption is currently based on the Encrypt-then-MAC construction,
+AEAD modes
+~~~~~~~~~~
+
+Modes: --encryption (repokey|keyfile)-[blake2-](aes-ocb|chacha20-poly1305)
+
+Supported: borg 1.3+
+
+Encryption with these modes is based on AEAD ciphers (authenticated encryption
+with associated data) and session keys.
+
+Depending on the chosen mode (see :ref:`borg_init`) different AEAD ciphers are used:
+
+- AES-256-OCB - super fast, single-pass algorithm IF you have hw accelerated AES.
+- chacha20-poly1305 - very fast, purely software based AEAD cipher.
+
+The chunk ID is derived via a MAC over the plaintext (mac key taken from borg key):
+
+- HMAC-SHA256 - super fast IF you have hw accelerated SHA256.
+- Blake2b - very fast, purely software based algorithm.
+
+For each borg invocation, a new session id is generated by `os.urandom`_.
+
+From that session id, the initial key material (ikm, taken from the borg key)
+and an application and cipher specific salt, borg derives a session key via HKDF.
+
+For each session key, IVs (nonces) are generated by a counter which increments for
+each encrypted message.
+
+Session::
+
+    sessionid = os.urandom(24)
+    ikm = enc_key || enc_hmac_key
+    salt = "borg-session-key-CIPHERNAME"
+    sessionkey = HKDF(ikm, sessionid, salt)
+    message_iv = 0
+
+Encryption::
+
+    id = MAC(id_key, data)
+    compressed = compress(data)
+
+    header = type-byte || 00h || message_iv || sessionid
+    aad = id || header
+    message_iv++
+    encrypted, auth_tag = AEAD_encrypt(session_key, message_iv, compressed, aad)
+    authenticated = header || auth_tag || encrypted
+
+Decryption::
+
+    # Given: input *authenticated* data and a *chunk-id* to assert
+    type-byte, past_message_iv, past_sessionid, auth_tag, encrypted = SPLIT(authenticated)
+
+    ASSERT(type-byte is correct)
+
+    past_key = HKDF(ikm, past_sessionid, salt)
+    decrypted = AEAD_decrypt(past_key, past_message_iv, authenticated)
+
+    decompressed = decompress(decrypted)
+
+    ASSERT( CONSTANT-TIME-COMPARISON( chunk-id, MAC(id_key, decompressed) ) )
+
+Notable:
+
+- More modern and often faster AEAD ciphers instead of self-assembled stuff.
+- Due to the usage of session keys, IVs (nonces) do not need special care here as
+  they did for the legacy encryption modes.
+- The id is now also input into the authentication tag computation.
+  This strongly associates the id with the written data (== associates the key with
+  the value). When later reading the data for some id, authentication will only
+  succeed if what we get was really written by us for that id.
+
+
+Legacy modes
+~~~~~~~~~~~~
+
+Modes: --encryption (repokey|keyfile)-[blake2]
+
+Supported: all borg versions, blake2 since 1.1
+
+DEPRECATED. We strongly suggest you use the safer AEAD modes, see above.
+
+Encryption with these modes is based on the Encrypt-then-MAC construction,
 which is generally seen as the most robust way to create an authenticated
 encryption scheme from encryption and message authentication primitives.
 
@@ -137,7 +218,7 @@ in the future.
 
 Depending on the chosen mode (see :ref:`borg_init`) different primitives are used:
 
-- The actual encryption is currently always AES-256 in CTR mode. The
+- Legacy encryption modes use AES-256 in CTR mode. The
   counter is added in plaintext, since it is needed for decryption,
   and is also tracked locally on the client to avoid counter reuse.
 
@@ -253,7 +334,7 @@ Implementations used
 We do not implement cryptographic primitives ourselves, but rely
 on widely used libraries providing them:
 
-- AES-CTR and HMAC-SHA-256 from OpenSSL 1.0 / 1.1 are used,
+- AES-CTR, AES-OCB, CHACHA20-POLY1305 and HMAC-SHA-256 from OpenSSL 1.1 are used,
   which is also linked into the static binaries we provide.
   We think this is not an additional risk, since we don't ever
   use OpenSSL's networking, TLS or X.509 code, but only their
@@ -268,7 +349,8 @@ on widely used libraries providing them:
 
 Implemented cryptographic constructions are:
 
-- Encrypt-then-MAC based on AES-256-CTR and either HMAC-SHA-256
+- AEAD modes: AES-OCB and CHACHA20-POLY1305 are straight from OpenSSL.
+- Legacy modes: Encrypt-then-MAC based on AES-256-CTR and either HMAC-SHA-256
   or keyed BLAKE2b256 as described above under Encryption_.
 - Encrypt-and-MAC based on AES-256-CTR and HMAC-SHA-256
   as described above under `Offline key security`_.

docs/quickstart.rst

@@ -387,6 +387,7 @@ For automated backups the passphrase can be specified using the
     A backup inside of the backup that is encrypted with that key/passphrase
     won't help you with that, of course.
 
+    Only applies to repos using legacy encryption modes:
     In case you lose your repository and the security information, but have an
     older copy of it to restore from, don't use that later for creating new
     backups – you would run into security issues (reuse of nonce counter

docs/usage/init.rst

@@ -4,16 +4,19 @@ Examples
 ~~~~~~~~
 ::
 
-    # Local repository, repokey encryption, BLAKE2b (often faster, since Borg 1.1)
-    $ borg init --encryption=repokey-blake2 /path/to/repo
+    # Local repository, recommended repokey AEAD crypto modes
+    $ borg init --encryption=repokey-aes-ocb /path/to/repo
+    $ borg init --encryption=repokey-chacha20-poly1305 /path/to/repo
+    $ borg init --encryption=repokey-blake2-aes-ocb /path/to/repo
+    $ borg init --encryption=repokey-blake2-chacha20-poly1305 /path/to/repo
 
-    # Local repository (no encryption)
+    # Local repository (no encryption), not recommended
     $ borg init --encryption=none /path/to/repo
 
     # Remote repository (accesses a remote borg via ssh)
     # repokey: stores the (encrypted) key into <REPO_DIR>/config
-    $ borg init --encryption=repokey-blake2 user@hostname:backup
+    $ borg init --encryption=repokey-aes-ocb user@hostname:backup
 
     # Remote repository (accesses a remote borg via ssh)
     # keyfile: stores the (encrypted) key into ~/.config/borg/keys/
-    $ borg init --encryption=keyfile user@hostname:backup
+    $ borg init --encryption=keyfile-aes-ocb user@hostname:backup

src/borg/archive.py

@@ -1789,7 +1789,7 @@ class ArchiveChecker:
 
         def add_callback(chunk):
             id_ = self.key.id_hash(chunk)
-            cdata = self.key.encrypt(chunk)
+            cdata = self.key.encrypt(id_, chunk)
             add_reference(id_, len(chunk), len(cdata), cdata)
             return id_
 
@@ -1811,7 +1811,7 @@ class ArchiveChecker:
             def replacement_chunk(size):
                 chunk = Chunk(None, allocation=CH_ALLOC, size=size)
                 chunk_id, data = cached_hash(chunk, self.key.id_hash)
-                cdata = self.key.encrypt(data)
+                cdata = self.key.encrypt(chunk_id, data)
                 csize = len(cdata)
                 return chunk_id, size, csize, cdata
 
@@ -1998,7 +1998,7 @@ class ArchiveChecker:
                 archive.items = items_buffer.chunks
                 data = msgpack.packb(archive.as_dict())
                 new_archive_id = self.key.id_hash(data)
-                cdata = self.key.encrypt(data)
+                cdata = self.key.encrypt(new_archive_id, data)
                 add_reference(new_archive_id, len(data), len(cdata), cdata)
                 self.manifest.archives[info.name] = (new_archive_id, info.ts)
             pi.finish()

src/borg/archiver.py

@@ -604,9 +604,9 @@ class Archiver:
         if not is_libressl:
             tests.extend([
                 ("aes-256-ocb", lambda: AES256_OCB(
-                    None, key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
+                    key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
                 ("chacha20-poly1305", lambda: CHACHA20_POLY1305(
-                    None, key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
+                    key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
             ])
         for spec, func in tests:
             print(f"{spec:<24} {size:<10} {timeit(func, number=100):.3f}s")
@@ -4184,22 +4184,19 @@ class Archiver:
         Encryption mode TLDR
         ++++++++++++++++++++
 
-        The encryption mode can only be configured when creating a new repository -
-        you can neither configure it on a per-archive basis nor change the
-        encryption mode of an existing repository.
+        The encryption mode can only be configured when creating a new repository - you can
+        neither configure it on a per-archive basis nor change the mode of an existing repository.
+        This example will likely NOT give optimum performance on your machine (performance
+        tips will come below):
 
-        Use ``repokey``::
+        ::
 
             borg init --encryption repokey /path/to/repo
 
-        Or ``repokey-blake2`` depending on which is faster on your client machines (see below)::
-
-            borg init --encryption repokey-blake2 /path/to/repo
-
         Borg will:
 
         1. Ask you to come up with a passphrase.
-        2. Create a borg key (which contains 3 random secrets. See :ref:`key_files`).
+        2. Create a borg key (which contains some random secrets. See :ref:`key_files`).
         3. Encrypt the key with your passphrase.
         4. Store the encrypted borg key inside the repository directory (in the repo config).
            This is why it is essential to use a secure passphrase.
@@ -4235,79 +4232,53 @@ class Archiver:
         You can change your passphrase for existing repos at any time, it won't affect
         the encryption/decryption key or other secrets.
 
-        More encryption modes
-        +++++++++++++++++++++
+        Choosing an encryption mode
+        +++++++++++++++++++++++++++
 
-        Only use ``--encryption none`` if you are OK with anyone who has access to
-        your repository being able to read your backups and tamper with their
-        contents without you noticing.
+        Depending on your hardware, hashing and crypto performance may vary widely.
+        The easiest way to find out about what's fastest is to run ``borg benchmark cpu``.
 
-        If you want "passphrase and having-the-key" security, use ``--encryption keyfile``.
-        The key will be stored in your home directory (in ``~/.config/borg/keys``).
+        `repokey` modes: if you want ease-of-use and "passphrase" security is good enough -
+        the key will be stored in the repository (in ``repo_dir/config``).
 
-        If you do **not** want to encrypt the contents of your backups, but still
-        want to detect malicious tampering use ``--encryption authenticated``.
+        `keyfile` modes: if you rather want "passphrase and having-the-key" security -
+        the key will be stored in your home directory (in ``~/.config/borg/keys``).
 
-        If ``BLAKE2b`` is faster than ``SHA-256`` on your hardware, use ``--encryption authenticated-blake2``,
-        ``--encryption repokey-blake2`` or ``--encryption keyfile-blake2``. Note: for remote backups
-        the hashing is done on your local machine.
+        The following table is roughly sorted in order of preference, the better ones are
+        in the upper part of the table, in the lower part is the old and/or unsafe(r) stuff:
 
         .. nanorst: inline-fill
 
-        +----------+---------------+------------------------+--------------------------+
-        | Hash/MAC | Not encrypted | Not encrypted,         | Encrypted (AEAD w/ AES)  |
-        |          | no auth       | but authenticated      | and authenticated        |
-        +----------+---------------+------------------------+--------------------------+
-        | SHA-256  | none          | `authenticated`        | repokey                  |
-        |          |               |                        | keyfile                  |
-        +----------+---------------+------------------------+--------------------------+
-        | BLAKE2b  | n/a           | `authenticated-blake2` | `repokey-blake2`         |
-        |          |               |                        | `keyfile-blake2`         |
-        +----------+---------------+------------------------+--------------------------+
+        +---------------------------------+---------------+---------------+------------------+-------+
+        |**mode (* = keyfile or repokey)**|**ID-Hash**    |**Encryption** |**Authentication**|**V>=**|
+        +---------------------------------+---------------+---------------+------------------+-------+
+        | ``*-blake2-chacha20-poly1305``  | BLAKE2b       | CHACHA20      | POLY1305         | 1.3   |
+        +---------------------------------+---------------+---------------+------------------+-------+
+        | ``*-chacha20-poly1305``         | HMAC-SHA-256  | CHACHA20      | POLY1305         | 1.3   |
+        +---------------------------------+---------------+---------------+------------------+-------+
+        | ``*-blake2-aes-ocb``            | BLAKE2b       | AES256-OCB    | AES256-OCB       | 1.3   |
+        +---------------------------------+---------------+---------------+------------------+-------+
+        | ``*-aes-ocb``                   | HMAC-SHA-256  | AES256-OCB    | AES256-OCB       | 1.3   |
+        +---------------------------------+---------------+---------------+------------------+-------+
+        | ``*-blake2``                    | BLAKE2b       | AES256-CTR    | BLAKE2b          | 1.1   |
+        +---------------------------------+---------------+---------------+------------------+-------+
+        | ``*``                           | HMAC-SHA-256  | AES256-CTR    | HMAC-SHA256      | any   |
+        +---------------------------------+---------------+---------------+------------------+-------+
+        | authenticated-blake2            | BLAKE2b       | none          | BLAKE2b          | 1.1   |
+        +---------------------------------+---------------+---------------+------------------+-------+
+        | authenticated                   | HMAC-SHA-256  | none          | HMAC-SHA256      | 1.1   |
+        +---------------------------------+---------------+---------------+------------------+-------+
+        | none                            | SHA-256       | none          | none             | any   |
+        +---------------------------------+---------------+---------------+------------------+-------+
 
         .. nanorst: inline-replace
 
-        Modes `marked like this` in the above table are new in Borg 1.1 and are not
-        backwards-compatible with Borg 1.0.x.
-
-        On modern Intel/AMD CPUs (except very cheap ones), AES is usually
-        hardware-accelerated.
-        BLAKE2b is faster than SHA256 on Intel/AMD 64-bit CPUs
-        (except AMD Ryzen and future CPUs with SHA extensions),
-        which makes `authenticated-blake2` faster than `none` and `authenticated`.
-
-        On modern ARM CPUs, NEON provides hardware acceleration for SHA256 making it faster
-        than BLAKE2b-256 there. NEON accelerates AES as well.
-
-        Hardware acceleration is always used automatically when available.
-
-        `repokey` and `keyfile` use AES-CTR-256 for encryption and HMAC-SHA256 for
-        authentication in an encrypt-then-MAC (EtM) construction. The chunk ID hash
-        is HMAC-SHA256 as well (with a separate key).
-        These modes are compatible with Borg 1.0.x.
-
-        `repokey-blake2` and `keyfile-blake2` are also authenticated encryption modes,
-        but use BLAKE2b-256 instead of HMAC-SHA256 for authentication. The chunk ID
-        hash is a keyed BLAKE2b-256 hash.
-        These modes are new and *not* compatible with Borg 1.0.x.
-
-        `authenticated` mode uses no encryption, but authenticates repository contents
-        through the same HMAC-SHA256 hash as the `repokey` and `keyfile` modes (it uses it
-        as the chunk ID hash). The key is stored like `repokey`.
-        This mode is new and *not* compatible with Borg 1.0.x.
-
-        `authenticated-blake2` is like `authenticated`, but uses the keyed BLAKE2b-256 hash
-        from the other blake2 modes.
-        This mode is new and *not* compatible with Borg 1.0.x.
-
-        `none` mode uses no encryption and no authentication. It uses SHA256 as chunk
-        ID hash. This mode is not recommended, you should rather consider using an authenticated
-        or authenticated/encrypted mode. This mode has possible denial-of-service issues
-        when running ``borg create`` on contents controlled by an attacker.
-        Use it only for new repositories where no encryption is wanted **and** when compatibility
-        with 1.0.x is important. If compatibility with 1.0.x is not important, use
-        `authenticated-blake2` or `authenticated` instead.
-        This mode is compatible with Borg 1.0.x.
+        `none` mode uses no encryption and no authentication. You're advised to NOT use this mode
+        as it would expose you to all sorts of issues (DoS, confidentiality, tampering, ...) in
+        case of malicious activity in the repository.
+
+        If you do **not** want to encrypt the contents of your backups, but still want to detect
+        malicious tampering use an `authenticated` mode. It's like `repokey` minus encryption.
         """)
         subparser = subparsers.add_parser('init', parents=[common_parser], add_help=False,
                                           description=self.do_init.__doc__, epilog=init_epilog,

src/borg/cache.py

@@ -942,7 +942,7 @@ class LocalCache(CacheStatsMixin):
         refcount = self.seen_chunk(id, size)
         if refcount and not overwrite:
             return self.chunk_incref(id, stats)
-        data = self.key.encrypt(chunk)
+        data = self.key.encrypt(id, chunk)
         csize = len(data)
         self.repository.put(id, data, wait=wait)
         self.chunks.add(id, 1, size, csize)
@@ -1107,7 +1107,7 @@ Chunk index:    {0.total_unique_chunks:20d}             unknown"""
         refcount = self.seen_chunk(id, size)
         if refcount:
             return self.chunk_incref(id, stats, size=size)
-        data = self.key.encrypt(chunk)
+        data = self.key.encrypt(id, chunk)
         csize = len(data)
         self.repository.put(id, data, wait=wait)
         self.chunks.add(id, 1, size, csize)

src/borg/constants.py

@@ -111,6 +111,8 @@ class KeyBlobStorage:
 
 
 class KeyType:
+    # legacy crypto
+    # upper 4 bits are ciphersuite, 0 == legacy AES-CTR
     KEYFILE = 0x00
     # repos with PASSPHRASE mode could not be created any more since borg 1.0, see #97.
     # in borg 1.3 all of its code and also the "borg key migrate-to-repokey" command was removed.
@@ -123,6 +125,16 @@ class KeyType:
     BLAKE2REPO = 0x05
     BLAKE2AUTHENTICATED = 0x06
     AUTHENTICATED = 0x07
+    # new crypto
+    # upper 4 bits are ciphersuite, lower 4 bits are keytype
+    AESOCBKEYFILE = 0x10
+    AESOCBREPO = 0x11
+    CHPOKEYFILE = 0x20
+    CHPOREPO = 0x21
+    BLAKE2AESOCBKEYFILE = 0x30
+    BLAKE2AESOCBREPO = 0x31
+    BLAKE2CHPOKEYFILE = 0x40
+    BLAKE2CHPOREPO = 0x41
 
 
 REPOSITORY_README = """This is a Borg Backup repository.

src/borg/crypto/key.py

@@ -23,7 +23,7 @@ from ..platform import SaveFile
 
 from .nonces import NonceManager
 from .low_level import AES, bytes_to_long, long_to_bytes, bytes_to_int, num_cipher_blocks, hmac_sha256, blake2b_256, hkdf_hmac_sha512
-from .low_level import AES256_CTR_HMAC_SHA256, AES256_CTR_BLAKE2b
+from .low_level import AES256_CTR_HMAC_SHA256, AES256_CTR_BLAKE2b, AES256_OCB, CHACHA20_POLY1305
 
 
 class UnsupportedPayloadError(Error):
@@ -156,8 +156,9 @@ class KeyBase:
     def id_hash(self, data):
         """Return HMAC hash using the "id" HMAC key
         """
+        raise NotImplementedError
 
-    def encrypt(self, chunk):
+    def encrypt(self, id, data):
         pass
 
     def decrypt(self, id, data, decompress=True):
@@ -263,8 +264,8 @@ class PlaintextKey(KeyBase):
     def id_hash(self, data):
         return sha256(data).digest()
 
-    def encrypt(self, chunk):
-        data = self.compressor.compress(chunk)
+    def encrypt(self, id, data):
+        data = self.compressor.compress(data)
         return b''.join([self.TYPE_STR, data])
 
     def decrypt(self, id, data, decompress=True):
@@ -335,12 +336,12 @@ class AESKeyBase(KeyBase):
 
     PAYLOAD_OVERHEAD = 1 + 32 + 8  # TYPE + HMAC + NONCE
 
-    CIPHERSUITE = AES256_CTR_HMAC_SHA256
+    CIPHERSUITE = None  # override in derived class
 
     logically_encrypted = True
 
-    def encrypt(self, chunk):
-        data = self.compressor.compress(chunk)
+    def encrypt(self, id, data):
+        data = self.compressor.compress(data)
         next_iv = self.nonce_manager.ensure_reservation(self.cipher.next_iv(),
                                                         self.cipher.block_count(len(data)))
         return self.cipher.encrypt(data, header=self.TYPE_STR, iv=next_iv)
@@ -382,7 +383,9 @@ class AESKeyBase(KeyBase):
         self.nonce_manager = NonceManager(self.repository, nonce)
 
 
-class FlexiKeyBase(AESKeyBase):
+class FlexiKey:
+    FILE_ID = 'BORG_KEY'
+
     @classmethod
     def detect(cls, repository, manifest_data):
         key = cls(repository)
@@ -405,12 +408,6 @@ class FlexiKeyBase(AESKeyBase):
         key._passphrase = passphrase
         return key
 
-    def find_key(self):
-        raise NotImplementedError
-
-    def load(self, target, passphrase):
-        raise NotImplementedError
-
     def _load(self, key_data, passphrase):
         cdata = a2b_base64(key_data)
         data = self.decrypt_key_file(cdata, passphrase)
@@ -488,18 +485,6 @@ class FlexiKeyBase(AESKeyBase):
         logger.info('Keep this key safe. Your data will be inaccessible without it.')
         return key
 
-    def save(self, target, passphrase, create=False):
-        raise NotImplementedError
-
-    def get_new_target(self, args):
-        raise NotImplementedError
-
-
-class FlexiKey(ID_HMAC_SHA_256, FlexiKeyBase):
-    TYPES_ACCEPTABLE = {KeyType.KEYFILE, KeyType.REPO, KeyType.PASSPHRASE}
-
-    FILE_ID = 'BORG_KEY'
-
     def sanity_check(self, filename, id):
         file_id = self.FILE_ID.encode() + b' '
         repo_id = hexlify(id)
@@ -624,40 +609,43 @@ class FlexiKey(ID_HMAC_SHA_256, FlexiKeyBase):
             raise TypeError('Unsupported borg key storage type')
 
 
-class KeyfileKey(FlexiKey):
+class KeyfileKey(ID_HMAC_SHA_256, AESKeyBase, FlexiKey):
+    TYPES_ACCEPTABLE = {KeyType.KEYFILE, KeyType.REPO, KeyType.PASSPHRASE}
     TYPE = KeyType.KEYFILE
     NAME = 'key file'
     ARG_NAME = 'keyfile'
     STORAGE = KeyBlobStorage.KEYFILE
+    CIPHERSUITE = AES256_CTR_HMAC_SHA256
 
 
-class RepoKey(FlexiKey):
+class RepoKey(ID_HMAC_SHA_256, AESKeyBase, FlexiKey):
+    TYPES_ACCEPTABLE = {KeyType.KEYFILE, KeyType.REPO, KeyType.PASSPHRASE}
     TYPE = KeyType.REPO
     NAME = 'repokey'
     ARG_NAME = 'repokey'
     STORAGE = KeyBlobStorage.REPO
+    CIPHERSUITE = AES256_CTR_HMAC_SHA256
 
 
-class Blake2FlexiKey(ID_BLAKE2b_256, FlexiKey):
+class Blake2KeyfileKey(ID_BLAKE2b_256, AESKeyBase, FlexiKey):
     TYPES_ACCEPTABLE = {KeyType.BLAKE2KEYFILE, KeyType.BLAKE2REPO}
-    CIPHERSUITE = AES256_CTR_BLAKE2b
-
-
-class Blake2KeyfileKey(Blake2FlexiKey):
     TYPE = KeyType.BLAKE2KEYFILE
     NAME = 'key file BLAKE2b'
     ARG_NAME = 'keyfile-blake2'
     STORAGE = KeyBlobStorage.KEYFILE
+    CIPHERSUITE = AES256_CTR_BLAKE2b
 
 
-class Blake2RepoKey(Blake2FlexiKey):
+class Blake2RepoKey(ID_BLAKE2b_256, AESKeyBase, FlexiKey):
+    TYPES_ACCEPTABLE = {KeyType.BLAKE2KEYFILE, KeyType.BLAKE2REPO}
     TYPE = KeyType.BLAKE2REPO
     NAME = 'repokey BLAKE2b'
     ARG_NAME = 'repokey-blake2'
     STORAGE = KeyBlobStorage.REPO
+    CIPHERSUITE = AES256_CTR_BLAKE2b
 
 
-class AuthenticatedKeyBase(FlexiKey):
+class AuthenticatedKeyBase(AESKeyBase, FlexiKey):
     STORAGE = KeyBlobStorage.REPO
 
     # It's only authenticated, not encrypted.
@@ -676,8 +664,8 @@ class AuthenticatedKeyBase(FlexiKey):
         if manifest_data is not None:
             self.assert_type(manifest_data[0])
 
-    def encrypt(self, chunk):
-        data = self.compressor.compress(chunk)
+    def encrypt(self, id, data):
+        data = self.compressor.compress(data)
         return b''.join([self.TYPE_STR, data])
 
     def decrypt(self, id, data, decompress=True):
@@ -690,7 +678,7 @@ class AuthenticatedKeyBase(FlexiKey):
         return data
 
 
-class AuthenticatedKey(AuthenticatedKeyBase):
+class AuthenticatedKey(ID_HMAC_SHA_256, AuthenticatedKeyBase):
     TYPE = KeyType.AUTHENTICATED
     TYPES_ACCEPTABLE = {TYPE}
     NAME = 'authenticated'
@@ -704,8 +692,173 @@ class Blake2AuthenticatedKey(ID_BLAKE2b_256, AuthenticatedKeyBase):
     ARG_NAME = 'authenticated-blake2'
 
 
+# ------------ new crypto ------------
+
+
+class AEADKeyBase(KeyBase):
+    """
+    Chunks are encrypted and authenticated using some AEAD ciphersuite
+
+    Layout: suite:4 keytype:4 reserved:8 messageIV:48 sessionID:192 auth_tag:128 payload:... [bits]
+            ^-------------------- AAD ----------------------------^
+    Offsets:0                 1          2            8             32           48 [bytes]
+
+    suite: 1010b for new AEAD crypto, 0000b is old crypto
+    keytype: see constants.KeyType (suite+keytype)
+    reserved: all-zero, for future use
+    messageIV: a counter starting from 0 for all new encrypted messages of one session
+    sessionID: 192bit random, computed once per session (the session key is derived from this)
+    auth_tag: authentication tag output of the AEAD cipher (computed over payload and AAD)
+    payload: encrypted chunk data
+    """
+
+    PAYLOAD_OVERHEAD = 1 + 1 + 6 + 24 + 16  # [bytes], see Layout
+
+    CIPHERSUITE = None  # override in subclass
+
+    logically_encrypted = True
+
+    MAX_IV = 2 ** 48 - 1
+
+    def encrypt(self, id, data):
+        # to encrypt new data in this session we use always self.cipher and self.sessionid
+        data = self.compressor.compress(data)
+        reserved = b'\0'
+        iv = self.cipher.next_iv()
+        if iv > self.MAX_IV:  # see the data-structures docs about why the IV range is enough
+            raise IntegrityError("IV overflow, should never happen.")
+        iv_48bit = iv.to_bytes(6, 'big')
+        header = self.TYPE_STR + reserved + iv_48bit + self.sessionid
+        return self.cipher.encrypt(data, header=header, iv=iv, aad=id)
+
+    def decrypt(self, id, data, decompress=True):
+        # to decrypt existing data, we need to get a cipher configured for the sessionid and iv from header
+        self.assert_type(data[0], id)
+        iv_48bit = data[2:8]
+        sessionid = data[8:32]
+        iv = int.from_bytes(iv_48bit, 'big')
+        cipher = self._get_cipher(sessionid, iv)
+        try:
+            payload = cipher.decrypt(data, aad=id)
+        except IntegrityError as e:
+            raise IntegrityError(f"Chunk {bin_to_hex(id)}: Could not decrypt [{str(e)}]")
+        if not decompress:
+            return payload
+        data = self.decompress(payload)
+        self.assert_id(id, data)
+        return data
+
+    def init_from_random_data(self):
+        data = os.urandom(100)
+        self.enc_key = data[0:32]
+        self.enc_hmac_key = data[32:64]
+        self.id_key = data[64:96]
+        self.chunk_seed = bytes_to_int(data[96:100])
+        # Convert to signed int32
+        if self.chunk_seed & 0x80000000:
+            self.chunk_seed = self.chunk_seed - 0xffffffff - 1
+
+    def _get_session_key(self, sessionid):
+        assert len(sessionid) == 24  # 192bit
+        key = hkdf_hmac_sha512(
+            ikm=self.enc_key + self.enc_hmac_key,
+            salt=sessionid,
+            info=b'borg-session-key-' + self.CIPHERSUITE.__name__.encode(),
+            output_length=32
+        )
+        return key
+
+    def _get_cipher(self, sessionid, iv):
+        assert isinstance(iv, int)
+        key = self._get_session_key(sessionid)
+        cipher = self.CIPHERSUITE(key=key, iv=iv, header_len=1+1+6+24, aad_offset=0)
+        return cipher
+
+    def init_ciphers(self, manifest_data=None, iv=0):
+        # in every new session we start with a fresh sessionid and at iv == 0, manifest_data and iv params are ignored
+        self.sessionid = os.urandom(24)
+        self.cipher = self._get_cipher(self.sessionid, iv=0)
+
+
+class AESOCBKeyfileKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey):
+    TYPES_ACCEPTABLE = {KeyType.AESOCBKEYFILE, KeyType.AESOCBREPO}
+    TYPE = KeyType.AESOCBKEYFILE
+    NAME = 'key file AES-OCB'
+    ARG_NAME = 'keyfile-aes-ocb'
+    STORAGE = KeyBlobStorage.KEYFILE
+    CIPHERSUITE = AES256_OCB
+
+
+class AESOCBRepoKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey):
+    TYPES_ACCEPTABLE = {KeyType.AESOCBKEYFILE, KeyType.AESOCBREPO}
+    TYPE = KeyType.AESOCBREPO
+    NAME = 'repokey AES-OCB'
+    ARG_NAME = 'repokey-aes-ocb'
+    STORAGE = KeyBlobStorage.REPO
+    CIPHERSUITE = AES256_OCB
+
+
+class CHPOKeyfileKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey):
+    TYPES_ACCEPTABLE = {KeyType.CHPOKEYFILE, KeyType.CHPOREPO}
+    TYPE = KeyType.CHPOKEYFILE
+    NAME = 'key file ChaCha20-Poly1305'
+    ARG_NAME = 'keyfile-chacha20-poly1305'
+    STORAGE = KeyBlobStorage.KEYFILE
+    CIPHERSUITE = CHACHA20_POLY1305
+
+
+class CHPORepoKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey):
+    TYPES_ACCEPTABLE = {KeyType.CHPOKEYFILE, KeyType.CHPOREPO}
+    TYPE = KeyType.CHPOREPO
+    NAME = 'repokey ChaCha20-Poly1305'
+    ARG_NAME = 'repokey-chacha20-poly1305'
+    STORAGE = KeyBlobStorage.REPO
+    CIPHERSUITE = CHACHA20_POLY1305
+
+
+class Blake2AESOCBKeyfileKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey):
+    TYPES_ACCEPTABLE = {KeyType.BLAKE2AESOCBKEYFILE, KeyType.BLAKE2AESOCBREPO}
+    TYPE = KeyType.BLAKE2AESOCBKEYFILE
+    NAME = 'key file Blake2b AES-OCB'
+    ARG_NAME = 'keyfile-blake2-aes-ocb'
+    STORAGE = KeyBlobStorage.KEYFILE
+    CIPHERSUITE = AES256_OCB
+
+
+class Blake2AESOCBRepoKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey):
+    TYPES_ACCEPTABLE = {KeyType.BLAKE2AESOCBKEYFILE, KeyType.BLAKE2AESOCBREPO}
+    TYPE = KeyType.BLAKE2AESOCBREPO
+    NAME = 'repokey Blake2b AES-OCB'
+    ARG_NAME = 'repokey-blake2-aes-ocb'
+    STORAGE = KeyBlobStorage.REPO
+    CIPHERSUITE = AES256_OCB
+
+
+class Blake2CHPOKeyfileKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey):
+    TYPES_ACCEPTABLE = {KeyType.BLAKE2CHPOKEYFILE, KeyType.BLAKE2CHPOREPO}
+    TYPE = KeyType.BLAKE2CHPOKEYFILE
+    NAME = 'key file Blake2b ChaCha20-Poly1305'
+    ARG_NAME = 'keyfile-blake2-chacha20-poly1305'
+    STORAGE = KeyBlobStorage.KEYFILE
+    CIPHERSUITE = CHACHA20_POLY1305
+
+
+class Blake2CHPORepoKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey):
+    TYPES_ACCEPTABLE = {KeyType.BLAKE2CHPOKEYFILE, KeyType.BLAKE2CHPOREPO}
+    TYPE = KeyType.BLAKE2CHPOREPO
+    NAME = 'repokey Blake2b ChaCha20-Poly1305'
+    ARG_NAME = 'repokey-blake2-chacha20-poly1305'
+    STORAGE = KeyBlobStorage.REPO
+    CIPHERSUITE = CHACHA20_POLY1305
+
+
 AVAILABLE_KEY_TYPES = (
     PlaintextKey,
     KeyfileKey, RepoKey, AuthenticatedKey,
     Blake2KeyfileKey, Blake2RepoKey, Blake2AuthenticatedKey,
+    # new crypto
+    AESOCBKeyfileKey, AESOCBRepoKey,
+    CHPOKeyfileKey, CHPORepoKey,
+    Blake2AESOCBKeyfileKey, Blake2AESOCBRepoKey,
+    Blake2CHPOKeyfileKey, Blake2CHPORepoKey,
 )

src/borg/crypto/low_level.pyx

@@ -42,7 +42,7 @@ from cpython cimport PyMem_Malloc, PyMem_Free
 from cpython.buffer cimport PyBUF_SIMPLE, PyObject_GetBuffer, PyBuffer_Release
 from cpython.bytes cimport PyBytes_FromStringAndSize
 
-API_VERSION = '1.2_01'
+API_VERSION = '1.3_01'
 
 cdef extern from "openssl/crypto.h":
     int CRYPTO_memcmp(const void *a, const void *b, size_t len)
@@ -77,9 +77,9 @@ cdef extern from "openssl/evp.h":
     int EVP_DecryptFinal_ex(EVP_CIPHER_CTX *ctx, unsigned char *out, int *outl)
 
     int EVP_CIPHER_CTX_ctrl(EVP_CIPHER_CTX *ctx, int type, int arg, void *ptr)
-    int EVP_CTRL_GCM_GET_TAG
-    int EVP_CTRL_GCM_SET_TAG
-    int EVP_CTRL_GCM_SET_IVLEN
+    int EVP_CTRL_AEAD_GET_TAG
+    int EVP_CTRL_AEAD_SET_TAG
+    int EVP_CTRL_AEAD_SET_IVLEN
 
     const EVP_MD *EVP_sha256() nogil
 
@@ -152,7 +152,7 @@ class UNENCRYPTED:
         self.header_len = header_len
         self.set_iv(iv)
 
-    def encrypt(self, data, header=b'', iv=None):
+    def encrypt(self, data, header=b'', iv=None, aad=None):
         """
         IMPORTANT: it is called encrypt to satisfy the crypto api naming convention,
         but this does NOT encrypt and it does NOT compute and store a MAC either.
@@ -162,7 +162,7 @@ class UNENCRYPTED:
         assert self.iv is not None, 'iv needs to be set before encrypt is called'
         return header + data
 
-    def decrypt(self, envelope):
+    def decrypt(self, envelope, aad=None):
         """
         IMPORTANT: it is called decrypt to satisfy the crypto api naming convention,
         but this does NOT decrypt and it does NOT verify a MAC either, because data
@@ -184,10 +184,10 @@ class UNENCRYPTED:
 
 
 cdef class AES256_CTR_BASE:
-    # Layout: HEADER + MAC 32 + IV 8 + CT (same as attic / borg < 1.2 IF HEADER = TYPE_BYTE, no AAD)
+    # Layout: HEADER + MAC 32 + IV 8 + CT (same as attic / borg < 1.3 IF HEADER = TYPE_BYTE, no AAD)
 
     cdef EVP_CIPHER_CTX *ctx
-    cdef unsigned char *enc_key
+    cdef unsigned char enc_key[32]
     cdef int cipher_blk_len
     cdef int iv_len, iv_len_short
     cdef int aad_offset
@@ -235,7 +235,7 @@ cdef class AES256_CTR_BASE:
         """
         raise NotImplementedError
 
-    def encrypt(self, data, header=b'', iv=None):
+    def encrypt(self, data, header=b'', iv=None, aad=None):
         """
         encrypt data, compute mac over aad + iv + cdata, prepend header.
         aad_offset is the offset into the header where aad starts.
@@ -252,7 +252,7 @@ cdef class AES256_CTR_BASE:
                                                                   ilen + self.cipher_blk_len)  # play safe, 1 extra blk
         if not odata:
             raise MemoryError
-        cdef int olen
+        cdef int olen = 0
         cdef int offset
         cdef Py_buffer idata = ro_buffer(data)
         cdef Py_buffer hdata = ro_buffer(header)
@@ -264,15 +264,12 @@ cdef class AES256_CTR_BASE:
             offset += self.mac_len
             self.store_iv(odata+offset, self.iv)
             offset += self.iv_len_short
-            rc = EVP_EncryptInit_ex(self.ctx, EVP_aes_256_ctr(), NULL, self.enc_key, self.iv)
-            if not rc:
+            if not EVP_EncryptInit_ex(self.ctx, EVP_aes_256_ctr(), NULL, self.enc_key, self.iv):
                 raise CryptoError('EVP_EncryptInit_ex failed')
-            rc = EVP_EncryptUpdate(self.ctx, odata+offset, &olen, <const unsigned char*> idata.buf, ilen)
-            if not rc:
+            if not EVP_EncryptUpdate(self.ctx, odata+offset, &olen, <const unsigned char*> idata.buf, ilen):
                 raise CryptoError('EVP_EncryptUpdate failed')
             offset += olen
-            rc = EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen)
-            if not rc:
+            if not EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen):
                 raise CryptoError('EVP_EncryptFinal_ex failed')
             offset += olen
             self.mac_compute(<const unsigned char *> hdata.buf+aoffset, alen,
@@ -285,19 +282,18 @@ cdef class AES256_CTR_BASE:
             PyBuffer_Release(&hdata)
             PyBuffer_Release(&idata)
 
-    def decrypt(self, envelope):
+    def decrypt(self, envelope, aad=None):
         """
         authenticate aad + iv + cdata, decrypt cdata, ignore header bytes up to aad_offset.
         """
         cdef int ilen = len(envelope)
         cdef int hlen = self.header_len
-        assert hlen == self.header_len
         cdef int aoffset = self.aad_offset
         cdef int alen = hlen - aoffset
         cdef unsigned char *odata = <unsigned char *>PyMem_Malloc(ilen + self.cipher_blk_len)  # play safe, 1 extra blk
         if not odata:
             raise MemoryError
-        cdef int olen
+        cdef int olen = 0
         cdef int offset
         cdef unsigned char mac_buf[32]
         assert sizeof(mac_buf) == self.mac_len
@@ -311,14 +307,12 @@ cdef class AES256_CTR_BASE:
             if not EVP_DecryptInit_ex(self.ctx, EVP_aes_256_ctr(), NULL, self.enc_key, iv):
                 raise CryptoError('EVP_DecryptInit_ex failed')
             offset = 0
-            rc = EVP_DecryptUpdate(self.ctx, odata+offset, &olen,
-                                   <const unsigned char*> idata.buf+hlen+self.mac_len+self.iv_len_short,
-                                   ilen-hlen-self.mac_len-self.iv_len_short)
-            if not rc:
+            if not EVP_DecryptUpdate(self.ctx, odata+offset, &olen,
+                                     <const unsigned char*> idata.buf+hlen+self.mac_len+self.iv_len_short,
+                                     ilen-hlen-self.mac_len-self.iv_len_short):
                 raise CryptoError('EVP_DecryptUpdate failed')
             offset += olen
-            rc = EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen)
-            if rc <= 0:
+            if not EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen):
                 raise CryptoError('EVP_DecryptFinal_ex failed')
             offset += olen
             self.blocks += self.block_count(offset)
@@ -335,8 +329,7 @@ cdef class AES256_CTR_BASE:
         if isinstance(iv, int):
             iv = iv.to_bytes(self.iv_len, byteorder='big')
         assert isinstance(iv, bytes) and len(iv) == self.iv_len
-        for i in range(self.iv_len):
-            self.iv[i] = iv[i]
+        self.iv = iv
         self.blocks = 0  # how many AES blocks got encrypted with this IV?
 
     def next_iv(self):
@@ -360,7 +353,7 @@ cdef class AES256_CTR_BASE:
 
 
 cdef class AES256_CTR_HMAC_SHA256(AES256_CTR_BASE):
-    cdef unsigned char *mac_key
+    cdef unsigned char mac_key[32]
 
     def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
         assert isinstance(mac_key, bytes) and len(mac_key) == 32
@@ -377,7 +370,7 @@ cdef class AES256_CTR_HMAC_SHA256(AES256_CTR_BASE):
                      const unsigned char *data2, int data2_len,
                      unsigned char *mac_buf):
         data = data1[:data1_len] + data2[:data2_len]
-        mac = hmac.HMAC(self.mac_key, data, hashlib.sha256).digest()
+        mac = hmac.digest(self.mac_key[:self.mac_len], data, 'sha256')
         for i in range(self.mac_len):
             mac_buf[i] = mac[i]
 
@@ -390,7 +383,7 @@ cdef class AES256_CTR_HMAC_SHA256(AES256_CTR_BASE):
 
 
 cdef class AES256_CTR_BLAKE2b(AES256_CTR_BASE):
-    cdef unsigned char *mac_key
+    cdef unsigned char mac_key[128]
 
     def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
         assert isinstance(mac_key, bytes) and len(mac_key) == 128
@@ -423,15 +416,16 @@ ctypedef const EVP_CIPHER * (* CIPHER)()
 
 
 cdef class _AEAD_BASE:
-    # Layout: HEADER + MAC 16 + IV 12 + CT
+    # new crypto used in borg >= 1.3
+    # Layout: HEADER + MAC 16 + CT
 
     cdef CIPHER cipher
     cdef EVP_CIPHER_CTX *ctx
-    cdef unsigned char *enc_key
+    cdef unsigned char key[32]
     cdef int cipher_blk_len
     cdef int iv_len
     cdef int aad_offset
-    cdef int header_len
+    cdef int header_len_expected
     cdef int mac_len
     cdef unsigned char iv[12]
     cdef long long blocks
@@ -441,83 +435,87 @@ cdef class _AEAD_BASE:
         """check whether library requirements for this ciphersuite are satisfied"""
         raise NotImplemented  # override / implement in child class
 
-    def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
-        assert mac_key is None
-        assert isinstance(enc_key, bytes) and len(enc_key) == 32
+    def __init__(self, key, iv=None, header_len=0, aad_offset=0):
+        """
+        init AEAD crypto
+
+        :param key: 256bit encrypt-then-mac key
+        :param iv: 96bit initialisation vector / nonce
+        :param header_len: expected length of header
+        :param aad_offset: where in the header the authenticated data starts
+        """
+        assert isinstance(key, bytes) and len(key) == 32
         self.iv_len = sizeof(self.iv)
-        self.header_len = 1
+        self.header_len_expected = header_len
         assert aad_offset <= header_len
         self.aad_offset = aad_offset
-        self.header_len = header_len
         self.mac_len = 16
-        self.enc_key = enc_key
+        self.key = key
         if iv is not None:
             self.set_iv(iv)
         else:
             self.blocks = -1  # make sure set_iv is called before encrypt
 
-    def __cinit__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
+    def __cinit__(self, key, iv=None, header_len=0, aad_offset=0):
         self.ctx = EVP_CIPHER_CTX_new()
 
     def __dealloc__(self):
         EVP_CIPHER_CTX_free(self.ctx)
 
-    def encrypt(self, data, header=b'', iv=None):
+    def encrypt(self, data, header=b'', iv=None, aad=b''):
         """
-        encrypt data, compute mac over aad + iv + cdata, prepend header.
-        aad_offset is the offset into the header where aad starts.
+        encrypt data, compute auth tag over aad + header + cdata.
+        return header + auth tag + cdata.
+        aad_offset is the offset into the header where the authenticated header part starts.
+        aad is additional authenticated data, which won't be included in the returned data,
+        but only used for the auth tag computation.
         """
         if iv is not None:
             self.set_iv(iv)
         assert self.blocks == 0, 'iv needs to be set before encrypt is called'
-        # AES-GCM, AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte)
+        # AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte)
         # IV we provide, thus we must not encrypt more than 2^32 cipher blocks with same IV).
         block_count = self.block_count(len(data))
         if block_count > 2**32:
             raise ValueError('too much data, would overflow internal 32bit counter')
         cdef int ilen = len(data)
         cdef int hlen = len(header)
-        assert hlen == self.header_len
+        assert hlen == self.header_len_expected
         cdef int aoffset = self.aad_offset
         cdef int alen = hlen - aoffset
-        cdef unsigned char *odata = <unsigned char *>PyMem_Malloc(hlen + self.mac_len + self.iv_len +
+        cdef int aadlen = len(aad)
+        cdef unsigned char *odata = <unsigned char *>PyMem_Malloc(hlen + self.mac_len +
                                                                   ilen + self.cipher_blk_len)
         if not odata:
             raise MemoryError
-        cdef int olen
+        cdef int olen = 0
         cdef int offset
         cdef Py_buffer idata = ro_buffer(data)
         cdef Py_buffer hdata = ro_buffer(header)
+        cdef Py_buffer aadata = ro_buffer(aad)
         try:
             offset = 0
             for i in range(hlen):
                 odata[offset+i] = header[i]
             offset += hlen
             offset += self.mac_len
-            self.store_iv(odata+offset, self.iv)
-            rc = EVP_EncryptInit_ex(self.ctx, self.cipher(), NULL, NULL, NULL)
-            if not rc:
+            if not EVP_EncryptInit_ex(self.ctx, self.cipher(), NULL, NULL, NULL):
                 raise CryptoError('EVP_EncryptInit_ex failed')
-            if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_SET_IVLEN, self.iv_len, NULL):
+            if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_SET_IVLEN, self.iv_len, NULL):
                 raise CryptoError('EVP_CIPHER_CTX_ctrl SET IVLEN failed')
-            rc = EVP_EncryptInit_ex(self.ctx, NULL, NULL, self.enc_key, self.iv)
-            if not rc:
+            if not EVP_EncryptInit_ex(self.ctx, NULL, NULL, self.key, self.iv):
                 raise CryptoError('EVP_EncryptInit_ex failed')
-            rc = EVP_EncryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> hdata.buf+aoffset, alen)
-            if not rc:
+            if not EVP_EncryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> aadata.buf, aadlen):
                 raise CryptoError('EVP_EncryptUpdate failed')
-            if not EVP_EncryptUpdate(self.ctx, NULL, &olen, odata+offset, self.iv_len):
+            if not EVP_EncryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> hdata.buf+aoffset, alen):
                 raise CryptoError('EVP_EncryptUpdate failed')
-            offset += self.iv_len
-            rc = EVP_EncryptUpdate(self.ctx, odata+offset, &olen, <const unsigned char*> idata.buf, ilen)
-            if not rc:
+            if not EVP_EncryptUpdate(self.ctx, odata+offset, &olen, <const unsigned char*> idata.buf, ilen):
                 raise CryptoError('EVP_EncryptUpdate failed')
             offset += olen
-            rc = EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen)
-            if not rc:
+            if not EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen):
                 raise CryptoError('EVP_EncryptFinal_ex failed')
             offset += olen
-            if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_GET_TAG, self.mac_len, odata+hlen):
+            if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_GET_TAG, self.mac_len, odata + hlen):
                 raise CryptoError('EVP_CIPHER_CTX_ctrl GET TAG failed')
             self.blocks = block_count
             return odata[:offset]
@@ -525,53 +523,50 @@ cdef class _AEAD_BASE:
             PyMem_Free(odata)
             PyBuffer_Release(&hdata)
             PyBuffer_Release(&idata)
+            PyBuffer_Release(&aadata)
 
-    def decrypt(self, envelope):
+    def decrypt(self, envelope, aad=b''):
         """
-        authenticate aad + iv + cdata, decrypt cdata, ignore header bytes up to aad_offset.
+        authenticate aad + header + cdata (from envelope), ignore header bytes up to aad_offset.,
+        return decrypted cdata.
         """
-        # AES-GCM, AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte)
+        # AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte)
         # IV we provide, thus we must not decrypt more than 2^32 cipher blocks with same IV):
         approx_block_count = self.block_count(len(envelope))  # sloppy, but good enough for borg
         if approx_block_count > 2**32:
             raise ValueError('too much data, would overflow internal 32bit counter')
         cdef int ilen = len(envelope)
-        cdef int hlen = self.header_len
-        assert hlen == self.header_len
+        cdef int hlen = self.header_len_expected
         cdef int aoffset = self.aad_offset
         cdef int alen = hlen - aoffset
+        cdef int aadlen = len(aad)
         cdef unsigned char *odata = <unsigned char *>PyMem_Malloc(ilen + self.cipher_blk_len)
         if not odata:
             raise MemoryError
-        cdef int olen
+        cdef int olen = 0
         cdef int offset
         cdef Py_buffer idata = ro_buffer(envelope)
+        cdef Py_buffer aadata = ro_buffer(aad)
         try:
             if not EVP_DecryptInit_ex(self.ctx, self.cipher(), NULL, NULL, NULL):
                 raise CryptoError('EVP_DecryptInit_ex failed')
-            iv = self.fetch_iv(<unsigned char *> idata.buf+hlen+self.mac_len)
-            self.set_iv(iv)
-            if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_SET_IVLEN, self.iv_len, NULL):
+            if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_SET_IVLEN, self.iv_len, NULL):
                 raise CryptoError('EVP_CIPHER_CTX_ctrl SET IVLEN failed')
-            if not EVP_DecryptInit_ex(self.ctx, NULL, NULL, self.enc_key, iv):
+            if not EVP_DecryptInit_ex(self.ctx, NULL, NULL, self.key, self.iv):
                 raise CryptoError('EVP_DecryptInit_ex failed')
-            if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_SET_TAG, self.mac_len, <unsigned char *> idata.buf+hlen):
-                raise CryptoError('EVP_CIPHER_CTX_ctrl SET TAG failed')
-            rc = EVP_DecryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> idata.buf+aoffset, alen)
-            if not rc:
+            if not EVP_DecryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> aadata.buf, aadlen):
                 raise CryptoError('EVP_DecryptUpdate failed')
-            if not EVP_DecryptUpdate(self.ctx, NULL, &olen,
-                                     <const unsigned char*> idata.buf+hlen+self.mac_len, self.iv_len):
+            if not EVP_DecryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> idata.buf+aoffset, alen):
                 raise CryptoError('EVP_DecryptUpdate failed')
             offset = 0
-            rc = EVP_DecryptUpdate(self.ctx, odata+offset, &olen,
-                                   <const unsigned char*> idata.buf+hlen+self.mac_len+self.iv_len,
-                                   ilen-hlen-self.mac_len-self.iv_len)
-            if not rc:
+            if not EVP_DecryptUpdate(self.ctx, odata+offset, &olen,
+                                     <const unsigned char*> idata.buf+hlen+self.mac_len,
+                                     ilen-hlen-self.mac_len):
                 raise CryptoError('EVP_DecryptUpdate failed')
             offset += olen
-            rc = EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen)
-            if rc <= 0:
+            if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_SET_TAG, self.mac_len, <unsigned char *> idata.buf + hlen):
+                raise CryptoError('EVP_CIPHER_CTX_ctrl SET TAG failed')
+            if not EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen):
                 # a failure here means corrupted or tampered tag (mac) or data.
                 raise IntegrityError('Authentication / EVP_DecryptFinal_ex failed')
             offset += olen
@@ -580,6 +575,7 @@ cdef class _AEAD_BASE:
         finally:
             PyMem_Free(odata)
             PyBuffer_Release(&idata)
+            PyBuffer_Release(&aadata)
 
     def block_count(self, length):
         return num_cipher_blocks(length, self.cipher_blk_len)
@@ -590,71 +586,48 @@ cdef class _AEAD_BASE:
         if isinstance(iv, int):
             iv = iv.to_bytes(self.iv_len, byteorder='big')
         assert isinstance(iv, bytes) and len(iv) == self.iv_len
-        for i in range(self.iv_len):
-            self.iv[i] = iv[i]
+        self.iv = iv
         self.blocks = 0  # number of cipher blocks encrypted with this IV
 
     def next_iv(self):
         # call this after encrypt() to get the next iv (int) for the next encrypt() call
-        # AES-GCM, AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit
+        # AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit
         # (12 byte) IV we provide, thus we only need to increment the IV by 1.
         iv = int.from_bytes(self.iv[:self.iv_len], byteorder='big')
         return iv + 1
 
-    cdef fetch_iv(self, unsigned char * iv_in):
-        return iv_in[0:self.iv_len]
-
-    cdef store_iv(self, unsigned char * iv_out, unsigned char * iv):
-        cdef int i
-        for i in range(self.iv_len):
-            iv_out[i] = iv[i]
-
-    def extract_iv(self, envelope):
-        offset = self.header_len + self.mac_len
-        return bytes_to_long(envelope[offset:offset+self.iv_len])
-
-
-cdef class _AES_BASE(_AEAD_BASE):
-    def __init__(self, *args, **kwargs):
-        self.cipher_blk_len = 16
-        super().__init__(*args, **kwargs)
-
-
-cdef class _CHACHA_BASE(_AEAD_BASE):
-    def __init__(self, *args, **kwargs):
-        self.cipher_blk_len = 64
-        super().__init__(*args, **kwargs)
 
-
-cdef class AES256_OCB(_AES_BASE):
+cdef class AES256_OCB(_AEAD_BASE):
     @classmethod
     def requirements_check(cls):
         if is_libressl:
             raise ValueError('AES OCB is not implemented by LibreSSL (yet?).')
 
-    def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
+    def __init__(self, key, iv=None, header_len=0, aad_offset=0):
         self.requirements_check()
         self.cipher = EVP_aes_256_ocb
-        super().__init__(mac_key, enc_key, iv=iv, header_len=header_len, aad_offset=aad_offset)
+        self.cipher_blk_len = 16
+        super().__init__(key, iv=iv, header_len=header_len, aad_offset=aad_offset)
 
 
-cdef class CHACHA20_POLY1305(_CHACHA_BASE):
+cdef class CHACHA20_POLY1305(_AEAD_BASE):
     @classmethod
     def requirements_check(cls):
         if is_libressl:
             raise ValueError('CHACHA20-POLY1305 is not implemented by LibreSSL (yet?).')
 
-    def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
+    def __init__(self, key, iv=None, header_len=0, aad_offset=0):
         self.requirements_check()
         self.cipher = EVP_chacha20_poly1305
-        super().__init__(mac_key, enc_key, iv=iv, header_len=header_len, aad_offset=aad_offset)
+        self.cipher_blk_len = 64
+        super().__init__(key, iv=iv, header_len=header_len, aad_offset=aad_offset)
 
 
 cdef class AES:
     """A thin wrapper around the OpenSSL EVP cipher API - for legacy code, like key file encryption"""
     cdef CIPHER cipher
     cdef EVP_CIPHER_CTX *ctx
-    cdef unsigned char *enc_key
+    cdef unsigned char enc_key[32]
     cdef int cipher_blk_len
     cdef int iv_len
     cdef unsigned char iv[16]
@@ -721,7 +694,7 @@ cdef class AES:
             if not EVP_DecryptUpdate(self.ctx, odata, &olen, <const unsigned char*> idata.buf, ilen):
                 raise Exception('EVP_DecryptUpdate failed')
             offset += olen
-            if EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen) <= 0:
+            if not EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen):
                 # this error check is very important for modes with padding or
                 # authentication. for them, a failure here means corrupted data.
                 # CTR mode does not use padding nor authentication.
@@ -742,8 +715,7 @@ cdef class AES:
         if isinstance(iv, int):
             iv = iv.to_bytes(self.iv_len, byteorder='big')
         assert isinstance(iv, bytes) and len(iv) == self.iv_len
-        for i in range(self.iv_len):
-            self.iv[i] = iv[i]
+        self.iv = iv
         self.blocks = 0  # number of cipher blocks encrypted with this IV
 
     def next_iv(self):
@@ -752,7 +724,6 @@ cdef class AES:
         return iv + self.blocks
 
 
-
 def hmac_sha256(key, data):
     return hmac.digest(key, data, 'sha256')
 
@@ -779,7 +750,7 @@ def hkdf_hmac_sha512(ikm, salt, info, output_length):
     # Step 1. HKDF-Extract (ikm, salt) -> prk
     if salt is None:
         salt = bytes(64)
-    prk = hmac.HMAC(salt, ikm, hashlib.sha512).digest()
+    prk = hmac.digest(salt, ikm, 'sha512')
 
     # Step 2. HKDF-Expand (prk, info, output_length) -> output key
     n = ceil(output_length / digest_length)
@@ -787,6 +758,6 @@ def hkdf_hmac_sha512(ikm, salt, info, output_length):
     output = b''
     for i in range(n):
         msg = t_n + info + (i + 1).to_bytes(1, 'little')
-        t_n = hmac.HMAC(prk, msg, hashlib.sha512).digest()
+        t_n = hmac.digest(prk, msg, 'sha512')
         output += t_n
     return output[:output_length]

src/borg/helpers/checks.py

@@ -31,7 +31,7 @@ def check_extension_modules():
         raise ExtensionModuleError
     if compress.API_VERSION != '1.2_02':
         raise ExtensionModuleError
-    if borg.crypto.low_level.API_VERSION != '1.2_01':
+    if borg.crypto.low_level.API_VERSION != '1.3_01':
         raise ExtensionModuleError
     if item.API_VERSION != '1.2_01':
         raise ExtensionModuleError

src/borg/helpers/manifest.py

@@ -261,4 +261,4 @@ class Manifest:
         self.tam_verified = True
         data = self.key.pack_and_authenticate_metadata(manifest.as_dict())
         self.id = self.key.id_hash(data)
-        self.repository.put(self.MANIFEST_ID, self.key.encrypt(data))
+        self.repository.put(self.MANIFEST_ID, self.key.encrypt(self.MANIFEST_ID, data))

src/borg/selftest.py

@@ -29,7 +29,7 @@ SELFTEST_CASES = [
     ChunkerTestCase,
 ]
 
-SELFTEST_COUNT = 35
+SELFTEST_COUNT = 36
 
 
 class SelfTestResult(TestResult):

src/borg/testsuite/archiver.py

@@ -36,7 +36,7 @@ from ..cache import Cache, LocalCache
 from ..chunker import has_seek_hole
 from ..constants import *  # NOQA
 from ..crypto.low_level import bytes_to_long, num_cipher_blocks
-from ..crypto.key import FlexiKeyBase, RepoKey, KeyfileKey, Passphrase, TAMRequiredError
+from ..crypto.key import FlexiKey, RepoKey, KeyfileKey, Passphrase, TAMRequiredError
 from ..crypto.keymanager import RepoIdMismatch, NotABorgKeyFile
 from ..crypto.file_integrity import FileIntegrityError
 from ..helpers import Location, get_security_dir
@@ -2882,7 +2882,7 @@ class ArchiverTestCase(ArchiverTestCaseBase):
         def raise_eof(*args):
             raise EOFError
 
-        with patch.object(FlexiKeyBase, 'create', raise_eof):
+        with patch.object(FlexiKey, 'create', raise_eof):
             self.cmd('init', '--encryption=repokey', self.repository_location, exit_code=1)
         assert not os.path.exists(self.repository_location)
 
@@ -3806,7 +3806,7 @@ class ArchiverCheckTestCase(ArchiverTestCaseBase):
                 'version': 1,
             })
             archive_id = key.id_hash(archive)
-            repository.put(archive_id, key.encrypt(archive))
+            repository.put(archive_id, key.encrypt(archive_id, archive))
             repository.commit(compact=False)
         self.cmd('check', self.repository_location, exit_code=1)
         self.cmd('check', '--repair', self.repository_location, exit_code=0)
@@ -3894,7 +3894,7 @@ class ManifestAuthenticationTest(ArchiverTestCaseBase):
     def spoof_manifest(self, repository):
         with repository:
             _, key = Manifest.load(repository, Manifest.NO_OPERATION_CHECK)
-            repository.put(Manifest.MANIFEST_ID, key.encrypt(msgpack.packb({
+            repository.put(Manifest.MANIFEST_ID, key.encrypt(Manifest.MANIFEST_ID, msgpack.packb({
                 'version': 1,
                 'archives': {},
                 'config': {},
@@ -3907,7 +3907,7 @@ class ManifestAuthenticationTest(ArchiverTestCaseBase):
         repository = Repository(self.repository_path, exclusive=True)
         with repository:
             manifest, key = Manifest.load(repository, Manifest.NO_OPERATION_CHECK)
-            repository.put(Manifest.MANIFEST_ID, key.encrypt(msgpack.packb({
+            repository.put(Manifest.MANIFEST_ID, key.encrypt(Manifest.MANIFEST_ID, msgpack.packb({
                 'version': 1,
                 'archives': {},
                 'timestamp': (datetime.utcnow() + timedelta(days=1)).strftime(ISO_FORMAT),
@@ -3929,7 +3929,7 @@ class ManifestAuthenticationTest(ArchiverTestCaseBase):
 
             manifest = msgpack.unpackb(key.decrypt(None, repository.get(Manifest.MANIFEST_ID)))
             del manifest[b'tam']
-            repository.put(Manifest.MANIFEST_ID, key.encrypt(msgpack.packb(manifest)))
+            repository.put(Manifest.MANIFEST_ID, key.encrypt(Manifest.MANIFEST_ID, msgpack.packb(manifest)))
             repository.commit(compact=False)
         output = self.cmd('list', '--debug', self.repository_location)
         assert 'archive1234' in output

src/borg/testsuite/crypto.py

@@ -91,11 +91,10 @@ class CryptoTestCase(BaseTestCase):
 
     def test_AE(self):
         # used in legacy-like layout (1 type byte, no aad)
-        mac_key = None
-        enc_key = b'X' * 32
-        iv = 0
+        key = b'X' * 32
+        iv_int = 0
         data = b'foo' * 10
-        header = b'\x23'
+        header = b'\x23' + iv_int.to_bytes(12, 'big')
         tests = [
             # (ciphersuite class, exp_mac, exp_cdata)
         ]
@@ -111,11 +110,11 @@ class CryptoTestCase(BaseTestCase):
         for cs_cls, exp_mac, exp_cdata in tests:
             # print(repr(cs_cls))
             # encrypt/mac
-            cs = cs_cls(mac_key, enc_key, iv, header_len=1, aad_offset=1)
+            cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
             hdr_mac_iv_cdata = cs.encrypt(data, header=header)
             hdr = hdr_mac_iv_cdata[0:1]
-            mac = hdr_mac_iv_cdata[1:17]
-            iv = hdr_mac_iv_cdata[17:29]
+            iv = hdr_mac_iv_cdata[1:13]
+            mac = hdr_mac_iv_cdata[13:29]
             cdata = hdr_mac_iv_cdata[29:]
             self.assert_equal(hexlify(hdr), b'23')
             self.assert_equal(hexlify(mac), exp_mac)
@@ -123,23 +122,22 @@ class CryptoTestCase(BaseTestCase):
             self.assert_equal(hexlify(cdata), exp_cdata)
             self.assert_equal(cs.next_iv(), 1)
             # auth/decrypt
-            cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1)
+            cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
             pdata = cs.decrypt(hdr_mac_iv_cdata)
             self.assert_equal(data, pdata)
             self.assert_equal(cs.next_iv(), 1)
             # auth-failure due to corruption (corrupted data)
-            cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1)
+            cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
             hdr_mac_iv_cdata_corrupted = hdr_mac_iv_cdata[:29] + b'\0' + hdr_mac_iv_cdata[30:]
             self.assert_raises(IntegrityError,
                                lambda: cs.decrypt(hdr_mac_iv_cdata_corrupted))
 
     def test_AEAD(self):
         # test with aad
-        mac_key = None
-        enc_key = b'X' * 32
-        iv = 0
+        key = b'X' * 32
+        iv_int = 0
         data = b'foo' * 10
-        header = b'\x12\x34\x56'
+        header = b'\x12\x34\x56' + iv_int.to_bytes(12, 'big')
         tests = [
             # (ciphersuite class, exp_mac, exp_cdata)
         ]
@@ -155,11 +153,11 @@ class CryptoTestCase(BaseTestCase):
         for cs_cls, exp_mac, exp_cdata in tests:
             # print(repr(cs_cls))
             # encrypt/mac
-            cs = cs_cls(mac_key, enc_key, iv, header_len=3, aad_offset=1)
+            cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
             hdr_mac_iv_cdata = cs.encrypt(data, header=header)
             hdr = hdr_mac_iv_cdata[0:3]
-            mac = hdr_mac_iv_cdata[3:19]
-            iv = hdr_mac_iv_cdata[19:31]
+            iv = hdr_mac_iv_cdata[3:15]
+            mac = hdr_mac_iv_cdata[15:31]
             cdata = hdr_mac_iv_cdata[31:]
             self.assert_equal(hexlify(hdr), b'123456')
             self.assert_equal(hexlify(mac), exp_mac)
@@ -167,16 +165,38 @@ class CryptoTestCase(BaseTestCase):
             self.assert_equal(hexlify(cdata), exp_cdata)
             self.assert_equal(cs.next_iv(), 1)
             # auth/decrypt
-            cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1)
+            cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
             pdata = cs.decrypt(hdr_mac_iv_cdata)
             self.assert_equal(data, pdata)
             self.assert_equal(cs.next_iv(), 1)
             # auth-failure due to corruption (corrupted aad)
-            cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1)
+            cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
             hdr_mac_iv_cdata_corrupted = hdr_mac_iv_cdata[:1] + b'\0' + hdr_mac_iv_cdata[2:]
             self.assert_raises(IntegrityError,
                                lambda: cs.decrypt(hdr_mac_iv_cdata_corrupted))
 
+    def test_AEAD_with_more_AAD(self):
+        # test giving extra aad to the .encrypt() and .decrypt() calls
+        key = b'X' * 32
+        iv_int = 0
+        data = b'foo' * 10
+        header = b'\x12\x34'
+        tests = []
+        if not is_libressl:
+            tests += [AES256_OCB, CHACHA20_POLY1305]
+        for cs_cls in tests:
+            # encrypt/mac
+            cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=0)
+            hdr_mac_iv_cdata = cs.encrypt(data, header=header, aad=b'correct_chunkid')
+            # successful auth/decrypt (correct aad)
+            cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=0)
+            pdata = cs.decrypt(hdr_mac_iv_cdata, aad=b'correct_chunkid')
+            self.assert_equal(data, pdata)
+            # unsuccessful auth (incorrect aad)
+            cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=0)
+            self.assert_raises(IntegrityError,
+                               lambda: cs.decrypt(hdr_mac_iv_cdata, aad=b'incorrect_chunkid'))
+
     # These test vectors come from https://www.kullo.net/blog/hkdf-sha-512-test-vectors/
     # who claims to have verified these against independent Python and C++ implementations.
 

src/borg/testsuite/key.py

@@ -8,7 +8,8 @@ import pytest
 
 from ..crypto.key import bin_to_hex
 from ..crypto.key import PlaintextKey, AuthenticatedKey, RepoKey, KeyfileKey, \
-    Blake2KeyfileKey, Blake2RepoKey, Blake2AuthenticatedKey
+    Blake2KeyfileKey, Blake2RepoKey, Blake2AuthenticatedKey, \
+    AESOCBKeyfileKey, AESOCBRepoKey, CHPOKeyfileKey, CHPORepoKey
 from ..crypto.key import ID_HMAC_SHA_256, ID_BLAKE2b_256
 from ..crypto.key import TAMRequiredError, TAMInvalid, TAMUnsupportedSuiteError, UnsupportedManifestError
 from ..crypto.key import identify_key
@@ -80,6 +81,8 @@ class TestKey:
         Blake2KeyfileKey,
         Blake2RepoKey,
         Blake2AuthenticatedKey,
+        AESOCBKeyfileKey, AESOCBRepoKey,
+        CHPOKeyfileKey, CHPORepoKey,
     ))
     def key(self, request, monkeypatch):
         monkeypatch.setenv('BORG_PASSPHRASE', 'test')
@@ -111,18 +114,21 @@ class TestKey:
     def test_plaintext(self):
         key = PlaintextKey.create(None, None)
         chunk = b'foo'
-        assert hexlify(key.id_hash(chunk)) == b'2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae'
-        assert chunk == key.decrypt(key.id_hash(chunk), key.encrypt(chunk))
+        id = key.id_hash(chunk)
+        assert hexlify(id) == b'2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae'
+        assert chunk == key.decrypt(id, key.encrypt(id, chunk))
 
     def test_keyfile(self, monkeypatch, keys_dir):
         monkeypatch.setenv('BORG_PASSPHRASE', 'test')
         key = KeyfileKey.create(self.MockRepository(), self.MockArgs())
         assert key.cipher.next_iv() == 0
-        manifest = key.encrypt(b'ABC')
+        chunk = b'ABC'
+        id = key.id_hash(chunk)
+        manifest = key.encrypt(id, chunk)
         assert key.cipher.extract_iv(manifest) == 0
-        manifest2 = key.encrypt(b'ABC')
+        manifest2 = key.encrypt(id, chunk)
         assert manifest != manifest2
-        assert key.decrypt(None, manifest) == key.decrypt(None, manifest2)
+        assert key.decrypt(id, manifest) == key.decrypt(id, manifest2)
         assert key.cipher.extract_iv(manifest2) == 1
         iv = key.cipher.extract_iv(manifest)
         key2 = KeyfileKey.detect(self.MockRepository(), manifest)
@@ -131,7 +137,8 @@ class TestKey:
         assert len({key2.id_key, key2.enc_key, key2.enc_hmac_key}) == 3
         assert key2.chunk_seed != 0
         chunk = b'foo'
-        assert chunk == key2.decrypt(key.id_hash(chunk), key.encrypt(chunk))
+        id = key.id_hash(chunk)
+        assert chunk == key2.decrypt(id, key.encrypt(id, chunk))
 
     def test_keyfile_nonce_rollback_protection(self, monkeypatch, keys_dir):
         monkeypatch.setenv('BORG_PASSPHRASE', 'test')
@@ -139,9 +146,11 @@ class TestKey:
         with open(os.path.join(get_security_dir(repository.id_str), 'nonce'), "w") as fd:
             fd.write("0000000000002000")
         key = KeyfileKey.create(repository, self.MockArgs())
-        data = key.encrypt(b'ABC')
+        chunk = b'ABC'
+        id = key.id_hash(chunk)
+        data = key.encrypt(id, chunk)
         assert key.cipher.extract_iv(data) == 0x2000
-        assert key.decrypt(None, data) == b'ABC'
+        assert key.decrypt(id, data) == chunk
 
     def test_keyfile_kfenv(self, tmpdir, monkeypatch):
         keyfile = tmpdir.join('keyfile')
@@ -152,7 +161,7 @@ class TestKey:
         assert keyfile.exists()
         chunk = b'ABC'
         chunk_id = key.id_hash(chunk)
-        chunk_cdata = key.encrypt(chunk)
+        chunk_cdata = key.encrypt(chunk_id, chunk)
         key = KeyfileKey.detect(self.MockRepository(), chunk_cdata)
         assert chunk == key.decrypt(chunk_id, chunk_cdata)
         keyfile.remove()
@@ -209,18 +218,20 @@ class TestKey:
     def test_roundtrip(self, key):
         repository = key.repository
         plaintext = b'foo'
-        encrypted = key.encrypt(plaintext)
+        id = key.id_hash(plaintext)
+        encrypted = key.encrypt(id, plaintext)
         identified_key_class = identify_key(encrypted)
         assert identified_key_class == key.__class__
         loaded_key = identified_key_class.detect(repository, encrypted)
-        decrypted = loaded_key.decrypt(None, encrypted)
+        decrypted = loaded_key.decrypt(id, encrypted)
         assert decrypted == plaintext
 
     def test_decrypt_decompress(self, key):
         plaintext = b'123456789'
-        encrypted = key.encrypt(plaintext)
-        assert key.decrypt(None, encrypted, decompress=False) != plaintext
-        assert key.decrypt(None, encrypted) == plaintext
+        id = key.id_hash(plaintext)
+        encrypted = key.encrypt(id, plaintext)
+        assert key.decrypt(id, encrypted, decompress=False) != plaintext
+        assert key.decrypt(id, encrypted) == plaintext
 
     def test_assert_id(self, key):
         plaintext = b'123456789'
@@ -240,7 +251,8 @@ class TestKey:
         assert AuthenticatedKey.id_hash is ID_HMAC_SHA_256.id_hash
         assert len(key.id_key) == 32
         plaintext = b'123456789'
-        authenticated = key.encrypt(plaintext)
+        id = key.id_hash(plaintext)
+        authenticated = key.encrypt(id, plaintext)
         # 0x07 is the key TYPE, \x0000 identifies no compression.
         assert authenticated == b'\x07\x00\x00' + plaintext
 
@@ -250,7 +262,8 @@ class TestKey:
         assert Blake2AuthenticatedKey.id_hash is ID_BLAKE2b_256.id_hash
         assert len(key.id_key) == 128
         plaintext = b'123456789'
-        authenticated = key.encrypt(plaintext)
+        id = key.id_hash(plaintext)
+        authenticated = key.encrypt(id, plaintext)
         # 0x06 is the key TYPE, 0x0000 identifies no compression.
         assert authenticated == b'\x06\x00\x00' + plaintext
 

src/borg/testsuite/remote.py

@@ -165,7 +165,7 @@ class TestRepositoryCache:
 
     def _put_encrypted_object(self, key, repository, data):
         id_ = key.id_hash(data)
-        repository.put(id_, key.encrypt(data))
+        repository.put(id_, key.encrypt(id_, data))
         return id_
 
     @pytest.fixture