Merge pull request #7792 from ThomasWaldmann/rel126

Release 1.2.6

docs/changes.rst

@@ -35,21 +35,23 @@ borg version that has the relevant security patches for this vulnerability appli
 
 Steps you must take to upgrade a repository:
 
-1. Upgrade all clients using this repository to borg 1.2.5.
+1. Upgrade all clients using this repository to borg 1.2.6.
    Note: it is not required to upgrade a server, except if the server-side borg
    is also used as a client (and not just for "borg serve").
 
-   Do **not** run ``borg check`` with borg 1.2.5 before completing the upgrade steps.
+   Do **not** run ``borg check`` with borg > 1.2.4 before completing the upgrade steps.
 
-2. Run ``borg info --debug <repository> 2>&1 | grep TAM | grep -i manifest``.
+2. Run ``BORG_WORKAROUNDS=ignore_invalid_archive_tam borg info --debug <repo> 2>&1 | grep TAM | grep -i manifest``.
 
    a) If you get "TAM-verified manifest", continue with 3.
    b) If you get "Manifest TAM not found and not required", run
       ``borg upgrade --tam --force <repository>`` *on every client*.
 
-3. Run ``borg list --format='{name} {time} tam:{tam}{NL}' <repository>``.
+3. Run ``BORG_WORKAROUNDS=ignore_invalid_archive_tam borg list --format='{name} {time} tam:{tam}{NL}' <repo>``.
    "tam:verified" means that the archive has a valid TAM authentication.
    "tam:none" is expected as output for archives created by borg <1.0.9.
+   "tam:none" is also expected for archives resulting from a borg rename
+   or borg recreate operation (see #7791).
    "tam:none" could also come from archives created by an attacker.
    You should verify that "tam:none" archives are authentic and not malicious
    (== have good content, have correct timestamp, can be extracted successfully).
@@ -57,13 +59,15 @@ Steps you must take to upgrade a repository:
    In low-risk, trusted environments, you may decide on your own risk to skip step 3
    and just trust in everything being OK.
 
-4. If there are no tam:non archives left at this point, you can skip this step.
-   Run ``borg upgrade --archives-tam <repository>``.
-   This will make sure all archives are TAM authenticated (an archive TAM will be added
-   for all archives still missing one).
-   ``borg check`` would consider TAM-less archives as garbage or a potential attack.
-   Optionally run the same command as in step 3 to see that all archives now are "tam:verified".
+4. If there are no tam:none archives left at this point, you can skip this step.
+   Run ``BORG_WORKAROUNDS=ignore_invalid_archive_tam borg upgrade --archives-tam <repo>``.
+   This will unconditionally add a correct archive TAM to all archives not having one.
+   ``borg check`` would consider TAM-less or invalid-TAM archives as garbage or a potential attack.
+   To see that all archives now are "tam:verified" run: ``borg list --format='{name} {time} tam:{tam}{NL}' <repo>``
 
+5. Please note that you should never use BORG_WORKAROUNDS=ignore_invalid_archive_tam
+   for normal production operations - it is only needed once to get the archives in a
+   repository into a good state. All archives have a valid TAM now.
 
 Vulnerability time line:
 
@@ -71,7 +75,8 @@ Vulnerability time line:
 * 2023-06-13...: Work on fixing the issue, upgrade procedure, docs.
 * 2023-06-30: CVE was assigned via Github CNA
 * 2023-06-30 .. 2023-08-29: Fixed issue, code review, docs, testing.
-* 2023-08-30: Released fixed version 1.2.5
+* 2023-08-30: Released fixed version 1.2.5 (broken upgrade procedure for some repos)
+* 2023-08-31: Released fixed version 1.2.6 (fixes upgrade procedure)
 
 .. _hashindex_set_bug:
 
@@ -369,7 +374,7 @@ Compatibility notes:
 Change Log
 ==========
 
-Version 1.2.5 (2023-08-30)
+Version 1.2.6 (2023-08-31)
 --------------------------
 
 For upgrade and compatibility hints, please also read the section "Upgrade Notes"
@@ -377,8 +382,33 @@ above.
 
 Fixes:
 
+- The upgrade procedure docs as published with borg 1.2.5 did not work, if the
+  repository had archives resulting from a borg rename or borg recreate operation.
+
+  The updated docs now use BORG_WORKAROUNDS=ignore_invalid_archive_tam at some
+  places to avoid that issue, #7791.
+
+  See: fix pre-1.2.5 archives spoofing vulnerability (CVE-2023-36811),
+  details and necessary upgrade procedure described above.
+
+Other changes:
+
+- updated 1.2.5 changelog entry: 1.2.5 already has the fix for rename/recreate.
+- remove cython restrictions. recommended is to build with cython 0.29.latest,
+  because borg 1.2.x uses this since years and it is very stable.
+  you can also try to build with cython 3.0.x, there is a good chance that it works.
+  as a 3rd option, we also bundle the `*.c` files cython outputs in the release
+  pypi package, so you can also just use these and not need cython at all.
+
+
+Version 1.2.5 (2023-08-30)
+--------------------------
+
+Fixes:
+
 - Security: fix pre-1.2.5 archives spoofing vulnerability (CVE-2023-36811),
   see details and necessary upgrade procedure described above.
+- rename/recreate: correctly update resulting archive's TAM, see #7791
 - create: do not try to read parent dir of recursion root, #7746
 - extract: fix false warning about pattern never matching, #4110
 - diff: remove surrogates before output, #7535

docs/usage/general/environment.rst.inc

@@ -116,6 +116,14 @@ General:
 
             Now you can init a fresh repo. Make sure you do not use the workaround any more.
 
+        ignore_invalid_archive_tam
+            Work around invalid archive TAMs created by borg < 1.2.5, see :issue:`7791`.
+
+            This workaround likely needs to get used only once when following the upgrade
+            instructions for CVE-2023-36811, see :ref:`archives_tam_vuln`.
+
+            In normal production operations, this workaround should never be used.
+
 Some automatic "answerers" (if set, they automatically answer confirmation questions):
     BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=no (or =yes)
         For "Warning: Attempting to access a previously unknown unencrypted repository"

pyproject.toml

@@ -1,3 +1,3 @@
 [build-system]
-requires = ["setuptools", "pkgconfig", "Cython<3", "setuptools_scm>=1.7"]
+requires = ["setuptools", "pkgconfig", "Cython", "setuptools_scm>=1.7"]
 build-backend = "setuptools.build_meta"

requirements.d/development.txt

@@ -8,7 +8,7 @@ pytest
 pytest-xdist
 pytest-cov
 pytest-benchmark
-Cython <3
+Cython
 twine
 python-dateutil
 pre-commit

src/borg/crypto/key.py

@@ -285,7 +285,7 @@ class KeyBase:
         return unpacked, True
 
     def unpack_and_verify_archive(self, data, force_tam_not_required=False):
-        """Unpack msgpacked *data* and return (object, did_verify)."""
+        """Unpack msgpacked *data* and return (object, did_verify, salt)."""
         tam_required = self.tam_required
         if force_tam_not_required and tam_required:
             # for a long time, borg only checked manifest for "tam_required" and
@@ -322,7 +322,11 @@ class KeyBase:
         tam_key = self._tam_key(tam_salt, context=b'archive')
         calculated_hmac = hmac.digest(tam_key, data, 'sha512')
         if not hmac.compare_digest(calculated_hmac, tam_hmac):
-            raise ArchiveTAMInvalid()
+            if 'ignore_invalid_archive_tam' in workarounds:
+                logger.debug('ignoring invalid archive TAM due to BORG_WORKAROUNDS')
+                return unpacked, False, None  # same as if no TAM is present
+            else:
+                raise ArchiveTAMInvalid()
         logger.debug('TAM-verified archive')
         return unpacked, True, tam_salt