Merge pull request #5920 from ThomasWaldmann/rel1117

release 1.1.17

Vagrantfile

@@ -299,17 +299,18 @@ def run_tests(boxname)
     cd /vagrant/borg/borg
     . ../borg-env/bin/activate
     if which pyenv 2> /dev/null; then
-      # for testing, use the earliest point releases of the supported python versions:
-      pyenv global 3.5.3 3.6.2 3.7.11 3.8.0 3.9.0 3.10-dev
-      pyenv local 3.5.3 3.6.2 3.7.11 3.8.0 3.9.0 3.10-dev
+      # for testing, use the earliest point releases of the supported python versions.
+      # on some dists, 3.10-dev does not compile, so if pyenv fails due to this, try without 3.10.
+      pyenv global 3.5.3 3.6.2 3.7.11 3.8.0 3.9.0 3.10-dev || pyenv global 3.5.3 3.6.2 3.7.11 3.8.0 3.9.0
+      pyenv local 3.5.3 3.6.2 3.7.11 3.8.0 3.9.0 3.10-dev || pyenv local 3.5.3 3.6.2 3.7.11 3.8.0 3.9.0
     fi
     # otherwise: just use the system python
     if which fakeroot 2> /dev/null; then
       echo "Running tox WITH fakeroot -u"
-      fakeroot -u tox --skip-missing-interpreters -e py35,py36,py37,py38,py39
+      fakeroot -u tox --skip-missing-interpreters -e py35,py36,py37,py38,py39,py310
     else
       echo "Running tox WITHOUT fakeroot -u"
-      tox --skip-missing-interpreters -e py35,py36,py37,py38,py39
+      tox --skip-missing-interpreters -e py35,py36,py37,py38,py39,py310
     fi
   EOF
 end

docs/changes.rst

@@ -217,7 +217,7 @@ The best check that everything is ok is to run a dry-run extraction::
 Changelog
 =========
 
-Version 1.1.17 (2021-07-xx)
+Version 1.1.17 (2021-07-12)
 ---------------------------
 
 Compatibility notes:
@@ -244,7 +244,7 @@ Compatibility notes:
   In case you have scripts expecting rc == 2 for a signal exit, you need to
   update them to check for >= 128 (as documented since long).
 - 1.1.15 drops python 3.4 support, minimum requirement is 3.5 now.
-- 1.1.17 install_requires now the "packaging" pypi package
+- 1.1.17 install_requires the "packaging" pypi package now.
 
 Fixes:
 
@@ -265,22 +265,25 @@ New features:
   (sys+user) per borg command invocation.
 - implement BORG_LIBC env variable to give the libc filename, #5870.
   you can use this if a borg does not find your libc.
-- check: add progress indicator for archive check
+- check: add progress indicator for archive check.
 - allow --files-cache=size (not recommended, make sure you know what you do)
 
 Other changes:
 
+- Python 3.10 now officially supported!
+  we test on py310-dev on github CI since a while and now also on the vagrant
+  machines, so it should work ok.
+- github CI: test on py310 (again)
 - get rid of distutils, use packaging and setuptools.
   distutils is deprecated and gives warnings on py 3.10.
 - setup.py: rename "clean" to "clean2" to avoid shadowing the "clean" command.
 - remove libc filename fallback for the BSDs (there is no "usual" name)
-- github CI: test on py310 (again)
-- cleanup flake8 checks, fix some pep8 violations
+- cleanup flake8 checks, fix some pep8 violations.
 - docs building: replace deprecated function ".add_stylesheet()" for Sphinx 4 compatibility
 - docs:
 
   - add a hint on sleeping computer and ssh connections, #5301
-  - update the documentation on hacked backup client, #5480.
+  - update the documentation on hacked backup client, #5480
   - improve docs/FAQ about append-only remote repos, #5497
   - complement the documentation for pattern files and exclude files, #5520
   - "filename with spaces" example added to exclude file, #5236
@@ -294,6 +297,14 @@ Other changes:
   - pull mode: add some warnings, #5827
   - mention tar --compare (compare archive to fs files), #5880
   - fix typos, backport of #5597
+- vagrant:
+
+  - add py3.7.11 for binary build, also add 3.10-dev.
+  - use latest Cython 0.29.23 for py310 compat fixes.
+  - more RAM for openindiana upgrade plan resolver, it just hangs (swaps?) if
+    there is too little RAM.
+  - fix install_pyenv to adapt to recent changes in pyenv (same as in master now).
+  - use generic/netbsd9 box, copied from master branch.
 
 Version 1.1.16 (2021-03-23)
 ---------------------------

docs/man/borg-benchmark-crud.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-BENCHMARK-CRUD 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-BENCHMARK-CRUD 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-benchmark-crud \- Benchmark Create, Read, Update, Delete for archives.
 .

docs/man/borg-benchmark.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-BENCHMARK 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-BENCHMARK 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-benchmark \- benchmark command
 .

docs/man/borg-break-lock.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-BREAK-LOCK 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-BREAK-LOCK 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-break-lock \- Break the repository lock (e.g. in case it was left by a dead borg.
 .

docs/man/borg-change-passphrase.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CHANGE-PASSPHRASE 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-CHANGE-PASSPHRASE 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-change-passphrase \- Change repository key file passphrase
 .

docs/man/borg-check.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CHECK 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-CHECK 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-check \- Check repository consistency
 .
@@ -40,6 +40,11 @@ The check command verifies the consistency of a repository and the corresponding
 check \-\-repair is a potentially dangerous function and might lead to data loss
 (for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL!
 .sp
+Pursuant to the previous warning it is also highly recommended to test the
+reliability of the hardware running this software with stress testing software
+such as memory testers. Unreliable hardware can also lead to data loss especially
+when this command is run in repair mode.
+.sp
 First, the underlying repository data files are checked:
 .INDENT 0.0
 .IP \(bu 2
@@ -80,6 +85,11 @@ In repair mode, when all the archives were checked, orphaned chunks are deleted
 from the repo. One cause of orphaned chunks are input file related errors (like
 read errors) in the archive creation process.
 .IP \(bu 2
+In verify\-data mode, a complete cryptographic verification of the archive data
+integrity is performed. This conflicts with \fB\-\-repository\-only\fP as this mode
+only makes sense if the archive checks are enabled. The full details of this mode
+are documented below.
+.IP \(bu 2
 If checking a remote repo via \fBssh:\fP, the archive check is executed on the
 client machine because it requires decryption, and this is always done client\-side
 as key access is needed.

docs/man/borg-common.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-COMMON 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-COMMON 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-common \- Common options of Borg commands
 .

docs/man/borg-compression.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-COMPRESSION 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-COMPRESSION 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-compression \- Details regarding compression
 .

docs/man/borg-config.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CONFIG 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-CONFIG 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-config \- get, set, and delete values in a repository or cache config file
 .

docs/man/borg-create.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CREATE 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-CREATE 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-create \- Create new archive
 .

docs/man/borg-delete.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-DELETE 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-DELETE 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-delete \- Delete an existing repository or archives
 .

docs/man/borg-diff.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-DIFF 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-DIFF 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-diff \- Diff contents of two archives
 .

docs/man/borg-export-tar.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-EXPORT-TAR 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-EXPORT-TAR 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-export-tar \- Export archive contents as a tarball
 .

docs/man/borg-extract.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-EXTRACT 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-EXTRACT 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-extract \- Extract archive contents
 .

docs/man/borg-info.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-INFO 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-INFO 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-info \- Show archive details such as disk space used
 .

docs/man/borg-init.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-INIT 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-INIT 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-init \- Initialize an empty repository
 .

docs/man/borg-key-change-passphrase.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-CHANGE-PASSPHRASE 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-KEY-CHANGE-PASSPHRASE 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-key-change-passphrase \- Change repository key file passphrase
 .

docs/man/borg-key-export.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-EXPORT 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-KEY-EXPORT 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-key-export \- Export the repository key for backup
 .

docs/man/borg-key-import.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-IMPORT 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-KEY-IMPORT 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-key-import \- Import the repository key from backup
 .

docs/man/borg-key-migrate-to-repokey.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-MIGRATE-TO-REPOKEY 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-KEY-MIGRATE-TO-REPOKEY 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-key-migrate-to-repokey \- Migrate passphrase -> repokey
 .

docs/man/borg-key.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-KEY 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-key \- Manage a keyfile or repokey of a repository
 .

docs/man/borg-list.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-LIST 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-LIST 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-list \- List archive or repository contents
 .

docs/man/borg-mount.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-MOUNT 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-MOUNT 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-mount \- Mount archive or an entire repository as a FUSE filesystem
 .

docs/man/borg-patterns.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PATTERNS 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-PATTERNS 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-patterns \- Details regarding patterns
 .
@@ -168,9 +168,9 @@ $ cat >exclude.txt <<EOF
 /home/*/junk
 *.tmp
 fm:aa:something/*
-re:^home/[^/]\e.tmp/
-sh:home/*/.thumbnails
- # Example with spaces, no need to escape as it is processed by borg
+re:^/home/[^/]+\e.tmp/
+sh:/home/*/.thumbnails
+# Example with spaces, no need to escape as it is processed by borg
 some file with spaces.txt
 EOF
 $ borg create \-\-exclude\-from exclude.txt backup /
@@ -190,7 +190,7 @@ with the prefix \-, an exclude\-norecurse rule starts with !, all followed by a
 .INDENT 3.5
 Via \fB\-\-pattern\fP or \fB\-\-patterns\-from\fP you can define BOTH inclusion and exclusion
 of files using pattern prefixes \fB+\fP and \fB\-\fP\&. With \fB\-\-exclude\fP and
-\fB\-\-exlude\-from\fP ONLY excludes are defined.
+\fB\-\-exclude\-from\fP ONLY excludes are defined.
 .UNINDENT
 .UNINDENT
 .sp

docs/man/borg-placeholders.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PLACEHOLDERS 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-PLACEHOLDERS 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-placeholders \- Details regarding placeholders
 .

docs/man/borg-prune.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PRUNE 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-PRUNE 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-prune \- Prune repository archives according to specified rules
 .

docs/man/borg-recreate.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-RECREATE 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-RECREATE 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-recreate \- Re-create archives
 .

docs/man/borg-rename.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-RENAME 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-RENAME 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-rename \- Rename an existing archive
 .

docs/man/borg-serve.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-SERVE 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-SERVE 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-serve \- Start in server mode. This command is usually not used manually.
 .

docs/man/borg-umount.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-UMOUNT 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-UMOUNT 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-umount \- un-mount the FUSE filesystem
 .

docs/man/borg-upgrade.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-UPGRADE 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-UPGRADE 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-upgrade \- upgrade a repository from a previous version
 .

docs/man/borg-with-lock.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-WITH-LOCK 1 "2021-03-22" "" "borg backup tool"
+.TH BORG-WITH-LOCK 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borg-with-lock \- run a user specified command with the repository lock held
 .

docs/man/borg.1

@@ -456,6 +456,21 @@ When set to no (default: yes), system information (like OS, Python version, ...)
 exceptions is not shown.
 Please only use for good reasons as it makes issues harder to analyze.
 .TP
+.B BORG_LIBC
+borg uses \fBctypes.util.find_library\fP to locate the \(aqc\(aq library (aka libc).
+find_library needs a shell and will invoke some tools like ldconfig, gcc/cc or objdump.
+If a shell or these tools are not available, you can give the name of your libc via
+BORG_LIBC=libc.so.6 (for example) and borg will not try the \fBfind_library\fP call.
+.TP
+.B BORG_SELFTEST
+This can be used to influence borg\(aqs builtin self\-tests. The default is to execute the tests
+at the beginning of each borg command invocation.
+.sp
+BORG_SELFTEST=disabled can be used to switch off the tests and rather save some time.
+Disabling is not recommended for normal borg users, but large scale borg storage providers can
+use this to optimize production servers after at least doing a one\-time test borg (with
+selftests not disabled) when installing or upgrading machines / OS / borg.
+.TP
 .B BORG_WORKAROUNDS
 A list of comma separated strings that trigger workarounds in borg,
 e.g. to work around bugs in other software.

docs/man/borgfs.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORGFS 1 "2021-03-22" "" "borg backup tool"
+.TH BORGFS 1 "2021-07-12" "" "borg backup tool"
 .SH NAME
 borgfs \- Mount archive or an entire repository as a FUSE filesystem
 .

docs/usage/help.rst.inc

@@ -133,9 +133,9 @@ Examples::
     /home/*/junk
     *.tmp
     fm:aa:something/*
-    re:^home/[^/]\.tmp/
-    sh:home/*/.thumbnails
-     # Example with spaces, no need to escape as it is processed by borg
+    re:^/home/[^/]+\.tmp/
+    sh:/home/*/.thumbnails
+    # Example with spaces, no need to escape as it is processed by borg
     some file with spaces.txt
     EOF
     $ borg create --exclude-from exclude.txt backup /
@@ -153,7 +153,7 @@ Examples::
 
         Via ``--pattern`` or ``--patterns-from`` you can define BOTH inclusion and exclusion
         of files using pattern prefixes ``+`` and ``-``. With ``--exclude`` and
-        ``--exlude-from`` ONLY excludes are defined.
+        ``--exclude-from`` ONLY excludes are defined.
 
     Inclusion patterns are useful to include paths that are contained in an excluded
     path. The first matching pattern is used so if an include pattern matches before