Merge pull request #3744 from ThomasWaldmann/rel115

release 1.1.5

docs/changes.rst

@@ -171,8 +171,8 @@ The best check that everything is ok is to run a dry-run extraction::
 Changelog
 =========
 
-Version 1.1.5 (not released yet)
---------------------------------
+Version 1.1.5 (2018-04-01)
+--------------------------
 
 Compatibility notes:
 
@@ -215,11 +215,15 @@ Fixes:
 - config: add some validation, #3566
 - repository config: add validation for max_segment_size, #3592
 - set cache previous_location on load instead of save
+- remove platform.uname() call which caused library mismatch issues, #3732
+- add exception handler around deprecated platform.linux_distribution() call
+- use same datetime object for {now} and {utcnow}, #3548
 
 New features:
 
 - create: implement --stdin-name, #3533
 - add chunker_params to borg archive info (--json)
+- BORG_SHOW_SYSINFO=no to hide system information from exceptions
 
 Other changes:
 
@@ -253,8 +257,8 @@ Other changes:
 
   - xenial64 box now uses username 'vagrant', #3707
   - move cleanup steps to fs_init, #3706
-
-
+  - the boxcutter wheezy boxes are 404, use local ones
+  - update to Python 3.5.5 (for binary builds)
 
 
 Version 1.1.4 (2017-12-31)

docs/man/borg-benchmark-crud.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-BENCHMARK-CRUD 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-BENCHMARK-CRUD 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-benchmark-crud \- Benchmark Create, Read, Update, Delete for archives.
 .

docs/man/borg-benchmark.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-BENCHMARK 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-BENCHMARK 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-benchmark \- benchmark command
 .

docs/man/borg-break-lock.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-BREAK-LOCK 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-BREAK-LOCK 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-break-lock \- Break the repository lock (e.g. in case it was left by a dead borg.
 .

docs/man/borg-change-passphrase.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CHANGE-PASSPHRASE 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-CHANGE-PASSPHRASE 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-change-passphrase \- Change repository key file passphrase
 .

docs/man/borg-check.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CHECK 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-CHECK 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-check \- Check repository consistency
 .

docs/man/borg-common.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-COMMON 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-COMMON 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-common \- Common options of Borg commands
 .

docs/man/borg-compression.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-COMPRESSION 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-COMPRESSION 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-compression \- Details regarding compression
 .

docs/man/borg-config.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CONFIG 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-CONFIG 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-config \- get, set, and delete values in a repository or cache config file
 .

docs/man/borg-create.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CREATE 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-CREATE 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-create \- Create new archive
 .
@@ -151,6 +151,9 @@ experimental: do not synchronize the cache. Implies not using the files cache.
 .TP
 .B \-\-no\-files\-cache
 do not load/update the file metadata cache used to detect unchanged files
+.TP
+.BI \-\-stdin\-name \ NAME
+use NAME in archive for stdin data (default: "stdin")
 .UNINDENT
 .SS Exclusion options
 .INDENT 0.0

docs/man/borg-delete.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-DELETE 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-DELETE 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-delete \- Delete an existing repository or archives
 .

docs/man/borg-diff.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-DIFF 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-DIFF 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-diff \- Diff contents of two archives
 .

docs/man/borg-export-tar.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-EXPORT-TAR 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-EXPORT-TAR 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-export-tar \- Export archive contents as a tarball
 .

docs/man/borg-extract.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-EXTRACT 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-EXTRACT 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-extract \- Extract archive contents
 .

docs/man/borg-info.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-INFO 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-INFO 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-info \- Show archive details such as disk space used
 .

docs/man/borg-init.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-INIT 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-INIT 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-init \- Initialize an empty repository
 .

docs/man/borg-key-change-passphrase.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-CHANGE-PASSPHRASE 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-KEY-CHANGE-PASSPHRASE 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-key-change-passphrase \- Change repository key file passphrase
 .

docs/man/borg-key-export.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-EXPORT 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-KEY-EXPORT 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-key-export \- Export the repository key for backup
 .

docs/man/borg-key-import.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-IMPORT 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-KEY-IMPORT 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-key-import \- Import the repository key from backup
 .

docs/man/borg-key-migrate-to-repokey.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-MIGRATE-TO-REPOKEY 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-KEY-MIGRATE-TO-REPOKEY 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-key-migrate-to-repokey \- Migrate passphrase -> repokey
 .

docs/man/borg-key.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-KEY 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-key \- Manage a keyfile or repokey of a repository
 .

docs/man/borg-list.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-LIST 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-LIST 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-list \- List archive or repository contents
 .

docs/man/borg-mount.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-MOUNT 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-MOUNT 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-mount \- Mount archive or an entire repository as a FUSE filesystem
 .

docs/man/borg-patterns.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PATTERNS 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-PATTERNS 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-patterns \- Details regarding patterns
 .
@@ -32,6 +32,15 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH DESCRIPTION
 .sp
+The path/filenames used as input for the pattern matching start from the
+currently active recursion root. You usually give the recursion root(s)
+when invoking borg and these can be either relative or absolute paths.
+.sp
+So, when you give \fIrelative/\fP as root, the paths going into the matcher
+will look like \fIrelative/.../file.ext\fP\&. When you give \fI/absolute/\fP as root,
+they will look like \fI/absolute/.../file.ext\fP\&. This is meant when we talk
+about "full path" below.
+.sp
 File patterns support these styles: fnmatch, shell, regular expressions,
 path prefixes and path full\-matches. By default, fnmatch is used for
 \fB\-\-exclude\fP patterns and shell\-style is used for the experimental \fB\-\-pattern\fP
@@ -52,8 +61,8 @@ matching any character not specified. For the purpose of these patterns,
 the path separator (\(aq\(aq for Windows and \(aq/\(aq on other systems) is not
 treated specially. Wrap meta\-characters in brackets for a literal
 match (i.e. \fI[?]\fP to match the literal character \fI?\fP). For a path
-to match a pattern, it must completely match from start to end, or
-must match from the start to just before a path separator. Except
+to match a pattern, the full path must match, or it must match
+from the start of the full path to just before a path separator. Except
 for the root path, paths will never end in the path separator when
 matching is attempted.  Thus, if a given pattern ends in a path
 separator, a \(aq*\(aq is appended before matching is attempted.
@@ -67,7 +76,7 @@ exception of any path separator.
 .TP
 .B Regular expressions, selector \fIre:\fP
 Regular expressions similar to those found in Perl are supported. Unlike
-shell patterns regular expressions are not required to match the complete
+shell patterns regular expressions are not required to match the full
 path and any substring match is sufficient. It is strongly recommended to
 anchor patterns to the start (\(aq^\(aq), to the end (\(aq$\(aq) or both. Path
 separators (\(aq\(aq for Windows and \(aq/\(aq on other systems) in paths are
@@ -77,13 +86,13 @@ the re module\fP\&.
 .TP
 .B Path prefix, selector \fIpp:\fP
 This pattern style is useful to match whole sub\-directories. The pattern
-\fIpp:/data/bar\fP matches \fI/data/bar\fP and everything therein.
+\fIpp:root/somedir\fP matches \fIroot/somedir\fP and everything therein.
 .TP
 .B Path full\-match, selector \fIpf:\fP
-This pattern style is useful to match whole paths.
+This pattern style is (only) useful to match full paths.
 This is kind of a pseudo pattern as it can not have any variable or
-unspecified parts \- the full, precise path must be given.
-\fIpf:/data/foo.txt\fP matches \fI/data/foo.txt\fP only.
+unspecified parts \- the full path must be given.
+\fIpf:root/file.ext\fP matches \fIroot/file.txt\fP only.
 .sp
 Implementation note: this is implemented via very time\-efficient O(1)
 hashtable lookups (this means you can have huge amounts of such patterns
@@ -175,7 +184,24 @@ Patterns (\fB\-\-pattern\fP) and excludes (\fB\-\-exclude\fP) from the command l
 considered first (in the order of appearance). Then patterns from \fB\-\-patterns\-from\fP
 are added. Exclusion patterns from \fB\-\-exclude\-from\fP files are appended last.
 .sp
-An example \fB\-\-patterns\-from\fP file could look like that:
+Examples:
+.INDENT 0.0
+.INDENT 3.5
+.sp
+.nf
+.ft C
+# backup pics, but not the ones from 2018, except the good ones:
+# note: using = is essential to avoid cmdline argument parsing issues.
+borg create \-\-pattern=+pics/2018/good \-\-pattern=\-pics/2018 repo::arch pics
+
+# use a file with patterns:
+borg create \-\-patterns\-from patterns.lst repo::arch
+.ft P
+.fi
+.UNINDENT
+.UNINDENT
+.sp
+The patterns.lst file could look like that:
 .INDENT 0.0
 .INDENT 3.5
 .sp

docs/man/borg-placeholders.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PLACEHOLDERS 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-PLACEHOLDERS 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-placeholders \- Details regarding placeholders
 .

docs/man/borg-prune.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PRUNE 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-PRUNE 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-prune \- Prune repository archives according to specified rules
 .

docs/man/borg-recreate.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-RECREATE 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-RECREATE 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-recreate \- Re-create archives
 .
@@ -151,8 +151,8 @@ manually specify the archive creation date/time (UTC, yyyy\-mm\-ddThh:mm:ss form
 .BI \-C \ COMPRESSION\fP,\fB \ \-\-compression \ COMPRESSION
 select compression algorithm, see the output of the "borg help compression" command for details.
 .TP
-.B \-\-recompress
-recompress data chunks according to \fB\-\-compression\fP if \fIif\-different\fP\&. When \fIalways\fP, chunks that are already compressed that way are not skipped, but compressed again. Only the algorithm is considered for \fIif\-different\fP, not the compression level (if any).
+.BI \-\-recompress \ MODE
+recompress data chunks according to \fB\-\-compression\fP\&. MODE \fIif\-different\fP: recompress if current compression is with a different compression algorithm (the level is not considered). MODE \fIalways\fP: recompress even if current compression is with the same compression algorithm (use this to change the compression level). MODE \fInever\fP (default): do not recompress.
 .TP
 .BI \-\-chunker\-params \ PARAMS
 specify the chunker parameters (CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or \fIdefault\fP to use the current defaults. default: 19,23,21,4095

docs/man/borg-rename.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-RENAME 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-RENAME 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-rename \- Rename an existing archive
 .

docs/man/borg-serve.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-SERVE 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-SERVE 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-serve \- Start in server mode. This command is usually not used manually.
 .

docs/man/borg-umount.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-UMOUNT 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-UMOUNT 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-umount \- un-mount the FUSE filesystem
 .

docs/man/borg-upgrade.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-UPGRADE 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-UPGRADE 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-upgrade \- upgrade a repository from a previous version
 .

docs/man/borg-with-lock.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-WITH-LOCK 1 "2017-12-30" "" "borg backup tool"
+.TH BORG-WITH-LOCK 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borg-with-lock \- run a user specified command with the repository lock held
 .

docs/man/borg.1

@@ -274,7 +274,26 @@ name).
 .sp
 If you have set BORG_REPO (see above) and an archive location is needed, use
 \fB::archive_name\fP \- the repo URL part is then read from BORG_REPO.
-.SS Type of log output
+.SS Logging
+.sp
+Borg writes all log output to stderr by default. But please note that something
+showing up on stderr does \fInot\fP indicate an error condition just because it is
+on stderr. Please check the log levels of the messages and the return code of
+borg for determining error, warning or success conditions.
+.sp
+If you want to capture the log output to a file, just redirect it:
+.INDENT 0.0
+.INDENT 3.5
+.sp
+.nf
+.ft C
+borg create repo::archive myfiles 2>> logfile
+.ft P
+.fi
+.UNINDENT
+.UNINDENT
+.sp
+Custom logging configurations can be implemented via BORG_LOGGING_CONF.
 .sp
 The log level of the builtin logging configuration defaults to WARNING.
 This is because we want Borg to be mostly silent and only output
@@ -408,6 +427,11 @@ When set to a numeric value, this determines the maximum "time to live" for the
 entries (default: 20). The files cache is used to quickly determine whether a file is unchanged.
 The FAQ explains this more detailed in: \fIalways_chunking\fP
 .TP
+.B BORG_SHOW_SYSINFO
+When set to no (default: yes), system information (like OS, Python version, ...) in
+exceptions is not shown.
+Please only use for good reasons as it makes issues harder to analyze.
+.TP
 .B TMPDIR
 where temporary files are stored (might need a lot of temporary space for some operations)
 .UNINDENT
@@ -545,12 +569,21 @@ server to get the approximate resource usage.
 .INDENT 0.0
 .TP
 .B CPU client:
-borg create: does chunking, hashing, compression, crypto (high CPU usage)
-chunks cache sync: quite heavy on CPU, doing lots of hashtable operations.
-borg extract: crypto, decompression (medium to high CPU usage)
-borg check: similar to extract, but depends on options given.
-borg prune / borg delete archive: low to medium CPU usage
-borg delete repo: done on the server
+.INDENT 7.0
+.IP \(bu 2
+\fBborg create:\fP does chunking, hashing, compression, crypto (high CPU usage)
+.IP \(bu 2
+\fBchunks cache sync:\fP quite heavy on CPU, doing lots of hashtable operations.
+.IP \(bu 2
+\fBborg extract:\fP crypto, decompression (medium to high CPU usage)
+.IP \(bu 2
+\fBborg check:\fP similar to extract, but depends on options given.
+.IP \(bu 2
+\fBborg prune / borg delete archive:\fP low to medium CPU usage
+.IP \(bu 2
+\fBborg delete repo:\fP done on the server
+.UNINDENT
+.sp
 It won\(aqt go beyond 100% of 1 core as the code is currently single\-threaded.
 Especially higher zlib and lzma compression levels use significant amounts
 of CPU cycles. Crypto might be cheap on the CPU (if hardware accelerated) or

docs/man/borgfs.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORGFS 1 "2017-12-30" "" "borg backup tool"
+.TH BORGFS 1 "2018-03-29" "" "borg backup tool"
 .SH NAME
 borgfs \- Mount archive or an entire repository as a FUSE filesystem
 .

docs/usage/create.rst.inc

@@ -35,6 +35,8 @@ borg create
     +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
     |                                                       | ``--no-files-cache``                              | do not load/update the file metadata cache used to detect unchanged files                                                                         |
     +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--stdin-name NAME``                             | use NAME in archive for stdin data (default: "stdin")                                                                                             |
+    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
     | .. class:: borg-common-opt-ref                                                                                                                                                                                                                                |
     |                                                                                                                                                                                                                                                               |
     | :ref:`common_options`                                                                                                                                                                                                                                         |
@@ -114,6 +116,7 @@ borg create
         --json          output stats as JSON. Implies ``--stats``.
         --no-cache-sync    experimental: do not synchronize the cache. Implies not using the files cache.
         --no-files-cache    do not load/update the file metadata cache used to detect unchanged files
+        --stdin-name NAME    use NAME in archive for stdin data (default: "stdin")
 
 
     :ref:`common_options`

docs/usage/help.rst.inc

@@ -6,6 +6,15 @@ borg help patterns
 ~~~~~~~~~~~~~~~~~~
 
 
+The path/filenames used as input for the pattern matching start from the
+currently active recursion root. You usually give the recursion root(s)
+when invoking borg and these can be either relative or absolute paths.
+
+So, when you give `relative/` as root, the paths going into the matcher
+will look like `relative/.../file.ext`. When you give `/absolute/` as root,
+they will look like `/absolute/.../file.ext`. This is meant when we talk
+about "full path" below.
+
 File patterns support these styles: fnmatch, shell, regular expressions,
 path prefixes and path full-matches. By default, fnmatch is used for
 ``--exclude`` patterns and shell-style is used for the experimental ``--pattern``
@@ -25,8 +34,8 @@ two alphanumeric characters followed by a colon (i.e. `aa:something/*`).
     the path separator ('\' for Windows and '/' on other systems) is not
     treated specially. Wrap meta-characters in brackets for a literal
     match (i.e. `[?]` to match the literal character `?`). For a path
-    to match a pattern, it must completely match from start to end, or
-    must match from the start to just before a path separator. Except
+    to match a pattern, the full path must match, or it must match
+    from the start of the full path to just before a path separator. Except
     for the root path, paths will never end in the path separator when
     matching is attempted.  Thus, if a given pattern ends in a path
     separator, a '\*' is appended before matching is attempted.
@@ -40,7 +49,7 @@ Shell-style patterns, selector `sh:`
 
 Regular expressions, selector `re:`
     Regular expressions similar to those found in Perl are supported. Unlike
-    shell patterns regular expressions are not required to match the complete
+    shell patterns regular expressions are not required to match the full
     path and any substring match is sufficient. It is strongly recommended to
     anchor patterns to the start ('^'), to the end ('$') or both. Path
     separators ('\' for Windows and '/' on other systems) in paths are
@@ -50,13 +59,13 @@ Regular expressions, selector `re:`
 
 Path prefix, selector `pp:`
     This pattern style is useful to match whole sub-directories. The pattern
-    `pp:/data/bar` matches `/data/bar` and everything therein.
+    `pp:root/somedir` matches `root/somedir` and everything therein.
 
 Path full-match, selector `pf:`
-    This pattern style is useful to match whole paths.
+    This pattern style is (only) useful to match full paths.
     This is kind of a pseudo pattern as it can not have any variable or
-    unspecified parts - the full, precise path must be given.
-    `pf:/data/foo.txt` matches `/data/foo.txt` only.
+    unspecified parts - the full path must be given.
+    `pf:root/file.ext` matches `root/file.txt` only.
 
     Implementation note: this is implemented via very time-efficient O(1)
     hashtable lookups (this means you can have huge amounts of such patterns
@@ -139,7 +148,16 @@ Examples::
     considered first (in the order of appearance). Then patterns from ``--patterns-from``
     are added. Exclusion patterns from ``--exclude-from`` files are appended last.
 
-    An example ``--patterns-from`` file could look like that::
+    Examples::
+
+        # backup pics, but not the ones from 2018, except the good ones:
+        # note: using = is essential to avoid cmdline argument parsing issues.
+        borg create --pattern=+pics/2018/good --pattern=-pics/2018 repo::arch pics
+
+        # use a file with patterns:
+        borg create --patterns-from patterns.lst repo::arch
+
+    The patterns.lst file could look like that::
 
         # "sh:" pattern style is the default, so the following line is not needed:
         P sh

docs/usage/recreate.rst.inc

@@ -12,59 +12,59 @@ borg recreate
 
     .. class:: borg-options-table
 
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    | **positional arguments**                                                                                                                                                                                                                                                                                                                                                        |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``REPOSITORY_OR_ARCHIVE``                         | repository/archive to recreate                                                                                                                                                                                                                                      |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``PATH``                                          | paths to recreate; patterns are supported                                                                                                                                                                                                                           |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    | **optional arguments**                                                                                                                                                                                                                                                                                                                                                          |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--list``                                        | output verbose list of items (files, dirs, ...)                                                                                                                                                                                                                     |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--filter STATUSCHARS``                          | only display items with the given status characters (listed in borg create --help)                                                                                                                                                                                  |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``-n``, ``--dry-run``                             | do not change anything                                                                                                                                                                                                                                              |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``-s``, ``--stats``                               | print statistics at end                                                                                                                                                                                                                                             |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    | .. class:: borg-common-opt-ref                                                                                                                                                                                                                                                                                                                                                  |
-    |                                                                                                                                                                                                                                                                                                                                                                                 |
-    | :ref:`common_options`                                                                                                                                                                                                                                                                                                                                                           |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    | **Exclusion options**                                                                                                                                                                                                                                                                                                                                                           |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``-e PATTERN``, ``--exclude PATTERN``             | exclude paths matching PATTERN                                                                                                                                                                                                                                      |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--exclude-from EXCLUDEFILE``                    | read exclude patterns from EXCLUDEFILE, one per line                                                                                                                                                                                                                |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--pattern PATTERN``                             | experimental: include/exclude paths matching PATTERN                                                                                                                                                                                                                |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--patterns-from PATTERNFILE``                   | experimental: read include/exclude patterns from PATTERNFILE, one per line                                                                                                                                                                                          |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--exclude-caches``                              | exclude directories that contain a CACHEDIR.TAG file (http://www.brynosaurus.com/cachedir/spec.html)                                                                                                                                                                |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--exclude-if-present NAME``                     | exclude directories that are tagged by containing a filesystem object with the given NAME                                                                                                                                                                           |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--keep-exclude-tags``, ``--keep-tag-files``     | if tag objects are specified with ``--exclude-if-present``, don't omit the tag objects themselves from the backup archive                                                                                                                                           |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    | **Archive options**                                                                                                                                                                                                                                                                                                                                                             |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--target TARGET``                               | create a new archive with the name ARCHIVE, do not replace existing archive (only applies for a single archive)                                                                                                                                                     |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``-c SECONDS``, ``--checkpoint-interval SECONDS`` | write checkpoint every SECONDS seconds (Default: 1800)                                                                                                                                                                                                              |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--comment COMMENT``                             | add a comment text to the archive                                                                                                                                                                                                                                   |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--timestamp TIMESTAMP``                         | manually specify the archive creation date/time (UTC, yyyy-mm-ddThh:mm:ss format). alternatively, give a reference file/directory.                                                                                                                                  |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``-C COMPRESSION``, ``--compression COMPRESSION`` | select compression algorithm, see the output of the "borg help compression" command for details.                                                                                                                                                                    |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--recompress``                                  | recompress data chunks according to ``--compression`` if `if-different`. When `always`, chunks that are already compressed that way are not skipped, but compressed again. Only the algorithm is considered for `if-different`, not the compression level (if any). |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--chunker-params PARAMS``                       | specify the chunker parameters (CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or `default` to use the current defaults. default: 19,23,21,4095                                                                                                    |
-    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    | **positional arguments**                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``REPOSITORY_OR_ARCHIVE``                         | repository/archive to recreate                                                                                                                                                                                                                                                                                                                                             |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``PATH``                                          | paths to recreate; patterns are supported                                                                                                                                                                                                                                                                                                                                  |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    | **optional arguments**                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--list``                                        | output verbose list of items (files, dirs, ...)                                                                                                                                                                                                                                                                                                                            |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--filter STATUSCHARS``                          | only display items with the given status characters (listed in borg create --help)                                                                                                                                                                                                                                                                                         |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``-n``, ``--dry-run``                             | do not change anything                                                                                                                                                                                                                                                                                                                                                     |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``-s``, ``--stats``                               | print statistics at end                                                                                                                                                                                                                                                                                                                                                    |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    | .. class:: borg-common-opt-ref                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+    |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+    | :ref:`common_options`                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    | **Exclusion options**                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``-e PATTERN``, ``--exclude PATTERN``             | exclude paths matching PATTERN                                                                                                                                                                                                                                                                                                                                             |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--exclude-from EXCLUDEFILE``                    | read exclude patterns from EXCLUDEFILE, one per line                                                                                                                                                                                                                                                                                                                       |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--pattern PATTERN``                             | experimental: include/exclude paths matching PATTERN                                                                                                                                                                                                                                                                                                                       |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--patterns-from PATTERNFILE``                   | experimental: read include/exclude patterns from PATTERNFILE, one per line                                                                                                                                                                                                                                                                                                 |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--exclude-caches``                              | exclude directories that contain a CACHEDIR.TAG file (http://www.brynosaurus.com/cachedir/spec.html)                                                                                                                                                                                                                                                                       |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--exclude-if-present NAME``                     | exclude directories that are tagged by containing a filesystem object with the given NAME                                                                                                                                                                                                                                                                                  |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--keep-exclude-tags``, ``--keep-tag-files``     | if tag objects are specified with ``--exclude-if-present``, don't omit the tag objects themselves from the backup archive                                                                                                                                                                                                                                                  |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    | **Archive options**                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--target TARGET``                               | create a new archive with the name ARCHIVE, do not replace existing archive (only applies for a single archive)                                                                                                                                                                                                                                                            |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``-c SECONDS``, ``--checkpoint-interval SECONDS`` | write checkpoint every SECONDS seconds (Default: 1800)                                                                                                                                                                                                                                                                                                                     |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--comment COMMENT``                             | add a comment text to the archive                                                                                                                                                                                                                                                                                                                                          |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--timestamp TIMESTAMP``                         | manually specify the archive creation date/time (UTC, yyyy-mm-ddThh:mm:ss format). alternatively, give a reference file/directory.                                                                                                                                                                                                                                         |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``-C COMPRESSION``, ``--compression COMPRESSION`` | select compression algorithm, see the output of the "borg help compression" command for details.                                                                                                                                                                                                                                                                           |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--recompress MODE``                             | recompress data chunks according to ``--compression``. MODE `if-different`: recompress if current compression is with a different compression algorithm (the level is not considered). MODE `always`: recompress even if current compression is with the same compression algorithm (use this to change the compression level). MODE `never` (default): do not recompress. |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--chunker-params PARAMS``                       | specify the chunker parameters (CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or `default` to use the current defaults. default: 19,23,21,4095                                                                                                                                                                                                           |
+    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 
     .. raw:: html
 
@@ -108,7 +108,7 @@ borg recreate
         --comment COMMENT                             add a comment text to the archive
         --timestamp TIMESTAMP                         manually specify the archive creation date/time (UTC, yyyy-mm-ddThh:mm:ss format). alternatively, give a reference file/directory.
         -C COMPRESSION, --compression COMPRESSION     select compression algorithm, see the output of the "borg help compression" command for details.
-        --recompress                                  recompress data chunks according to ``--compression`` if `if-different`. When `always`, chunks that are already compressed that way are not skipped, but compressed again. Only the algorithm is considered for `if-different`, not the compression level (if any).
+        --recompress MODE                             recompress data chunks according to ``--compression``. MODE `if-different`: recompress if current compression is with a different compression algorithm (the level is not considered). MODE `always`: recompress even if current compression is with the same compression algorithm (use this to change the compression level). MODE `never` (default): do not recompress.
         --chunker-params PARAMS                       specify the chunker parameters (CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or `default` to use the current defaults. default: 19,23,21,4095
 
 

docs/usage_general.rst.inc

@@ -330,6 +330,7 @@ CPU client:
     - **borg check:** similar to extract, but depends on options given.
     - **borg prune / borg delete archive:** low to medium CPU usage
     - **borg delete repo:** done on the server
+
     It won't go beyond 100% of 1 core as the code is currently single-threaded.
     Especially higher zlib and lzma compression levels use significant amounts
     of CPU cycles. Crypto might be cheap on the CPU (if hardware accelerated) or