Merge pull request #3076 from borgbackup/rel110rc4

release 1.1.0rc4

docs/man/borg-benchmark-crud.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-BENCHMARK-CRUD 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-BENCHMARK-CRUD 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-benchmark-crud \- Benchmark Create, Read, Update, Delete for archives.
 .

docs/man/borg-benchmark.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-BENCHMARK 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-BENCHMARK 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-benchmark \- benchmark command
 .

docs/man/borg-break-lock.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-BREAK-LOCK 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-BREAK-LOCK 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-break-lock \- Break the repository lock (e.g. in case it was left by a dead borg.
 .

docs/man/borg-change-passphrase.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CHANGE-PASSPHRASE 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-CHANGE-PASSPHRASE 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-change-passphrase \- Change repository key file passphrase
 .
@@ -37,6 +37,11 @@ borg [common options] change\-passphrase [options] [REPOSITORY]
 .sp
 The key files used for repository encryption are optionally passphrase
 protected. This command can be used to change this passphrase.
+.sp
+Please note that this command only changes the passphrase, but not any
+secret protected by it (like e.g. encryption/MAC keys or chunker seed).
+Thus, changing the passphrase after passphrase and borg key got compromised
+does not protect future (nor past) backups to the same repository.
 .SH OPTIONS
 .sp
 See \fIborg\-common(1)\fP for common options of Borg commands.

docs/man/borg-check.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CHECK 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-CHECK 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-check \- Check repository consistency
 .

docs/man/borg-common.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-COMMON 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-COMMON 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-common \- Common options of Borg commands
 .

docs/man/borg-compression.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-COMPRESSION 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-COMPRESSION 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-compression \- Details regarding compression
 .

docs/man/borg-create.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CREATE 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-CREATE 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-create \- Create new archive
 .
@@ -54,13 +54,50 @@ In the archive name, you may use the following placeholders:
 {now}, {utcnow}, {fqdn}, {hostname}, {user} and some others.
 .sp
 Backup speed is increased by not reprocessing files that are already part of
-existing archives and weren\(aqt modified. Normally, detecting file modifications
-will take inode information into consideration. This is problematic for files
-located on sshfs and similar network file systems which do not provide stable
-inode numbers, such files will always be considered modified. The
-\fB\-\-ignore\-inode\fP flag can be used to prevent this and improve performance.
-This flag will reduce reliability of change detection however, with files
-considered unmodified as long as their size and modification time are unchanged.
+existing archives and weren\(aqt modified. The detection of unmodified files is
+done by comparing multiple file metadata values with previous values kept in
+the files cache.
+.sp
+This comparison can operate in different modes as given by \fB\-\-files\-cache\fP:
+.INDENT 0.0
+.IP \(bu 2
+ctime,size,inode (default)
+.IP \(bu 2
+mtime,size,inode (default behaviour of borg versions older than 1.1.0rc4)
+.IP \(bu 2
+ctime,size (ignore the inode number)
+.IP \(bu 2
+mtime,size (ignore the inode number)
+.IP \(bu 2
+rechunk,ctime (all files are considered modified \- rechunk, cache ctime)
+.IP \(bu 2
+rechunk,mtime (all files are considered modified \- rechunk, cache mtime)
+.IP \(bu 2
+disabled (disable the files cache, all files considered modified \- rechunk)
+.UNINDENT
+.sp
+inode number: better safety, but often unstable on network filesystems
+.sp
+Normally, detecting file modifications will take inode information into
+consideration to improve the reliability of file change detection.
+This is problematic for files located on sshfs and similar network file
+systems which do not provide stable inode numbers, such files will always
+be considered modified. You can use modes without \fIinode\fP in this case to
+improve performance, but reliability of change detection might be reduced.
+.sp
+ctime vs. mtime: safety vs. speed
+.INDENT 0.0
+.IP \(bu 2
+ctime is a rather safe way to detect changes to a file (metadata and contents)
+as it can not be set from userspace. But, a metadata\-only change will already
+update the ctime, so there might be some unnecessary chunking/hashing even
+without content changes. Some filesystems do not support ctime (change time).
+.IP \(bu 2
+mtime usually works and only updates if file contents were changed. But mtime
+can be arbitrarily set from userspace, e.g. to set mtime back to the same value
+it had before a content change happened. This can be used maliciously as well as
+well\-meant, but in both cases mtime based cache modes can be problematic.
+.UNINDENT
 .sp
 The mount points of filesystems or filesystem snapshots should be the same for every
 creation of a new archive to ensure fast operation. This is because the file cache that
@@ -104,7 +141,7 @@ only display items with the given status characters (see description)
 output stats as JSON. Implies \fB\-\-stats\fP\&.
 .TP
 .B \-\-no\-cache\-sync
-experimental: do not synchronize the cache. Implies \fB\-\-no\-files\-cache\fP\&.
+experimental: do not synchronize the cache. Implies not using the files cache.
 .UNINDENT
 .SS Exclusion options
 .INDENT 0.0
@@ -148,6 +185,9 @@ do not store ctime into archive
 .B \-\-ignore\-inode
 ignore inode data in the file metadata cache used to detect unchanged files.
 .TP
+.BI \-\-files\-cache \ MODE
+operate files cache in MODE. default: ctime,size,inode
+.TP
 .B \-\-read\-special
 open and read block and char device files as well as FIFOs as if they were regular files. Also follows symlinks pointing to these kinds of files.
 .UNINDENT
@@ -212,18 +252,21 @@ $ borg create \-\-chunker\-params 10,23,16,4095 /path/to/repo::small /smallstuff
 # Backup a raw device (must not be active/in use/mounted at that time)
 $ dd if=/dev/sdx bs=10M | borg create /path/to/repo::my\-sdx \-
 
-# No compression (default)
-$ borg create /path/to/repo::arch ~
+# No compression (none)
+$ borg create \-\-compression none /path/to/repo::arch ~
 
-# Super fast, low compression
-$ borg create \-\-compression lz4 /path/to/repo::arch ~
+# Super fast, low compression (lz4, default)
+$ borg create /path/to/repo::arch ~
 
-# Less fast, higher compression (N = 0..9)
+# Less fast, higher compression (zlib, N = 0..9)
 $ borg create \-\-compression zlib,N /path/to/repo::arch ~
 
-# Even slower, even higher compression (N = 0..9)
+# Even slower, even higher compression (lzma, N = 0..9)
 $ borg create \-\-compression lzma,N /path/to/repo::arch ~
 
+# Only compress compressible data with lzma,N (N = 0..9)
+$ borg create \-\-compression auto,lzma,N /path/to/repo::arch ~
+
 # Use short hostname, user name and current time in archive name
 $ borg create /path/to/repo::{hostname}\-{user}\-{now} ~
 # Similar, use the same datetime format as borg 1.1 will have as default

docs/man/borg-delete.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-DELETE 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-DELETE 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-delete \- Delete an existing repository or archives
 .

docs/man/borg-diff.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-DIFF 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-DIFF 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-diff \- Diff contents of two archives
 .

docs/man/borg-export-tar.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-EXPORT-TAR 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-EXPORT-TAR 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-export-tar \- Export archive contents as a tarball
 .

docs/man/borg-extract.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-EXTRACT 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-EXTRACT 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-extract \- Extract archive contents
 .

docs/man/borg-info.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-INFO 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-INFO 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-info \- Show archive details such as disk space used
 .

docs/man/borg-init.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-INIT 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-INIT 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-init \- Initialize an empty repository
 .

docs/man/borg-key-change-passphrase.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-CHANGE-PASSPHRASE 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-KEY-CHANGE-PASSPHRASE 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-key-change-passphrase \- Change repository key file passphrase
 .
@@ -37,6 +37,11 @@ borg [common options] key change\-passphrase [options] [REPOSITORY]
 .sp
 The key files used for repository encryption are optionally passphrase
 protected. This command can be used to change this passphrase.
+.sp
+Please note that this command only changes the passphrase, but not any
+secret protected by it (like e.g. encryption/MAC keys or chunker seed).
+Thus, changing the passphrase after passphrase and borg key got compromised
+does not protect future (nor past) backups to the same repository.
 .SH OPTIONS
 .sp
 See \fIborg\-common(1)\fP for common options of Borg commands.

docs/man/borg-key-export.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-EXPORT 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-KEY-EXPORT 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-key-export \- Export the repository key for backup
 .

docs/man/borg-key-import.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-IMPORT 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-KEY-IMPORT 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-key-import \- Import the repository key from backup
 .

docs/man/borg-key-migrate-to-repokey.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-MIGRATE-TO-REPOKEY 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-KEY-MIGRATE-TO-REPOKEY 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-key-migrate-to-repokey \- Migrate passphrase -> repokey
 .

docs/man/borg-key.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-KEY 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-key \- Manage a keyfile or repokey of a repository
 .

docs/man/borg-list.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-LIST 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-LIST 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-list \- List archive or repository contents
 .

docs/man/borg-mount.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-MOUNT 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-MOUNT 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-mount \- Mount archive or an entire repository as a FUSE filesystem
 .

docs/man/borg-patterns.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PATTERNS 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-PATTERNS 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-patterns \- Details regarding patterns
 .

docs/man/borg-placeholders.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PLACEHOLDERS 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-PLACEHOLDERS 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-placeholders \- Details regarding placeholders
 .

docs/man/borg-prune.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PRUNE 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-PRUNE 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-prune \- Prune repository archives according to specified rules
 .

docs/man/borg-recreate.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-RECREATE 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-RECREATE 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-recreate \- Re-create archives
 .

docs/man/borg-rename.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-RENAME 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-RENAME 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-rename \- Rename an existing archive
 .

docs/man/borg-serve.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-SERVE 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-SERVE 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-serve \- Start in server mode. This command is usually not used manually.
 .

docs/man/borg-umount.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-UMOUNT 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-UMOUNT 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-umount \- un-mount the FUSE filesystem
 .

docs/man/borg-upgrade.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-UPGRADE 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-UPGRADE 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-upgrade \- upgrade a repository from a previous version
 .

docs/man/borg-with-lock.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-WITH-LOCK 1 "2017-09-09" "" "borg backup tool"
+.TH BORG-WITH-LOCK 1 "2017-10-01" "" "borg backup tool"
 .SH NAME
 borg-with-lock \- run a user specified command with the repository lock held
 .

docs/usage/change-passphrase.rst.inc

@@ -43,4 +43,9 @@ Description
 ~~~~~~~~~~~
 
 The key files used for repository encryption are optionally passphrase
-protected. This command can be used to change this passphrase.
+protected. This command can be used to change this passphrase.
+
+Please note that this command only changes the passphrase, but not any
+secret protected by it (like e.g. encryption/MAC keys or chunker seed).
+Thus, changing the passphrase after passphrase and borg key got compromised
+does not protect future (nor past) backups to the same repository.

docs/usage/create.rst.inc

@@ -31,7 +31,7 @@ borg create
     +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
     |                                                       | ``--json``                                        | output stats as JSON. Implies ``--stats``.                                                                                                        |
     +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--no-cache-sync``                               | experimental: do not synchronize the cache. Implies ``--no-files-cache``.                                                                         |
+    |                                                       | ``--no-cache-sync``                               | experimental: do not synchronize the cache. Implies not using the files cache.                                                                    |
     +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
     | .. class:: borg-common-opt-ref                                                                                                                                                                                                                                |
     |                                                                                                                                                                                                                                                               |
@@ -65,6 +65,8 @@ borg create
     +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
     |                                                       | ``--ignore-inode``                                | ignore inode data in the file metadata cache used to detect unchanged files.                                                                      |
     +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--files-cache MODE``                            | operate files cache in MODE. default: ctime,size,inode                                                                                            |
+    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
     |                                                       | ``--read-special``                                | open and read block and char device files as well as FIFOs as if they were regular files. Also follows symlinks pointing to these kinds of files. |
     +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
     | **Archive options**                                                                                                                                                                                                                                           |
@@ -102,7 +104,7 @@ borg create
         --list          output verbose list of items (files, dirs, ...)
         --filter STATUSCHARS    only display items with the given status characters (see description)
         --json          output stats as JSON. Implies ``--stats``.
-        --no-cache-sync    experimental: do not synchronize the cache. Implies ``--no-files-cache``.
+        --no-cache-sync    experimental: do not synchronize the cache. Implies not using the files cache.
 
 
     :ref:`common_options`
@@ -124,6 +126,7 @@ borg create
         --noatime                 do not store atime into archive
         --noctime                 do not store ctime into archive
         --ignore-inode            ignore inode data in the file metadata cache used to detect unchanged files.
+        --files-cache MODE        operate files cache in MODE. default: ctime,size,inode
         --read-special            open and read block and char device files as well as FIFOs as if they were regular files. Also follows symlinks pointing to these kinds of files.
 
 
@@ -157,13 +160,39 @@ In the archive name, you may use the following placeholders:
 {now}, {utcnow}, {fqdn}, {hostname}, {user} and some others.
 
 Backup speed is increased by not reprocessing files that are already part of
-existing archives and weren't modified. Normally, detecting file modifications
-will take inode information into consideration. This is problematic for files
-located on sshfs and similar network file systems which do not provide stable
-inode numbers, such files will always be considered modified. The
-``--ignore-inode`` flag can be used to prevent this and improve performance.
-This flag will reduce reliability of change detection however, with files
-considered unmodified as long as their size and modification time are unchanged.
+existing archives and weren't modified. The detection of unmodified files is
+done by comparing multiple file metadata values with previous values kept in
+the files cache.
+
+This comparison can operate in different modes as given by ``--files-cache``:
+
+- ctime,size,inode (default)
+- mtime,size,inode (default behaviour of borg versions older than 1.1.0rc4)
+- ctime,size (ignore the inode number)
+- mtime,size (ignore the inode number)
+- rechunk,ctime (all files are considered modified - rechunk, cache ctime)
+- rechunk,mtime (all files are considered modified - rechunk, cache mtime)
+- disabled (disable the files cache, all files considered modified - rechunk)
+
+inode number: better safety, but often unstable on network filesystems
+
+Normally, detecting file modifications will take inode information into
+consideration to improve the reliability of file change detection.
+This is problematic for files located on sshfs and similar network file
+systems which do not provide stable inode numbers, such files will always
+be considered modified. You can use modes without `inode` in this case to
+improve performance, but reliability of change detection might be reduced.
+
+ctime vs. mtime: safety vs. speed
+
+- ctime is a rather safe way to detect changes to a file (metadata and contents)
+  as it can not be set from userspace. But, a metadata-only change will already
+  update the ctime, so there might be some unnecessary chunking/hashing even
+  without content changes. Some filesystems do not support ctime (change time).
+- mtime usually works and only updates if file contents were changed. But mtime
+  can be arbitrarily set from userspace, e.g. to set mtime back to the same value
+  it had before a content change happened. This can be used maliciously as well as
+  well-meant, but in both cases mtime based cache modes can be problematic.
 
 The mount points of filesystems or filesystem snapshots should be the same for every
 creation of a new archive to ensure fast operation. This is because the file cache that

docs/usage/key_change-passphrase.rst.inc

@@ -43,4 +43,9 @@ Description
 ~~~~~~~~~~~
 
 The key files used for repository encryption are optionally passphrase
-protected. This command can be used to change this passphrase.
+protected. This command can be used to change this passphrase.
+
+Please note that this command only changes the passphrase, but not any
+secret protected by it (like e.g. encryption/MAC keys or chunker seed).
+Thus, changing the passphrase after passphrase and borg key got compromised
+does not protect future (nor past) backups to the same repository.