docs: ran build_man, build_usage

docs/man/borg-benchmark-crud.1

@@ -0,0 +1,101 @@
+.\" Man page generated from reStructuredText.
+.
+.TH BORG-BENCHMARK-CRUD 1 "2017-05-17" "" "borg backup tool"
+.SH NAME
+borg-benchmark-crud \- Benchmark Create, Read, Update, Delete for archives.
+.
+.nr rst2man-indent-level 0
+.
+.de1 rstReportMargin
+\\$1 \\n[an-margin]
+level \\n[rst2man-indent-level]
+level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
+-
+\\n[rst2man-indent0]
+\\n[rst2man-indent1]
+\\n[rst2man-indent2]
+..
+.de1 INDENT
+.\" .rstReportMargin pre:
+. RS \\$1
+. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
+. nr rst2man-indent-level +1
+.\" .rstReportMargin post:
+..
+.de UNINDENT
+. RE
+.\" indent \\n[an-margin]
+.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
+.nr rst2man-indent-level -1
+.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
+.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
+..
+.SH SYNOPSIS
+.sp
+borg [common options] benchmark crud <options> REPO PATH
+.SH DESCRIPTION
+.sp
+This command benchmarks borg CRUD (create, read, update, delete) operations.
+.sp
+It creates input data below the given PATH and backups this data into the given REPO.
+The REPO must already exist (it could be a fresh empty repo or an existing repo, the
+command will create / read / update / delete some archives named borg\-test\-data* there.
+.sp
+Make sure you have free space there, you\(aqll need about 1GB each (+ overhead).
+.sp
+If your repository is encrypted and borg needs a passphrase to unlock the key, use:
+.sp
+BORG_PASSPHRASE=mysecret borg benchmark crud REPO PATH
+.sp
+Measurements are done with different input file sizes and counts.
+The file contents are very artificial (either all zero or all random),
+thus the measurement results do not necessarily reflect performance with real data.
+Also, due to the kind of content used, no compression is used in these benchmarks.
+.INDENT 0.0
+.TP
+.B C\- == borg create (1st archive creation, no compression, do not use files cache)
+C\-Z\- == all\-zero files. full dedup, this is primarily measuring reader/chunker/hasher.
+C\-R\- == random files. no dedup, measuring throughput through all processing stages.
+.TP
+.B R\- == borg extract (extract archive, dry\-run, do everything, but do not write files to disk)
+R\-Z\- == all zero files. Measuring heavily duplicated files.
+R\-R\- == random files. No duplication here, measuring throughput through all processing
+.IP "System Message: ERROR/3 (docs/virtmanpage.rst:, line 56)"
+Unexpected indentation.
+.INDENT 7.0
+.INDENT 3.5
+stages, except writing to disk.
+.UNINDENT
+.UNINDENT
+.TP
+.B U\- == borg create (2nd archive creation of unchanged input files, measure files cache speed)
+The throughput value is kind of virtual here, it does not actually read the file.
+U\-Z\- == needs to check the 2 all\-zero chunks\(aq existence in the repo.
+U\-R\- == needs to check existence of a lot of different chunks in the repo.
+.TP
+.B D\- == borg delete archive (delete last remaining archive, measure deletion + compaction)
+D\-Z\- == few chunks to delete / few segments to compact/remove.
+D\-R\- == many chunks to delete / many segments to compact/remove.
+.UNINDENT
+.sp
+Please note that there might be quite some variance in these measurements.
+Try multiple measurements and having a otherwise idle machine (and network, if you use it).
+.SH OPTIONS
+.sp
+See \fIborg\-common(1)\fP for common options of Borg commands.
+.SS arguments
+.INDENT 0.0
+.TP
+.B REPO
+repo to use for benchmark (must exist)
+.TP
+.B PATH
+path were to create benchmark input data
+.UNINDENT
+.SH SEE ALSO
+.sp
+\fIborg\-common(1)\fP
+.SH AUTHOR
+The Borg Collective
+.\" Generated by docutils manpage writer.
+.

docs/man/borg-benchmark.1

@@ -0,0 +1,47 @@
+.\" Man page generated from reStructuredText.
+.
+.TH BORG-BENCHMARK 1 "2017-05-17" "" "borg backup tool"
+.SH NAME
+borg-benchmark \- benchmark command
+.
+.nr rst2man-indent-level 0
+.
+.de1 rstReportMargin
+\\$1 \\n[an-margin]
+level \\n[rst2man-indent-level]
+level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
+-
+\\n[rst2man-indent0]
+\\n[rst2man-indent1]
+\\n[rst2man-indent2]
+..
+.de1 INDENT
+.\" .rstReportMargin pre:
+. RS \\$1
+. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
+. nr rst2man-indent-level +1
+.\" .rstReportMargin post:
+..
+.de UNINDENT
+. RE
+.\" indent \\n[an-margin]
+.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
+.nr rst2man-indent-level -1
+.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
+.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
+..
+.SH SYNOPSIS
+.nf
+borg [common options] benchmark crud ...
+.fi
+.sp
+.SH DESCRIPTION
+.sp
+These commands do various benchmarks.
+.SH SEE ALSO
+.sp
+\fIborg\-common(1)\fP, \fIborg\-benchmark\-crud(1)\fP
+.SH AUTHOR
+The Borg Collective
+.\" Generated by docutils manpage writer.
+.

docs/man/borg-break-lock.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-BREAK-LOCK 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-BREAK-LOCK 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-break-lock \- Break the repository lock (e.g. in case it was left by a dead borg.
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg break\-lock <options> REPOSITORY
+borg [common options] break\-lock <options> REPOSITORY
 .SH DESCRIPTION
 .sp
 This command breaks the repository and cache locks.

docs/man/borg-change-passphrase.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CHANGE-PASSPHRASE 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-CHANGE-PASSPHRASE 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-change-passphrase \- Change repository key file passphrase
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg change\-passphrase <options> REPOSITORY
+borg [common options] change\-passphrase <options> REPOSITORY
 .SH DESCRIPTION
 .sp
 The key files used for repository encryption are optionally passphrase

docs/man/borg-check.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CHECK 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-CHECK 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-check \- Check repository consistency
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg check <options> REPOSITORY_OR_ARCHIVE
+borg [common options] check <options> REPOSITORY_OR_ARCHIVE
 .SH DESCRIPTION
 .sp
 The check command verifies the consistency of a repository and the corresponding archives.
@@ -120,9 +120,6 @@ attempt to repair any inconsistencies found
 .TP
 .B \-\-save\-space
 work slower, but using less space
-.TP
-.B \-p\fP,\fB  \-\-progress
-show progress display while checking
 .UNINDENT
 .SS filters
 .INDENT 0.0

docs/man/borg-common.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-COMMON 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-COMMON 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-common \- Common options of Borg commands
 .
@@ -54,6 +54,9 @@ enable debug output, work on log level DEBUG
 .BI \-\-debug\-topic \ TOPIC
 enable TOPIC debugging (can be specified multiple times). The logger path is borg.debug.<TOPIC> if TOPIC is not fully qualified.
 .TP
+.B \-p\fP,\fB  \-\-progress
+show progress information
+.TP
 .B \-\-log\-json
 Output one JSON object per log line instead of formatted text.
 .TP
@@ -80,6 +83,9 @@ set remote network upload rate limit in kiByte/s (default: 0=unlimited)
 .TP
 .B \-\-consider\-part\-files
 treat part files like normal files (e.g. to list/extract them)
+.TP
+.BI \-\-debug\-profile \ FILE
+Write execution profile in Borg format into FILE. For local use a Python\-compatible file can be generated by suffixing FILE with ".pyprof".
 .UNINDENT
 .SH SEE ALSO
 .sp

docs/man/borg-compression.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-COMPRESSION 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-COMPRESSION 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-compression \- Details regarding compression
 .

docs/man/borg-create.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-CREATE 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-CREATE 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-create \- Create new archive
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg create <options> ARCHIVE PATH
+borg [common options] create <options> ARCHIVE PATH
 .SH DESCRIPTION
 .sp
 This command creates a backup archive containing all files found while recursively
@@ -63,6 +63,10 @@ creation of a new archive to ensure fast operation. This is because the file cac
 is used to determine changed files quickly uses absolute filenames.
 If this is not possible, consider creating a bind mount to a stable location.
 .sp
+The \-\-progress option shows (from left to right) Original, Compressed and Deduplicated
+(O, C and D, respectively), then the Number of files (N) processed so far, followed by
+the currently processed path.
+.sp
 See the output of the "borg help patterns" command for more help on exclude patterns.
 See the output of the "borg help placeholders" command for more help on placeholders.
 .SH OPTIONS
@@ -86,9 +90,6 @@ do not create a backup archive
 .B \-s\fP,\fB  \-\-stats
 print statistics for the created archive
 .TP
-.B \-p\fP,\fB  \-\-progress
-show progress display while creating the archive, showing Original, Compressed and Deduplicated sizes, followed by the Number of files seen and the path being processed, default: False
-.TP
 .B \-\-list
 output verbose list of items (files, dirs, ...)
 .TP

docs/man/borg-delete.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-DELETE 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-DELETE 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-delete \- Delete an existing repository or archives
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg delete <options> TARGET
+borg [common options] delete <options> TARGET
 .SH DESCRIPTION
 .sp
 This command deletes an archive from the repository or the complete repository.
@@ -50,9 +50,6 @@ archive or repository to delete
 .SS optional arguments
 .INDENT 0.0
 .TP
-.B \-p\fP,\fB  \-\-progress
-show progress display while deleting a single archive
-.TP
 .B \-s\fP,\fB  \-\-stats
 print statistics for the deleted archive
 .TP

docs/man/borg-diff.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-DIFF 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-DIFF 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-diff \- Diff contents of two archives
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg diff <options> REPO_ARCHIVE1 ARCHIVE2 PATH
+borg [common options] diff <options> REPO_ARCHIVE1 ARCHIVE2 PATH
 .SH DESCRIPTION
 .sp
 This command finds differences (file contents, user/group/mode) between archives.

docs/man/borg-export-tar.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-EXPORT-TAR 1 "2017-05-16" "" "borg backup tool"
+.TH BORG-EXPORT-TAR 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-export-tar \- Export archive contents as a tarball
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg export\-tar <options> ARCHIVE FILE PATH
+borg [common options] export\-tar <options> ARCHIVE FILE PATH
 .SH DESCRIPTION
 .sp
 This command creates a tarball from an archive.

docs/man/borg-extract.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-EXTRACT 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-EXTRACT 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-extract \- Extract archive contents
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg extract <options> ARCHIVE PATH
+borg [common options] extract <options> ARCHIVE PATH
 .SH DESCRIPTION
 .sp
 This command extracts the contents of an archive. By default the entire
@@ -45,6 +45,9 @@ See the output of the "borg help patterns" command for more help on exclude patt
 By using \fB\-\-dry\-run\fP, you can do all extraction steps except actually writing the
 output data: reading metadata and data chunks from the repo, checking the hash/hmac,
 decrypting, decompressing.
+.sp
+\fB\-\-progress\fP can be slower than no progress display, since it makes one additional
+pass over the archive metadata.
 .SH OPTIONS
 .sp
 See \fIborg\-common(1)\fP for common options of Borg commands.
@@ -60,9 +63,6 @@ paths to extract; patterns are supported
 .SS optional arguments
 .INDENT 0.0
 .TP
-.B \-p\fP,\fB  \-\-progress
-show progress while extracting (may be slower)
-.TP
 .B \-\-list
 output verbose list of items (files, dirs, ...)
 .TP

docs/man/borg-info.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-INFO 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-INFO 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-info \- Show archive details such as disk space used
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg info <options> REPOSITORY_OR_ARCHIVE
+borg [common options] info <options> REPOSITORY_OR_ARCHIVE
 .SH DESCRIPTION
 .sp
 This command displays detailed information about the specified archive or repository.

docs/man/borg-init.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-INIT 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-INIT 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-init \- Initialize an empty repository
 .
@@ -32,13 +32,13 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg init <options> REPOSITORY
+borg [common options] init <options> REPOSITORY
 .SH DESCRIPTION
 .sp
 This command initializes an empty repository. A repository is a filesystem
 directory containing the deduplicated data from zero or more archives.
 .sp
-Encryption can be enabled at repository init time.
+Encryption can be enabled at repository init time. It cannot be changed later.
 .sp
 It is not recommended to work without encryption. Repository encryption protects
 you e.g. against the case that an attacker has access to your backup repository.
@@ -94,8 +94,8 @@ These modes are new and \fInot\fP compatible with borg 1.0.x.
 .sp
 \fIauthenticated\fP mode uses no encryption, but authenticates repository contents
 through the same keyed BLAKE2b\-256 hash as the other blake2 modes (it uses it
-as chunk ID hash). The key is stored like repokey.
-This mode is new and not compatible with borg 1.0.x.
+as the chunk ID hash). The key is stored like repokey.
+This mode is new and \fInot\fP compatible with borg 1.0.x.
 .sp
 \fInone\fP mode uses no encryption and no authentication. It uses sha256 as chunk
 ID hash. Not recommended, rather consider using an authenticated or
@@ -123,7 +123,7 @@ repository to create
 .INDENT 0.0
 .TP
 .B \-e\fP,\fB  \-\-encryption
-select encryption key mode
+select encryption key mode \fB(required)\fP
 .TP
 .B \-a\fP,\fB  \-\-append\-only
 create an append\-only mode repository

docs/man/borg-key-change-passphrase.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-CHANGE-PASSPHRASE 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-KEY-CHANGE-PASSPHRASE 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-key-change-passphrase \- Change repository key file passphrase
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg key change\-passphrase <options> REPOSITORY
+borg [common options] key change\-passphrase <options> REPOSITORY
 .SH DESCRIPTION
 .sp
 The key files used for repository encryption are optionally passphrase

docs/man/borg-key-export.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-EXPORT 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-KEY-EXPORT 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-key-export \- Export the repository key for backup
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg key export <options> REPOSITORY PATH
+borg [common options] key export <options> REPOSITORY PATH
 .SH DESCRIPTION
 .sp
 If repository encryption is used, the repository is inaccessible

docs/man/borg-key-import.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-IMPORT 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-KEY-IMPORT 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-key-import \- Import the repository key from backup
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg key import <options> REPOSITORY PATH
+borg [common options] key import <options> REPOSITORY PATH
 .SH DESCRIPTION
 .sp
 This command allows to restore a key previously backed up with the

docs/man/borg-key-migrate-to-repokey.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY-MIGRATE-TO-REPOKEY 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-KEY-MIGRATE-TO-REPOKEY 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-key-migrate-to-repokey \- Migrate passphrase -> repokey
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg key migrate\-to\-repokey <options> REPOSITORY
+borg [common options] key migrate\-to\-repokey <options> REPOSITORY
 .SH DESCRIPTION
 .sp
 This command migrates a repository from passphrase mode (removed in Borg 1.0)

docs/man/borg-key.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-KEY 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-KEY 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-key \- Manage a keyfile or repokey of a repository
 .
@@ -32,10 +32,10 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .nf
-borg key export ...
-borg key import ...
-borg key change\-passphrase ...
-borg key migrate\-to\-repokey ...
+borg [common options] key export ...
+borg [common options] key import ...
+borg [common options] key change\-passphrase ...
+borg [common options] key migrate\-to\-repokey ...
 .fi
 .sp
 .SH SEE ALSO

docs/man/borg-list.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-LIST 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-LIST 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-list \- List archive or repository contents
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg list <options> REPOSITORY_OR_ARCHIVE PATH
+borg [common options] list <options> REPOSITORY_OR_ARCHIVE PATH
 .SH DESCRIPTION
 .sp
 This command lists the contents of a repository or an archive.
@@ -61,7 +61,10 @@ specify format for file listing
 (default: "{mode} {user:6} {group:6} {size:8d} {isomtime} {path}{extra}{NL}")
 .TP
 .B \-\-json
-format output as JSON. The form of \-\-format is ignored, but keys used in it are added to the JSON output. Some keys are always present. Note: JSON can only represent text. A "bpath" key is therefore not available.
+Only valid for listing repository contents. Format output as JSON. The form of \-\-format is ignored, but keys used in it are added to the JSON output. Some keys are always present. Note: JSON can only represent text. A "barchive" key is therefore not available.
+.TP
+.B \-\-json\-lines
+Only valid for listing archive contents. Format output as JSON Lines. The form of \-\-format is ignored, but keys used in it are added to the JSON output. Some keys are always present. Note: JSON can only represent text. A "bpath" key is therefore not available.
 .UNINDENT
 .SS filters
 .INDENT 0.0

docs/man/borg-mount.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-MOUNT 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-MOUNT 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-mount \- Mount archive or an entire repository as a FUSE filesystem
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg mount <options> REPOSITORY_OR_ARCHIVE MOUNTPOINT
+borg [common options] mount <options> REPOSITORY_OR_ARCHIVE MOUNTPOINT
 .SH DESCRIPTION
 .sp
 This command mounts an archive as a FUSE filesystem. This can be useful for

docs/man/borg-patterns.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PATTERNS 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-PATTERNS 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-patterns \- Details regarding patterns
 .

docs/man/borg-placeholders.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PLACEHOLDERS 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-PLACEHOLDERS 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-placeholders \- Details regarding placeholders
 .

docs/man/borg-prune.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-PRUNE 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-PRUNE 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-prune \- Prune repository archives according to specified rules
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg prune <options> REPOSITORY
+borg [common options] prune <options> REPOSITORY
 .SH DESCRIPTION
 .sp
 The prune command prunes a repository by deleting all archives not matching
@@ -91,9 +91,6 @@ do not change repository
 .B \-\-force
 force pruning of corrupted archives
 .TP
-.B \-p\fP,\fB  \-\-progress
-show progress display while deleting archives
-.TP
 .B \-s\fP,\fB  \-\-stats
 print statistics for the deleted archive
 .TP

docs/man/borg-recreate.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-RECREATE 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-RECREATE 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-recreate \- Re-create archives
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg recreate <options> REPOSITORY_OR_ARCHIVE PATH
+borg [common options] recreate <options> REPOSITORY_OR_ARCHIVE PATH
 .SH DESCRIPTION
 .sp
 Recreate the contents of existing archives.
@@ -92,9 +92,6 @@ output verbose list of items (files, dirs, ...)
 .BI \-\-filter \ STATUSCHARS
 only display items with the given status characters
 .TP
-.B \-p\fP,\fB  \-\-progress
-show progress display while recreating archives
-.TP
 .B \-n\fP,\fB  \-\-dry\-run
 do not change anything
 .TP

docs/man/borg-rename.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-RENAME 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-RENAME 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-rename \- Rename an existing archive
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg rename <options> ARCHIVE NEWNAME
+borg [common options] rename <options> ARCHIVE NEWNAME
 .SH DESCRIPTION
 .sp
 This command renames an archive in the repository.

docs/man/borg-serve.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-SERVE 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-SERVE 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-serve \- Start in server mode. This command is usually not used manually.
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg serve <options>
+borg [common options] serve <options>
 .SH DESCRIPTION
 .sp
 This command starts a repository server process. This command is usually not used manually.

docs/man/borg-umount.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-UMOUNT 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-UMOUNT 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-umount \- un-mount the FUSE filesystem
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg umount <options> MOUNTPOINT
+borg [common options] umount <options> MOUNTPOINT
 .SH DESCRIPTION
 .sp
 This command un\-mounts a FUSE filesystem that was mounted with \fBborg mount\fP\&.

docs/man/borg-upgrade.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-UPGRADE 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-UPGRADE 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-upgrade \- upgrade a repository from a previous version
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg upgrade <options> REPOSITORY
+borg [common options] upgrade <options> REPOSITORY
 .SH DESCRIPTION
 .sp
 Upgrade an existing, local Borg repository.
@@ -129,9 +129,6 @@ path to the repository to be upgraded
 .SS optional arguments
 .INDENT 0.0
 .TP
-.B \-p\fP,\fB  \-\-progress
-show progress display while upgrading the repository
-.TP
 .B \-n\fP,\fB  \-\-dry\-run
 do not change repository
 .TP

docs/man/borg-with-lock.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH BORG-WITH-LOCK 1 "2017-04-29" "" "borg backup tool"
+.TH BORG-WITH-LOCK 1 "2017-05-17" "" "borg backup tool"
 .SH NAME
 borg-with-lock \- run a user specified command with the repository lock held
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg with\-lock <options> REPOSITORY COMMAND ARGS
+borg [common options] with\-lock <options> REPOSITORY COMMAND ARGS
 .SH DESCRIPTION
 .sp
 This command runs a user\-specified command while the repository lock is held.

docs/man/borg.1

@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-borg <command> [options] [arguments]
+borg [common options] <command> [options] [arguments]
 .SH DESCRIPTION
 .\" we don't include the README.rst here since we want to keep this terse.
 .

docs/usage/benchmark_crud.rst.inc

@@ -0,0 +1,59 @@
+.. IMPORTANT: this file is auto-generated from borg's built-in help, do not edit!
+
+.. _borg_benchmark_crud:
+
+borg benchmark crud
+-------------------
+::
+
+    borg [common options] benchmark crud <options> REPO PATH
+
+positional arguments
+    REPO
+        repo to use for benchmark (must exist)
+    PATH
+        path were to create benchmark input data
+
+`Common options`_
+    |
+
+Description
+~~~~~~~~~~~
+
+This command benchmarks borg CRUD (create, read, update, delete) operations.
+
+It creates input data below the given PATH and backups this data into the given REPO.
+The REPO must already exist (it could be a fresh empty repo or an existing repo, the
+command will create / read / update / delete some archives named borg-test-data* there.
+
+Make sure you have free space there, you'll need about 1GB each (+ overhead).
+
+If your repository is encrypted and borg needs a passphrase to unlock the key, use:
+
+BORG_PASSPHRASE=mysecret borg benchmark crud REPO PATH
+
+Measurements are done with different input file sizes and counts.
+The file contents are very artificial (either all zero or all random),
+thus the measurement results do not necessarily reflect performance with real data.
+Also, due to the kind of content used, no compression is used in these benchmarks.
+
+C- == borg create (1st archive creation, no compression, do not use files cache)
+      C-Z- == all-zero files. full dedup, this is primarily measuring reader/chunker/hasher.
+      C-R- == random files. no dedup, measuring throughput through all processing stages.
+
+R- == borg extract (extract archive, dry-run, do everything, but do not write files to disk)
+      R-Z- == all zero files. Measuring heavily duplicated files.
+      R-R- == random files. No duplication here, measuring throughput through all processing
+              stages, except writing to disk.
+
+U- == borg create (2nd archive creation of unchanged input files, measure files cache speed)
+      The throughput value is kind of virtual here, it does not actually read the file.
+      U-Z- == needs to check the 2 all-zero chunks' existence in the repo.
+      U-R- == needs to check existence of a lot of different chunks in the repo.
+
+D- == borg delete archive (delete last remaining archive, measure deletion + compaction)
+      D-Z- == few chunks to delete / few segments to compact/remove.
+      D-R- == many chunks to delete / many segments to compact/remove.
+
+Please note that there might be quite some variance in these measurements.
+Try multiple measurements and having a otherwise idle machine (and network, if you use it).

docs/usage/break-lock.rst.inc

@@ -6,7 +6,7 @@ borg break-lock
 ---------------
 ::
 
-    borg break-lock <options> REPOSITORY
+    borg [common options] break-lock <options> REPOSITORY
 
 positional arguments
     REPOSITORY

docs/usage/change-passphrase.rst.inc

@@ -6,7 +6,7 @@ borg change-passphrase
 ----------------------
 ::
 
-    borg change-passphrase <options> REPOSITORY
+    borg [common options] change-passphrase <options> REPOSITORY
 
 positional arguments
     REPOSITORY

docs/usage/check.rst.inc

@@ -6,7 +6,7 @@ borg check
 ----------
 ::
 
-    borg check <options> REPOSITORY_OR_ARCHIVE
+    borg [common options] check <options> REPOSITORY_OR_ARCHIVE
 
 positional arguments
     REPOSITORY_OR_ARCHIVE
@@ -23,8 +23,6 @@ optional arguments
         | attempt to repair any inconsistencies found
     ``--save-space``
         | work slower, but using less space
-    ``-p``, ``--progress``
-        | show progress display while checking
 
 `Common options`_
     |

docs/usage/common-options.rst.inc

@@ -12,6 +12,8 @@
         | enable debug output, work on log level DEBUG
     ``--debug-topic TOPIC``
         | enable TOPIC debugging (can be specified multiple times). The logger path is borg.debug.<TOPIC> if TOPIC is not fully qualified.
+    ``-p``, ``--progress``
+        | show progress information
     ``--log-json``
         | Output one JSON object per log line instead of formatted text.
     ``--lock-wait N``
@@ -29,4 +31,6 @@
     ``--remote-ratelimit rate``
         | set remote network upload rate limit in kiByte/s (default: 0=unlimited)
     ``--consider-part-files``
-        | treat part files like normal files (e.g. to list/extract them)
+        | treat part files like normal files (e.g. to list/extract them)
+    ``--debug-profile FILE``
+        | Write execution profile in Borg format into FILE. For local use a Python-compatible file can be generated by suffixing FILE with ".pyprof".

docs/usage/create.rst.inc

@@ -6,7 +6,7 @@ borg create
 -----------
 ::
 
-    borg create <options> ARCHIVE PATH
+    borg [common options] create <options> ARCHIVE PATH
 
 positional arguments
     ARCHIVE
@@ -19,8 +19,6 @@ optional arguments
         | do not create a backup archive
     ``-s``, ``--stats``
         | print statistics for the created archive
-    ``-p``, ``--progress``
-        | show progress display while creating the archive, showing Original, Compressed and Deduplicated sizes, followed by the Number of files seen and the path being processed, default: False
     ``--list``
         | output verbose list of items (files, dirs, ...)
     ``--filter STATUSCHARS``
@@ -104,6 +102,10 @@ creation of a new archive to ensure fast operation. This is because the file cac
 is used to determine changed files quickly uses absolute filenames.
 If this is not possible, consider creating a bind mount to a stable location.
 
+The --progress option shows (from left to right) Original, Compressed and Deduplicated
+(O, C and D, respectively), then the Number of files (N) processed so far, followed by
+the currently processed path.
+
 See the output of the "borg help patterns" command for more help on exclude patterns.
 See the output of the "borg help placeholders" command for more help on placeholders.
 

docs/usage/delete.rst.inc

@@ -6,15 +6,13 @@ borg delete
 -----------
 ::
 
-    borg delete <options> TARGET
+    borg [common options] delete <options> TARGET
 
 positional arguments
     TARGET
         archive or repository to delete
 
 optional arguments
-    ``-p``, ``--progress``
-        | show progress display while deleting a single archive
     ``-s``, ``--stats``
         | print statistics for the deleted archive
     ``-c``, ``--cache-only``

docs/usage/diff.rst.inc

@@ -6,7 +6,7 @@ borg diff
 ---------
 ::
 
-    borg diff <options> REPO_ARCHIVE1 ARCHIVE2 PATH
+    borg [common options] diff <options> REPO_ARCHIVE1 ARCHIVE2 PATH
 
 positional arguments
     REPO_ARCHIVE1

docs/usage/export-tar.rst.inc

@@ -6,7 +6,7 @@ borg export-tar
 ---------------
 ::
 
-    borg export-tar <options> ARCHIVE FILE PATH
+    borg [common options] export-tar <options> ARCHIVE FILE PATH
 
 positional arguments
     ARCHIVE

docs/usage/extract.rst.inc

@@ -6,7 +6,7 @@ borg extract
 ------------
 ::
 
-    borg extract <options> ARCHIVE PATH
+    borg [common options] extract <options> ARCHIVE PATH
 
 positional arguments
     ARCHIVE
@@ -15,8 +15,6 @@ positional arguments
         paths to extract; patterns are supported
 
 optional arguments
-    ``-p``, ``--progress``
-        | show progress while extracting (may be slower)
     ``--list``
         | output verbose list of items (files, dirs, ...)
     ``-n``, ``--dry-run``
@@ -53,4 +51,7 @@ See the output of the "borg help patterns" command for more help on exclude patt
 
 By using ``--dry-run``, you can do all extraction steps except actually writing the
 output data: reading metadata and data chunks from the repo, checking the hash/hmac,
-decrypting, decompressing.
+decrypting, decompressing.
+
+``--progress`` can be slower than no progress display, since it makes one additional
+pass over the archive metadata.

docs/usage/info.rst.inc

@@ -6,7 +6,7 @@ borg info
 ---------
 ::
 
-    borg info <options> REPOSITORY_OR_ARCHIVE
+    borg [common options] info <options> REPOSITORY_OR_ARCHIVE
 
 positional arguments
     REPOSITORY_OR_ARCHIVE

docs/usage/init.rst.inc

@@ -6,7 +6,7 @@ borg init
 ---------
 ::
 
-    borg init <options> REPOSITORY
+    borg [common options] init <options> REPOSITORY
 
 positional arguments
     REPOSITORY
@@ -14,7 +14,7 @@ positional arguments
 
 optional arguments
     ``-e``, ``--encryption``
-        | select encryption key mode
+        | select encryption key mode **(required)**
     ``-a``, ``--append-only``
         | create an append-only mode repository
 
@@ -27,7 +27,7 @@ Description
 This command initializes an empty repository. A repository is a filesystem
 directory containing the deduplicated data from zero or more archives.
 
-Encryption can be enabled at repository init time.
+Encryption can be enabled at repository init time. It cannot be changed later.
 
 It is not recommended to work without encryption. Repository encryption protects
 you e.g. against the case that an attacker has access to your backup repository.
@@ -82,8 +82,8 @@ These modes are new and *not* compatible with borg 1.0.x.
 
 `authenticated` mode uses no encryption, but authenticates repository contents
 through the same keyed BLAKE2b-256 hash as the other blake2 modes (it uses it
-as chunk ID hash). The key is stored like repokey.
-This mode is new and not compatible with borg 1.0.x.
+as the chunk ID hash). The key is stored like repokey.
+This mode is new and *not* compatible with borg 1.0.x.
 
 `none` mode uses no encryption and no authentication. It uses sha256 as chunk
 ID hash. Not recommended, rather consider using an authenticated or

docs/usage/key_change-passphrase.rst.inc

@@ -6,7 +6,7 @@ borg key change-passphrase
 --------------------------
 ::
 
-    borg key change-passphrase <options> REPOSITORY
+    borg [common options] key change-passphrase <options> REPOSITORY
 
 positional arguments
     REPOSITORY

docs/usage/key_export.rst.inc

@@ -6,7 +6,7 @@ borg key export
 ---------------
 ::
 
-    borg key export <options> REPOSITORY PATH
+    borg [common options] key export <options> REPOSITORY PATH
 
 positional arguments
     REPOSITORY

docs/usage/key_import.rst.inc

@@ -6,7 +6,7 @@ borg key import
 ---------------
 ::
 
-    borg key import <options> REPOSITORY PATH
+    borg [common options] key import <options> REPOSITORY PATH
 
 positional arguments
     REPOSITORY

docs/usage/key_migrate-to-repokey.rst.inc

@@ -6,7 +6,7 @@ borg key migrate-to-repokey
 ---------------------------
 ::
 
-    borg key migrate-to-repokey <options> REPOSITORY
+    borg [common options] key migrate-to-repokey <options> REPOSITORY
 
 positional arguments
     REPOSITORY

docs/usage/list.rst.inc

@@ -6,7 +6,7 @@ borg list
 ---------
 ::
 
-    borg list <options> REPOSITORY_OR_ARCHIVE PATH
+    borg [common options] list <options> REPOSITORY_OR_ARCHIVE PATH
 
 positional arguments
     REPOSITORY_OR_ARCHIVE
@@ -21,7 +21,9 @@ optional arguments
         | specify format for file listing
         |                                 (default: "{mode} {user:6} {group:6} {size:8d} {isomtime} {path}{extra}{NL}")
     ``--json``
-        | format output as JSON. The form of --format is ignored, but keys used in it are added to the JSON output. Some keys are always present. Note: JSON can only represent text. A "bpath" key is therefore not available.
+        | Only valid for listing repository contents. Format output as JSON. The form of --format is ignored, but keys used in it are added to the JSON output. Some keys are always present. Note: JSON can only represent text. A "barchive" key is therefore not available.
+    ``--json-lines``
+        | Only valid for listing archive contents. Format output as JSON Lines. The form of --format is ignored, but keys used in it are added to the JSON output. Some keys are always present. Note: JSON can only represent text. A "bpath" key is therefore not available.
 
 `Common options`_
     |

docs/usage/mount.rst.inc

@@ -6,7 +6,7 @@ borg mount
 ----------
 ::
 
-    borg mount <options> REPOSITORY_OR_ARCHIVE MOUNTPOINT
+    borg [common options] mount <options> REPOSITORY_OR_ARCHIVE MOUNTPOINT
 
 positional arguments
     REPOSITORY_OR_ARCHIVE

docs/usage/prune.rst.inc

@@ -6,7 +6,7 @@ borg prune
 ----------
 ::
 
-    borg prune <options> REPOSITORY
+    borg [common options] prune <options> REPOSITORY
 
 positional arguments
     REPOSITORY
@@ -17,8 +17,6 @@ optional arguments
         | do not change repository
     ``--force``
         | force pruning of corrupted archives
-    ``-p``, ``--progress``
-        | show progress display while deleting archives
     ``-s``, ``--stats``
         | print statistics for the deleted archive
     ``--list``

docs/usage/recreate.rst.inc

@@ -6,7 +6,7 @@ borg recreate
 -------------
 ::
 
-    borg recreate <options> REPOSITORY_OR_ARCHIVE PATH
+    borg [common options] recreate <options> REPOSITORY_OR_ARCHIVE PATH
 
 positional arguments
     REPOSITORY_OR_ARCHIVE
@@ -19,8 +19,6 @@ optional arguments
         | output verbose list of items (files, dirs, ...)
     ``--filter STATUSCHARS``
         | only display items with the given status characters
-    ``-p``, ``--progress``
-        | show progress display while recreating archives
     ``-n``, ``--dry-run``
         | do not change anything
     ``-s``, ``--stats``

docs/usage/rename.rst.inc

@@ -6,7 +6,7 @@ borg rename
 -----------
 ::
 
-    borg rename <options> ARCHIVE NEWNAME
+    borg [common options] rename <options> ARCHIVE NEWNAME
 
 positional arguments
     ARCHIVE

docs/usage/serve.rst.inc

@@ -6,7 +6,7 @@ borg serve
 ----------
 ::
 
-    borg serve <options>
+    borg [common options] serve <options>
 
 optional arguments
     ``--restrict-to-path PATH``

docs/usage/umount.rst.inc

@@ -6,7 +6,7 @@ borg umount
 -----------
 ::
 
-    borg umount <options> MOUNTPOINT
+    borg [common options] umount <options> MOUNTPOINT
 
 positional arguments
     MOUNTPOINT

docs/usage/upgrade.rst.inc

@@ -6,15 +6,13 @@ borg upgrade
 ------------
 ::
 
-    borg upgrade <options> REPOSITORY
+    borg [common options] upgrade <options> REPOSITORY
 
 positional arguments
     REPOSITORY
         path to the repository to be upgraded
 
 optional arguments
-    ``-p``, ``--progress``
-        | show progress display while upgrading the repository
     ``-n``, ``--dry-run``
         | do not change repository
     ``-i``, ``--inplace``

docs/usage/with-lock.rst.inc

@@ -6,7 +6,7 @@ borg with-lock
 --------------
 ::
 
-    borg with-lock <options> REPOSITORY COMMAND ARGS
+    borg [common options] with-lock <options> REPOSITORY COMMAND ARGS
 
 positional arguments
     REPOSITORY