Merge branch 'master' into multithreading

Some tests are failing, some block, one new test needed skipping.

.gitignore

@@ -2,7 +2,7 @@ MANIFEST
 docs/_build
 build
 dist
-env
+borg-env
 .tox
 hashindex.c
 chunker.c
@@ -16,6 +16,7 @@ platform_linux.c
 *.pyo
 *.so
 docs/usage/*.inc
+docs/api.rst
 .idea/
 .cache/
 borg/_version.py

.travis.yml

@@ -17,6 +17,9 @@ matrix:
         - python: 3.4
           os: linux
           env: TOXENV=py34
+        - python: 3.5
+          os: linux
+          env: TOXENV=py35
         - language: generic
           os: osx
           osx_image: xcode6.4
@@ -29,6 +32,10 @@ matrix:
           os: osx
           osx_image: xcode6.4
           env: TOXENV=py34
+        - language: generic
+          os: osx
+          osx_image: xcode6.4
+          env: TOXENV=py35
 
 install:
     - ./.travis/install.sh

.travis/install.sh

@@ -30,6 +30,10 @@ if [[ "$(uname -s)" == 'Darwin' ]]; then
             pyenv install 3.4.3
             pyenv global 3.4.3
             ;;
+        py35)
+            pyenv install 3.5.0
+            pyenv global 3.5.0
+            ;;
     esac
     pyenv rehash
     python -m pip install --user virtualenv

.travis/run.sh

@@ -17,7 +17,7 @@ source ~/.venv/bin/activate
 
 if [[ "$(uname -s)" == "Darwin" ]]; then
     # no fakeroot on OS X
-    sudo tox -e $TOXENV
+    sudo tox -e $TOXENV -r
 else
-    fakeroot -u tox
+    fakeroot -u tox -r
 fi

AUTHORS

@@ -1,10 +1,14 @@
-Borg Developers / Contributors ("The Borg Collective")
-``````````````````````````````````````````````````````
+Contributors ("The Borg Collective")
+====================================
+
 - Thomas Waldmann <tw@waldmann-edv.de>
-- Antoine Beaupré
+- Antoine Beaupré <anarcat@debian.org>
 - Radek Podgorny <radek@podgorny.cz>
 - Yuri D'Elia
 
+Attic authors
+-------------
+
 Borg is a fork of Attic. Attic is written and maintained
 by Jonas Borgström and various contributors:
 

CHANGES.rst

@@ -1,510 +0,0 @@
-Borg Changelog
-==============
-
-Version 0.27.0
---------------
-
-New features:
-
-- "borg upgrade" command - attic -> borg one time converter / migration, #21
-- temporary hack to avoid using lots of disk space for chunks.archive.d, #235:
-  To use it: rm -rf chunks.archive.d ; touch chunks.archive.d
-- respect XDG_CACHE_HOME, attic #181
-- add support for arbitrary SSH commands, attic #99
-- borg delete --cache-only REPO (only delete cache, not REPO), attic #123
-
-
-Bug fixes:
-
-- use Debian 7 (wheezy) to build pyinstaller borgbackup binaries, fixes slow
-  down observed when running the Centos6-built binary on Ubuntu, #222
-- do not crash on empty lock.roster, fixes #232
-- fix multiple issues with the cache config version check, #234
-- fix segment entry header size check, attic #352
-  plus other error handling improvements / code deduplication there.
-- always give segment and offset in repo IntegrityErrors
-
-
-Other changes:
-
-- stop producing binary wheels, remove docs about it, #147
-- docs:
-  - add warning about prune
-  - generate usage include files only as needed
-  - development docs: add Vagrant section
-  - update / improve / reformat FAQ
-  - hint to single-file pyinstaller binaries from README
-
-
-Version 0.26.1
---------------
-
-This is a minor update, just docs and new pyinstaller binaries.
-
-- docs update about python and binary requirements
-- better docs for --read-special, fix #220
-- re-built the binaries, fix #218 and #213 (glibc version issue)
-- update web site about single-file pyinstaller binaries
-
-Note: if you did a python-based installation, there is no need to upgrade.
-
-
-Version 0.26.0
---------------
-
-New features:
-
-- Faster cache sync (do all in one pass, remove tar/compression stuff), #163
-- BORG_REPO env var to specify the default repo, #168
-- read special files as if they were regular files, #79
-- implement borg create --dry-run, attic issue #267
-- Normalize paths before pattern matching on OS X, #143
-- support OpenBSD and NetBSD (except xattrs/ACLs)
-- support / run tests on Python 3.5
-
-Bug fixes:
-
-- borg mount repo: use absolute path, attic #200, attic #137
-- chunker: use off_t to get 64bit on 32bit platform, #178
-- initialize chunker fd to -1, so it's not equal to STDIN_FILENO (0)
-- fix reaction to "no" answer at delete repo prompt, #182
-- setup.py: detect lz4.h header file location
-- to support python < 3.2.4, add less buggy argparse lib from 3.2.6 (#194)
-- fix for obtaining 'char *' from temporary Python value (old code causes
-  a compile error on Mint 17.2)
-- llfuse 0.41 install troubles on some platforms, require < 0.41
-  (UnicodeDecodeError exception due to non-ascii llfuse setup.py)
-- cython code: add some int types to get rid of unspecific python add /
-  subtract operations (avoid undefined symbol FPE_... error on some platforms)
-- fix verbose mode display of stdin backup
-- extract: warn if a include pattern never matched, fixes #209,
-  implement counters for Include/ExcludePatterns
-- archive names with slashes are invalid, attic issue #180
-- chunker: add a check whether the POSIX_FADV_DONTNEED constant is defined -
-  fixes building on OpenBSD.
-
-Other changes:
-
-- detect inconsistency / corruption / hash collision, #170
-- replace versioneer with setuptools_scm, #106
-- docs:
-
-  - pkg-config is needed for llfuse installation
-  - be more clear about pruning, attic issue #132
-- unit tests:
-
-  - xattr: ignore security.selinux attribute showing up
-  - ext3 seems to need a bit more space for a sparse file
-  - do not test lzma level 9 compression (avoid MemoryError)
-  - work around strange mtime granularity issue on netbsd, fixes #204
-  - ignore st_rdev if file is not a block/char device, fixes #203
-  - stay away from the setgid and sticky mode bits
-- use Vagrant to do easy cross-platform testing (#196), currently:
-
-  - Debian 7 "wheezy" 32bit, Debian 8 "jessie" 64bit
-  - Ubuntu 12.04 32bit, Ubuntu 14.04 64bit
-  - Centos 7 64bit
-  - FreeBSD 10.2 64bit
-  - OpenBSD 5.7 64bit
-  - NetBSD 6.1.5 64bit
-  - Darwin (OS X Yosemite)
-
-
-Version 0.25.0
---------------
-
-Compatibility notes:
-
-- lz4 compression library (liblz4) is a new requirement (#156)
-- the new compression code is very compatible: as long as you stay with zlib
-  compression, older borg releases will still be able to read data from a
-  repo/archive made with the new code (note: this is not the case for the
-  default "none" compression, use "zlib,0" if you want a "no compression" mode
-  that can be read by older borg). Also the new code is able to read repos and
-  archives made with older borg versions (for all zlib levels  0..9).
-
-Deprecations:
-
-- --compression N (with N being a number, as in 0.24) is deprecated.
-  We keep the --compression 0..9 for now to not break scripts, but it is
-  deprecated and will be removed later, so better fix your scripts now:
-  --compression 0 (as in 0.24) is the same as --compression zlib,0 (now).
-  BUT: if you do not want compression, you rather want --compression none
-  (which is the default).
-  --compression 1 (in 0.24) is the same as --compression zlib,1 (now)
-  --compression 9 (in 0.24) is the same as --compression zlib,9 (now)
-
-New features:
-
-- create --compression none (default, means: do not compress, just pass through
-  data "as is". this is more efficient than zlib level 0 as used in borg 0.24)
-- create --compression lz4 (super-fast, but not very high compression)
-- create --compression zlib,N (slower, higher compression, default for N is 6)
-- create --compression lzma,N (slowest, highest compression, default N is 6)
-- honor the nodump flag (UF_NODUMP) and do not backup such items
-- list --short just outputs a simple list of the files/directories in an archive
-
-Bug fixes:
-
-- fixed --chunker-params parameter order confusion / malfunction, fixes #154
-- close fds of segments we delete (during compaction)
-- close files which fell out the lrucache
-- fadvise DONTNEED now is only called for the byte range actually read, not for
-  the whole file, fixes #158.
-- fix issue with negative "all archives" size, fixes #165
-- restore_xattrs: ignore if setxattr fails with EACCES, fixes #162
-
-Other changes:
-
-- remove fakeroot requirement for tests, tests run faster without fakeroot
-  (test setup does not fail any more without fakeroot, so you can run with or
-  without fakeroot), fixes #151 and #91.
-- more tests for archiver
-- recover_segment(): don't assume we have an fd for segment
-- lrucache refactoring / cleanup, add dispose function, py.test tests
-- generalize hashindex code for any key length (less hardcoding)
-- lock roster: catch file not found in remove() method and ignore it
-- travis CI: use requirements file
-- improved docs:
-
-  - replace hack for llfuse with proper solution (install libfuse-dev)
-  - update docs about compression
-  - update development docs about fakeroot
-  - internals: add some words about lock files / locking system
-  - support: mention BountySource and for what it can be used
-  - theme: use a lighter green
-  - add pypi, wheel, dist package based install docs
-  - split install docs into system-specific preparations and generic instructions
-
-
-Version 0.24.0
---------------
-
-Incompatible changes (compared to 0.23):
-
-- borg now always issues --umask NNN option when invoking another borg via ssh
-  on the repository server. By that, it's making sure it uses the same umask
-  for remote repos as for local ones. Because of this, you must upgrade both
-  server and client(s) to 0.24.
-- the default umask is 077 now (if you do not specify via --umask) which might
-  be a different one as you used previously. The default umask avoids that
-  you accidentally give access permissions for group and/or others to files
-  created by borg (e.g. the repository).
-
-Deprecations:
-
-- "--encryption passphrase" mode is deprecated, see #85 and #97.
-  See the new "--encryption repokey" mode for a replacement.
-
-New features:
-
-- borg create --chunker-params ... to configure the chunker, fixes #16
-  (attic #302, attic #300, and somehow also #41).
-  This can be used to reduce memory usage caused by chunk management overhead,
-  so borg does not create a huge chunks index/repo index and eats all your RAM
-  if you back up lots of data in huge files (like VM disk images).
-  See docs/misc/create_chunker-params.txt for more information.
-- borg info now reports chunk counts in the chunk index.
-- borg create --compression 0..9 to select zlib compression level, fixes #66
-  (attic #295).
-- borg init --encryption repokey (to store the encryption key into the repo),
-  fixes #85
-- improve at-end error logging, always log exceptions and set exit_code=1
-- LoggedIO: better error checks / exceptions / exception handling
-- implement --remote-path to allow non-default-path borg locations, #125
-- implement --umask M and use 077 as default umask for better security, #117
-- borg check: give a named single archive to it, fixes #139
-- cache sync: show progress indication
-- cache sync: reimplement the chunk index merging in C
-
-Bug fixes:
-
-- fix segfault that happened for unreadable files (chunker: n needs to be a
-  signed size_t), #116
-- fix the repair mode, #144
-- repo delete: add destroy to allowed rpc methods, fixes issue #114
-- more compatible repository locking code (based on mkdir), maybe fixes #92
-  (attic #317, attic #201).
-- better Exception msg if no Borg is installed on the remote repo server, #56
-- create a RepositoryCache implementation that can cope with >2GiB,
-  fixes attic #326.
-- fix Traceback when running check --repair, attic #232
-- clarify help text, fixes #73.
-- add help string for --no-files-cache, fixes #140
-
-Other changes:
-
-- improved docs:
-
-  - added docs/misc directory for misc. writeups that won't be included
-    "as is" into the html docs.
-  - document environment variables and return codes (attic #324, attic #52)
-  - web site: add related projects, fix web site url, IRC #borgbackup
-  - Fedora/Fedora-based install instructions added to docs
-  - Cygwin-based install instructions added to docs
-  - updated AUTHORS
-  - add FAQ entries about redundancy / integrity
-  - clarify that borg extract uses the cwd as extraction target
-  - update internals doc about chunker params, memory usage and compression
-  - added docs about development
-  - add some words about resource usage in general
-  - document how to backup a raw disk
-  - add note about how to run borg from virtual env
-  - add solutions for (ll)fuse installation problems
-  - document what borg check does, fixes #138
-  - reorganize borgbackup.github.io sidebar, prev/next at top
-  - deduplicate and refactor the docs / README.rst
-
-- use borg-tmp as prefix for temporary files / directories
-- short prune options without "keep-" are deprecated, do not suggest them
-- improved tox configuration
-- remove usage of unittest.mock, always use mock from pypi
-- use entrypoints instead of scripts, for better use of the wheel format and
-  modern installs
-- add requirements.d/development.txt and modify tox.ini
-- use travis-ci for testing based on Linux and (new) OS X
-- use coverage.py, pytest-cov and codecov.io for test coverage support
-
-I forgot to list some stuff already implemented in 0.23.0, here they are:
-
-New features:
-
-- efficient archive list from manifest, meaning a big speedup for slow
-  repo connections and "list <repo>", "delete <repo>", "prune" (attic #242,
-  attic #167)
-- big speedup for chunks cache sync (esp. for slow repo connections), fixes #18
-- hashindex: improve error messages
-
-Other changes:
-
-- explicitly specify binary mode to open binary files
-- some easy micro optimizations
-
-
-Version 0.23.0
---------------
-
-Incompatible changes (compared to attic, fork related):
-
-- changed sw name and cli command to "borg", updated docs
-- package name (and name in urls) uses "borgbackup" to have less collisions
-- changed repo / cache internal magic strings from ATTIC* to BORG*,
-  changed cache location to .cache/borg/ - this means that it currently won't
-  accept attic repos (see issue #21 about improving that)
-
-Bug fixes:
-
-- avoid defect python-msgpack releases, fixes attic #171, fixes attic #185
-- fix traceback when trying to do unsupported passphrase change, fixes attic #189
-- datetime does not like the year 10.000, fixes attic #139
-- fix "info" all archives stats, fixes attic #183
-- fix parsing with missing microseconds, fixes attic #282
-- fix misleading hint the fuse ImportError handler gave, fixes attic #237
-- check unpacked data from RPC for tuple type and correct length, fixes attic #127
-- fix Repository._active_txn state when lock upgrade fails
-- give specific path to xattr.is_enabled(), disable symlink setattr call that
-  always fails
-- fix test setup for 32bit platforms, partial fix for attic #196
-- upgraded versioneer, PEP440 compliance, fixes attic #257
-
-New features:
-
-- less memory usage: add global option --no-cache-files
-- check --last N (only check the last N archives)
-- check: sort archives in reverse time order
-- rename repo::oldname newname (rename repository)
-- create -v output more informative
-- create --progress (backup progress indicator)
-- create --timestamp (utc string or reference file/dir)
-- create: if "-" is given as path, read binary from stdin
-- extract: if --stdout is given, write all extracted binary data to stdout
-- extract --sparse (simple sparse file support)
-- extra debug information for 'fread failed'
-- delete <repo> (deletes whole repo + local cache)
-- FUSE: reflect deduplication in allocated blocks
-- only allow whitelisted RPC calls in server mode
-- normalize source/exclude paths before matching
-- use posix_fadvise to not spoil the OS cache, fixes attic #252
-- toplevel error handler: show tracebacks for better error analysis
-- sigusr1 / sigint handler to print current file infos - attic PR #286
-- RPCError: include the exception args we get from remote
-
-Other changes:
-
-- source: misc. cleanups, pep8, style
-- docs and faq improvements, fixes, updates
-- cleanup crypto.pyx, make it easier to adapt to other AES modes
-- do os.fsync like recommended in the python docs
-- source: Let chunker optionally work with os-level file descriptor.
-- source: Linux: remove duplicate os.fsencode calls
-- source: refactor _open_rb code a bit, so it is more consistent / regular
-- source: refactor indicator (status) and item processing
-- source: use py.test for better testing, flake8 for code style checks
-- source: fix tox >=2.0 compatibility (test runner)
-- pypi package: add python version classifiers, add FreeBSD to platforms
-
-
-Attic Changelog
-===============
-
-Here you can see the full list of changes between each Attic release until Borg
-forked from Attic:
-
-Version 0.17
-------------
-
-(bugfix release, released on X)
-- Fix hashindex ARM memory alignment issue (#309)
-- Improve hashindex error messages (#298)
-
-Version 0.16
-------------
-
-(bugfix release, released on May 16, 2015)
-- Fix typo preventing the security confirmation prompt from working (#303)
-- Improve handling of systems with improperly configured file system encoding (#289)
-- Fix "All archives" output for attic info. (#183)
-- More user friendly error message when repository key file is not found (#236)
-- Fix parsing of iso 8601 timestamps with zero microseconds (#282)
-
-Version 0.15
-------------
-
-(bugfix release, released on Apr 15, 2015)
-- xattr: Be less strict about unknown/unsupported platforms (#239)
-- Reduce repository listing memory usage (#163).
-- Fix BrokenPipeError for remote repositories (#233)
-- Fix incorrect behavior with two character directory names (#265, #268)
-- Require approval before accessing relocated/moved repository (#271)
-- Require approval before accessing previously unknown unencrypted repositories (#271)
-- Fix issue with hash index files larger than 2GB.
-- Fix Python 3.2 compatibility issue with noatime open() (#164)
-- Include missing pyx files in dist files (#168)
-
-Version 0.14
-------------
-
-(feature release, released on Dec 17, 2014)
-- Added support for stripping leading path segments (#95)
-  "attic extract --strip-segments X"
-- Add workaround for old Linux systems without acl_extended_file_no_follow (#96)
-- Add MacPorts' path to the default openssl search path (#101)
-- HashIndex improvements, eliminates unnecessary IO on low memory systems.
-- Fix "Number of files" output for attic info. (#124)
-- limit create file permissions so files aren't read while restoring
-- Fix issue with empty xattr values (#106)
-
-Version 0.13
-------------
-
-(feature release, released on Jun 29, 2014)
-
-- Fix sporadic "Resource temporarily unavailable" when using remote repositories
-- Reduce file cache memory usage (#90)
-- Faster AES encryption (utilizing AES-NI when available)
-- Experimental Linux, OS X and FreeBSD ACL support (#66)
-- Added support for backup and restore of BSDFlags (OSX, FreeBSD) (#56)
-- Fix bug where xattrs on symlinks were not correctly restored
-- Added cachedir support. CACHEDIR.TAG compatible cache directories
-  can now be excluded using ``--exclude-caches`` (#74)
-- Fix crash on extreme mtime timestamps (year 2400+) (#81)
-- Fix Python 3.2 specific lockf issue (EDEADLK)
-
-Version 0.12
-------------
-
-(feature release, released on April 7, 2014)
-
-- Python 3.4 support (#62)
-- Various documentation improvements a new style
-- ``attic mount`` now supports mounting an entire repository not only
-  individual archives (#59)
-- Added option to restrict remote repository access to specific path(s):
-  ``attic serve --restrict-to-path X`` (#51)
-- Include "all archives" size information in "--stats" output. (#54)
-- Added ``--stats`` option to ``attic delete`` and ``attic prune``
-- Fixed bug where ``attic prune`` used UTC instead of the local time zone
-  when determining which archives to keep.
-- Switch to SI units (Power of 1000 instead 1024) when printing file sizes
-
-Version 0.11
-------------
-
-(feature release, released on March 7, 2014)
-
-- New "check" command for repository consistency checking (#24)
-- Documentation improvements
-- Fix exception during "attic create" with repeated files (#39)
-- New "--exclude-from" option for attic create/extract/verify.
-- Improved archive metadata deduplication.
-- "attic verify" has been deprecated. Use "attic extract --dry-run" instead.
-- "attic prune --hourly|daily|..." has been deprecated.
-  Use "attic prune --keep-hourly|daily|..." instead.
-- Ignore xattr errors during "extract" if not supported by the filesystem. (#46)
-
-Version 0.10
-------------
-
-(bugfix release, released on Jan 30, 2014)
-
-- Fix deadlock when extracting 0 sized files from remote repositories
-- "--exclude" wildcard patterns are now properly applied to the full path
-  not just the file name part (#5).
-- Make source code endianness agnostic (#1)
-
-Version 0.9
------------
-
-(feature release, released on Jan 23, 2014)
-
-- Remote repository speed and reliability improvements.
-- Fix sorting of segment names to ignore NFS left over files. (#17)
-- Fix incorrect display of time (#13)
-- Improved error handling / reporting. (#12)
-- Use fcntl() instead of flock() when locking repository/cache. (#15)
-- Let ssh figure out port/user if not specified so we don't override .ssh/config (#9)
-- Improved libcrypto path detection (#23).
-
-Version 0.8.1
--------------
-
-(bugfix release, released on Oct 4, 2013)
-
-- Fix segmentation fault issue.
-
-Version 0.8
------------
-
-(feature release, released on Oct 3, 2013)
-
-- Fix xattr issue when backing up sshfs filesystems (#4)
-- Fix issue with excessive index file size (#6)
-- Support access of read only repositories.
-- New syntax to enable repository encryption:
-    attic init --encryption="none|passphrase|keyfile".
-- Detect and abort if repository is older than the cache.
-
-
-Version 0.7
------------
-
-(feature release, released on Aug 5, 2013)
-
-- Ported to FreeBSD
-- Improved documentation
-- Experimental: Archives mountable as fuse filesystems.
-- The "user." prefix is no longer stripped from xattrs on Linux
-
-
-Version 0.6.1
--------------
-
-(bugfix release, released on July 19, 2013)
-
-- Fixed an issue where mtime was not always correctly restored.
-
-
-Version 0.6
------------
-
-First public release on July 9, 2013

CHANGES.rst

@@ -0,0 +1 @@
+docs/changes.rst

README.rst

@@ -1,5 +1,12 @@
+|screencast|
+
+.. |screencast| image:: https://asciinema.org/a/28691.png
+        :alt: BorgBackup Installation and Basic Usage
+        :target: https://asciinema.org/a/28691?autoplay=1&speed=2
+
+
 What is BorgBackup?
--------------------
+===================
 BorgBackup (short: Borg) is a deduplicating backup program.
 Optionally, it supports compression and authenticated encryption.
 
@@ -9,11 +16,13 @@ since only changes are stored.
 The authenticated encryption technique makes it suitable for backups to not
 fully trusted targets.
 
-`Borg Installation docs <http://borgbackup.github.io/borgbackup/installation.html>`_
+See the `installation manual`_ or, if you have already
+downloaded Borg, ``docs/installation.rst`` to get started with Borg.
 
+.. _installation manual: https://borgbackup.readthedocs.org/installation.html
 
 Main features
-~~~~~~~~~~~~~
+-------------
 **Space efficient storage**
   Deduplication based on content-defined chunking is used to reduce the number
   of bytes stored: each file is split into a number of variable length chunks
@@ -63,16 +72,16 @@ Main features
     Backup archives are mountable as userspace filesystems for easy interactive
     backup examination and restores (e.g. by using a regular file manager).
 
-**Easy installation**
-    For Linux, Mac OS X and FreeBSD, we offer a single-file pyinstaller binary
-    that does not require installing anything - you can just run it.
+**Easy installation on multiple platforms**
+    We offer single-file binaries
+    that does not require installing anything - you can just run it on
+    the supported platforms:
 
-**Platforms Borg works on**
-  * Linux
-  * Mac OS X
-  * FreeBSD
-  * OpenBSD and NetBSD (for both: no xattrs/ACLs support yet)
-  * Cygwin (unsupported)
+    * Linux
+    * Mac OS X
+    * FreeBSD
+    * OpenBSD and NetBSD (no xattrs/ACLs support or binaries yet)
+    * Cygwin (not supported, no binaries yet)
 
 **Free and Open Source Software**
   * security and functionality can be audited independently
@@ -80,7 +89,7 @@ Main features
 
 
 Easy to use
-~~~~~~~~~~~
+-----------
 Initialize a new backup repository and create a backup archive::
 
     $ borg init /mnt/backup
@@ -88,7 +97,7 @@ Initialize a new backup repository and create a backup archive::
 
 Now doing another backup, just to show off the great deduplication::
 
-    $ borg create --stats /mnt/backup::Tuesday ~/Documents
+    $ borg create --stats -C zlib,6 /mnt/backup::Tuesday ~/Documents
 
     Archive name: Tuesday
     Archive fingerprint: 387a5e3f9b0e792e91c...
@@ -100,29 +109,68 @@ Now doing another backup, just to show off the great deduplication::
     This archive:          57.16 MB           46.78 MB            151.67 kB  <--- !
     All archives:         114.02 MB           93.46 MB             44.81 MB
 
-For a graphical frontend refer to our complementary project
-`BorgWeb <https://github.com/borgbackup/borgweb>`_.
+For a graphical frontend refer to our complementary project `BorgWeb`_.
+
+Links
+=====
+
+ * `Main Web Site <https://borgbackup.readthedocs.org/>`_
+ * `Releases <https://github.com/borgbackup/borg/releases>`_
+ * `PyPI packages <https://pypi.python.org/pypi/borgbackup>`_
+ * `ChangeLog <https://github.com/borgbackup/borg/blob/master/CHANGES.rst>`_
+ * `GitHub <https://github.com/borgbackup/borg>`_
+ * `Issue Tracker <https://github.com/borgbackup/borg/issues>`_
+ * `Bounties & Fundraisers <https://www.bountysource.com/teams/borgbackup>`_
+ * `Mailing List <http://librelist.com/browser/borgbackup/>`_
+ * `License <https://borgbackup.github.io/borgbackup/authors.html#license>`_
+
+Related Projects
+----------------
 
+ * `BorgWeb <https://borgbackup.github.io/borgweb/>`_
+ * `Atticmatic <https://github.com/witten/atticmatic/>`_
+ * `Attic <https://github.com/jborg/attic>`_
 
 Notes
 -----
 
-Borg is a fork of `Attic <https://github.com/jborg/attic>`_ and maintained by
-"`The Borg Collective <https://github.com/borgbackup/borg/blob/master/AUTHORS>`_".
+Borg is a fork of `Attic`_ and maintained by "`The Borg collective`_".
+
+.. _The Borg collective: https://borgbackup.readthedocs.org/authors.html
+
+Differences between Attic and Borg
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Here's a (incomplete) list of some major changes:
+
+ * more open, faster paced development (see `issue #1 <https://github.com/borgbackup/borg/issues/1>`_)
+ * lots of attic issues fixed (see `issue #5 <https://github.com/borgbackup/borg/issues/5>`_)
+ * less chunk management overhead via --chunker-params option (less memory and disk usage)
+ * faster remote cache resync (useful when backing up multiple machines into same repo)
+ * compression: no, lz4, zlib or lzma compression, adjustable compression levels
+ * repokey replaces problematic passphrase mode (you can't change the passphrase nor the pbkdf2 iteration count in "passphrase" mode)
+ * simple sparse file support, great for virtual machine disk files
+ * can read special files (e.g. block devices) or from stdin, write to stdout
+ * mkdir-based locking is more compatible than attic's posix locking
+ * uses fadvise to not spoil / blow up the fs cache
+ * better error messages / exception handling
+ * better output for verbose mode, progress indication
+ * tested on misc. Linux systems, 32 and 64bit, FreeBSD, OpenBSD, NetBSD, Mac OS X
+
+Please read the `ChangeLog`_ (or ``CHANGES.rst`` in the source distribution) for more
+information.
 
-Read `issue #1 <https://github.com/borgbackup/borg/issues/1>`_ about the initial
-considerations regarding project goals and policy of the Borg project.
+BORG IS NOT COMPATIBLE WITH ORIGINAL ATTIC (but there is a one-way conversion).
 
-BORG IS NOT COMPATIBLE WITH ORIGINAL ATTIC.
 EXPECT THAT WE WILL BREAK COMPATIBILITY REPEATEDLY WHEN MAJOR RELEASE NUMBER
-CHANGES (like when going from 0.x.y to 1.0.0). Please read CHANGES document.
+CHANGES (like when going from 0.x.y to 1.0.0).
 
 NOT RELEASED DEVELOPMENT VERSIONS HAVE UNKNOWN COMPATIBILITY PROPERTIES.
 
 THIS IS SOFTWARE IN DEVELOPMENT, DECIDE YOURSELF WHETHER IT FITS YOUR NEEDS.
 
-For more information, please also see the
-`LICENSE  <https://github.com/borgbackup/borg/blob/master/LICENSE>`_.
+Borg is distributed under a 3-clause BSD license, see `License`_
+for the complete license.
 
 |build| |coverage|
 

Vagrantfile

@@ -22,6 +22,7 @@ def packages_debianoid
     apt-get update
     # for building borgbackup and dependencies:
     apt-get install -y libssl-dev libacl1-dev liblz4-dev libfuse-dev fuse pkg-config
+    usermod -a -G fuse vagrant
     apt-get install -y fakeroot build-essential git
     apt-get install -y python3-dev python3-setuptools
     # for building python:
@@ -137,7 +138,7 @@ end
 def install_pyenv(boxname)
   return <<-EOF
     curl -s -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
-    echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> ~/.bash_profile
+    echo 'export PATH="$HOME/.pyenv/bin:/vagrant/borg:$PATH"' >> ~/.bash_profile
     echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
     echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile
     echo 'export PYTHON_CONFIGURE_OPTS="--enable-shared"' >> ~/.bash_profile
@@ -232,7 +233,7 @@ def build_binary_with_pyinstaller(boxname)
     cd /vagrant/borg
     . borg-env/bin/activate
     cd borg
-    pyinstaller -F -n borg --hidden-import=logging.config borg/__main__.py
+    pyinstaller -F -n borg.exe --distpath=/vagrant/borg --clean --hidden-import=logging.config borg/__main__.py
   EOF
 end
 
@@ -247,8 +248,10 @@ def run_tests(boxname)
     fi
     # otherwise: just use the system python
     if which fakeroot > /dev/null; then
+      echo "Running tox WITH fakeroot -u"
       fakeroot -u tox --skip-missing-interpreters
     else
+      echo "Running tox WITHOUT fakeroot -u"
       tox --skip-missing-interpreters
     fi
   EOF

borg/archive.py

@@ -1,11 +1,16 @@
+from binascii import hexlify
 from datetime import datetime
 from getpass import getuser
 from itertools import groupby
 import errno
 import threading
+import logging
+
+from .logger import create_logger
+logger = create_logger()
+
 from .key import key_factory
 from .remote import cache_if_remote
-import msgpack
 from multiprocessing import cpu_count
 import os
 import socket
@@ -14,12 +19,17 @@ import sys
 import time
 from io import BytesIO
 from . import xattr
-from .platform import acl_get, acl_set
-from .chunker import Chunker
-from .hashindex import ChunkIndex
-from .helpers import parse_timestamp, Error, uid2user, user2uid, gid2group, group2gid, \
-    Manifest, Statistics, decode_dict, st_mtime_ns, make_path_safe, StableDict, int_to_bigint, bigint_to_int, \
-    make_queue, TerminatedQueue
+from .helpers import parse_timestamp, Error, uid2user, user2uid, gid2group, group2gid, format_timedelta, \
+    Manifest, Statistics, decode_dict, make_path_safe, StableDict, int_to_bigint, bigint_to_int, have_cython, \
+    st_atime_ns, st_ctime_ns, st_mtime_ns, make_queue, TerminatedQueue
+if have_cython():
+    from .platform import acl_get, acl_set
+    from .chunker import Chunker
+    from .hashindex import ChunkIndex
+    import msgpack
+else:
+    import mock
+    msgpack = mock.Mock()
 
 ITEMS_BUFFER = 1024 * 1024
 
@@ -317,7 +327,8 @@ class Archive:
 
     def __init__(self, repository, key, manifest, name, cache=None, create=False,
                  checkpoint_interval=300, numeric_owner=False, progress=False,
-                 chunker_params=CHUNKER_PARAMS):
+                 chunker_params=CHUNKER_PARAMS,
+                 start=datetime.now(), end=datetime.now()):
         self.cwd = os.getcwd()
         self.key = key
         self.repository = repository
@@ -330,6 +341,8 @@ class Archive:
         self.name = name
         self.checkpoint_interval = checkpoint_interval
         self.numeric_owner = numeric_owner
+        self.start = start
+        self.end = end
         self.pipeline = DownloadPipeline(self.repository, self.key)
         if create:
             self.pp = ParallelProcessor(self)
@@ -375,6 +388,22 @@ class Archive:
         """Timestamp of archive creation in UTC"""
         return parse_timestamp(self.metadata[b'time'])
 
+    @property
+    def fpr(self):
+        return hexlify(self.id).decode('ascii')
+
+    @property
+    def duration(self):
+        return format_timedelta(self.end-self.start)
+
+    def __str__(self):
+        return '''Archive name: {0.name}
+Archive fingerprint: {0.fpr}
+Start time: {0.start:%c}
+End time: {0.end:%c}
+Duration: {0.duration}
+Number of files: {0.stats.nfiles}'''.format(self)
+
     def __repr__(self):
         return 'Archive(%r)' % self.name
 
@@ -565,12 +594,17 @@ class Archive:
         elif has_lchmod:  # Not available on Linux
             os.lchmod(path, item[b'mode'])
         mtime = bigint_to_int(item[b'mtime'])
+        if b'atime' in item:
+            atime = bigint_to_int(item[b'atime'])
+        else:
+            # old archives only had mtime in item metadata
+            atime = mtime
         if fd and utime_supports_fd:  # Python >= 3.3
-            os.utime(fd, None, ns=(mtime, mtime))
+            os.utime(fd, None, ns=(atime, mtime))
         elif utime_supports_follow_symlinks:  # Python >= 3.3
-            os.utime(path, None, ns=(mtime, mtime), follow_symlinks=False)
+            os.utime(path, None, ns=(atime, mtime), follow_symlinks=False)
         elif not symlink:
-            os.utime(path, (mtime / 1e9, mtime / 1e9))
+            os.utime(path, (atime / 1e9, mtime / 1e9))
         acl_set(path, item, self.numeric_owner)
         # Only available on OS X and FreeBSD
         if has_lchflags and b'bsdflags' in item:
@@ -609,7 +643,9 @@ class Archive:
             b'mode': st.st_mode,
             b'uid': st.st_uid, b'user': uid2user(st.st_uid),
             b'gid': st.st_gid, b'group': gid2group(st.st_gid),
-            b'mtime': int_to_bigint(st_mtime_ns(st))
+            b'atime': int_to_bigint(st_atime_ns(st)),
+            b'ctime': int_to_bigint(st_ctime_ns(st)),
+            b'mtime': int_to_bigint(st_mtime_ns(st)),
         }
         if self.numeric_owner:
             item[b'user'] = item[b'group'] = None
@@ -677,7 +713,10 @@ class Archive:
             else:
                 self.hard_links[st.st_ino, st.st_dev] = safe_path
         path_hash = self.key.id_hash(os.path.join(self.cwd, path).encode('utf-8', 'surrogateescape'))
+        first_run = not cache.files
         ids = cache.file_known_and_unchanged(path_hash, st)
+        if first_run:
+            logger.info('processing files')
         chunks = None
         if ids is not None:
             # Make sure all ids are available
@@ -713,7 +752,7 @@ class Archive:
     @staticmethod
     def _open_rb(path, st):
         flags_normal = os.O_RDONLY | getattr(os, 'O_BINARY', 0)
-        flags_noatime = flags_normal | getattr(os, 'NO_ATIME', 0)
+        flags_noatime = flags_normal | getattr(os, 'O_NOATIME', 0)
         euid = None
 
         def open_simple(p, s):
@@ -833,7 +872,7 @@ class ArchiveChecker:
         self.orphan_chunks_check()
         self.finish()
         if not self.error_found:
-            self.report_progress('Archive consistency check complete, no problems found.')
+            logger.info('Archive consistency check complete, no problems found.')
         return self.repair or not self.error_found
 
     def init_chunks(self):
@@ -855,7 +894,7 @@ class ArchiveChecker:
     def report_progress(self, msg, error=False):
         if error:
             self.error_found = True
-        print(msg, file=sys.stderr if error else sys.stdout)
+        logger.log(logging.ERROR if error else logging.WARNING, msg)
 
     def identify_key(self, repository):
         cdata = repository.get(next(self.chunks.iteritems())[0])
@@ -982,7 +1021,7 @@ class ArchiveChecker:
             num_archives = 1
             end = 1
         for i, (name, info) in enumerate(archive_items[:end]):
-            self.report_progress('Analyzing archive {} ({}/{})'.format(name, num_archives - i, num_archives))
+            logger.info('Analyzing archive {} ({}/{})'.format(name, num_archives - i, num_archives))
             archive_id = info[b'id']
             if archive_id not in self.chunks:
                 self.report_progress('Archive metadata block is missing', error=True)
@@ -994,7 +1033,8 @@ class ArchiveChecker:
             archive = StableDict(msgpack.unpackb(data))
             if archive[b'version'] != 1:
                 raise Exception('Unknown archive metadata version')
-            decode_dict(archive, (b'name', b'hostname', b'username', b'time'))  # fixme: argv
+            decode_dict(archive, (b'name', b'hostname', b'username', b'time'))
+            archive[b'cmdline'] = [arg.decode('utf-8', 'surrogateescape') for arg in archive[b'cmdline']]
             items_buffer = ChunkBuffer(self.key)
             items_buffer.write_chunk = add_callback
             for item in robust_iterator(archive):

borg/archiver.py

@@ -15,17 +15,21 @@ import textwrap
 import traceback
 
 from . import __version__
-from .archive import Archive, ArchiveChecker, CHUNKER_PARAMS
-from .compress import Compressor, COMPR_BUFFER
-from .upgrader import AtticRepositoryUpgrader
-from .repository import Repository
-from .cache import Cache
-from .key import key_creator
 from .helpers import Error, location_validator, format_time, format_file_size, \
     format_file_mode, ExcludePattern, IncludePattern, exclude_path, adjust_patterns, to_localtime, timestamp, \
     get_cache_dir, get_keys_dir, format_timedelta, prune_within, prune_split, \
     Manifest, remove_surrogates, update_excludes, format_archive, check_extension_modules, Statistics, \
-    is_cachedir, bigint_to_int, ChunkerParams, CompressionSpec
+    is_cachedir, bigint_to_int, ChunkerParams, CompressionSpec, have_cython, \
+    EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR
+from .logger import create_logger, setup_logging
+logger = create_logger()
+if have_cython():
+    from .compress import Compressor, COMPR_BUFFER
+    from .upgrader import AtticRepositoryUpgrader
+    from .repository import Repository
+    from .cache import Cache
+    from .key import key_creator
+from .archive import Archive, ArchiveChecker, CHUNKER_PARAMS
 from .remote import RepositoryServer, RemoteRepository
 
 has_lchflags = hasattr(os, 'lchflags')
@@ -33,8 +37,9 @@ has_lchflags = hasattr(os, 'lchflags')
 
 class Archiver:
 
-    def __init__(self):
-        self.exit_code = 0
+    def __init__(self, verbose=False):
+        self.exit_code = EXIT_SUCCESS
+        self.verbose = verbose
 
     def open_repository(self, location, create=False, exclusive=False):
         if location.proto == 'ssh':
@@ -46,16 +51,21 @@ class Archiver:
 
     def print_error(self, msg, *args):
         msg = args and msg % args or msg
-        self.exit_code = 1
-        print('borg: ' + msg, file=sys.stderr)
+        self.exit_code = EXIT_ERROR
+        logger.error(msg)
+
+    def print_warning(self, msg, *args):
+        msg = args and msg % args or msg
+        self.exit_code = EXIT_WARNING  # we do not terminate here, so it is a warning
+        logger.warning(msg)
 
-    def print_verbose(self, msg, *args, **kw):
+    def print_info(self, msg, *args):
         if self.verbose:
             msg = args and msg % args or msg
-            if kw.get('newline', True):
-                print(msg)
-            else:
-                print(msg, end=' ')
+            logger.info(msg)
+
+    def print_status(self, status, path):
+        self.print_info("%1s %s", status, remove_surrogates(path))
 
     def do_serve(self, args):
         """Start in server mode. This command is usually not used manually.
@@ -64,7 +74,7 @@ class Archiver:
 
     def do_init(self, args):
         """Initialize an empty repository"""
-        print('Initializing repository at "%s"' % args.repository.orig)
+        logger.info('Initializing repository at "%s"' % args.repository.orig)
         repository = self.open_repository(args.repository, create=True, exclusive=True)
         key = key_creator(repository, args)
         manifest = Manifest(key, repository)
@@ -79,29 +89,29 @@ class Archiver:
         repository = self.open_repository(args.repository, exclusive=args.repair)
         if args.repair:
             while not os.environ.get('BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'):
-                self.print_error("""Warning: 'check --repair' is an experimental feature that might result
+                self.print_warning("""'check --repair' is an experimental feature that might result
 in data loss.
 
 Type "Yes I am sure" if you understand this and want to continue.\n""")
                 if input('Do you want to continue? ') == 'Yes I am sure':
                     break
         if not args.archives_only:
-            print('Starting repository check...')
+            logger.info('Starting repository check...')
             if repository.check(repair=args.repair):
-                print('Repository check complete, no problems found.')
+                logger.info('Repository check complete, no problems found.')
             else:
-                return 1
+                return EXIT_WARNING
         if not args.repo_only and not ArchiveChecker().check(
                 repository, repair=args.repair, archive=args.repository.archive, last=args.last):
-            return 1
-        return 0
+            return EXIT_WARNING
+        return EXIT_SUCCESS
 
     def do_change_passphrase(self, args):
         """Change repository key file passphrase"""
         repository = self.open_repository(args.repository)
         manifest, key = Manifest.load(repository)
         key.change_passphrase()
-        return 0
+        return EXIT_SUCCESS
 
     def do_create(self, args):
         """Create new archive"""
@@ -117,7 +127,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
             archive = Archive(repository, key, manifest, args.archive.archive, cache=cache,
                               create=True, checkpoint_interval=args.checkpoint_interval,
                               numeric_owner=args.numeric_owner, progress=args.progress,
-                              chunker_params=args.chunker_params)
+                              chunker_params=args.chunker_params, start=t0)
         else:
             archive = cache = None
         try:
@@ -142,17 +152,18 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
                         try:
                             status = archive.process_stdin(path, cache)
                         except IOError as e:
-                            self.print_error('%s: %s', path, e)
+                            status = 'E'
+                            self.print_warning('%s: %s', path, e)
                     else:
                         status = '-'
-                    self.print_verbose("%1s %s", status, path)
+                    self.print_status(status, path)
                     continue
                 path = os.path.normpath(path)
-                if args.dontcross:
+                if args.one_file_system:
                     try:
                         restrict_dev = os.lstat(path).st_dev
                     except OSError as e:
-                        self.print_error('%s: %s', path, e)
+                        self.print_warning('%s: %s', path, e)
                         continue
                 else:
                     restrict_dev = None
@@ -163,16 +174,12 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
                 if args.progress:
                     archive.stats.show_progress(final=True)
                 if args.stats:
-                    t = datetime.now()
-                    diff = t - t0
+                    archive.end = datetime.now()
                     print('-' * 78)
-                    print('Archive name: %s' % args.archive.archive)
-                    print('Archive fingerprint: %s' % hexlify(archive.id).decode('ascii'))
-                    print('Start time: %s' % t0.strftime('%c'))
-                    print('End time: %s' % t.strftime('%c'))
-                    print('Duration: %s' % format_timedelta(diff))
-                    print('Number of files: %d' % archive.stats.nfiles)
-                    archive.stats.print_('This archive:', cache)
+                    print(str(archive))
+                    print()
+                    print(str(archive.stats))
+                    print(str(cache))
                     print('-' * 78)
         finally:
             if not dry_run:
@@ -186,7 +193,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         try:
             st = os.lstat(path)
         except OSError as e:
-            self.print_error('%s: %s', path, e)
+            self.print_warning('%s: %s', path, e)
             return
         if (st.st_ino, st.st_dev) in skip_inodes:
             return
@@ -203,7 +210,8 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
                 try:
                     status = archive.process_file(path, st, cache)
                 except IOError as e:
-                    self.print_error('%s: %s', path, e)
+                    status = 'E'
+                    self.print_warning('%s: %s', path, e)
         elif stat.S_ISDIR(st.st_mode):
             if exclude_caches and is_cachedir(path):
                 return
@@ -212,7 +220,8 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
             try:
                 entries = os.listdir(path)
             except OSError as e:
-                self.print_error('%s: %s', path, e)
+                status = 'E'
+                self.print_warning('%s: %s', path, e)
             else:
                 for filename in sorted(entries):
                     entry_path = os.path.normpath(os.path.join(path, filename))
@@ -232,7 +241,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
             # Ignore unix sockets
             return
         else:
-            self.print_error('Unknown file type: %s', path)
+            self.print_warning('Unknown file type: %s', path)
             return
         # Status output
         # A lowercase character means a file type other than a regular file,
@@ -249,13 +258,13 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
                 status = '-'  # dry run, item was not backed up
         # output ALL the stuff - it can be easily filtered using grep.
         # even stuff considered unchanged might be interesting.
-        self.print_verbose("%1s %s", status, remove_surrogates(path))
+        self.print_status(status, path)
 
     def do_extract(self, args):
         """Extract archive contents"""
         # be restrictive when restoring files, restore permissions later
         if sys.getfilesystemencoding() == 'ascii':
-            print('Warning: File system encoding is "ascii", extracting non-ascii filenames will not be supported.')
+            logger.warning('Warning: File system encoding is "ascii", extracting non-ascii filenames will not be supported.')
         repository = self.open_repository(args.archive)
         manifest, key = Manifest.load(repository)
         archive = Archive(repository, key, manifest, args.archive.archive,
@@ -272,7 +281,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
                 item[b'path'] = os.sep.join(orig_path.split(os.sep)[strip_components:])
                 if not item[b'path']:
                     continue
-            self.print_verbose(remove_surrogates(orig_path))
+            self.print_info(remove_surrogates(orig_path))
             try:
                 if dry_run:
                     archive.extract_item(item, dry_run=True)
@@ -283,7 +292,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
                     else:
                         archive.extract_item(item, stdout=stdout, sparse=sparse)
             except IOError as e:
-                self.print_error('%s: %s', remove_surrogates(orig_path), e)
+                self.print_warning('%s: %s', remove_surrogates(orig_path), e)
 
         if not args.dry_run:
             # need to set each directory's timestamps AFTER all files in it are
@@ -293,7 +302,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
                 archive.extract_item(dirs.pop(-1))
         for pattern in (patterns or []):
             if isinstance(pattern, IncludePattern) and  pattern.match_count == 0:
-                self.print_error("Warning: Include pattern '%s' never matched.", pattern)
+                self.print_warning("Include pattern '%s' never matched.", pattern)
         return self.exit_code
 
     def do_rename(self, args):
@@ -321,21 +330,23 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
             repository.commit()
             cache.commit()
             if args.stats:
-                stats.print_('Deleted data:', cache)
+                logger.info(stats.summary.format(label='Deleted data:', stats=stats))
+                logger.info(str(cache))
         else:
             if not args.cache_only:
-                print("You requested to completely DELETE the repository *including* all archives it contains:")
+                print("You requested to completely DELETE the repository *including* all archives it contains:", file=sys.stderr)
                 for archive_info in manifest.list_archive_infos(sort_by='ts'):
-                    print(format_archive(archive_info))
+                    print(format_archive(archive_info), file=sys.stderr)
                 if not os.environ.get('BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'):
-                    print("""Type "YES" if you understand this and want to continue.\n""")
+                    print("""Type "YES" if you understand this and want to continue.\n""", file=sys.stderr)
+                    # XXX: prompt may end up on stdout, but we'll assume that input() does the right thing
                     if input('Do you want to continue? ') != 'YES':
-                        self.exit_code = 1
+                        self.exit_code = EXIT_ERROR
                         return self.exit_code
                 repository.destroy()
-                print("Repository deleted.")
+                logger.info("Repository deleted.")
             cache.destroy()
-            print("Cache deleted.")
+            logger.info("Cache deleted.")
         return self.exit_code
 
     def do_mount(self, args):
@@ -343,7 +354,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         try:
             from .fuse import FuseOperations
         except ImportError as e:
-            self.print_error('loading fuse support failed [ImportError: %s]' % str(e))
+            self.print_error('Loading fuse support failed [ImportError: %s]' % str(e))
             return self.exit_code
 
         if not os.path.isdir(args.mountpoint) or not os.access(args.mountpoint, os.R_OK | os.W_OK | os.X_OK):
@@ -351,18 +362,21 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
             return self.exit_code
 
         repository = self.open_repository(args.src)
-        manifest, key = Manifest.load(repository)
-        if args.src.archive:
-            archive = Archive(repository, key, manifest, args.src.archive)
-        else:
-            archive = None
-        operations = FuseOperations(key, repository, manifest, archive)
-        self.print_verbose("Mounting filesystem")
         try:
-            operations.mount(args.mountpoint, args.options, args.foreground)
-        except RuntimeError:
-            # Relevant error message already printed to stderr by fuse
-            self.exit_code = 1
+            manifest, key = Manifest.load(repository)
+            if args.src.archive:
+                archive = Archive(repository, key, manifest, args.src.archive)
+            else:
+                archive = None
+            operations = FuseOperations(key, repository, manifest, archive)
+            self.print_info("Mounting filesystem")
+            try:
+                operations.mount(args.mountpoint, args.options, args.foreground)
+            except RuntimeError:
+                # Relevant error message already printed to stderr by fuse
+                self.exit_code = EXIT_ERROR
+        finally:
+            repository.close()
         return self.exit_code
 
     def do_list(self, args):
@@ -421,7 +435,9 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         print('Time: %s' % to_localtime(archive.ts).strftime('%c'))
         print('Command line:', remove_surrogates(' '.join(archive.metadata[b'cmdline'])))
         print('Number of files: %d' % stats.nfiles)
-        stats.print_('This archive:', cache)
+        print()
+        print(str(stats))
+        print(str(cache))
         return self.exit_code
 
     def do_prune(self, args):
@@ -433,7 +449,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         if args.hourly + args.daily + args.weekly + args.monthly + args.yearly == 0 and args.within is None:
             self.print_error('At least one of the "within", "keep-hourly", "keep-daily", "keep-weekly", '
                              '"keep-monthly" or "keep-yearly" settings must be specified')
-            return 1
+            return self.exit_code
         if args.prefix:
             archives = [archive for archive in archives if archive.name.startswith(args.prefix)]
         keep = []
@@ -454,19 +470,20 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         to_delete = [a for a in archives if a not in keep]
         stats = Statistics()
         for archive in keep:
-            self.print_verbose('Keeping archive: %s' % format_archive(archive))
+            self.print_info('Keeping archive: %s' % format_archive(archive))
         for archive in to_delete:
             if args.dry_run:
-                self.print_verbose('Would prune:     %s' % format_archive(archive))
+                self.print_info('Would prune:     %s' % format_archive(archive))
             else:
-                self.print_verbose('Pruning archive: %s' % format_archive(archive))
+                self.print_info('Pruning archive: %s' % format_archive(archive))
                 Archive(repository, key, manifest, archive.name, cache).delete(stats)
         if to_delete and not args.dry_run:
             manifest.write()
             repository.commit()
             cache.commit()
         if args.stats:
-            stats.print_('Deleted data:', cache)
+            logger.info(stats.summary.format(label='Deleted data:', stats=stats))
+            logger.info(str(cache))
         return self.exit_code
 
     def do_upgrade(self, args):
@@ -482,7 +499,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         # XXX: should auto-detect if it is an attic repository here
         repo = AtticRepositoryUpgrader(args.repository.path, create=False)
         try:
-            repo.upgrade(args.dry_run)
+            repo.upgrade(args.dry_run, inplace=args.inplace)
         except NotImplementedError as e:
             print("warning: %s" % e)
         return self.exit_code
@@ -540,7 +557,9 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
             ('--daily', '--keep-daily', 'Warning: "--daily" has been deprecated. Use "--keep-daily" instead.'),
             ('--weekly', '--keep-weekly', 'Warning: "--weekly" has been deprecated. Use "--keep-weekly" instead.'),
             ('--monthly', '--keep-monthly', 'Warning: "--monthly" has been deprecated. Use "--keep-monthly" instead.'),
-            ('--yearly', '--keep-yearly', 'Warning: "--yearly" has been deprecated. Use "--keep-yearly" instead.')
+            ('--yearly', '--keep-yearly', 'Warning: "--yearly" has been deprecated. Use "--keep-yearly" instead.'),
+            ('--do-not-cross-mountpoints', '--one-file-system',
+             'Warning:  "--do-no-cross-mountpoints" has been deprecated. Use "--one-file-system" instead.'),
         ]
         if args and args[0] == 'verify':
             print('Warning: "borg verify" has been deprecated. Use "borg extract --dry-run" instead.')
@@ -552,26 +571,9 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
                     print(warning)
         return args
 
-    def run(self, args=None):
-        check_extension_modules()
-        keys_dir = get_keys_dir()
-        if not os.path.exists(keys_dir):
-            os.makedirs(keys_dir)
-            os.chmod(keys_dir, stat.S_IRWXU)
-        cache_dir = get_cache_dir()
-        if not os.path.exists(cache_dir):
-            os.makedirs(cache_dir)
-            os.chmod(cache_dir, stat.S_IRWXU)
-            with open(os.path.join(cache_dir, 'CACHEDIR.TAG'), 'w') as fd:
-                fd.write(textwrap.dedent("""
-                    Signature: 8a477f597d28d172789f06886806bc55
-                    # This file is a cache directory tag created by Borg.
-                    # For information about cache directory tags, see:
-                    #       http://www.brynosaurus.com/cachedir/
-                    """).lstrip())
-        common_parser = argparse.ArgumentParser(add_help=False)
-        common_parser.add_argument('-v', '--verbose', dest='verbose', action='store_true',
-                                   default=False,
+    def build_parser(self, args=None, prog=None):
+        common_parser = argparse.ArgumentParser(add_help=False, prog=prog)
+        common_parser.add_argument('-v', '--verbose', dest='verbose', action='store_true', default=False,
                                    help='verbose output')
         common_parser.add_argument('--no-files-cache', dest='cache_files', action='store_false',
                                    help='do not load/update the file metadata cache used to detect unchanged files')
@@ -580,11 +582,9 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         common_parser.add_argument('--remote-path', dest='remote_path', default=RemoteRepository.remote_path, metavar='PATH',
                                    help='set remote path to executable (default: "%(default)s")')
 
-        # We can't use argparse for "serve" since we don't want it to show up in "Available commands"
-        if args:
-            args = self.preprocess_args(args)
-
-        parser = argparse.ArgumentParser(description='Borg %s - Deduplicated Backups' % __version__)
+        parser = argparse.ArgumentParser(prog=prog, description='Borg - Deduplicated Backups')
+        parser.add_argument('-V', '--version', action='version', version='%(prog)s ' + __version__,
+                                   help='show version number and exit')
         subparsers = parser.add_subparsers(title='Available commands')
 
         serve_epilog = textwrap.dedent("""
@@ -699,9 +699,11 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         subparser.add_argument('-s', '--stats', dest='stats',
                                action='store_true', default=False,
                                help='print statistics for the created archive')
-        subparser.add_argument('-p', '--progress', dest='progress',
-                               action='store_true', default=False,
-                               help='print progress while creating the archive')
+        subparser.add_argument('-p', '--progress', dest='progress', const=not sys.stderr.isatty(),
+                               action='store_const', default=sys.stdin.isatty(),
+                               help="""toggle progress display while creating the archive, showing Original,
+                               Compressed and Deduplicated sizes, followed by the Number of files seen
+                               and the path being processed, default: %(default)s""")
         subparser.add_argument('-e', '--exclude', dest='excludes',
                                type=ExcludePattern, action='append',
                                metavar="PATTERN", help='exclude paths matching PATTERN')
@@ -714,9 +716,9 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         subparser.add_argument('-c', '--checkpoint-interval', dest='checkpoint_interval',
                                type=int, default=300, metavar='SECONDS',
                                help='write checkpoint every SECONDS seconds (Default: 300)')
-        subparser.add_argument('--do-not-cross-mountpoints', dest='dontcross',
+        subparser.add_argument('-x', '--one-file-system', dest='one_file_system',
                                action='store_true', default=False,
-                               help='do not cross mount points')
+                               help='stay in same file system, do not cross mount points')
         subparser.add_argument('--numeric-owner', dest='numeric_owner',
                                action='store_true', default=False,
                                help='only store numeric user and group identifiers')
@@ -925,7 +927,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
                                help='repository to prune')
 
         upgrade_epilog = textwrap.dedent("""
-        upgrade an existing Borg repository in place. this currently
+        upgrade an existing Borg repository. this currently
         only support converting an Attic repository, but may
         eventually be extended to cover major Borg upgrades as well.
 
@@ -940,13 +942,6 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         the first backup after the conversion takes longer than expected
         due to the cache resync.
 
-        it is recommended you run this on a copy of the Attic
-        repository, in case something goes wrong, for example:
-
-            cp -a attic borg
-            borg upgrade -n borg
-            borg upgrade borg
-
         upgrade should be able to resume if interrupted, although it
         will still iterate over all segments. if you want to start
         from scratch, use `borg delete` over the copied repository to
@@ -954,11 +949,19 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
 
             borg delete borg
 
-        the conversion can PERMANENTLY DAMAGE YOUR REPOSITORY! Attic
-        will also NOT BE ABLE TO READ THE BORG REPOSITORY ANYMORE, as
-        the magic strings will have changed.
-
-        you have been warned.""")
+        unless ``--inplace`` is specified, the upgrade process first
+        creates a backup copy of the repository, in
+        REPOSITORY.upgrade-DATETIME, using hardlinks. this takes
+        longer than in place upgrades, but is much safer and gives
+        progress information (as opposed to ``cp -al``). once you are
+        satisfied with the conversion, you can safely destroy the
+        backup copy.
+
+        WARNING: running the upgrade in place will make the current
+        copy unusable with older version, with no way of going back
+        to previous versions. this can PERMANENTLY DAMAGE YOUR
+        REPOSITORY!  Attic CAN NOT READ BORG REPOSITORIES, as the
+        magic strings have changed. you have been warned.""")
         subparser = subparsers.add_parser('upgrade', parents=[common_parser],
                                           description=self.do_upgrade.__doc__,
                                           epilog=upgrade_epilog,
@@ -967,6 +970,10 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         subparser.add_argument('-n', '--dry-run', dest='dry_run',
                                default=False, action='store_true',
                                help='do not change repository')
+        subparser.add_argument('-i', '--inplace', dest='inplace',
+                               default=False, action='store_true',
+                               help="""rewrite repository in place, with no chance of going back to older
+                               versions of the repository.""")
         subparser.add_argument('repository', metavar='REPOSITORY', nargs='?', default='',
                                type=location_validator(archive=False),
                                help='path to the repository to be upgraded')
@@ -980,9 +987,34 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         subparser.set_defaults(func=functools.partial(self.do_help, parser, subparsers.choices))
         subparser.add_argument('topic', metavar='TOPIC', type=str, nargs='?',
                                help='additional help on TOPIC')
+        return parser
+
+    def run(self, args=None):
+        check_extension_modules()
+        keys_dir = get_keys_dir()
+        if not os.path.exists(keys_dir):
+            os.makedirs(keys_dir)
+            os.chmod(keys_dir, stat.S_IRWXU)
+        cache_dir = get_cache_dir()
+        if not os.path.exists(cache_dir):
+            os.makedirs(cache_dir)
+            os.chmod(cache_dir, stat.S_IRWXU)
+            with open(os.path.join(cache_dir, 'CACHEDIR.TAG'), 'w') as fd:
+                fd.write(textwrap.dedent("""
+                    Signature: 8a477f597d28d172789f06886806bc55
+                    # This file is a cache directory tag created by Borg.
+                    # For information about cache directory tags, see:
+                    #       http://www.brynosaurus.com/cachedir/
+                    """).lstrip())
+
+        # We can't use argparse for "serve" since we don't want it to show up in "Available commands"
+        if args:
+            args = self.preprocess_args(args)
+        parser = self.build_parser(args)
 
         args = parser.parse_args(args or ['-h'])
         self.verbose = args.verbose
+        setup_logging()
         os.umask(args.umask)
         RemoteRepository.remote_path = args.remote_path
         RemoteRepository.umask = args.umask
@@ -1001,7 +1033,7 @@ def sig_info_handler(signum, stack):  # pragma: no cover
                 total = loc['st'].st_size
             except Exception:
                 pos, total = 0, 0
-            print("{0} {1}/{2}".format(path, format_file_size(pos), format_file_size(total)))
+            logger.info("{0} {1}/{2}".format(path, format_file_size(pos), format_file_size(total)))
             break
         if func in ('extract_item', ):  # extract op
             path = loc['item'][b'path']
@@ -1009,7 +1041,7 @@ def sig_info_handler(signum, stack):  # pragma: no cover
                 pos = loc['fd'].tell()
             except Exception:
                 pos = 0
-            print("{0} {1}/???".format(path, format_file_size(pos)))
+            logger.info("{0} {1}/???".format(path, format_file_size(pos)))
             break
 
 
@@ -1031,22 +1063,34 @@ def main():  # pragma: no cover
     setup_signal_handlers()
     archiver = Archiver()
     try:
+        msg = None
         exit_code = archiver.run(sys.argv[1:])
     except Error as e:
-        archiver.print_error(e.get_message() + "\n%s" % traceback.format_exc())
+        msg = e.get_message() + "\n%s" % traceback.format_exc()
         exit_code = e.exit_code
     except RemoteRepository.RPCError as e:
-        archiver.print_error('Error: Remote Exception.\n%s' % str(e))
-        exit_code = 1
+        msg = 'Remote Exception.\n%s' % str(e)
+        exit_code = EXIT_ERROR
     except Exception:
-        archiver.print_error('Error: Local Exception.\n%s' % traceback.format_exc())
-        exit_code = 1
+        msg = 'Local Exception.\n%s' % traceback.format_exc()
+        exit_code = EXIT_ERROR
     except KeyboardInterrupt:
-        archiver.print_error('Error: Keyboard interrupt.\n%s' % traceback.format_exc())
-        exit_code = 1
-    if exit_code:
-        archiver.print_error('Exiting with failure status due to previous errors')
+        msg = 'Keyboard interrupt.\n%s' % traceback.format_exc()
+        exit_code = EXIT_ERROR
+    if msg:
+        logger.error(msg)
+    exit_msg = 'terminating with %s status, rc %d'
+    if exit_code == EXIT_SUCCESS:
+        logger.info(exit_msg % ('success', exit_code))
+    elif exit_code == EXIT_WARNING:
+        logger.warning(exit_msg % ('warning', exit_code))
+    elif exit_code == EXIT_ERROR:
+        logger.error(exit_msg % ('error', exit_code))
+    else:
+        # if you see 666 in output, it usually means exit_code was None
+        logger.error(exit_msg % ('abnormal', exit_code or 666))
     sys.exit(exit_code)
 
+
 if __name__ == '__main__':
     main()

borg/cache.py

@@ -1,7 +1,7 @@
 import configparser
 from .remote import cache_if_remote
+from collections import namedtuple
 import errno
-import msgpack
 import os
 import stat
 import sys
@@ -12,11 +12,16 @@ import tarfile
 import tempfile
 
 from .key import PlaintextKey
+from .logger import create_logger
+logger = create_logger()
 from .helpers import Error, get_cache_dir, decode_dict, st_mtime_ns, unhexlify, int_to_bigint, \
-    bigint_to_int
+    bigint_to_int, format_file_size, have_cython
 from .locking import UpgradableLock
 from .hashindex import ChunkIndex
 
+if have_cython():
+    import msgpack
+
 
 class Cache:
     """Client Side cache
@@ -47,6 +52,7 @@ class Cache:
         self.manifest = manifest
         self.path = path or os.path.join(get_cache_dir(), hexlify(repository.id).decode('ascii'))
         self.do_files = do_files
+        logger.info('initializing cache')
         # Warn user before sending data to a never seen before unencrypted repository
         if not os.path.exists(self.path):
             if warn_if_unencrypted and isinstance(key, PlaintextKey):
@@ -68,16 +74,33 @@ class Cache:
             # Make sure an encrypted repository has not been swapped for an unencrypted repository
             if self.key_type is not None and self.key_type != str(key.TYPE):
                 raise self.EncryptionMethodMismatch()
+            logger.info('synchronizing cache')
             self.sync()
             self.commit()
 
     def __del__(self):
         self.close()
 
+    def __str__(self):
+        fmt = """\
+All archives:   {0.total_size:>20s} {0.total_csize:>20s} {0.unique_csize:>20s}
+
+                       Unique chunks         Total chunks
+Chunk index:    {0.total_unique_chunks:20d} {0.total_chunks:20d}"""
+        return fmt.format(self.format_tuple())
+
+    def format_tuple(self):
+        # XXX: this should really be moved down to `hashindex.pyx`
+        Summary = namedtuple('Summary', ['total_size', 'total_csize', 'unique_size', 'unique_csize', 'total_unique_chunks', 'total_chunks'])
+        stats = Summary(*self.chunks.summarize())._asdict()
+        for field in ['total_size', 'total_csize', 'unique_csize']:
+            stats[field] = format_file_size(stats[field])
+        return Summary(**stats)
+
     def _confirm(self, message, env_var_override=None):
         print(message, file=sys.stderr)
         if env_var_override and os.environ.get(env_var_override):
-            print("Yes (From {})".format(env_var_override))
+            print("Yes (From {})".format(env_var_override), file=sys.stderr)
             return True
         if not sys.stdin.isatty():
             return False
@@ -146,6 +169,7 @@ class Cache:
     def _read_files(self):
         self.files = {}
         self._newest_mtime = 0
+        logger.info('reading files cache')
         with open(os.path.join(self.path, 'files'), 'rb') as fd:
             u = msgpack.Unpacker(use_list=True)
             while True:
@@ -267,7 +291,7 @@ class Cache:
                 unpacker.feed(data)
                 for item in unpacker:
                     if not isinstance(item, dict):
-                        print('Error: Did not get expected metadata dict - archive corrupted!')
+                        logger.error('Error: Did not get expected metadata dict - archive corrupted!')
                         continue
                     if b'chunks' in item:
                         for chunk_id, size, csize in item[b'chunks']:
@@ -289,10 +313,10 @@ class Cache:
                     return name
 
         def create_master_idx(chunk_idx):
-            print('Synchronizing chunks cache...')
+            logger.info('Synchronizing chunks cache...')
             cached_ids = cached_archives()
             archive_ids = repo_archives()
-            print('Archives: %d, w/ cached Idx: %d, w/ outdated Idx: %d, w/o cached Idx: %d.' % (
+            logger.info('Archives: %d, w/ cached Idx: %d, w/ outdated Idx: %d, w/o cached Idx: %d.' % (
                 len(archive_ids), len(cached_ids),
                 len(cached_ids - archive_ids), len(archive_ids - cached_ids), ))
             # deallocates old hashindex, creates empty hashindex:
@@ -304,12 +328,12 @@ class Cache:
                     archive_name = lookup_name(archive_id)
                     if archive_id in cached_ids:
                         archive_chunk_idx_path = mkpath(archive_id)
-                        print("Reading cached archive chunk index for %s ..." % archive_name)
+                        logger.info("Reading cached archive chunk index for %s ..." % archive_name)
                         archive_chunk_idx = ChunkIndex.read(archive_chunk_idx_path)
                     else:
-                        print('Fetching and building archive index for %s ...' % archive_name)
+                        logger.info('Fetching and building archive index for %s ...' % archive_name)
                         archive_chunk_idx = fetch_and_build_idx(archive_id, repository, self.key)
-                    print("Merging into master chunks index ...")
+                    logger.info("Merging into master chunks index ...")
                     if chunk_idx is None:
                         # we just use the first archive's idx as starting point,
                         # to avoid growing the hash table from 0 size and also
@@ -317,7 +341,7 @@ class Cache:
                         chunk_idx = archive_chunk_idx
                     else:
                         chunk_idx.merge(archive_chunk_idx)
-            print('Done.')
+            logger.info('Done.')
             return chunk_idx
 
         def legacy_cleanup():

borg/fuse.py

@@ -2,17 +2,19 @@ from collections import defaultdict
 import errno
 import io
 import llfuse
-import msgpack
 import os
 import stat
 import tempfile
 import time
 from .archive import Archive
-from .helpers import daemonize
+from .helpers import daemonize, have_cython
 from .remote import cache_if_remote
 
+if have_cython():
+    import msgpack
+
 # Does this version of llfuse support ns precision?
-have_fuse_mtime_ns = hasattr(llfuse.EntryAttributes, 'st_mtime_ns')
+have_fuse_xtime_ns = hasattr(llfuse.EntryAttributes, 'st_mtime_ns')
 
 
 class ItemCache:
@@ -153,14 +155,15 @@ class FuseOperations(llfuse.Operations):
         entry.st_size = size
         entry.st_blksize = 512
         entry.st_blocks = dsize / 512
-        if have_fuse_mtime_ns:
-            entry.st_atime_ns = item[b'mtime']
+        # note: older archives only have mtime (not atime nor ctime)
+        if have_fuse_xtime_ns:
+            entry.st_atime_ns = item.get(b'atime') or item[b'mtime']
             entry.st_mtime_ns = item[b'mtime']
-            entry.st_ctime_ns = item[b'mtime']
+            entry.st_ctime_ns = item.get(b'ctime') or item[b'mtime']
         else:
-            entry.st_atime = item[b'mtime'] / 1e9
+            entry.st_atime = (item.get(b'atime') or item[b'mtime']) / 1e9
             entry.st_mtime = item[b'mtime'] / 1e9
-            entry.st_ctime = item[b'mtime'] / 1e9
+            entry.st_ctime = (item.get(b'ctime') or item[b'mtime']) / 1e9
         return entry
 
     def listxattr(self, inode):

borg/helpers.py

@@ -1,5 +1,4 @@
-from .support import argparse  # see support/__init__.py docstring
-                               # DEPRECATED - remove after requiring py 3.4
+from .support import argparse  # see support/__init__.py docstring, DEPRECATED - remove after requiring py 3.4
 
 import binascii
 from collections import namedtuple
@@ -9,6 +8,12 @@ import os
 import pwd
 import queue
 import re
+try:
+    from shutil import get_terminal_size
+except ImportError:
+    def get_terminal_size(fallback=(80, 24)):
+        TerminalSize = namedtuple('TerminalSize', ['columns', 'lines'])
+        return TerminalSize(int(os.environ.get('COLUMNS', fallback[0])), int(os.environ.get('LINES', fallback[1])))
 import sys
 import time
 import unicodedata
@@ -17,11 +22,34 @@ from datetime import datetime, timezone, timedelta
 from fnmatch import translate
 from operator import attrgetter
 
-import msgpack
 
-from . import hashindex
-from . import chunker
-from . import crypto
+def have_cython():
+    """allow for a way to disable Cython includes
+
+    this is used during usage docs build, in setup.py. It is to avoid
+    loading the Cython libraries which are built, but sometimes not in
+    the search path (namely, during Tox runs).
+
+    we simply check an environment variable (``BORG_CYTHON_DISABLE``)
+    which, when set (to anything) will disable includes of Cython
+    libraries in key places to enable usage docs to be built.
+
+    :returns: True if Cython is available, False otherwise.
+    """
+    return not os.environ.get('BORG_CYTHON_DISABLE')
+
+if have_cython():
+    from . import hashindex
+    from . import chunker
+    from . import crypto
+    import msgpack
+
+
+# return codes returned by borg command
+# when borg is killed by signal N, rc = 128 + N
+EXIT_SUCCESS = 0  # everything done, no problems
+EXIT_WARNING = 1  # reached normal end of operation, but there were issues
+EXIT_ERROR = 2  # terminated abruptly, did not reach end of operation
 
 QUEUE_DEBUG = False
 
@@ -29,10 +57,17 @@ QUEUE_DEBUG = False
 class Error(Exception):
     """Error base class"""
 
-    exit_code = 1
+    # if we raise such an Error and it is only catched by the uppermost
+    # exception handler (that exits short after with the given exit_code),
+    # it is always a (fatal and abrupt) EXIT_ERROR, never just a warning.
+    exit_code = EXIT_ERROR
 
     def get_message(self):
-        return 'Error: ' + type(self).__doc__.format(*self.args)
+        return type(self).__doc__.format(*self.args)
+
+
+class IntegrityError(Error):
+    """Data integrity error"""
 
 
 class ExtensionModuleError(Error):
@@ -144,27 +179,41 @@ class Statistics:
         if unique:
             self.usize += csize
 
-    def print_(self, label, cache):
-        total_size, total_csize, unique_size, unique_csize, total_unique_chunks, total_chunks = cache.chunks.summarize()
-        print()
-        print('                       Original size      Compressed size    Deduplicated size')
-        print('%-15s %20s %20s %20s' % (label, format_file_size(self.osize), format_file_size(self.csize), format_file_size(self.usize)))
-        print('All archives:   %20s %20s %20s' % (format_file_size(total_size), format_file_size(total_csize), format_file_size(unique_csize)))
-        print()
-        print('                       Unique chunks         Total chunks')
-        print('Chunk index:    %20d %20d' % (total_unique_chunks, total_chunks))
-
-    def show_progress(self, item=None, final=False):
+    summary = """\
+                       Original size      Compressed size    Deduplicated size
+{label:15} {stats.osize_fmt:>20s} {stats.csize_fmt:>20s} {stats.usize_fmt:>20s}"""
+
+    def __str__(self):
+        return self.summary.format(stats=self, label='This archive:')
+
+    def __repr__(self):
+        return "<{cls} object at {hash:#x} ({self.osize}, {self.csize}, {self.usize})>".format(cls=type(self).__name__, hash=id(self), self=self)
+
+    @property
+    def osize_fmt(self):
+        return format_file_size(self.osize)
+
+    @property
+    def usize_fmt(self):
+        return format_file_size(self.usize)
+
+    @property
+    def csize_fmt(self):
+        return format_file_size(self.csize)
+
+    def show_progress(self, item=None, final=False, stream=None):
+        columns, lines = get_terminal_size()
         if not final:
+            msg = '{0.osize_fmt} O {0.csize_fmt} C {0.usize_fmt} D {0.nfiles} N '.format(self)
             path = remove_surrogates(item[b'path']) if item else ''
-            if len(path) > 43:
-                path = '%s...%s' % (path[:20], path[-20:])
-            msg = '%9s O %9s C %9s D %-43s' % (
-                format_file_size(self.osize), format_file_size(self.csize), format_file_size(self.usize), path)
+            space = columns - len(msg)
+            if space < len('...') + len(path):
+                path = '%s...%s' % (path[:(space//2)-len('...')], path[-space//2:])
+            msg += "{0:<{space}}".format(path, space=space)
         else:
-            msg = ' ' * 79
-        print(msg, end='\r')
-        sys.stdout.flush()
+            msg = ' ' * columns
+        print(msg, file=stream or sys.stderr, end="\r")
+        (stream or sys.stderr).flush()
 
 
 def get_keys_dir():
@@ -230,7 +279,7 @@ def exclude_path(path, patterns):
 
 def normalized(func):
     """ Decorator for the Pattern match methods, returning a wrapper that
-    normalizes OSX paths to match the normalized pattern on OSX, and 
+    normalizes OSX paths to match the normalized pattern on OSX, and
     returning the original method on other platforms"""
     @wraps(func)
     def normalize_wrapper(self, path):
@@ -426,27 +475,33 @@ def format_file_mode(mod):
     return '%s%s%s' % (x(mod // 64), x(mod // 8), x(mod))
 
 
-def format_file_size(v):
+def format_file_size(v, precision=2):
     """Format file size into a human friendly format
     """
-    if abs(v) > 10**12:
-        return '%.2f TB' % (v / 10**12)
-    elif abs(v) > 10**9:
-        return '%.2f GB' % (v / 10**9)
-    elif abs(v) > 10**6:
-        return '%.2f MB' % (v / 10**6)
-    elif abs(v) > 10**3:
-        return '%.2f kB' % (v / 10**3)
-    else:
-        return '%d B' % v
+    return sizeof_fmt_decimal(v, suffix='B', sep=' ', precision=precision)
 
 
-def format_archive(archive):
-    return '%-36s %s' % (archive.name, to_localtime(archive.ts).strftime('%c'))
+def sizeof_fmt(num, suffix='B', units=None, power=None, sep='', precision=2):
+    for unit in units[:-1]:
+        if abs(round(num, precision)) < power:
+            if isinstance(num, int):
+                return "{}{}{}{}".format(num, sep, unit, suffix)
+            else:
+                return "{:3.{}f}{}{}{}".format(num, precision, sep, unit, suffix)
+        num /= float(power)
+    return "{:.{}f}{}{}{}".format(num, precision, sep, units[-1], suffix)
 
 
-class IntegrityError(Error):
-    """Data integrity error"""
+def sizeof_fmt_iec(num, suffix='B', sep='', precision=2):
+    return sizeof_fmt(num, suffix=suffix, sep=sep, precision=precision, units=['', 'Ki', 'Mi', 'Gi', 'Ti', 'Pi', 'Ei', 'Zi', 'Yi'], power=1024)
+
+
+def sizeof_fmt_decimal(num, suffix='B', sep='', precision=2):
+    return sizeof_fmt(num, suffix=suffix, sep=sep, precision=precision, units=['', 'k', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y'], power=1000)
+
+
+def format_archive(archive):
+    return '%-36s %s' % (archive.name, to_localtime(archive.ts).strftime('%c'))
 
 
 def memoize(function):
@@ -498,14 +553,24 @@ def posix_acl_use_stored_uid_gid(acl):
     """Replace the user/group field with the stored uid/gid
     """
     entries = []
-    for entry in acl.decode('ascii').split('\n'):
+    for entry in safe_decode(acl).split('\n'):
         if entry:
             fields = entry.split(':')
             if len(fields) == 4:
                 entries.append(':'.join([fields[0], fields[3], fields[2]]))
             else:
                 entries.append(entry)
-    return ('\n'.join(entries)).encode('ascii')
+    return safe_encode('\n'.join(entries))
+
+
+def safe_decode(s, coding='utf-8', errors='surrogateescape'):
+    """decode bytes to str, with round-tripping "invalid" bytes"""
+    return s.decode(coding, errors)
+
+
+def safe_encode(s, coding='utf-8', errors='surrogateescape'):
+    """encode str to bytes, with round-tripping "invalid" bytes"""
+    return s.encode(coding, errors)
 
 
 class Location:
@@ -685,7 +750,13 @@ class StableDict(dict):
 
 
 if sys.version < '3.3':
-    # st_mtime_ns attribute only available in 3.3+
+    # st_xtime_ns attributes only available in 3.3+
+    def st_atime_ns(st):
+        return int(st.st_atime * 1e9)
+
+    def st_ctime_ns(st):
+        return int(st.st_ctime * 1e9)
+
     def st_mtime_ns(st):
         return int(st.st_mtime * 1e9)
 
@@ -695,6 +766,12 @@ if sys.version < '3.3':
             data = data.encode('ascii')
         return binascii.unhexlify(data)
 else:
+    def st_atime_ns(st):
+        return st.st_atime_ns
+
+    def st_ctime_ns(st):
+        return st.st_ctime_ns
+
     def st_mtime_ns(st):
         return st.st_mtime_ns
 

borg/key.py

@@ -2,14 +2,18 @@ from binascii import hexlify, a2b_base64, b2a_base64
 import configparser
 import getpass
 import os
-import msgpack
 import textwrap
 import hmac
 from hashlib import sha256
 
-from .crypto import pbkdf2_sha256, get_random_bytes, AES, bytes_to_long, long_to_bytes, bytes_to_int, num_aes_blocks
-from .compress import Compressor, COMPR_BUFFER
-from .helpers import IntegrityError, get_keys_dir, Error
+from .helpers import IntegrityError, get_keys_dir, Error, have_cython
+from .logger import create_logger
+logger = create_logger()
+
+if have_cython():
+    from .crypto import pbkdf2_sha256, get_random_bytes, AES, bytes_to_long, long_to_bytes, bytes_to_int, num_aes_blocks
+    from .compress import Compressor, COMPR_BUFFER
+    import msgpack
 
 PREFIX = b'\0' * 8
 
@@ -88,7 +92,7 @@ class PlaintextKey(KeyBase):
 
     @classmethod
     def create(cls, repository, args):
-        print('Encryption NOT enabled.\nUse the "--encryption=repokey|keyfile|passphrase" to enable encryption.')
+        logger.info('Encryption NOT enabled.\nUse the "--encryption=repokey|keyfile|passphrase" to enable encryption.')
         return cls(repository)
 
     @classmethod
@@ -190,12 +194,12 @@ class Passphrase(str):
             if allow_empty or passphrase:
                 passphrase2 = cls.getpass('Enter same passphrase again: ')
                 if passphrase == passphrase2:
-                    print('Remember your passphrase. Your data will be inaccessible without it.')
+                    logger.info('Remember your passphrase. Your data will be inaccessible without it.')
                     return passphrase
                 else:
-                    print('Passphrases do not match')
+                    print('Passphrases do not match', file=sys.stderr)
             else:
-                print('Passphrase must not be blank')
+                print('Passphrase must not be blank', file=sys.stderr)
 
     def __repr__(self):
         return '<Passphrase "***hidden***">'
@@ -215,8 +219,8 @@ class PassphraseKey(AESKeyBase):
     @classmethod
     def create(cls, repository, args):
         key = cls(repository)
-        print('WARNING: "passphrase" mode is deprecated and will be removed in 1.0.')
-        print('If you want something similar (but with less issues), use "repokey" mode.')
+        logger.warning('WARNING: "passphrase" mode is deprecated and will be removed in 1.0.')
+        logger.warning('If you want something similar (but with less issues), use "repokey" mode.')
         passphrase = Passphrase.new(allow_empty=False)
         key.init(repository, passphrase)
         return key
@@ -324,7 +328,7 @@ class KeyfileKeyBase(AESKeyBase):
     def change_passphrase(self):
         passphrase = Passphrase.new(allow_empty=True)
         self.save(self.target, passphrase)
-        print('Key updated')
+        logger.info('Key updated')
 
     @classmethod
     def create(cls, repository, args):
@@ -335,8 +339,8 @@ class KeyfileKeyBase(AESKeyBase):
         key.init_ciphers()
         target = key.get_new_target(args)
         key.save(target, passphrase)
-        print('Key in "%s" created.' % target)
-        print('Keep this key safe. Your data will be inaccessible without it.')
+        logger.info('Key in "%s" created.' % target)
+        logger.info('Keep this key safe. Your data will be inaccessible without it.')
         return key
 
     def save(self, target, passphrase):

borg/locking.py

@@ -2,7 +2,6 @@ import errno
 import json
 import os
 import socket
-import threading
 import time
 
 from borg.helpers import Error
@@ -10,13 +9,17 @@ from borg.helpers import Error
 ADD, REMOVE = 'add', 'remove'
 SHARED, EXCLUSIVE = 'shared', 'exclusive'
 
+# only determine the PID and hostname once.
+# for FUSE mounts, we fork a child process that needs to release
+# the lock made by the parent, so it needs to use the same PID for that.
+_pid = os.getpid()
+_hostname = socket.gethostname()
+
 
 def get_id():
     """Get identification tuple for 'us'"""
-    hostname = socket.gethostname()
-    pid = os.getpid()
-    tid = threading.current_thread().ident & 0xffffffff
-    return hostname, pid, tid
+    thread_id = 0
+    return _hostname, _pid, thread_id
 
 
 class TimeoutTimer:

borg/logger.py

@@ -0,0 +1,85 @@
+"""logging facilities
+
+The way to use this is as follows:
+
+* each module declares its own logger, using:
+
+    from .logger import create_logger
+    logger = create_logger()
+
+* then each module uses logger.info/warning/debug/etc according to the
+  level it believes is appropriate:
+
+    logger.debug('debugging info for developers or power users')
+    logger.info('normal, informational output')
+    logger.warning('warn about a non-fatal error or sth else')
+    logger.error('a fatal error')
+
+  ... and so on. see the `logging documentation
+  <https://docs.python.org/3/howto/logging.html#when-to-use-logging>`_
+  for more information
+
+* console interaction happens on stderr, that includes interactive
+  reporting functions like `help`, `info` and `list`
+
+* ...except ``input()`` is special, because we can't control the
+  stream it is using, unfortunately. we assume that it won't clutter
+  stdout, because interaction would be broken then anyways
+
+* what is output on INFO level is additionally controlled by commandline
+  flags
+"""
+
+import inspect
+import logging
+import sys
+
+
+def setup_logging(stream=None):
+    """setup logging module according to the arguments provided
+
+    this sets up a stream handler logger on stderr (by default, if no
+    stream is provided).
+    """
+    logging.raiseExceptions = False
+    l = logging.getLogger('')
+    sh = logging.StreamHandler(stream)
+    # other formatters will probably want this, but let's remove
+    # clutter on stderr
+    # example:
+    # sh.setFormatter(logging.Formatter('%(name)s: %(message)s'))
+    l.addHandler(sh)
+    l.setLevel(logging.INFO)
+    return sh
+
+
+def find_parent_module():
+    """find the name of a the first module calling this module
+
+    if we cannot find it, we return the current module's name
+    (__name__) instead.
+    """
+    try:
+        frame = inspect.currentframe().f_back
+        module = inspect.getmodule(frame)
+        while module is None or module.__name__ == __name__:
+            frame = frame.f_back
+            module = inspect.getmodule(frame)
+        return module.__name__
+    except AttributeError:
+        # somehow we failed to find our module
+        # return the logger module name by default
+        return __name__
+
+
+def create_logger(name=None):
+    """create a Logger object with the proper path, which is returned by
+    find_parent_module() by default, or is provided via the commandline
+
+    this is really a shortcut for:
+
+        logger = logging.getLogger(__name__)
+
+    we use it to avoid errors and provide a more standard API.
+    """
+    return logging.getLogger(name or find_parent_module())

borg/platform_darwin.pyx

@@ -1,5 +1,5 @@
 import os
-from .helpers import user2uid, group2gid
+from .helpers import user2uid, group2gid, safe_decode, safe_encode
 
 API_VERSION = 2
 
@@ -20,7 +20,7 @@ def _remove_numeric_id_if_possible(acl):
     """Replace the user/group field with the local uid/gid if possible
     """
     entries = []
-    for entry in acl.decode('ascii').split('\n'):
+    for entry in safe_decode(acl).split('\n'):
         if entry:
             fields = entry.split(':')
             if fields[0] == 'user':
@@ -30,22 +30,22 @@ def _remove_numeric_id_if_possible(acl):
                 if group2gid(fields[2]) is not None:
                     fields[1] = fields[3] = ''
             entries.append(':'.join(fields))
-    return ('\n'.join(entries)).encode('ascii')
+    return safe_encode('\n'.join(entries))
 
 
 def _remove_non_numeric_identifier(acl):
     """Remove user and group names from the acl
     """
     entries = []
-    for entry in acl.split(b'\n'):
+    for entry in safe_decode(acl).split('\n'):
         if entry:
-            fields = entry.split(b':')
-            if fields[0] in (b'user', b'group'):
-                fields[2] = b''
-                entries.append(b':'.join(fields))
+            fields = entry.split(':')
+            if fields[0] in ('user', 'group'):
+                fields[2] = ''
+                entries.append(':'.join(fields))
             else:
                 entries.append(entry)
-    return b'\n'.join(entries)
+    return safe_encode('\n'.join(entries))
 
 
 def acl_get(path, item, st, numeric_owner=False):

borg/platform_freebsd.pyx

@@ -1,5 +1,5 @@
 import os
-from .helpers import posix_acl_use_stored_uid_gid
+from .helpers import posix_acl_use_stored_uid_gid, safe_encode, safe_decode
 
 API_VERSION = 2
 
@@ -78,14 +78,14 @@ cdef _nfs4_use_stored_uid_gid(acl):
     """Replace the user/group field with the stored uid/gid
     """
     entries = []
-    for entry in acl.decode('ascii').split('\n'):
+    for entry in safe_decode(acl).split('\n'):
         if entry:
             if entry.startswith('user:') or entry.startswith('group:'):
                 fields = entry.split(':')
                 entries.append(':'.join(fields[0], fields[5], *fields[2:-1]))
             else:
                 entries.append(entry)
-    return ('\n'.join(entries)).encode('ascii')
+    return safe_encode('\n'.join(entries))
 
 
 def acl_set(path, item, numeric_owner=False):

borg/platform_linux.pyx

@@ -1,7 +1,7 @@
 import os
 import re
 from stat import S_ISLNK
-from .helpers import posix_acl_use_stored_uid_gid, user2uid, group2gid
+from .helpers import posix_acl_use_stored_uid_gid, user2uid, group2gid, safe_decode, safe_encode
 
 API_VERSION = 2
 
@@ -31,22 +31,22 @@ def acl_use_local_uid_gid(acl):
     """Replace the user/group field with the local uid/gid if possible
     """
     entries = []
-    for entry in acl.decode('ascii').split('\n'):
+    for entry in safe_decode(acl).split('\n'):
         if entry:
             fields = entry.split(':')
             if fields[0] == 'user' and fields[1]:
-                fields[1] = user2uid(fields[1], fields[3])
+                fields[1] = str(user2uid(fields[1], fields[3]))
             elif fields[0] == 'group' and fields[1]:
-                fields[1] = group2gid(fields[1], fields[3])
-            entries.append(':'.join(entry.split(':')[:3]))
-    return ('\n'.join(entries)).encode('ascii')
+                fields[1] = str(group2gid(fields[1], fields[3]))
+            entries.append(':'.join(fields[:3]))
+    return safe_encode('\n'.join(entries))
 
 
 cdef acl_append_numeric_ids(acl):
     """Extend the "POSIX 1003.1e draft standard 17" format with an additional uid/gid field
     """
     entries = []
-    for entry in _comment_re.sub('', acl.decode('ascii')).split('\n'):
+    for entry in _comment_re.sub('', safe_decode(acl)).split('\n'):
         if entry:
             type, name, permission = entry.split(':')
             if name and type == 'user':
@@ -55,14 +55,14 @@ cdef acl_append_numeric_ids(acl):
                 entries.append(':'.join([type, name, permission, str(group2gid(name, name))]))
             else:
                 entries.append(entry)
-    return ('\n'.join(entries)).encode('ascii')
+    return safe_encode('\n'.join(entries))
 
 
 cdef acl_numeric_ids(acl):
     """Replace the "POSIX 1003.1e draft standard 17" user/group field with uid/gid
     """
     entries = []
-    for entry in _comment_re.sub('', acl.decode('ascii')).split('\n'):
+    for entry in _comment_re.sub('', safe_decode(acl)).split('\n'):
         if entry:
             type, name, permission = entry.split(':')
             if name and type == 'user':
@@ -73,7 +73,7 @@ cdef acl_numeric_ids(acl):
                 entries.append(':'.join([type, gid, permission, gid]))
             else:
                 entries.append(entry)
-    return ('\n'.join(entries)).encode('ascii')
+    return safe_encode('\n'.join(entries))
 
 
 def acl_get(path, item, st, numeric_owner=False):

borg/remote.py

@@ -1,6 +1,5 @@
 import errno
 import fcntl
-import msgpack
 import os
 import select
 import shlex
@@ -11,9 +10,12 @@ import traceback
 
 from . import __version__
 
-from .helpers import Error, IntegrityError
+from .helpers import Error, IntegrityError, have_cython
 from .repository import Repository
 
+if have_cython():
+    import msgpack
+
 BUFSIZE = 10 * 1024 * 1024
 
 

borg/repository.py

@@ -2,14 +2,18 @@ from configparser import RawConfigParser
 from binascii import hexlify
 from itertools import islice
 import errno
+import logging
+logger = logging.getLogger(__name__)
+
 import os
 import shutil
 import struct
 import sys
 from zlib import crc32
 
-from .hashindex import NSIndex
-from .helpers import Error, IntegrityError, read_msgpack, write_msgpack, unhexlify
+from .helpers import Error, IntegrityError, read_msgpack, write_msgpack, unhexlify, have_cython
+if have_cython():
+    from .hashindex import NSIndex
 from .locking import UpgradableLock
 from .lrucache import LRUCache
 
@@ -278,7 +282,7 @@ class Repository:
         def report_error(msg):
             nonlocal error_found
             error_found = True
-            print(msg, file=sys.stderr)
+            logger.error(msg)
 
         assert not self._active_txn
         try:
@@ -546,11 +550,10 @@ class LoggedIO:
     def recover_segment(self, segment, filename):
         if segment in self.fds:
             del self.fds[segment]
-        # FIXME: save a copy of the original file
         with open(filename, 'rb') as fd:
             data = memoryview(fd.read())
         os.rename(filename, filename + '.beforerecover')
-        print('attempting to recover ' + filename, file=sys.stderr)
+        logger.info('attempting to recover ' + filename)
         with open(filename, 'wb') as fd:
             fd.write(MAGIC)
             while len(data) >= self.header_fmt.size:

borg/testsuite/__init__.py

@@ -85,8 +85,9 @@ class BaseTestCase(unittest.TestCase):
                 if fuse and not have_fuse_mtime_ns:
                     d1.append(round(st_mtime_ns(s1), -4))
                     d2.append(round(st_mtime_ns(s2), -4))
-                d1.append(round(st_mtime_ns(s1), st_mtime_ns_round))
-                d2.append(round(st_mtime_ns(s2), st_mtime_ns_round))
+                else:
+                    d1.append(round(st_mtime_ns(s1), st_mtime_ns_round))
+                    d2.append(round(st_mtime_ns(s2), st_mtime_ns_round))
             d1.append(get_all(path1, follow_symlinks=False))
             d2.append(get_all(path2, follow_symlinks=False))
             self.assert_equal(d1, d2)

borg/testsuite/archiver.py

@@ -1,5 +1,6 @@
 from binascii import hexlify
 from configparser import RawConfigParser
+import errno
 import os
 from io import StringIO
 import stat
@@ -19,7 +20,7 @@ from ..archive import Archive, ChunkBuffer, CHUNK_MAX_EXP
 from ..archiver import Archiver
 from ..cache import Cache
 from ..crypto import bytes_to_long, num_aes_blocks
-from ..helpers import Manifest
+from ..helpers import Manifest, EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR, st_atime_ns, st_mtime_ns
 from ..remote import RemoteRepository, PathNotAllowed
 from ..repository import Repository
 from . import BaseTestCase
@@ -70,14 +71,83 @@ class environment_variable:
             else:
                 os.environ[k] = v
 
+def exec_cmd(*args, archiver=None, fork=False, exe=None, **kw):
+    if fork:
+        try:
+            if exe is None:
+                borg = (sys.executable, '-m', 'borg.archiver')
+            elif isinstance(exe, str):
+                borg = (exe, )
+            elif not isinstance(exe, tuple):
+                raise ValueError('exe must be None, a tuple or a str')
+            output = subprocess.check_output(borg + args, stderr=subprocess.STDOUT)
+            ret = 0
+        except subprocess.CalledProcessError as e:
+            output = e.output
+            ret = e.returncode
+        return ret, os.fsdecode(output)
+    else:
+        stdin, stdout, stderr = sys.stdin, sys.stdout, sys.stderr
+        try:
+            sys.stdin = StringIO()
+            sys.stdout = sys.stderr = output = StringIO()
+            if archiver is None:
+                archiver = Archiver()
+            ret = archiver.run(list(args))
+            return ret, output.getvalue()
+        finally:
+            sys.stdin, sys.stdout, sys.stderr = stdin, stdout, stderr
 
-class ArchiverTestCaseBase(BaseTestCase):
 
+# check if the binary "borg.exe" is available
+try:
+    exec_cmd('help', exe='borg.exe', fork=True)
+    BORG_EXES = ['python', 'binary', ]
+except (IOError, OSError) as err:
+    if err.errno != errno.ENOENT:
+        raise
+    BORG_EXES = ['python', ]
+
+
+@pytest.fixture(params=BORG_EXES)
+def cmd(request):
+    if request.param == 'python':
+        exe = None
+    elif request.param == 'binary':
+        exe = 'borg.exe'
+    else:
+        raise ValueError("param must be 'python' or 'binary'")
+    def exec_fn(*args, **kw):
+        return exec_cmd(*args, exe=exe, fork=True, **kw)
+    return exec_fn
+
+
+def test_return_codes(cmd, tmpdir):
+    repo = tmpdir.mkdir('repo')
+    input = tmpdir.mkdir('input')
+    output = tmpdir.mkdir('output')
+    input.join('test_file').write('content')
+    rc, out = cmd('init', '%s' % str(repo))
+    assert rc == EXIT_SUCCESS
+    rc, out = cmd('create', '%s::archive' % repo, str(input))
+    assert rc == EXIT_SUCCESS
+    with changedir(str(output)):
+        rc, out = cmd('extract', '%s::archive' % repo)
+        assert rc == EXIT_SUCCESS
+    rc, out = cmd('extract', '%s::archive' % repo, 'does/not/match')
+    assert rc == EXIT_WARNING  # pattern did not match
+    rc, out = cmd('create', '%s::archive' % repo, str(input))
+    assert rc == EXIT_ERROR  # duplicate archive name
+
+
+class ArchiverTestCaseBase(BaseTestCase):
+    EXE = None  # python source based
+    FORK_DEFAULT = False
     prefix = ''
 
     def setUp(self):
         os.environ['BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'] = '1'
-        self.archiver = Archiver()
+        self.archiver = not self.FORK_DEFAULT and Archiver() or None
         self.tmpdir = tempfile.mkdtemp()
         self.repository_path = os.path.join(self.tmpdir, 'repository')
         self.repository_location = self.prefix + self.repository_path
@@ -102,34 +172,15 @@ class ArchiverTestCaseBase(BaseTestCase):
         shutil.rmtree(self.tmpdir)
 
     def cmd(self, *args, **kw):
-        exit_code = kw.get('exit_code', 0)
-        fork = kw.get('fork', False)
-        if fork:
-            try:
-                output = subprocess.check_output((sys.executable, '-m', 'borg.archiver') + args)
-                ret = 0
-            except subprocess.CalledProcessError as e:
-                output = e.output
-                ret = e.returncode
-            output = os.fsdecode(output)
-            if ret != exit_code:
-                print(output)
-            self.assert_equal(exit_code, ret)
-            return output
-        args = list(args)
-        stdin, stdout, stderr = sys.stdin, sys.stdout, sys.stderr
-        try:
-            sys.stdin = StringIO()
-            output = StringIO()
-            sys.stdout = sys.stderr = output
-            ret = self.archiver.run(args)
-            sys.stdin, sys.stdout, sys.stderr = stdin, stdout, stderr
-            if ret != exit_code:
-                print(output.getvalue())
-            self.assert_equal(exit_code, ret)
-            return output.getvalue()
-        finally:
-            sys.stdin, sys.stdout, sys.stderr = stdin, stdout, stderr
+        exit_code = kw.pop('exit_code', 0)
+        fork = kw.pop('fork', None)
+        if fork is None:
+            fork = self.FORK_DEFAULT
+        ret, output = exec_cmd(*args, fork=fork, exe=self.EXE, archiver=self.archiver, **kw)
+        if ret != exit_code:
+            print(output)
+        self.assert_equal(ret, exit_code)
+        return output
 
     def create_src_archive(self, name):
         self.cmd('create', self.repository_location + '::' + name, src_dir)
@@ -231,9 +282,37 @@ class ArchiverTestCase(ArchiverTestCaseBase):
         shutil.rmtree(self.cache_path)
         with environment_variable(BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK='1'):
             info_output2 = self.cmd('info', self.repository_location + '::test')
-        # info_output2 starts with some "initializing cache" text but should
-        # end the same way as info_output
-        assert info_output2.endswith(info_output)
+
+        def filter(output):
+            # filter for interesting "info" output, ignore cache rebuilding related stuff
+            prefixes = ['Name:', 'Fingerprint:', 'Number of files:', 'This archive:',
+                        'All archives:', 'Chunk index:', ]
+            result = []
+            for line in output.splitlines():
+                for prefix in prefixes:
+                    if line.startswith(prefix):
+                        result.append(line)
+            return '\n'.join(result)
+
+        # the interesting parts of info_output2 and info_output should be same
+        self.assert_equal(filter(info_output), filter(info_output2))
+
+    def test_atime(self):
+        have_root = self.create_test_files()
+        atime, mtime = 123456780, 234567890
+        os.utime('input/file1', (atime, mtime))
+        self.cmd('init', self.repository_location)
+        self.cmd('create', self.repository_location + '::test', 'input')
+        with changedir('output'):
+            self.cmd('extract', self.repository_location + '::test')
+        sti = os.stat('input/file1')
+        sto = os.stat('output/input/file1')
+        assert st_mtime_ns(sti) == st_mtime_ns(sto) == mtime * 1e9
+        if hasattr(os, 'O_NOATIME'):
+            assert st_atime_ns(sti) == st_atime_ns(sto) == atime * 1e9
+        else:
+            # it touched the input file's atime while backing it up
+            assert st_atime_ns(sto) == atime * 1e9
 
     def _extract_repository_id(self, path):
         return Repository(self.repository_path).id
@@ -304,7 +383,10 @@ class ArchiverTestCase(ArchiverTestCaseBase):
         self.cmd('init', '--encryption=none', self.repository_location)
         self._set_repository_id(self.repository_path, repository_id)
         self.assert_equal(repository_id, self._extract_repository_id(self.repository_path))
-        self.assert_raises(Cache.EncryptionMethodMismatch, lambda: self.cmd('create', self.repository_location + '::test.2', 'input'))
+        if self.FORK_DEFAULT:
+            self.cmd('create', self.repository_location + '::test.2', 'input', exit_code=1)  # fails
+        else:
+            self.assert_raises(Cache.EncryptionMethodMismatch, lambda: self.cmd('create', self.repository_location + '::test.2', 'input'))
 
     def test_repository_swap_detection2(self):
         self.create_test_files()
@@ -314,7 +396,10 @@ class ArchiverTestCase(ArchiverTestCaseBase):
         self.cmd('create', self.repository_location + '_encrypted::test', 'input')
         shutil.rmtree(self.repository_path + '_encrypted')
         os.rename(self.repository_path + '_unencrypted', self.repository_path + '_encrypted')
-        self.assert_raises(Cache.RepositoryAccessAborted, lambda: self.cmd('create', self.repository_location + '_encrypted::test.2', 'input'))
+        if self.FORK_DEFAULT:
+            self.cmd('create', self.repository_location + '_encrypted::test.2', 'input', exit_code=1)  # fails
+        else:
+            self.assert_raises(Cache.RepositoryAccessAborted, lambda: self.cmd('create', self.repository_location + '_encrypted::test.2', 'input'))
 
     def test_strip_components(self):
         self.cmd('init', self.repository_location)
@@ -539,8 +624,12 @@ class ArchiverTestCase(ArchiverTestCaseBase):
         self.assert_in('bar-2015-08-12-20:00', output)
 
     def test_usage(self):
-        self.assert_raises(SystemExit, lambda: self.cmd())
-        self.assert_raises(SystemExit, lambda: self.cmd('-h'))
+        if self.FORK_DEFAULT:
+            self.cmd(exit_code=0)
+            self.cmd('-h', exit_code=0)
+        else:
+            self.assert_raises(SystemExit, lambda: self.cmd())
+            self.assert_raises(SystemExit, lambda: self.cmd('-h'))
 
     def test_help(self):
         assert 'Borg' in self.cmd('help')
@@ -627,6 +716,12 @@ class ArchiverTestCase(ArchiverTestCaseBase):
         self.verify_aes_counter_uniqueness('passphrase')
 
 
+@unittest.skipUnless('binary' in BORG_EXES, 'no borg.exe available')
+class ArchiverTestCaseBinary(ArchiverTestCase):
+    EXE = 'borg.exe'
+    FORK_DEFAULT = True
+
+
 class ArchiverCheckTestCase(ArchiverTestCaseBase):
 
     def setUp(self):
@@ -716,3 +811,16 @@ if 0:
                 self.cmd('init', self.repository_location + '_2')
             with patch.object(RemoteRepository, 'extra_test_args', ['--restrict-to-path', '/foo', '--restrict-to-path', path_prefix]):
                 self.cmd('init', self.repository_location + '_3')
+
+        # skip fuse tests here, they deadlock since this change in exec_cmd:
+        # -output = subprocess.check_output(borg + args, stderr=None)
+        # +output = subprocess.check_output(borg + args, stderr=subprocess.STDOUT)
+        # this was introduced because some tests expect stderr contents to show up
+        # in "output" also. Also, the non-forking exec_cmd catches both, too.
+        @unittest.skip('deadlock issues')
+        def test_fuse_mount_repository(self):
+            pass
+
+        @unittest.skip('deadlock issues')
+        def test_fuse_mount_archive(self):
+            pass

borg/testsuite/benchmark.py

@@ -0,0 +1,100 @@
+"""
+Do benchmarks using pytest-benchmark.
+
+Usage:
+
+    py.test --benchmark-only
+"""
+
+import os
+
+import pytest
+
+from .archiver import changedir, cmd
+
+
+@pytest.yield_fixture
+def repo_url(request, tmpdir):
+    os.environ['BORG_PASSPHRASE'] = '123456'
+    os.environ['BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'] = '1'
+    os.environ['BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK'] = '1'
+    os.environ['BORG_KEYS_DIR'] = str(tmpdir.join('keys'))
+    os.environ['BORG_CACHE_DIR'] = str(tmpdir.join('cache'))
+    yield str(tmpdir.join('repository'))
+    tmpdir.remove(rec=1)
+
+
+@pytest.fixture(params=["none", "passphrase"])
+def repo(request, cmd, repo_url):
+    cmd('init', '--encryption', request.param, repo_url)
+    return repo_url
+
+
+@pytest.yield_fixture(scope='session', params=["zeros", "random"])
+def testdata(request, tmpdir_factory):
+    count, size = 10, 1000*1000
+    p = tmpdir_factory.mktemp('data')
+    data_type = request.param
+    if data_type == 'zeros':
+        # do not use a binary zero (\0) to avoid sparse detection
+        data = lambda: b'0' * size
+    if data_type == 'random':
+        rnd = open('/dev/urandom', 'rb')
+        data = lambda: rnd.read(size)
+    for i in range(count):
+        with open(str(p.join(str(i))), "wb") as f:
+            f.write(data())
+    if data_type == 'random':
+        rnd.close()
+    yield str(p)
+    p.remove(rec=1)
+
+
+@pytest.fixture(params=['none', 'lz4'])
+def archive(request, cmd, repo, testdata):
+    archive_url = repo + '::test'
+    cmd('create', '--compression', request.param, archive_url, testdata)
+    return archive_url
+
+
+def test_create_none(benchmark, cmd, repo, testdata):
+    result, out = benchmark.pedantic(cmd, ('create', '--compression', 'none', repo + '::test', testdata))
+    assert result == 0
+
+
+def test_create_lz4(benchmark, cmd, repo, testdata):
+    result, out = benchmark.pedantic(cmd, ('create', '--compression', 'lz4', repo + '::test', testdata))
+    assert result == 0
+
+
+def test_extract(benchmark, cmd, archive, tmpdir):
+    with changedir(str(tmpdir)):
+        result, out = benchmark.pedantic(cmd, ('extract', archive))
+    assert result == 0
+
+
+def test_delete(benchmark, cmd, archive):
+    result, out = benchmark.pedantic(cmd, ('delete', archive))
+    assert result == 0
+
+
+def test_list(benchmark, cmd, archive):
+    result, out = benchmark(cmd, 'list', archive)
+    assert result == 0
+
+
+def test_info(benchmark, cmd, archive):
+    result, out = benchmark(cmd, 'info', archive)
+    assert result == 0
+
+
+def test_check(benchmark, cmd, archive):
+    repo = archive.split('::')[0]
+    result, out = benchmark(cmd, 'check', repo)
+    assert result == 0
+
+
+def test_help(benchmark, cmd):
+    result, out = benchmark(cmd, 'help')
+    assert result == 0
+

borg/testsuite/helpers.py

@@ -1,14 +1,15 @@
 import hashlib
 from time import mktime, strptime
 from datetime import datetime, timezone, timedelta
+from io import StringIO
 import os
 
 import pytest
 import sys
 import msgpack
 
-from ..helpers import adjust_patterns, exclude_path, Location, format_timedelta, IncludePattern, ExcludePattern, make_path_safe, \
-    prune_within, prune_split, get_cache_dir, \
+from ..helpers import adjust_patterns, exclude_path, Location, format_file_size, format_timedelta, IncludePattern, ExcludePattern, make_path_safe, \
+    prune_within, prune_split, get_cache_dir, Statistics, \
     StableDict, int_to_bigint, bigint_to_int, parse_timestamp, CompressionSpec, ChunkerParams
 from . import BaseTestCase
 
@@ -29,44 +30,44 @@ class TestLocationWithoutEnv:
     def test_ssh(self, monkeypatch):
         monkeypatch.delenv('BORG_REPO', raising=False)
         assert repr(Location('ssh://user@host:1234/some/path::archive')) == \
-               "Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive='archive')"
+            "Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive='archive')"
         assert repr(Location('ssh://user@host:1234/some/path')) == \
-               "Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive=None)"
+            "Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive=None)"
 
     def test_file(self, monkeypatch):
         monkeypatch.delenv('BORG_REPO', raising=False)
         assert repr(Location('file:///some/path::archive')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive='archive')"
+            "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive='archive')"
         assert repr(Location('file:///some/path')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive=None)"
+            "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive=None)"
 
     def test_scp(self, monkeypatch):
         monkeypatch.delenv('BORG_REPO', raising=False)
         assert repr(Location('user@host:/some/path::archive')) == \
-               "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive='archive')"
+            "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive='archive')"
         assert repr(Location('user@host:/some/path')) == \
-               "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive=None)"
+            "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive=None)"
 
     def test_folder(self, monkeypatch):
         monkeypatch.delenv('BORG_REPO', raising=False)
         assert repr(Location('path::archive')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='path', archive='archive')"
+            "Location(proto='file', user=None, host=None, port=None, path='path', archive='archive')"
         assert repr(Location('path')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='path', archive=None)"
+            "Location(proto='file', user=None, host=None, port=None, path='path', archive=None)"
 
     def test_abspath(self, monkeypatch):
         monkeypatch.delenv('BORG_REPO', raising=False)
         assert repr(Location('/some/absolute/path::archive')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive='archive')"
+            "Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive='archive')"
         assert repr(Location('/some/absolute/path')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive=None)"
+            "Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive=None)"
 
     def test_relpath(self, monkeypatch):
         monkeypatch.delenv('BORG_REPO', raising=False)
         assert repr(Location('some/relative/path::archive')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive='archive')"
+            "Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive='archive')"
         assert repr(Location('some/relative/path')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive=None)"
+            "Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive=None)"
 
     def test_underspecified(self, monkeypatch):
         monkeypatch.delenv('BORG_REPO', raising=False)
@@ -94,51 +95,51 @@ class TestLocationWithoutEnv:
                      'ssh://user@host:1234/some/path::archive']
         for location in locations:
             assert Location(location).canonical_path() == \
-                   Location(Location(location).canonical_path()).canonical_path()
+                Location(Location(location).canonical_path()).canonical_path()
 
 
 class TestLocationWithEnv:
     def test_ssh(self, monkeypatch):
         monkeypatch.setenv('BORG_REPO', 'ssh://user@host:1234/some/path')
         assert repr(Location('::archive')) == \
-               "Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive='archive')"
+            "Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive='archive')"
         assert repr(Location()) == \
-               "Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive=None)"
+            "Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive=None)"
 
     def test_file(self, monkeypatch):
         monkeypatch.setenv('BORG_REPO', 'file:///some/path')
         assert repr(Location('::archive')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive='archive')"
+            "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive='archive')"
         assert repr(Location()) == \
-               "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive=None)"
+            "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive=None)"
 
     def test_scp(self, monkeypatch):
         monkeypatch.setenv('BORG_REPO', 'user@host:/some/path')
         assert repr(Location('::archive')) == \
-               "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive='archive')"
+            "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive='archive')"
         assert repr(Location()) == \
-               "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive=None)"
+            "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive=None)"
 
     def test_folder(self, monkeypatch):
         monkeypatch.setenv('BORG_REPO', 'path')
         assert repr(Location('::archive')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='path', archive='archive')"
+            "Location(proto='file', user=None, host=None, port=None, path='path', archive='archive')"
         assert repr(Location()) == \
-               "Location(proto='file', user=None, host=None, port=None, path='path', archive=None)"
+            "Location(proto='file', user=None, host=None, port=None, path='path', archive=None)"
 
     def test_abspath(self, monkeypatch):
         monkeypatch.setenv('BORG_REPO', '/some/absolute/path')
         assert repr(Location('::archive')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive='archive')"
+            "Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive='archive')"
         assert repr(Location()) == \
-               "Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive=None)"
+            "Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive=None)"
 
     def test_relpath(self, monkeypatch):
         monkeypatch.setenv('BORG_REPO', 'some/relative/path')
         assert repr(Location('::archive')) == \
-               "Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive='archive')"
+            "Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive='archive')"
         assert repr(Location()) == \
-               "Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive=None)"
+            "Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive=None)"
 
     def test_no_slashes(self, monkeypatch):
         monkeypatch.setenv('BORG_REPO', '/some/absolute/path')
@@ -211,7 +212,7 @@ class PatternNonAsciiTestCase(BaseTestCase):
         assert i.match("ba\N{COMBINING ACUTE ACCENT}/foo")
         assert not e.match("b\N{LATIN SMALL LETTER A WITH ACUTE}/foo")
         assert e.match("ba\N{COMBINING ACUTE ACCENT}/foo")
-    
+
     def testInvalidUnicode(self):
         pattern = str(b'ba\x80', 'latin1')
         i = IncludePattern(pattern)
@@ -234,7 +235,7 @@ class OSXPatternNormalizationTestCase(BaseTestCase):
         assert i.match("ba\N{COMBINING ACUTE ACCENT}/foo")
         assert e.match("b\N{LATIN SMALL LETTER A WITH ACUTE}/foo")
         assert e.match("ba\N{COMBINING ACUTE ACCENT}/foo")
-    
+
     def testDecomposedUnicode(self):
         pattern = 'ba\N{COMBINING ACUTE ACCENT}'
         i = IncludePattern(pattern)
@@ -244,7 +245,7 @@ class OSXPatternNormalizationTestCase(BaseTestCase):
         assert i.match("ba\N{COMBINING ACUTE ACCENT}/foo")
         assert e.match("b\N{LATIN SMALL LETTER A WITH ACUTE}/foo")
         assert e.match("ba\N{COMBINING ACUTE ACCENT}/foo")
-    
+
     def testInvalidUnicode(self):
         pattern = str(b'ba\x80', 'latin1')
         i = IncludePattern(pattern)
@@ -399,3 +400,83 @@ def test_get_cache_dir():
     # reset old env
     if old_env is not None:
         os.environ['BORG_CACHE_DIR'] = old_env
+
+
+@pytest.fixture()
+def stats():
+    stats = Statistics()
+    stats.update(20, 10, unique=True)
+    return stats
+
+
+def test_stats_basic(stats):
+    assert stats.osize == 20
+    assert stats.csize == stats.usize == 10
+    stats.update(20, 10, unique=False)
+    assert stats.osize == 40
+    assert stats.csize == 20
+    assert stats.usize == 10
+
+
+def tests_stats_progress(stats, columns=80):
+    os.environ['COLUMNS'] = str(columns)
+    out = StringIO()
+    stats.show_progress(stream=out)
+    s = '20 B O 10 B C 10 B D 0 N '
+    buf = ' ' * (columns - len(s))
+    assert out.getvalue() == s + buf + "\r"
+
+    out = StringIO()
+    stats.update(10**3, 0, unique=False)
+    stats.show_progress(item={b'path': 'foo'}, final=False, stream=out)
+    s = '1.02 kB O 10 B C 10 B D 0 N foo'
+    buf = ' ' * (columns - len(s))
+    assert out.getvalue() == s + buf + "\r"
+    out = StringIO()
+    stats.show_progress(item={b'path': 'foo'*40}, final=False, stream=out)
+    s = '1.02 kB O 10 B C 10 B D 0 N foofoofoofoofoofoofoofo...oofoofoofoofoofoofoofoofoo'
+    buf = ' ' * (columns - len(s))
+    assert out.getvalue() == s + buf + "\r"
+
+
+def test_stats_format(stats):
+    assert str(stats) == """\
+                       Original size      Compressed size    Deduplicated size
+This archive:                   20 B                 10 B                 10 B"""
+    s = "{0.osize_fmt}".format(stats)
+    assert s == "20 B"
+    # kind of redundant, but id is variable so we can't match reliably
+    assert repr(stats) == '<Statistics object at {:#x} (20, 10, 10)>'.format(id(stats))
+
+
+def test_file_size():
+    """test the size formatting routines"""
+    si_size_map = {
+        0: '0 B',  # no rounding necessary for those
+        1: '1 B',
+        142: '142 B',
+        999: '999 B',
+        1000: '1.00 kB',  # rounding starts here
+        1001: '1.00 kB',  # should be rounded away
+        1234: '1.23 kB',  # should be rounded down
+        1235: '1.24 kB',  # should be rounded up
+        1010: '1.01 kB',  # rounded down as well
+        999990000: '999.99 MB',  # rounded down
+        999990001: '999.99 MB',  # rounded down
+        999995000: '1.00 GB',  # rounded up to next unit
+        10**6: '1.00 MB',  # and all the remaining units, megabytes
+        10**9: '1.00 GB',  # gigabytes
+        10**12: '1.00 TB',  # terabytes
+        10**15: '1.00 PB',  # petabytes
+        10**18: '1.00 EB',  # exabytes
+        10**21: '1.00 ZB',  # zottabytes
+        10**24: '1.00 YB',  # yottabytes
+    }
+    for size, fmt in si_size_map.items():
+        assert format_file_size(size) == fmt
+
+
+def test_file_size_precision():
+    assert format_file_size(1234, precision=1) == '1.2 kB'  # rounded down
+    assert format_file_size(1254, precision=1) == '1.3 kB'  # rounded up
+    assert format_file_size(999990000, precision=1) == '1.0 GB'  # and not 999.9 MB or 1000.0 MB

borg/testsuite/logger.py

@@ -0,0 +1,40 @@
+import logging
+from io import StringIO
+
+from mock import Mock
+import pytest
+
+from ..logger import find_parent_module, create_logger, setup_logging
+logger = create_logger()
+
+
+@pytest.fixture()
+def io_logger():
+    io = StringIO()
+    handler = setup_logging(io)
+    handler.setFormatter(logging.Formatter('%(name)s: %(message)s'))
+    logger.setLevel(logging.DEBUG)
+    return io
+
+
+def test_setup_logging(io_logger):
+    logger.info('hello world')
+    assert io_logger.getvalue() == "borg.testsuite.logger: hello world\n"
+
+
+def test_multiple_loggers(io_logger):
+    logger = logging.getLogger(__name__)
+    logger.info('hello world 1')
+    assert io_logger.getvalue() == "borg.testsuite.logger: hello world 1\n"
+    logger = logging.getLogger('borg.testsuite.logger')
+    logger.info('hello world 2')
+    assert io_logger.getvalue() == "borg.testsuite.logger: hello world 1\nborg.testsuite.logger: hello world 2\n"
+    io_logger.truncate(0)
+    io_logger.seek(0)
+    logger = logging.getLogger('borg.testsuite.logger')
+    logger.info('hello world 2')
+    assert io_logger.getvalue() == "borg.testsuite.logger: hello world 2\n"
+
+
+def test_parent_module():
+    assert find_parent_module() == __name__

borg/testsuite/platform.py

@@ -72,6 +72,42 @@ class PlatformLinuxTestCase(BaseTestCase):
         self.assert_equal(self.get_acl(self.tmpdir)[b'acl_access'], ACCESS_ACL)
         self.assert_equal(self.get_acl(self.tmpdir)[b'acl_default'], DEFAULT_ACL)
 
+    def test_non_ascii_acl(self):
+        # Testing non-ascii ACL processing to see whether our code is robust.
+        # I have no idea whether non-ascii ACLs are allowed by the standard,
+        # but in practice they seem to be out there and must not make our code explode.
+        file = tempfile.NamedTemporaryFile()
+        self.assert_equal(self.get_acl(file.name), {})
+        nothing_special = 'user::rw-\ngroup::r--\nmask::rw-\nother::---\n'.encode('ascii')
+        # TODO: can this be tested without having an existing system user übel with uid 666 gid 666?
+        user_entry = 'user:übel:rw-:666'.encode('utf-8')
+        user_entry_numeric = 'user:666:rw-:666'.encode('ascii')
+        group_entry = 'group:übel:rw-:666'.encode('utf-8')
+        group_entry_numeric = 'group:666:rw-:666'.encode('ascii')
+        acl = b'\n'.join([nothing_special, user_entry, group_entry])
+        self.set_acl(file.name, access=acl, numeric_owner=False)
+        acl_access = self.get_acl(file.name, numeric_owner=False)[b'acl_access']
+        self.assert_in(user_entry, acl_access)
+        self.assert_in(group_entry, acl_access)
+        acl_access_numeric = self.get_acl(file.name, numeric_owner=True)[b'acl_access']
+        self.assert_in(user_entry_numeric, acl_access_numeric)
+        self.assert_in(group_entry_numeric, acl_access_numeric)
+        file2 = tempfile.NamedTemporaryFile()
+        self.set_acl(file2.name, access=acl, numeric_owner=True)
+        acl_access = self.get_acl(file2.name, numeric_owner=False)[b'acl_access']
+        self.assert_in(user_entry, acl_access)
+        self.assert_in(group_entry, acl_access)
+        acl_access_numeric = self.get_acl(file.name, numeric_owner=True)[b'acl_access']
+        self.assert_in(user_entry_numeric, acl_access_numeric)
+        self.assert_in(group_entry_numeric, acl_access_numeric)
+
+    def test_utils(self):
+        from ..platform_linux import acl_use_local_uid_gid
+        self.assert_equal(acl_use_local_uid_gid(b'user:nonexistent1234:rw-:1234'), b'user:1234:rw-')
+        self.assert_equal(acl_use_local_uid_gid(b'group:nonexistent1234:rw-:1234'), b'group:1234:rw-')
+        self.assert_equal(acl_use_local_uid_gid(b'user:root:rw-:0'), b'user:0:rw-')
+        self.assert_equal(acl_use_local_uid_gid(b'group:root:rw-:0'), b'group:0:rw-')
+
 
 @unittest.skipUnless(sys.platform.startswith('darwin'), 'OS X only test')
 @unittest.skipIf(fakeroot_detected(), 'not compatible with fakeroot')

borg/testsuite/upgrader.py

@@ -1,6 +1,4 @@
 import os
-import shutil
-import tempfile
 
 import pytest
 
@@ -14,10 +12,8 @@ except ImportError:
 from ..upgrader import AtticRepositoryUpgrader, AtticKeyfileKey
 from ..helpers import get_keys_dir
 from ..key import KeyfileKey
-from ..repository import Repository, MAGIC
-
-pytestmark = pytest.mark.skipif(attic is None,
-                                reason='cannot find an attic install')
+from ..remote import RemoteRepository
+from ..repository import Repository
 
 
 def repo_valid(path):
@@ -64,7 +60,13 @@ def attic_repo(tmpdir):
     return attic_repo
 
 
-def test_convert_segments(tmpdir, attic_repo):
+@pytest.fixture(params=[True, False])
+def inplace(request):
+    return request.param
+
+
+@pytest.mark.skipif(attic is None, reason='cannot find an attic install')
+def test_convert_segments(tmpdir, attic_repo, inplace):
     """test segment conversion
 
     this will load the given attic repository, list all the segments
@@ -77,11 +79,10 @@ def test_convert_segments(tmpdir, attic_repo):
     """
     # check should fail because of magic number
     assert not repo_valid(tmpdir)
-    print("opening attic repository with borg and converting")
     repo = AtticRepositoryUpgrader(str(tmpdir), create=False)
     segments = [filename for i, filename in repo.io.segment_iterator()]
     repo.close()
-    repo.convert_segments(segments, dryrun=False)
+    repo.convert_segments(segments, dryrun=False, inplace=inplace)
     repo.convert_cache(dryrun=False)
     assert repo_valid(tmpdir)
 
@@ -124,6 +125,7 @@ def attic_key_file(attic_repo, tmpdir):
                                        MockArgs(keys_dir))
 
 
+@pytest.mark.skipif(attic is None, reason='cannot find an attic install')
 def test_keys(tmpdir, attic_repo, attic_key_file):
     """test key conversion
 
@@ -142,7 +144,8 @@ def test_keys(tmpdir, attic_repo, attic_key_file):
     assert key_valid(attic_key_file.path)
 
 
-def test_convert_all(tmpdir, attic_repo, attic_key_file):
+@pytest.mark.skipif(attic is None, reason='cannot find an attic install')
+def test_convert_all(tmpdir, attic_repo, attic_key_file, inplace):
     """test all conversion steps
 
     this runs everything. mostly redundant test, since everything is
@@ -156,8 +159,50 @@ def test_convert_all(tmpdir, attic_repo, attic_key_file):
     """
     # check should fail because of magic number
     assert not repo_valid(tmpdir)
-    print("opening attic repository with borg and converting")
+
+    def stat_segment(path):
+        return os.stat(os.path.join(path, 'data', '0', '0'))
+
+    def first_inode(path):
+        return stat_segment(path).st_ino
+
+    orig_inode = first_inode(attic_repo.path)
     repo = AtticRepositoryUpgrader(str(tmpdir), create=False)
-    repo.upgrade(dryrun=False)
+    # replicate command dispatch, partly
+    os.umask(RemoteRepository.umask)
+    backup = repo.upgrade(dryrun=False, inplace=inplace)
+    if inplace:
+        assert backup is None
+        assert first_inode(repo.path) == orig_inode
+    else:
+        assert backup
+        assert first_inode(repo.path) != first_inode(backup)
+        # i have seen cases where the copied tree has world-readable
+        # permissions, which is wrong
+        assert stat_segment(backup).st_mode & 0o007 == 0
+
     assert key_valid(attic_key_file.path)
     assert repo_valid(tmpdir)
+
+
+def test_hardlink(tmpdir, inplace):
+    """test that we handle hard links properly
+
+    that is, if we are in "inplace" mode, hardlinks should *not*
+    change (ie. we write to the file directly, so we do not rewrite the
+    whole file, and we do not re-create the file).
+
+    if we are *not* in inplace mode, then the inode should change, as
+    we are supposed to leave the original inode alone."""
+    a = str(tmpdir.join('a'))
+    with open(a, 'wb') as tmp:
+        tmp.write(b'aXXX')
+    b = str(tmpdir.join('b'))
+    os.link(a, b)
+    AtticRepositoryUpgrader.header_replace(b, b'a', b'b', inplace=inplace)
+    if not inplace:
+        assert os.stat(a).st_ino != os.stat(b).st_ino
+    else:
+        assert os.stat(a).st_ino == os.stat(b).st_ino
+    with open(b, 'rb') as tmp:
+        assert tmp.read() == b'bXXX'

borg/upgrader.py

@@ -1,6 +1,10 @@
 from binascii import hexlify
+import datetime
+import logging
+logger = logging.getLogger(__name__)
 import os
 import shutil
+import sys
 import time
 
 from .helpers import get_keys_dir, get_cache_dir
@@ -12,7 +16,7 @@ ATTIC_MAGIC = b'ATTICSEG'
 
 
 class AtticRepositoryUpgrader(Repository):
-    def upgrade(self, dryrun=True):
+    def upgrade(self, dryrun=True, inplace=False):
         """convert an attic repository to a borg repository
 
         those are the files that need to be upgraded here, from most
@@ -23,14 +27,20 @@ class AtticRepositoryUpgrader(Repository):
         we nevertheless do the order in reverse, as we prefer to do
         the fast stuff first, to improve interactivity.
         """
-        print("reading segments from attic repository using borg")
-        # we need to open it to load the configuration and other fields
+        backup = None
+        if not inplace:
+            backup = '{}.upgrade-{:%Y-%m-%d-%H:%M:%S}'.format(self.path, datetime.datetime.now())
+            logger.info('making a hardlink copy in %s', backup)
+            if not dryrun:
+                shutil.copytree(self.path, backup, copy_function=os.link)
+        logger.info("opening attic repository with borg and converting")
+        # we need to open the repo to load configuration, keyfiles and segments
         self.open(self.path, exclusive=False)
         segments = [filename for i, filename in self.io.segment_iterator()]
         try:
             keyfile = self.find_attic_keyfile()
         except KeyfileNotFoundError:
-            print("no key file found for repository")
+            logger.warning("no key file found for repository")
         else:
             self.convert_keyfiles(keyfile, dryrun)
         self.close()
@@ -39,13 +49,14 @@ class AtticRepositoryUpgrader(Repository):
                                    exclusive=True).acquire()
         try:
             self.convert_cache(dryrun)
-            self.convert_segments(segments, dryrun)
+            self.convert_segments(segments, dryrun=dryrun, inplace=inplace)
         finally:
             self.lock.release()
             self.lock = None
+        return backup
 
     @staticmethod
-    def convert_segments(segments, dryrun):
+    def convert_segments(segments, dryrun=True, inplace=False):
         """convert repository segments from attic to borg
 
         replacement pattern is `s/ATTICSEG/BORG_SEG/` in files in
@@ -53,26 +64,39 @@ class AtticRepositoryUpgrader(Repository):
 
         luckily the magic string length didn't change so we can just
         replace the 8 first bytes of all regular files in there."""
-        print("converting %d segments..." % len(segments))
+        logger.info("converting %d segments..." % len(segments))
         i = 0
         for filename in segments:
             i += 1
-            print("\rconverting segment %d/%d in place, %.2f%% done (%s)"
-                  % (i, len(segments), 100*float(i)/len(segments), filename), end='')
+            print("\rconverting segment %d/%d, %.2f%% done (%s)"
+                  % (i, len(segments), 100*float(i)/len(segments), filename),
+                  end='', file=sys.stderr)
             if dryrun:
                 time.sleep(0.001)
             else:
-                AtticRepositoryUpgrader.header_replace(filename, ATTIC_MAGIC, MAGIC)
-        print()
+                AtticRepositoryUpgrader.header_replace(filename, ATTIC_MAGIC, MAGIC, inplace=inplace)
+        print(file=sys.stderr)
 
     @staticmethod
-    def header_replace(filename, old_magic, new_magic):
+    def header_replace(filename, old_magic, new_magic, inplace=True):
         with open(filename, 'r+b') as segment:
             segment.seek(0)
             # only write if necessary
             if segment.read(len(old_magic)) == old_magic:
-                segment.seek(0)
-                segment.write(new_magic)
+                if inplace:
+                    segment.seek(0)
+                    segment.write(new_magic)
+                else:
+                    # rename the hardlink and rewrite the file. this works
+                    # because the file is still open. so even though the file
+                    # is renamed, we can still read it until it is closed.
+                    os.rename(filename, filename + '.tmp')
+                    with open(filename, 'wb') as new_segment:
+                        new_segment.write(new_magic)
+                        new_segment.write(segment.read())
+                    # the little dance with the .tmp file is necessary
+                    # because Windows won't allow overwriting an open file.
+                    os.unlink(filename + '.tmp')
 
     def find_attic_keyfile(self):
         """find the attic keyfiles
@@ -107,12 +131,12 @@ class AtticRepositoryUpgrader(Repository):
         key file because magic string length changed, but that's not a
         problem because the keyfiles are small (compared to, say,
         all the segments)."""
-        print("converting keyfile %s" % keyfile)
+        logger.info("converting keyfile %s" % keyfile)
         with open(keyfile, 'r') as f:
             data = f.read()
         data = data.replace(AtticKeyfileKey.FILE_ID, KeyfileKey.FILE_ID, 1)
         keyfile = os.path.join(get_keys_dir(), os.path.basename(keyfile))
-        print("writing borg keyfile to %s" % keyfile)
+        logger.info("writing borg keyfile to %s" % keyfile)
         if not dryrun:
             with open(keyfile, 'w') as f:
                 f.write(data)
@@ -135,12 +159,14 @@ class AtticRepositoryUpgrader(Repository):
           `Cache.open()`, edit in place and then `Cache.close()` to
           make sure we have locking right
         """
-        caches = []
         transaction_id = self.get_index_transaction_id()
         if transaction_id is None:
-            print('no index file found for repository %s' % self.path)
+            logger.warning('no index file found for repository %s' % self.path)
         else:
-            caches += [os.path.join(self.path, 'index.%d' % transaction_id).encode('utf-8')]
+            index = os.path.join(self.path, 'index.%d' % transaction_id).encode('utf-8')
+            logger.info("converting index index %s" % index)
+            if not dryrun:
+                AtticRepositoryUpgrader.header_replace(index, b'ATTICIDX', b'BORG_IDX')
 
         # copy of attic's get_cache_dir()
         attic_cache_dir = os.environ.get('ATTIC_CACHE_DIR',
@@ -160,23 +186,23 @@ class AtticRepositoryUpgrader(Repository):
             :params path: the basename of the cache file to copy
             (example: "files" or "chunks") as a string
 
-            :returns: the borg file that was created or None if non
-            was created.
+            :returns: the borg file that was created or None if no
+            Attic cache file was found.
 
             """
             attic_file = os.path.join(attic_cache_dir, path)
             if os.path.exists(attic_file):
                 borg_file = os.path.join(borg_cache_dir, path)
                 if os.path.exists(borg_file):
-                    print("borg cache file already exists in %s, skipping conversion of %s" % (borg_file, attic_file))
+                    logger.warning("borg cache file already exists in %s, not copying from Attic", borg_file)
                 else:
-                    print("copying attic cache file from %s to %s" % (attic_file, borg_file))
+                    logger.info("copying attic cache file from %s to %s" % (attic_file, borg_file))
                     if not dryrun:
                         shutil.copyfile(attic_file, borg_file)
-                    return borg_file
+                return borg_file
             else:
-                print("no %s cache file found in %s" % (path, attic_file))
-            return None
+                logger.warning("no %s cache file found in %s" % (path, attic_file))
+                return None
 
         # XXX: untested, because generating cache files is a PITA, see
         # Archiver.do_create() for proof
@@ -190,11 +216,10 @@ class AtticRepositoryUpgrader(Repository):
 
             # we need to convert the headers of those files, copy first
             for cache in ['chunks']:
-                copied = copy_cache_file(cache)
-                if copied:
-                    print("converting cache %s" % cache)
-                    if not dryrun:
-                        AtticRepositoryUpgrader.header_replace(cache, b'ATTICIDX', b'BORG_IDX')
+                cache = copy_cache_file(cache)
+                logger.info("converting cache %s" % cache)
+                if not dryrun:
+                    AtticRepositoryUpgrader.header_replace(cache, b'ATTICIDX', b'BORG_IDX')
 
 
 class AtticKeyfileKey(KeyfileKey):

docs/Makefile

@@ -36,7 +36,7 @@ help:
 clean:
 	-rm -rf $(BUILDDIR)/*
 
-html: usage api.rst
+html:
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
@@ -128,43 +128,3 @@ doctest:
 	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
 	@echo "Testing of doctests in the sources finished, look at the " \
 	      "results in $(BUILDDIR)/doctest/output.txt."
-
-gh-io: html
-	GH_IO_CLONE="`mktemp -d`" && \
-    git clone git@github.com:borgbackup/borgbackup.github.io.git $$GH_IO_CLONE && \
-	(cd $$GH_IO_CLONE && git rm -r *) && \
-	cp -r _build/html/* $$GH_IO_CLONE && \
-	(cd $$GH_IO_CLONE && git add -A && git commit -m 'Updated borgbackup.github.io' && git push) && \
-	rm -rf $$GH_IO_CLONE
-
-inotify: html
-	while inotifywait -r . --exclude usage.rst --exclude '_build/*' ; do make html ; done
-
-# generate list of targets
-usage: $(shell borg help | grep -A1 "Available commands:" | tail -1 | sed 's/[{} ]//g;s/,\|^/.rst.inc usage\//g;s/^.rst.inc//;s/usage\/help//')
-
-# generate help file based on usage
-usage/%.rst.inc: ../borg/archiver.py
-	@echo generating usage for $*
-	@printf ".. _borg_$*:\n\n" > $@
-	@printf "borg $*\n" >> $@
-	@echo -n borg $* | tr 'a-z- ' '-' >> $@
-	@printf "\n::\n\n" >> $@
-	@borg help $* --usage-only | sed -e 's/^/    /' >> $@
-	@printf "\nDescription\n~~~~~~~~~~~\n" >> $@
-	@borg help $* --epilog-only >> $@
-
-api.rst: Makefile
-	@echo "auto-generating API documentation"
-	@echo "Borg Backup API documentation" > $@
-	@echo "=============================" >> $@
-	@echo "" >> $@
-	@for mod in ../borg/*.pyx ../borg/*.py; do \
-		if echo "$$mod" | grep -q "/_"; then \
-			continue ; \
-		fi ; \
-		printf ".. automodule:: "; \
-		echo "$$mod" | sed "s!\.\./!!;s/\.pyx\?//;s!/!.!"; \
-		echo "    :members:"; \
-		echo "    :undoc-members:"; \
-	done >> $@

docs/_themes/local/sidebarlogo.html

@@ -1,5 +0,0 @@
-<div class="sidebarlogo">
-  <a href="{{ pathto('index') }}">
-  <div class="title">Borg</div>
-</a>
-</div>

docs/_themes/local/sidebarusefullinks.html

@@ -1,20 +0,0 @@
-<a href="https://github.com/borgbackup/borg"><img style="position: fixed; top: 0; right: 0; border: 0;"
-  src="https://s3.amazonaws.com/github/ribbons/forkme_right_gray_6d6d6d.png" alt="Fork me on GitHub"></a>
-
-<h3>Useful Links</h3>
-<ul>
-  <li><a href="https://borgbackup.github.io/borgbackup/">Main Web Site</a></li>
-  <li><a href="https://pypi.python.org/pypi/borgbackup">PyPI packages</a></li>
-  <li><a href="https://github.com/borgbackup/borg/issues/214">Binaries</a></li>
-  <li><a href="https://github.com/borgbackup/borg/blob/master/CHANGES.rst">Current ChangeLog</a></li>
-  <li><a href="https://github.com/borgbackup/borg">GitHub</a></li>
-  <li><a href="https://github.com/borgbackup/borg/issues">Issue Tracker</a></li>
-  <li><a href="https://www.bountysource.com/teams/borgbackup">Bounties &amp; Fundraisers</a></li>
-  <li><a href="http://librelist.com/browser/borgbackup/">Mailing List</a></li>
-</ul>
-
-<h3>Related Projects</h3>
-<ul>
-  <li><a href="https://borgbackup.github.io/borgweb/">BorgWeb</a></li>
-</ul>
-

docs/_themes/local/static/local.css_t

@@ -1,173 +0,0 @@
-@import url("basic.css");
-@import url(//fonts.googleapis.com/css?family=Black+Ops+One);
-
-body {
-  font-family: Arial, Helvetica, sans-serif;
-  background-color: black;
-  margin: 0;
-  padding: 0;
-  position: relative;
-}
-
-div.related {
-  display: none;
-  background-color: black;
-  padding: .4em;
-  width: 800px;
-  margin: 0 auto;
-}
-
-div.related a {
-  color: white;
-  text-decoration: none;
-}
-
-div.document {
-  width: 1030px;
-  margin: 0 auto;
-}
-
-div.documentwrapper {
-  float: right;
-  width: 760px;
-  padding: 0 20px 20px 20px;
-  color: #00cc00;
-  background-color: #000000;
-  margin-bottom: 2em;
-}
-
-div.sphinxsidebar {
-  margin-left: 0;
-  padding-right: 20px;
-  width: 230px;
-  background: #081008;
-  position: absolute;
-  top: 0;
-  min-height: 100%;
-}
-
-h1, h2, h3 {
-  font-weight: normal;
-  color: #33ff33;
-}
-
-h1 {
-  margin: .8em 0 .5em;
-  font-size: 200%;
-}
-
-h2 {
-  margin: 1.2em 0 .6em;
-  font-size: 140%;
-}
-
-h3 {
-  margin: 1.2em 0 .6em;
-  font-size: 110%;
-}
-
-ul {
-  padding-left: 1.2em;
-  margin-bottom: .3em;
-}
-
-ul ul {
-  font-size: 95%;
-}
-
-li {
-  margin: .1em 0;
-}
-
-a:link {
-  color: #dddd00;
-  text-decoration: none;
-}
-
-a:visited {
-  color: #990000;
-  text-decoration: none;
-}
-
-a:hover {
-  color: #dd0000;
-  border-bottom: 1px dotted #dd0000;
-}
-
-div.sphinxsidebar a:link, div.sphinxsidebar a:visited {
-  border-bottom: 1px dotted #555;
-}
-
-div.sphinxsidebar {
-  color: #00cc00;
-  background: 0000000;
-}
-
-div.sphinxsidebar input {
-  color: #00ff00;
-  background: 0000000;
-  border: 1px solid #444444;
-}
-
-pre {
-  padding: 10px 20px;
-  background: #101010;
-  color: #22cc22;
-  line-height: 1.5em;
-  border-bottom: 2px solid black;
-}
-
-pre a:link,
-pre a:visited {
-  color: #00b0e4;
-}
-
-div.sidebarlogo .title {
-  font-family: 'Black Ops One', cursive;
-  font-size: 500%;
-}
-
-div.sidebarlogo a {
-  color: #00dd00;
-}
-
-div.sidebarlogo .subtitle {
-  font-style: italic;
-  color: #777;
-}
-
-tt span.pre {
-  font-size: 110%;
-}
-
-dt {
-  font-size: 95%;
-}
-
-div.admonition p.admonition-title + p {
-  display: inline;
-}
-
-div.admonition p {
-  margin-bottom: 5px;
-}
-
-p.admonition-title {
-  display: inline;
-}
-
-p.admonition-title:after {
-  content: ":";
-}
-
-div.note {
-  background-color: #002211;
-  border-bottom: 2px solid #22dd22;
-}
-
-div.seealso {
-  background-color: #0fe;
-  border: 1px solid #0f6;
-  border-radius: .4em;
-  box-shadow: 2px 2px #dd6;
-}

docs/_themes/local/theme.conf

@@ -1,6 +0,0 @@
-[theme]
-inherit = basic
-stylesheet = local.css
-pygments_style = tango
-
-[options]

docs/authors.rst

@@ -0,0 +1,11 @@
+.. include:: global.rst.inc
+
+.. include:: ../AUTHORS
+
+License
+=======
+
+.. _license:
+
+.. include:: ../LICENSE
+   :literal:

docs/changes.rst

@@ -1,4 +1,576 @@
-.. include:: global.rst.inc
-.. _changelog:
+Changelog
+=========
 
-.. include:: ../CHANGES.rst
+Version 0.28.0
+--------------
+
+New features:
+
+- refactor return codes (exit codes), fixes #61
+- give a final status into the log output, including exit code, fixes #58
+- borg create backups atime and ctime additionally to mtime, fixes #317
+  - extract: support atime additionally to mtime
+  - FUSE: support ctime and atime additionally to mtime
+- support borg --version
+
+Bug fixes:
+
+- setup.py: fix bug related to BORG_LZ4_PREFIX processing
+- borg mount: fix unlocking of repository at umount time, fixes #331
+- fix reading files without touching their atime, #334
+- non-ascii ACL fixes for Linux, FreeBSD and OS X, #277
+- fix acl_use_local_uid_gid() and add a test for it, attic #359
+- borg upgrade: do not upgrade repositories in place by default, #299
+- fix cascading failure with the index conversion code, #269
+- borg check: implement 'cmdline' archive metadata value decoding, #311
+
+Other changes:
+
+- improve file size displays
+- convert to more flexible size formatters
+- explicitely commit to the units standard, #289
+- archiver: add E status (means that an error occured when processing this
+  (single) item
+- do binary releases via "github releases", closes #214
+- create: use -x and --one-file-system (was: --do-not-cross-mountpoints), #296
+- a lot of changes related to using "logging" module and screen output, #233
+- show progress display if on a tty, output more progress information, #303
+- factor out status output so it is consistent, fix surrogates removal,
+  maybe fixes #309
+- benchmarks: test create, extract, list, delete, info, check, help, fixes #146
+- benchmarks: test with both the binary and the python code
+- tests:
+
+  - travis: also run tests on Python 3.5
+  - travis: use tox -r so it rebuilds the tox environments
+  - test the generated pyinstaller-based binary by archiver unit tests, #215
+  - vagrant: tests: announce whether fakeroot is used or not
+  - vagrant: add vagrant user to fuse group for debianoid systems also
+  - vagrant: llfuse install on darwin needs pkgconfig installed
+  - archiver tests: test with both the binary and the python code, fixes #215
+- docs:
+
+  - moved docs to borgbackup.readthedocs.org, #155
+  - a lot of fixes and improvements, use mobile-friendly RTD standard theme
+  - use zlib,6 compression in some examples, fixes #275
+  - add missing rename usage to docs, closes #279
+  - include the help offered by borg help <topic> in the usage docs, fixes #293
+  - include a list of major changes compared to attic into README, fixes #224
+  - add OS X install instructions, #197
+  - more details about the release process, #260
+  - fix linux glibc requirement (binaries built on debian7 now)
+  - build: move usage and API generation to setup.py
+  - update docs about return codes, #61
+  - remove api docs (too much breakage on rtd)
+  - borgbackup install + basics presentation (asciinema)
+  - describe the current style guide in documentation
+    
+
+Version 0.27.0
+--------------
+
+New features:
+
+- "borg upgrade" command - attic -> borg one time converter / migration, #21
+- temporary hack to avoid using lots of disk space for chunks.archive.d, #235:
+  To use it: rm -rf chunks.archive.d ; touch chunks.archive.d
+- respect XDG_CACHE_HOME, attic #181
+- add support for arbitrary SSH commands, attic #99
+- borg delete --cache-only REPO (only delete cache, not REPO), attic #123
+
+
+Bug fixes:
+
+- use Debian 7 (wheezy) to build pyinstaller borgbackup binaries, fixes slow
+  down observed when running the Centos6-built binary on Ubuntu, #222
+- do not crash on empty lock.roster, fixes #232
+- fix multiple issues with the cache config version check, #234
+- fix segment entry header size check, attic #352
+  plus other error handling improvements / code deduplication there.
+- always give segment and offset in repo IntegrityErrors
+
+
+Other changes:
+
+- stop producing binary wheels, remove docs about it, #147
+- docs:
+  - add warning about prune
+  - generate usage include files only as needed
+  - development docs: add Vagrant section
+  - update / improve / reformat FAQ
+  - hint to single-file pyinstaller binaries from README
+
+
+Version 0.26.1
+--------------
+
+This is a minor update, just docs and new pyinstaller binaries.
+
+- docs update about python and binary requirements
+- better docs for --read-special, fix #220
+- re-built the binaries, fix #218 and #213 (glibc version issue)
+- update web site about single-file pyinstaller binaries
+
+Note: if you did a python-based installation, there is no need to upgrade.
+
+
+Version 0.26.0
+--------------
+
+New features:
+
+- Faster cache sync (do all in one pass, remove tar/compression stuff), #163
+- BORG_REPO env var to specify the default repo, #168
+- read special files as if they were regular files, #79
+- implement borg create --dry-run, attic issue #267
+- Normalize paths before pattern matching on OS X, #143
+- support OpenBSD and NetBSD (except xattrs/ACLs)
+- support / run tests on Python 3.5
+
+Bug fixes:
+
+- borg mount repo: use absolute path, attic #200, attic #137
+- chunker: use off_t to get 64bit on 32bit platform, #178
+- initialize chunker fd to -1, so it's not equal to STDIN_FILENO (0)
+- fix reaction to "no" answer at delete repo prompt, #182
+- setup.py: detect lz4.h header file location
+- to support python < 3.2.4, add less buggy argparse lib from 3.2.6 (#194)
+- fix for obtaining ``char *`` from temporary Python value (old code causes
+  a compile error on Mint 17.2)
+- llfuse 0.41 install troubles on some platforms, require < 0.41
+  (UnicodeDecodeError exception due to non-ascii llfuse setup.py)
+- cython code: add some int types to get rid of unspecific python add /
+  subtract operations (avoid ``undefined symbol FPE_``... error on some platforms)
+- fix verbose mode display of stdin backup
+- extract: warn if a include pattern never matched, fixes #209,
+  implement counters for Include/ExcludePatterns
+- archive names with slashes are invalid, attic issue #180
+- chunker: add a check whether the POSIX_FADV_DONTNEED constant is defined -
+  fixes building on OpenBSD.
+
+Other changes:
+
+- detect inconsistency / corruption / hash collision, #170
+- replace versioneer with setuptools_scm, #106
+- docs:
+
+  - pkg-config is needed for llfuse installation
+  - be more clear about pruning, attic issue #132
+- unit tests:
+
+  - xattr: ignore security.selinux attribute showing up
+  - ext3 seems to need a bit more space for a sparse file
+  - do not test lzma level 9 compression (avoid MemoryError)
+  - work around strange mtime granularity issue on netbsd, fixes #204
+  - ignore st_rdev if file is not a block/char device, fixes #203
+  - stay away from the setgid and sticky mode bits
+- use Vagrant to do easy cross-platform testing (#196), currently:
+
+  - Debian 7 "wheezy" 32bit, Debian 8 "jessie" 64bit
+  - Ubuntu 12.04 32bit, Ubuntu 14.04 64bit
+  - Centos 7 64bit
+  - FreeBSD 10.2 64bit
+  - OpenBSD 5.7 64bit
+  - NetBSD 6.1.5 64bit
+  - Darwin (OS X Yosemite)
+
+
+Version 0.25.0
+--------------
+
+Compatibility notes:
+
+- lz4 compression library (liblz4) is a new requirement (#156)
+- the new compression code is very compatible: as long as you stay with zlib
+  compression, older borg releases will still be able to read data from a
+  repo/archive made with the new code (note: this is not the case for the
+  default "none" compression, use "zlib,0" if you want a "no compression" mode
+  that can be read by older borg). Also the new code is able to read repos and
+  archives made with older borg versions (for all zlib levels  0..9).
+
+Deprecations:
+
+- --compression N (with N being a number, as in 0.24) is deprecated.
+  We keep the --compression 0..9 for now to not break scripts, but it is
+  deprecated and will be removed later, so better fix your scripts now:
+  --compression 0 (as in 0.24) is the same as --compression zlib,0 (now).
+  BUT: if you do not want compression, you rather want --compression none
+  (which is the default).
+  --compression 1 (in 0.24) is the same as --compression zlib,1 (now)
+  --compression 9 (in 0.24) is the same as --compression zlib,9 (now)
+
+New features:
+
+- create --compression none (default, means: do not compress, just pass through
+  data "as is". this is more efficient than zlib level 0 as used in borg 0.24)
+- create --compression lz4 (super-fast, but not very high compression)
+- create --compression zlib,N (slower, higher compression, default for N is 6)
+- create --compression lzma,N (slowest, highest compression, default N is 6)
+- honor the nodump flag (UF_NODUMP) and do not backup such items
+- list --short just outputs a simple list of the files/directories in an archive
+
+Bug fixes:
+
+- fixed --chunker-params parameter order confusion / malfunction, fixes #154
+- close fds of segments we delete (during compaction)
+- close files which fell out the lrucache
+- fadvise DONTNEED now is only called for the byte range actually read, not for
+  the whole file, fixes #158.
+- fix issue with negative "all archives" size, fixes #165
+- restore_xattrs: ignore if setxattr fails with EACCES, fixes #162
+
+Other changes:
+
+- remove fakeroot requirement for tests, tests run faster without fakeroot
+  (test setup does not fail any more without fakeroot, so you can run with or
+  without fakeroot), fixes #151 and #91.
+- more tests for archiver
+- recover_segment(): don't assume we have an fd for segment
+- lrucache refactoring / cleanup, add dispose function, py.test tests
+- generalize hashindex code for any key length (less hardcoding)
+- lock roster: catch file not found in remove() method and ignore it
+- travis CI: use requirements file
+- improved docs:
+
+  - replace hack for llfuse with proper solution (install libfuse-dev)
+  - update docs about compression
+  - update development docs about fakeroot
+  - internals: add some words about lock files / locking system
+  - support: mention BountySource and for what it can be used
+  - theme: use a lighter green
+  - add pypi, wheel, dist package based install docs
+  - split install docs into system-specific preparations and generic instructions
+
+
+Version 0.24.0
+--------------
+
+Incompatible changes (compared to 0.23):
+
+- borg now always issues --umask NNN option when invoking another borg via ssh
+  on the repository server. By that, it's making sure it uses the same umask
+  for remote repos as for local ones. Because of this, you must upgrade both
+  server and client(s) to 0.24.
+- the default umask is 077 now (if you do not specify via --umask) which might
+  be a different one as you used previously. The default umask avoids that
+  you accidentally give access permissions for group and/or others to files
+  created by borg (e.g. the repository).
+
+Deprecations:
+
+- "--encryption passphrase" mode is deprecated, see #85 and #97.
+  See the new "--encryption repokey" mode for a replacement.
+
+New features:
+
+- borg create --chunker-params ... to configure the chunker, fixes #16
+  (attic #302, attic #300, and somehow also #41).
+  This can be used to reduce memory usage caused by chunk management overhead,
+  so borg does not create a huge chunks index/repo index and eats all your RAM
+  if you back up lots of data in huge files (like VM disk images).
+  See docs/misc/create_chunker-params.txt for more information.
+- borg info now reports chunk counts in the chunk index.
+- borg create --compression 0..9 to select zlib compression level, fixes #66
+  (attic #295).
+- borg init --encryption repokey (to store the encryption key into the repo),
+  fixes #85
+- improve at-end error logging, always log exceptions and set exit_code=1
+- LoggedIO: better error checks / exceptions / exception handling
+- implement --remote-path to allow non-default-path borg locations, #125
+- implement --umask M and use 077 as default umask for better security, #117
+- borg check: give a named single archive to it, fixes #139
+- cache sync: show progress indication
+- cache sync: reimplement the chunk index merging in C
+
+Bug fixes:
+
+- fix segfault that happened for unreadable files (chunker: n needs to be a
+  signed size_t), #116
+- fix the repair mode, #144
+- repo delete: add destroy to allowed rpc methods, fixes issue #114
+- more compatible repository locking code (based on mkdir), maybe fixes #92
+  (attic #317, attic #201).
+- better Exception msg if no Borg is installed on the remote repo server, #56
+- create a RepositoryCache implementation that can cope with >2GiB,
+  fixes attic #326.
+- fix Traceback when running check --repair, attic #232
+- clarify help text, fixes #73.
+- add help string for --no-files-cache, fixes #140
+
+Other changes:
+
+- improved docs:
+
+  - added docs/misc directory for misc. writeups that won't be included
+    "as is" into the html docs.
+  - document environment variables and return codes (attic #324, attic #52)
+  - web site: add related projects, fix web site url, IRC #borgbackup
+  - Fedora/Fedora-based install instructions added to docs
+  - Cygwin-based install instructions added to docs
+  - updated AUTHORS
+  - add FAQ entries about redundancy / integrity
+  - clarify that borg extract uses the cwd as extraction target
+  - update internals doc about chunker params, memory usage and compression
+  - added docs about development
+  - add some words about resource usage in general
+  - document how to backup a raw disk
+  - add note about how to run borg from virtual env
+  - add solutions for (ll)fuse installation problems
+  - document what borg check does, fixes #138
+  - reorganize borgbackup.github.io sidebar, prev/next at top
+  - deduplicate and refactor the docs / README.rst
+
+- use borg-tmp as prefix for temporary files / directories
+- short prune options without "keep-" are deprecated, do not suggest them
+- improved tox configuration
+- remove usage of unittest.mock, always use mock from pypi
+- use entrypoints instead of scripts, for better use of the wheel format and
+  modern installs
+- add requirements.d/development.txt and modify tox.ini
+- use travis-ci for testing based on Linux and (new) OS X
+- use coverage.py, pytest-cov and codecov.io for test coverage support
+
+I forgot to list some stuff already implemented in 0.23.0, here they are:
+
+New features:
+
+- efficient archive list from manifest, meaning a big speedup for slow
+  repo connections and "list <repo>", "delete <repo>", "prune" (attic #242,
+  attic #167)
+- big speedup for chunks cache sync (esp. for slow repo connections), fixes #18
+- hashindex: improve error messages
+
+Other changes:
+
+- explicitly specify binary mode to open binary files
+- some easy micro optimizations
+
+
+Version 0.23.0
+--------------
+
+Incompatible changes (compared to attic, fork related):
+
+- changed sw name and cli command to "borg", updated docs
+- package name (and name in urls) uses "borgbackup" to have less collisions
+- changed repo / cache internal magic strings from ATTIC* to BORG*,
+  changed cache location to .cache/borg/ - this means that it currently won't
+  accept attic repos (see issue #21 about improving that)
+
+Bug fixes:
+
+- avoid defect python-msgpack releases, fixes attic #171, fixes attic #185
+- fix traceback when trying to do unsupported passphrase change, fixes attic #189
+- datetime does not like the year 10.000, fixes attic #139
+- fix "info" all archives stats, fixes attic #183
+- fix parsing with missing microseconds, fixes attic #282
+- fix misleading hint the fuse ImportError handler gave, fixes attic #237
+- check unpacked data from RPC for tuple type and correct length, fixes attic #127
+- fix Repository._active_txn state when lock upgrade fails
+- give specific path to xattr.is_enabled(), disable symlink setattr call that
+  always fails
+- fix test setup for 32bit platforms, partial fix for attic #196
+- upgraded versioneer, PEP440 compliance, fixes attic #257
+
+New features:
+
+- less memory usage: add global option --no-cache-files
+- check --last N (only check the last N archives)
+- check: sort archives in reverse time order
+- rename repo::oldname newname (rename repository)
+- create -v output more informative
+- create --progress (backup progress indicator)
+- create --timestamp (utc string or reference file/dir)
+- create: if "-" is given as path, read binary from stdin
+- extract: if --stdout is given, write all extracted binary data to stdout
+- extract --sparse (simple sparse file support)
+- extra debug information for 'fread failed'
+- delete <repo> (deletes whole repo + local cache)
+- FUSE: reflect deduplication in allocated blocks
+- only allow whitelisted RPC calls in server mode
+- normalize source/exclude paths before matching
+- use posix_fadvise to not spoil the OS cache, fixes attic #252
+- toplevel error handler: show tracebacks for better error analysis
+- sigusr1 / sigint handler to print current file infos - attic PR #286
+- RPCError: include the exception args we get from remote
+
+Other changes:
+
+- source: misc. cleanups, pep8, style
+- docs and faq improvements, fixes, updates
+- cleanup crypto.pyx, make it easier to adapt to other AES modes
+- do os.fsync like recommended in the python docs
+- source: Let chunker optionally work with os-level file descriptor.
+- source: Linux: remove duplicate os.fsencode calls
+- source: refactor _open_rb code a bit, so it is more consistent / regular
+- source: refactor indicator (status) and item processing
+- source: use py.test for better testing, flake8 for code style checks
+- source: fix tox >=2.0 compatibility (test runner)
+- pypi package: add python version classifiers, add FreeBSD to platforms
+
+
+Attic Changelog
+---------------
+
+Here you can see the full list of changes between each Attic release until Borg
+forked from Attic:
+
+Version 0.17
+~~~~~~~~~~~~
+
+(bugfix release, released on X)
+- Fix hashindex ARM memory alignment issue (#309)
+- Improve hashindex error messages (#298)
+
+Version 0.16
+~~~~~~~~~~~~
+
+(bugfix release, released on May 16, 2015)
+- Fix typo preventing the security confirmation prompt from working (#303)
+- Improve handling of systems with improperly configured file system encoding (#289)
+- Fix "All archives" output for attic info. (#183)
+- More user friendly error message when repository key file is not found (#236)
+- Fix parsing of iso 8601 timestamps with zero microseconds (#282)
+
+Version 0.15
+~~~~~~~~~~~~
+
+(bugfix release, released on Apr 15, 2015)
+- xattr: Be less strict about unknown/unsupported platforms (#239)
+- Reduce repository listing memory usage (#163).
+- Fix BrokenPipeError for remote repositories (#233)
+- Fix incorrect behavior with two character directory names (#265, #268)
+- Require approval before accessing relocated/moved repository (#271)
+- Require approval before accessing previously unknown unencrypted repositories (#271)
+- Fix issue with hash index files larger than 2GB.
+- Fix Python 3.2 compatibility issue with noatime open() (#164)
+- Include missing pyx files in dist files (#168)
+
+Version 0.14
+~~~~~~~~~~~~
+
+(feature release, released on Dec 17, 2014)
+
+- Added support for stripping leading path segments (#95)
+  "attic extract --strip-segments X"
+- Add workaround for old Linux systems without acl_extended_file_no_follow (#96)
+- Add MacPorts' path to the default openssl search path (#101)
+- HashIndex improvements, eliminates unnecessary IO on low memory systems.
+- Fix "Number of files" output for attic info. (#124)
+- limit create file permissions so files aren't read while restoring
+- Fix issue with empty xattr values (#106)
+
+Version 0.13
+~~~~~~~~~~~~
+
+(feature release, released on Jun 29, 2014)
+
+- Fix sporadic "Resource temporarily unavailable" when using remote repositories
+- Reduce file cache memory usage (#90)
+- Faster AES encryption (utilizing AES-NI when available)
+- Experimental Linux, OS X and FreeBSD ACL support (#66)
+- Added support for backup and restore of BSDFlags (OSX, FreeBSD) (#56)
+- Fix bug where xattrs on symlinks were not correctly restored
+- Added cachedir support. CACHEDIR.TAG compatible cache directories
+  can now be excluded using ``--exclude-caches`` (#74)
+- Fix crash on extreme mtime timestamps (year 2400+) (#81)
+- Fix Python 3.2 specific lockf issue (EDEADLK)
+
+Version 0.12
+~~~~~~~~~~~~
+
+(feature release, released on April 7, 2014)
+
+- Python 3.4 support (#62)
+- Various documentation improvements a new style
+- ``attic mount`` now supports mounting an entire repository not only
+  individual archives (#59)
+- Added option to restrict remote repository access to specific path(s):
+  ``attic serve --restrict-to-path X`` (#51)
+- Include "all archives" size information in "--stats" output. (#54)
+- Added ``--stats`` option to ``attic delete`` and ``attic prune``
+- Fixed bug where ``attic prune`` used UTC instead of the local time zone
+  when determining which archives to keep.
+- Switch to SI units (Power of 1000 instead 1024) when printing file sizes
+
+Version 0.11
+~~~~~~~~~~~~
+
+(feature release, released on March 7, 2014)
+
+- New "check" command for repository consistency checking (#24)
+- Documentation improvements
+- Fix exception during "attic create" with repeated files (#39)
+- New "--exclude-from" option for attic create/extract/verify.
+- Improved archive metadata deduplication.
+- "attic verify" has been deprecated. Use "attic extract --dry-run" instead.
+- "attic prune --hourly|daily|..." has been deprecated.
+  Use "attic prune --keep-hourly|daily|..." instead.
+- Ignore xattr errors during "extract" if not supported by the filesystem. (#46)
+
+Version 0.10
+~~~~~~~~~~~~
+
+(bugfix release, released on Jan 30, 2014)
+
+- Fix deadlock when extracting 0 sized files from remote repositories
+- "--exclude" wildcard patterns are now properly applied to the full path
+  not just the file name part (#5).
+- Make source code endianness agnostic (#1)
+
+Version 0.9
+~~~~~~~~~~~
+
+(feature release, released on Jan 23, 2014)
+
+- Remote repository speed and reliability improvements.
+- Fix sorting of segment names to ignore NFS left over files. (#17)
+- Fix incorrect display of time (#13)
+- Improved error handling / reporting. (#12)
+- Use fcntl() instead of flock() when locking repository/cache. (#15)
+- Let ssh figure out port/user if not specified so we don't override .ssh/config (#9)
+- Improved libcrypto path detection (#23).
+
+Version 0.8.1
+~~~~~~~~~~~~~
+
+(bugfix release, released on Oct 4, 2013)
+
+- Fix segmentation fault issue.
+
+Version 0.8
+~~~~~~~~~~~
+
+(feature release, released on Oct 3, 2013)
+
+- Fix xattr issue when backing up sshfs filesystems (#4)
+- Fix issue with excessive index file size (#6)
+- Support access of read only repositories.
+- New syntax to enable repository encryption:
+    attic init --encryption="none|passphrase|keyfile".
+- Detect and abort if repository is older than the cache.
+
+
+Version 0.7
+~~~~~~~~~~~
+
+(feature release, released on Aug 5, 2013)
+
+- Ported to FreeBSD
+- Improved documentation
+- Experimental: Archives mountable as fuse filesystems.
+- The "user." prefix is no longer stripped from xattrs on Linux
+
+
+Version 0.6.1
+~~~~~~~~~~~~~
+
+(bugfix release, released on July 19, 2013)
+
+- Fixed an issue where mtime was not always correctly restored.
+
+
+Version 0.6
+~~~~~~~~~~~
+
+First public release on July 9, 2013

docs/conf.py

@@ -19,6 +19,8 @@ sys.path.insert(0, os.path.abspath('..'))
 
 from borg import __version__ as sw_version
 
+on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
+
 # -- General configuration -----------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
@@ -92,7 +94,11 @@ pygments_style = 'sphinx'
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-html_theme = 'local'
+#html_theme = ''
+if not on_rtd:  # only import and set the theme if we're building docs locally
+    import sphinx_rtd_theme
+    html_theme = 'sphinx_rtd_theme'
+    html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
@@ -111,17 +117,17 @@ html_theme_path = ['_themes']
 
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
-#html_logo = None
+html_logo = '_static/favicon.ico'
 
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
 # pixels large.
-html_favicon = 'favicon.ico'
+html_favicon = '_static/favicon.ico'
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = []
+#html_static_path = ['_static']
 
 # If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
 # using the given strftime format.

docs/development.rst

@@ -9,6 +9,15 @@ This chapter will get you started with |project_name|' development.
 |project_name| is written in Python (with a little bit of Cython and C for
 the performance critical parts).
 
+Style guide
+-----------
+
+We generally follow `pep8
+<https://www.python.org/dev/peps/pep-0008/>`_, with 120 columns
+instead of 79. We do *not* use form-feed (``^L``) characters to
+separate sections either. The `flake8
+<https://flake8.readthedocs.org/>`_ commandline tool should be used to
+check for style errors before sending pull requests.
 
 Building a development environment
 ----------------------------------
@@ -68,6 +77,9 @@ Now run::
 
 Then point a web browser at docs/_build/html/index.html.
 
+The website is updated automatically through Github web hooks on the
+main repository.
+
 Using Vagrant
 -------------
 
@@ -91,49 +103,54 @@ Usage::
      vagrant scp OS:/vagrant/borg/borg/dist/borg .
 
 
-Creating a new release
-----------------------
+Creating standalone binaries
+----------------------------
 
-Checklist::
+Make sure you have everything built and installed (including llfuse and fuse).
+When using the Vagrant VMs, pyinstaller will already be installed.
 
-- all issues for this milestone closed?
-- any low hanging fruit left on the issue tracker?
-- run tox on all supported platforms via vagrant, check for test fails.
-- is Travis CI happy also?
-- update CHANGES.rst (compare to git log). check version number of upcoming release.
-- check MANIFEST.in and setup.py - are they complete?
-- tag the release::
+With virtual env activated::
 
-  git tag -s -m "tagged release" 0.26.0
+  pip install pyinstaller>=3.0  # or git checkout master
+  pyinstaller -F -n borg-PLATFORM --hidden-import=logging.config borg/__main__.py
+  for file in dist/borg-*; do gpg --armor --detach-sign $file; done
 
-- cd docs ; make html  # to update the usage include files
-- update website with the html
-- create a release on PyPi::
+If you encounter issues, see also our `Vagrantfile` for details.
 
-    python setup.py register sdist upload --identity="Thomas Waldmann" --sign
+.. note:: Standalone binaries built with pyinstaller are supposed to
+          work on same OS, same architecture (x86 32bit, amd64 64bit)
+          without external dependencies.
 
-- close release milestone.
-- announce on::
 
-  - mailing list
-  - Twitter
-  - IRC channel (topic)
+Creating a new release
+----------------------
 
-- create standalone binaries and link them from issue tracker: https://github.com/borgbackup/borg/issues/214
+Checklist:
 
+- make sure all issues for this milestone are closed or moved to the
+  next milestone
+- find and fix any low hanging fruit left on the issue tracker
+- run tox on all supported platforms via vagrant, check for test failures
+- check that Travis CI is also happy
+- update ``CHANGES.rst``, based on ``git log $PREVIOUS_RELEASE..``
+- check version number of upcoming release in ``CHANGES.rst``
+- verify that ``MANIFEST.in`` and ``setup.py`` are complete
+- tag the release::
 
-Creating standalone binaries
-----------------------------
+    git tag -s -m "tagged/signed release X.Y.Z" X.Y.Z
 
-Make sure you have everything built and installed (including llfuse and fuse).
+- build fresh docs and update the web site with them
+- create a release on PyPi::
 
-With virtual env activated::
+    python setup.py register sdist upload --identity="Thomas Waldmann" --sign
 
-  pip install pyinstaller>=3.0  # or git checkout master
-  pyinstaller -F -n borg-PLATFORM --hidden-import=logging.config borg/__main__.py
-  ls -l dist/*
+- close release milestone on Github
+- announce on:
 
-If you encounter issues, see also our `Vagrantfile` for details.
+ - `mailing list <mailto:borgbackup@librelist.org>`_
+ - Twitter (follow @ThomasJWaldmann for these tweets)
+ - `IRC channel <irc://irc.freenode.net/borgbackup>`_ (change ``/topic``)
 
-Note: Standalone binaries built with pyinstaller are supposed to work on same OS,
-      same architecture (x86 32bit, amd64 64bit) without external dependencies.
+- create a Github release, include:
+  * standalone binaries (see above for how to create them)
+  * a link to ``CHANGES.rst``

docs/faq.rst

@@ -5,24 +5,30 @@ Frequently asked questions
 ==========================
 
 Can I backup VM disk images?
-    Yes, the :ref:`deduplication <deduplication_def>` technique used by
-    |project_name| makes sure only the modified parts of the file are stored.
-    Also, we have optional simple sparse file support for extract.
+----------------------------
+
+Yes, the `deduplication`_ technique used by
+|project_name| makes sure only the modified parts of the file are stored.
+Also, we have optional simple sparse file support for extract.
 
 Can I backup from multiple servers into a single repository?
-    Yes, but in order for the deduplication used by |project_name| to work, it
-    needs to keep a local cache containing checksums of all file
-    chunks already stored in the repository. This cache is stored in
-    ``~/.cache/borg/``.  If |project_name| detects that a repository has been
-    modified since the local cache was updated it will need to rebuild
-    the cache. This rebuild can be quite time consuming.
-
-    So, yes it's possible. But it will be most efficient if a single
-    repository is only modified from one place. Also keep in mind that
-    |project_name| will keep an exclusive lock on the repository while creating
-    or deleting archives, which may make *simultaneous* backups fail.
+------------------------------------------------------------
+
+Yes, but in order for the deduplication used by |project_name| to work, it
+needs to keep a local cache containing checksums of all file
+chunks already stored in the repository. This cache is stored in
+``~/.cache/borg/``.  If |project_name| detects that a repository has been
+modified since the local cache was updated it will need to rebuild
+the cache. This rebuild can be quite time consuming.
+
+So, yes it's possible. But it will be most efficient if a single
+repository is only modified from one place. Also keep in mind that
+|project_name| will keep an exclusive lock on the repository while creating
+or deleting archives, which may make *simultaneous* backups fail.
 
 Which file types, attributes, etc. are preserved?
+-------------------------------------------------
+
     * Directories
     * Regular files
     * Hardlinks (considering all files in the same archive)
@@ -40,6 +46,8 @@ Which file types, attributes, etc. are preserved?
     * BSD flags on OS X and FreeBSD
 
 Which file types, attributes, etc. are *not* preserved?
+-------------------------------------------------------
+
     * UNIX domain sockets (because it does not make sense - they are
       meaningless without the running process that created them and the process
       needs to recreate them in any case). So, don't panic if your backup
@@ -50,91 +58,138 @@ Which file types, attributes, etc. are *not* preserved?
       Archive extraction has optional support to extract all-zero chunks as
       holes in a sparse file.
 
+Why is my backup bigger than with attic? Why doesn't |project_name| do compression by default?
+----------------------------------------------------------------------------------------------
+
+Attic was rather unflexible when it comes to compression, it always
+compressed using zlib level 6 (no way to switch compression off or
+adjust the level or algorithm).
+
+|project_name| offers a lot of different compression algorithms and
+levels. Which of them is the best for you pretty much depends on your
+use case, your data, your hardware - so you need to do an informed
+decision about whether you want to use compression, which algorithm
+and which level you want to use. This is why compression defaults to
+none.
+
 How can I specify the encryption passphrase programmatically?
-    The encryption passphrase can be specified programmatically using the
-    `BORG_PASSPHRASE` environment variable. This is convenient when setting up
-    automated encrypted backups. Another option is to use
-    key file based encryption with a blank passphrase. See
-    :ref:`encrypted_repos` for more details.
+-------------------------------------------------------------
+      
+The encryption passphrase can be specified programmatically using the
+`BORG_PASSPHRASE` environment variable. This is convenient when setting up
+automated encrypted backups. Another option is to use
+key file based encryption with a blank passphrase. See
+:ref:`encrypted_repos` for more details.
+
+.. _password_env:
+.. note:: Be careful how you set the environment; using the ``env``
+          command, a ``system()`` call or using inline shell scripts
+          might expose the credentials in the process list directly
+          and they will be readable to all users on a system. Using
+          ``export`` in a shell script file should be safe, however, as
+          the environment of a process is `accessible only to that
+          user
+          <http://security.stackexchange.com/questions/14000/environment-variable-accessibility-in-linux/14009#14009>`_.
 
 When backing up to remote encrypted repos, is encryption done locally?
-    Yes, file and directory metadata and data is locally encrypted, before
-    leaving the local machine. We do not mean the transport layer encryption
-    by that, but the data/metadata itself. Transport layer encryption (e.g.
-    when ssh is used as a transport) applies additionally.
+----------------------------------------------------------------------
+     
+Yes, file and directory metadata and data is locally encrypted, before
+leaving the local machine. We do not mean the transport layer encryption
+by that, but the data/metadata itself. Transport layer encryption (e.g.
+when ssh is used as a transport) applies additionally.
 
 When backing up to remote servers, do I have to trust the remote server?
-    Yes and No.
-    No, as far as data confidentiality is concerned - if you use encryption,
-    all your files/dirs data and metadata are stored in their encrypted form
-    into the repository.
-    Yes, as an attacker with access to the remote server could delete (or
-    otherwise make unavailable) all your backups.
+------------------------------------------------------------------------
+
+Yes and No.
+
+No, as far as data confidentiality is concerned - if you use encryption,
+all your files/dirs data and metadata are stored in their encrypted form
+into the repository.
+
+Yes, as an attacker with access to the remote server could delete (or
+otherwise make unavailable) all your backups.
 
 If a backup stops mid-way, does the already-backed-up data stay there?
-    Yes, |project_name| supports resuming backups.
-    During a backup a special checkpoint archive named ``<archive-name>.checkpoint``
-    is saved every checkpoint interval (the default value for this is 5
-    minutes) containing all the data backed-up until that point. This means
-    that at most <checkpoint interval> worth of data needs to be retransmitted
-    if a backup needs to be restarted.
-    Once your backup has finished successfully, you can delete all ``*.checkpoint``
-    archives.
+----------------------------------------------------------------------
+
+Yes, |project_name| supports resuming backups.
+
+During a backup a special checkpoint archive named ``<archive-name>.checkpoint``
+is saved every checkpoint interval (the default value for this is 5
+minutes) containing all the data backed-up until that point. This means
+that at most <checkpoint interval> worth of data needs to be retransmitted
+if a backup needs to be restarted.
+
+Once your backup has finished successfully, you can delete all ``*.checkpoint``
+archives.
 
 If it crashes with a UnicodeError, what can I do?
-    Check if your encoding is set correctly. For most POSIX-like systems, try::
+-------------------------------------------------
+
+Check if your encoding is set correctly. For most POSIX-like systems, try::
 
-        export LANG=en_US.UTF-8  # or similar, important is correct charset
+  export LANG=en_US.UTF-8  # or similar, important is correct charset
 
 I can't extract non-ascii filenames by giving them on the commandline!?
-    This might be due to different ways to represent some characters in unicode
-    or due to other non-ascii encoding issues.
-    If you run into that, try this:
+-----------------------------------------------------------------------
 
-    - avoid the non-ascii characters on the commandline by e.g. extracting
-      the parent directory (or even everything)
-    - mount the repo using FUSE and use some file manager
+This might be due to different ways to represent some characters in unicode
+or due to other non-ascii encoding issues.
+
+If you run into that, try this:
+
+- avoid the non-ascii characters on the commandline by e.g. extracting
+  the parent directory (or even everything)
+- mount the repo using FUSE and use some file manager
 
 Can |project_name| add redundancy to the backup data to deal with hardware malfunction?
-    No, it can't. While that at first sounds like a good idea to defend against
-    some defect HDD sectors or SSD flash blocks, dealing with this in a
-    reliable way needs a lot of low-level storage layout information and
-    control which we do not have (and also can't get, even if we wanted).
+---------------------------------------------------------------------------------------
+
+No, it can't. While that at first sounds like a good idea to defend against
+some defect HDD sectors or SSD flash blocks, dealing with this in a
+reliable way needs a lot of low-level storage layout information and
+control which we do not have (and also can't get, even if we wanted).
 
-    So, if you need that, consider RAID or a filesystem that offers redundant
-    storage or just make backups to different locations / different hardware.
+So, if you need that, consider RAID or a filesystem that offers redundant
+storage or just make backups to different locations / different hardware.
 
-    See also `ticket 225 <https://github.com/borgbackup/borg/issues/225>`_.
+See also `ticket 225 <https://github.com/borgbackup/borg/issues/225>`_.
 
 Can |project_name| verify data integrity of a backup archive?
-    Yes, if you want to detect accidental data damage (like bit rot), use the
-    ``check`` operation. It will notice corruption using CRCs and hashes.
-    If you want to be able to detect malicious tampering also, use a encrypted
-    repo. It will then be able to check using CRCs and HMACs.
+-------------------------------------------------------------
+
+Yes, if you want to detect accidental data damage (like bit rot), use the
+``check`` operation. It will notice corruption using CRCs and hashes.
+If you want to be able to detect malicious tampering also, use a encrypted
+repo. It will then be able to check using CRCs and HMACs.
 
 Why was Borg forked from Attic?
-    Borg was created in May 2015 in response to the difficulty of getting new
-    code or larger changes incorporated into Attic and establishing a bigger
-    developer community / more open development.
+-------------------------------
+
+Borg was created in May 2015 in response to the difficulty of getting new
+code or larger changes incorporated into Attic and establishing a bigger
+developer community / more open development.
 
-    More details can be found in `ticket 217
-    <https://github.com/jborg/attic/issues/217>`_ that led to the fork.
+More details can be found in `ticket 217
+<https://github.com/jborg/attic/issues/217>`_ that led to the fork.
 
-    Borg intends to be:
+Borg intends to be:
 
-    * simple:
+* simple:
 
-      * as simple as possible, but no simpler
-      * do the right thing by default, but offer options
-    * open:
+  * as simple as possible, but no simpler
+  * do the right thing by default, but offer options
+* open:
 
-      * welcome feature requests
-      * accept pull requests of good quality and coding style
-      * give feedback on PRs that can't be accepted "as is"
-      * discuss openly, don't work in the dark
-    * changing:
+  * welcome feature requests
+  * accept pull requests of good quality and coding style
+  * give feedback on PRs that can't be accepted "as is"
+  * discuss openly, don't work in the dark
+* changing:
 
-      * Borg is not compatible with Attic
-      * do not break compatibility accidentally, without a good reason
-        or without warning. allow compatibility breaking for other cases.
-      * if major version number changes, it may have incompatible changes
+  * Borg is not compatible with Attic
+  * do not break compatibility accidentally, without a good reason
+    or without warning. allow compatibility breaking for other cases.
+  * if major version number changes, it may have incompatible changes

docs/global.rst.inc

@@ -16,12 +16,12 @@
 .. _libattr: http://savannah.nongnu.org/projects/attr/
 .. _liblz4: https://github.com/Cyan4973/lz4
 .. _OpenSSL: https://www.openssl.org/
-.. _Python: http://www.python.org/
+.. _`Python 3`: http://www.python.org/
 .. _Buzhash: https://en.wikipedia.org/wiki/Buzhash
 .. _msgpack: http://msgpack.org/
 .. _`msgpack-python`: https://pypi.python.org/pypi/msgpack-python/
 .. _llfuse: https://pypi.python.org/pypi/llfuse/
-.. _homebrew: http://mxcl.github.io/homebrew/
+.. _homebrew: http://brew.sh/
 .. _userspace filesystems: https://en.wikipedia.org/wiki/Filesystem_in_Userspace
 .. _librelist: http://librelist.com/
 .. _Cython: http://cython.org/

docs/index.rst

@@ -4,10 +4,11 @@
 Borg Documentation
 ==================
 
+.. include:: ../README.rst
+
 .. toctree::
    :maxdepth: 2
 
-   intro
    installation
    quickstart
    usage
@@ -16,4 +17,4 @@ Borg Documentation
    changes
    internals
    development
-   api
+   authors

docs/installation.rst

@@ -4,153 +4,126 @@
 Installation
 ============
 
-|project_name| pyinstaller binary installation requires:
-
-* Linux: glibc >= 2.12 (ok for most supported Linux releases)
-* MacOS X: 10.10 (unknown whether it works for older releases)
-* FreeBSD: 10.2 (unknown whether it works for older releases)
-
-|project_name| non-binary installation requires:
-
-* Python_ >= 3.2.2
-* OpenSSL_ >= 1.0.0
-* libacl_ (that pulls in libattr_ also)
-* liblz4_
-* some python dependencies, see install_requires in setup.py
-
-General notes
--------------
-You need to do some platform specific preparation steps (to install libraries
-and tools) followed by the generic installation of |project_name| itself:
-
-Below, we describe different ways to install |project_name|.
-
-- **dist package** - easy and fast, needs a distribution and platform specific
-  binary package (for your Linux/*BSD/OS X/... distribution).
-- **pyinstaller binary** - easy and fast, we provide a ready-to-use binary file
-  that just works on the supported platforms
-- **pypi** - installing a source package from pypi needs more installation steps
-  and will need a compiler, development headers, etc..
+There are different ways to install |project_name|:
+
+- **distribution package** - easy and fast if a package is available for your
+  Linux/BSD distribution.
+- **PyInstaller binary** - easy and fast, we provide a ready-to-use binary file
+  that comes bundled with all dependencies.
+- **pip** - installing a source package with pip needs more installation steps
+  and requires all dependencies with development headers and a compiler.
 - **git** - for developers and power users who want to have the latest code or
   use revision control (each release is tagged).
 
-**Python 3**: Even though this is not the default Python version on many systems,
-it is usually available as an optional install.
 
-Virtualenv_ can be used to build and install |project_name| without affecting
-the system Python or requiring root access.
-
-Important:
-If you install into a virtual environment, you need to **activate**
-the virtual env first (``source borg-env/bin/activate``).
-Alternatively, directly run ``borg-env/bin/borg`` (or symlink that into some
-directory that is in your PATH so you can just run ``borg``).
-Using a virtual environment is optional, but recommended except for the most
-simple use cases.
-
-The llfuse_ python package is also required if you wish to mount an
-archive as a FUSE filesystem. Only FUSE >= 2.8.0 can support llfuse.
+Installation (Distribution Package)
+-----------------------------------
 
-You only need **Cython** to compile the .pyx files to the respective .c files
-when using |project_name| code from git. For |project_name| releases, the .c
-files will be bundled, so you won't need Cython to install a release.
+Some Linux and BSD distributions might offer a ready-to-use ``borgbackup``
+package which can be installed with the package manager.  As |project_name| is
+still a young project, such a package might be not available for your system
+yet. Please ask package maintainers to build a package or, if you can package /
+submit it yourself, please help us with that!
 
-Platform notes
---------------
-FreeBSD: You may need to get a recent enough OpenSSL version from FreeBSD ports.
+* On **Arch Linux**, there is a package available in the AUR_.
 
-Mac OS X: You may need to get a recent enough OpenSSL version from homebrew_.
+If a package is available, it might be interesting to check its version
+and compare that to our latest release and review the :doc:`changes`.
 
-Mac OS X: You need OS X FUSE >= 3.0.
+.. _AUR: https://aur.archlinux.org/packages/borgbackup/
 
 
-Installation (dist package)
----------------------------
-Some Linux, BSD and OS X distributions might offer a ready-to-use
-`borgbackup` package (which can be easily installed in the usual way).
-
-As |project_name| is still relatively new, such a package might be not
-available for your system yet. Please ask package maintainers to build a
-package or, if you can package / submit it yourself, please help us with
-that!
-
-If a package is available, it might be interesting for you to check its version
-and compare that to our latest release and review the change log (see links on
-our web site).
-
-
-Installation (pyinstaller binary)
+Installation (PyInstaller Binary)
 ---------------------------------
-For some platforms we offer a ready-to-use standalone borg binary.
 
-It is supposed to work without requiring installation or preparations.
+The |project_name| binary is available on the releases_ page for the following
+platforms:
 
-Check https://github.com/borgbackup/borg/issues/214 for available binaries.
+* **Linux**: glibc >= 2.13 (ok for most supported Linux releases)
+* **Mac OS X**: 10.10 (unknown whether it works for older releases)
+* **FreeBSD**: 10.2 (unknown whether it works for older releases)
 
+These binaries work without requiring specific installation steps. Just drop
+them into a directory in your ``PATH`` and then you can run ``borg``. If a new
+version is released, you will have to manually download it and replace the old
+version.
 
-Debian Jessie / Ubuntu 14.04 preparations (git/pypi)
-----------------------------------------------------
+.. _releases: https://github.com/borgbackup/borg/releases
 
-.. parsed-literal::
+Installing the Dependencies
+---------------------------
 
-    # Python 3.x (>= 3.2) + Headers, Py Package Installer, VirtualEnv
-    apt-get install python3 python3-dev python3-pip python-virtualenv
+To install |project_name| from a source package, you have to install the
+following dependencies first:
 
-    # we need OpenSSL + Headers for Crypto
-    apt-get install libssl-dev openssl
+* `Python 3`_ >= 3.2.2. Even though Python 3 is not the default Python version on
+  most systems, it is usually available as an optional install.
+* OpenSSL_ >= 1.0.0
+* libacl_ (that pulls in libattr_ also)
+* liblz4_
+* some Python dependencies, pip will automatically install them for you
+* optionally, the llfuse_ Python package is required if you wish to mount an
+  archive as a FUSE filesystem. FUSE >= 2.8.0 is required for llfuse.
 
-    # ACL support Headers + Library
-    apt-get install libacl1-dev libacl1
+In the following, the steps needed to install the dependencies are listed for a
+selection of platforms. If your distribution is not covered by these
+instructions, try to use your package manager to install the dependencies.  On
+FreeBSD, you may need to get a recent enough OpenSSL version from FreeBSD
+ports.
 
-    # lz4 super fast compression support Headers + Library
-    apt-get install liblz4-dev liblz4-1
+After you have installed the dependencies, you can proceed with steps outlined
+under :ref:`pip-installation`.
 
-    # if you do not have gcc / make / etc. yet
-    apt-get install build-essential
+Debian / Ubuntu
+~~~~~~~~~~~~~~~
 
-    # optional: FUSE support - to mount backup archives
-    # in case you get complaints about permission denied on /etc/fuse.conf:
-    # on ubuntu this means your user is not in the "fuse" group. just add
-    # yourself there, log out and log in again.
-    apt-get install libfuse-dev fuse pkg-config
+Install the dependencies with development headers::
 
-    # optional: for unit testing
-    apt-get install fakeroot
+    sudo apt-get install python3 python3-dev python3-pip python-virtualenv
+    sudo apt-get install libssl-dev openssl
+    sudo apt-get install libacl1-dev libacl1
+    sudo apt-get install liblz4-dev liblz4-1
+    sudo apt-get install build-essential
+    sudo apt-get install libfuse-dev fuse pkg-config    # optional, for FUSE support
 
+In case you get complaints about permission denied on ``/etc/fuse.conf``: on
+Ubuntu this means your user is not in the ``fuse`` group. Add yourself to that
+group, log out and log in again.
 
-Korora / Fedora 21 preparations (git/pypi)
-------------------------------------------
+Fedora / Korora
+~~~~~~~~~~~~~~~
 
-.. parsed-literal::
+Install the dependencies with development headers::
 
-    # Python 3.x (>= 3.2) + Headers, Py Package Installer, VirtualEnv
     sudo dnf install python3 python3-devel python3-pip python3-virtualenv
-
-    # we need OpenSSL + Headers for Crypto
     sudo dnf install openssl-devel openssl
-
-    # ACL support Headers + Library
     sudo dnf install libacl-devel libacl
-
-    # lz4 super fast compression support Headers + Library
     sudo dnf install lz4-devel
+    sudo dnf install fuse-devel fuse pkgconfig         # optional, for FUSE support
+
+
+Mac OS X
+~~~~~~~~
+
+Assuming you have installed homebrew_, the following steps will install all the
+dependencies::
 
-    # optional: FUSE support - to mount backup archives
-    sudo dnf install fuse-devel fuse pkgconfig
-    
-    # optional: for unit testing
-    sudo dnf install fakeroot
+    brew install python3 lz4 openssl
+    pip3 install virtualenv
 
+For FUSE support to mount the backup archives, you need at least version 3.0 of
+FUSE for OS X, which is available as a pre-release_.
 
-Cygwin preparations (git/pypi)
-------------------------------
+.. _pre-release: https://github.com/osxfuse/osxfuse/releases
 
-Please note that running under cygwin is rather experimental, stuff has been
-tested with CygWin (x86-64) v2.1.0.
+Cygwin
+~~~~~~
 
-You'll need at least (use the cygwin installer to fetch/install these):
+.. note::
+    Running under Cygwin is experimental and has only been tested with Cygwin
+    (x86-64) v2.1.0.
 
-::
+Use the Cygwin installer to install the dependencies::
 
     python3 python3-setuptools
     python3-cython  # not needed for releases
@@ -159,50 +132,55 @@ You'll need at least (use the cygwin installer to fetch/install these):
     liblz4_1 liblz4-devel  # from cygwinports.org
     git make openssh
 
-You can then install ``pip`` and ``virtualenv``:
-
-::
+You can then install ``pip`` and ``virtualenv``::
 
     easy_install-3.4 pip
     pip install virtualenv
 
-And now continue with the generic installation (see below).
+In case the creation of the virtual environment fails, try deleting this file::
 
-In case that creation of the virtual env fails, try deleting this file:
+    /usr/lib/python3.4/__pycache__/platform.cpython-34.pyc
 
-::
 
-    /usr/lib/python3.4/__pycache__/platform.cpython-34.pyc
+.. _pip-installation:
 
+Installation (pip)
+------------------
 
-Installation (pypi)
--------------------
+Virtualenv_ can be used to build and install |project_name| without affecting
+the system Python or requiring root access.  Using a virtual environment is
+optional, but recommended except for the most simple use cases.
 
-This uses the latest (source package) release from PyPi.
+.. note::
+    If you install into a virtual environment, you need to **activate** it
+    first (``source borg-env/bin/activate``), before running ``borg``.
+    Alternatively, symlink ``borg-env/bin/borg`` into some directory that is in
+    your ``PATH`` so you can just run ``borg``.
 
-.. parsed-literal::
+This will use ``pip`` to install the latest release from PyPi::
 
     virtualenv --python=python3 borg-env
-    source borg-env/bin/activate   # always before using!
+    source borg-env/bin/activate
 
-    # install borg + dependencies into virtualenv
+    # install Borg + Python dependencies into virtualenv
     pip install 'llfuse<0.41'  # optional, for FUSE support
                                # 0.41 and 0.41.1 have unicode issues at install time
     pip install borgbackup
 
-Note: we install into a virtual environment here, but this is not a requirement.
+To upgrade |project_name| to a new version later, run the following after
+activating your virtual environment::
+
+    pip install -U borgbackup
 
 
 Installation (git)
 ------------------
 
 This uses latest, unreleased development code from git.
-While we try not to break master, there are no guarantees on anything.
-
-.. parsed-literal::
+While we try not to break master, there are no guarantees on anything. ::
 
-    # get |project_name| from github, install it
-    git clone |git_url|
+    # get borg from github
+    git clone https://github.com/borgbackup/borg.git
 
     virtualenv --python=python3 borg-env
     source borg-env/bin/activate   # always before using!
@@ -216,6 +194,7 @@ While we try not to break master, there are no guarantees on anything.
     pip install -e .  # in-place editable mode
 
     # optional: run all the tests, on all supported Python versions
+    # requires fakeroot, available through your package manager
     fakeroot -u tox
 
-Note: as a developer or power user, you always want to use a virtual environment.
+.. note:: As a developer or power user, you always want to use a virtual environment.

docs/intro.rst

@@ -1,7 +0,0 @@
-.. include:: global.rst.inc
-.. _foreword:
-
-Introduction
-============
-
-.. include:: ../README.rst

docs/misc/asciinema/install_and_basics.json

@@ -0,0 +1,5550 @@
+{
+  "version": 1,
+  "width": 80,
+  "height": 24,
+  "duration": 332.0,
+  "command": "/bin/bash",
+  "title": "borgbackup - installation and basic usage",
+  "env": {
+    "TERM": "xterm",
+    "SHELL": "/bin/bash"
+  },
+  "stdout": [
+    [
+      0.083341,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      0.349103,
+      "#"
+    ],
+    [
+      0.338948,
+      " "
+    ],
+    [
+      0.185424,
+      "b"
+    ],
+    [
+      0.142971,
+      "o"
+    ],
+    [
+      0.091227,
+      "r"
+    ],
+    [
+      0.092867,
+      "g"
+    ],
+    [
+      0.222552,
+      "b"
+    ],
+    [
+      0.114706,
+      "a"
+    ],
+    [
+      0.125044,
+      "c"
+    ],
+    [
+      0.144755,
+      "k"
+    ],
+    [
+      0.241044,
+      "u"
+    ],
+    [
+      0.243681,
+      "p"
+    ],
+    [
+      0.265888,
+      " "
+    ],
+    [
+      0.345247,
+      "-"
+    ],
+    [
+      0.251918,
+      " "
+    ],
+    [
+      0.233420,
+      "i"
+    ],
+    [
+      0.078609,
+      "n"
+    ],
+    [
+      0.076809,
+      "s"
+    ],
+    [
+      0.070225,
+      "t"
+    ],
+    [
+      0.148413,
+      "a"
+    ],
+    [
+      0.077403,
+      "l"
+    ],
+    [
+      0.139884,
+      "l"
+    ],
+    [
+      0.084807,
+      "a"
+    ],
+    [
+      0.138823,
+      "t"
+    ],
+    [
+      0.068185,
+      "i"
+    ],
+    [
+      0.170422,
+      "o"
+    ],
+    [
+      0.161091,
+      "n"
+    ],
+    [
+      0.169247,
+      " "
+    ],
+    [
+      0.110722,
+      "a"
+    ],
+    [
+      0.113785,
+      "n"
+    ],
+    [
+      0.397895,
+      "d"
+    ],
+    [
+      0.305048,
+      " "
+    ],
+    [
+      0.211476,
+      "b"
+    ],
+    [
+      0.109865,
+      "a"
+    ],
+    [
+      0.230634,
+      "s"
+    ],
+    [
+      0.277915,
+      "i"
+    ],
+    [
+      0.206167,
+      "c"
+    ],
+    [
+      0.145265,
+      " "
+    ],
+    [
+      0.219619,
+      "u"
+    ],
+    [
+      0.139989,
+      "s"
+    ],
+    [
+      0.180240,
+      "a"
+    ],
+    [
+      0.200391,
+      "g"
+    ],
+    [
+      0.116961,
+      "e"
+    ],
+    [
+      0.172074,
+      "\r\n"
+    ],
+    [
+      0.000449,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      0.620909,
+      "#"
+    ],
+    [
+      0.217833,
+      " "
+    ],
+    [
+      0.592920,
+      "I"
+    ],
+    [
+      0.166726,
+      " "
+    ],
+    [
+      0.161953,
+      "h"
+    ],
+    [
+      0.072501,
+      "a"
+    ],
+    [
+      0.170951,
+      "v"
+    ],
+    [
+      0.154067,
+      "e"
+    ],
+    [
+      0.110535,
+      " "
+    ],
+    [
+      0.155235,
+      "a"
+    ],
+    [
+      0.130825,
+      "l"
+    ],
+    [
+      0.111834,
+      "r"
+    ],
+    [
+      0.142378,
+      "e"
+    ],
+    [
+      0.165867,
+      "a"
+    ],
+    [
+      0.062556,
+      "d"
+    ],
+    [
+      0.091778,
+      "y"
+    ],
+    [
+      0.216280,
+      " "
+    ],
+    [
+      0.169501,
+      "d"
+    ],
+    [
+      0.198240,
+      "o"
+    ],
+    [
+      0.092373,
+      "w"
+    ],
+    [
+      0.143405,
+      "n"
+    ],
+    [
+      0.207324,
+      "l"
+    ],
+    [
+      0.164248,
+      "o"
+    ],
+    [
+      0.088481,
+      "a"
+    ],
+    [
+      0.129191,
+      "d"
+    ],
+    [
+      0.179234,
+      "e"
+    ],
+    [
+      0.189248,
+      "d"
+    ],
+    [
+      0.145203,
+      " "
+    ],
+    [
+      0.221625,
+      "t"
+    ],
+    [
+      0.100064,
+      "h"
+    ],
+    [
+      0.133349,
+      "e"
+    ],
+    [
+      0.066501,
+      " "
+    ],
+    [
+      0.187004,
+      "f"
+    ],
+    [
+      0.142461,
+      "i"
+    ],
+    [
+      0.204723,
+      "l"
+    ],
+    [
+      0.068716,
+      "e"
+    ],
+    [
+      0.237576,
+      "s"
+    ],
+    [
+      0.128085,
+      ":"
+    ],
+    [
+      0.242282,
+      "\r\n"
+    ],
+    [
+      0.000327,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      1.796834,
+      "l"
+    ],
+    [
+      0.092545,
+      "s"
+    ],
+    [
+      0.210322,
+      " "
+    ],
+    [
+      0.189710,
+      "-"
+    ],
+    [
+      0.215532,
+      "l"
+    ],
+    [
+      0.852863,
+      "\r\n"
+    ],
+    [
+      0.002104,
+      "total 10620\r\n"
+    ],
+    [
+      0.000040,
+      "-rw-rw-r-- 1 tw tw 10869049 Oct 24 22:11 borg-linux64"
+    ],
+    [
+      0.000007,
+      "\r\n"
+    ],
+    [
+      0.000019,
+      "-rw-rw-r-- 1 tw tw      819 Oct 24 22:11 borg-linux64.asc\r\n"
+    ],
+    [
+      0.000431,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      0.513172,
+      "#"
+    ],
+    [
+      0.284059,
+      " "
+    ],
+    [
+      0.330931,
+      "b"
+    ],
+    [
+      0.118806,
+      "i"
+    ],
+    [
+      0.100553,
+      "n"
+    ],
+    [
+      0.259930,
+      "a"
+    ],
+    [
+      0.106715,
+      "r"
+    ],
+    [
+      0.276545,
+      "y"
+    ],
+    [
+      0.126132,
+      " "
+    ],
+    [
+      0.379724,
+      "+"
+    ],
+    [
+      0.199249,
+      " "
+    ],
+    [
+      0.295913,
+      "G"
+    ],
+    [
+      0.108970,
+      "P"
+    ],
+    [
+      0.080480,
+      "G"
+    ],
+    [
+      0.349293,
+      " "
+    ],
+    [
+      0.236785,
+      "s"
+    ],
+    [
+      0.105197,
+      "i"
+    ],
+    [
+      0.289951,
+      "g"
+    ],
+    [
+      0.351385,
+      "n"
+    ],
+    [
+      0.282003,
+      "a"
+    ],
+    [
+      0.206591,
+      "t"
+    ],
+    [
+      0.163963,
+      "u"
+    ],
+    [
+      0.082416,
+      "r"
+    ],
+    [
+      0.125432,
+      "e"
+    ],
+    [
+      0.369988,
+      "\r\n"
+    ],
+    [
+      0.000341,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      0.889617,
+      "#"
+    ],
+    [
+      0.226974,
+      " "
+    ],
+    [
+      0.218497,
+      "l"
+    ],
+    [
+      0.134545,
+      "e"
+    ],
+    [
+      0.103159,
+      "t"
+    ],
+    [
+      0.711682,
+      "'"
+    ],
+    [
+      0.185463,
+      "s"
+    ],
+    [
+      0.162130,
+      " "
+    ],
+    [
+      0.166049,
+      "v"
+    ],
+    [
+      0.183069,
+      "e"
+    ],
+    [
+      0.099764,
+      "r"
+    ],
+    [
+      0.234211,
+      "i"
+    ],
+    [
+      0.854328,
+      "f"
+    ],
+    [
+      0.203758,
+      "y"
+    ],
+    [
+      0.166681,
+      " "
+    ],
+    [
+      0.216715,
+      "t"
+    ],
+    [
+      0.560064,
+      "h"
+    ],
+    [
+      0.151837,
+      "a"
+    ],
+    [
+      0.194509,
+      "t"
+    ],
+    [
+      0.119665,
+      " "
+    ],
+    [
+      0.141089,
+      "t"
+    ],
+    [
+      0.096803,
+      "h"
+    ],
+    [
+      0.104718,
+      "e"
+    ],
+    [
+      0.106761,
+      " "
+    ],
+    [
+      0.229401,
+      "b"
+    ],
+    [
+      0.213802,
+      "i"
+    ],
+    [
+      0.075481,
+      "n"
+    ],
+    [
+      0.138720,
+      "a"
+    ],
+    [
+      0.062411,
+      "r"
+    ],
+    [
+      0.292719,
+      "y"
+    ],
+    [
+      0.482737,
+      " "
+    ],
+    [
+      0.211595,
+      "i"
+    ],
+    [
+      0.110964,
+      "s"
+    ],
+    [
+      0.102100,
+      " "
+    ],
+    [
+      0.143380,
+      "v"
+    ],
+    [
+      0.189214,
+      "a"
+    ],
+    [
+      0.099337,
+      "l"
+    ],
+    [
+      0.172757,
+      "i"
+    ],
+    [
+      0.082456,
+      "d"
+    ],
+    [
+      0.177514,
+      ":"
+    ],
+    [
+      0.622492,
+      "\r\n"
+    ],
+    [
+      0.000313,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      2.000000,
+      "g"
+    ],
+    [
+      0.261924,
+      "p"
+    ],
+    [
+      0.108570,
+      "g"
+    ],
+    [
+      0.247315,
+      " "
+    ],
+    [
+      0.277162,
+      "-"
+    ],
+    [
+      0.141397,
+      "-"
+    ],
+    [
+      0.143255,
+      "v"
+    ],
+    [
+      0.162858,
+      "e"
+    ],
+    [
+      0.040051,
+      "r"
+    ],
+    [
+      0.105941,
+      "i"
+    ],
+    [
+      0.144872,
+      "f"
+    ],
+    [
+      0.306497,
+      "y"
+    ],
+    [
+      0.468271,
+      " "
+    ],
+    [
+      2.000000,
+      "b"
+    ],
+    [
+      0.119390,
+      "o"
+    ],
+    [
+      0.463137,
+      "\u0007"
+    ],
+    [
+      0.000095,
+      "rg-linux64"
+    ],
+    [
+      0.341519,
+      "."
+    ],
+    [
+      0.146977,
+      "asc "
+    ],
+    [
+      0.186292,
+      "\r\n"
+    ],
+    [
+      0.100648,
+      "gpg: Signature made Wed 07 Oct 2015 02:41:38 PM CEST\r\n"
+    ],
+    [
+      0.000011,
+      "gpg:                using RSA key 243ACFA951F78E01\r\n"
+    ],
+    [
+      0.006906,
+      "gpg: Good signature from \"Thomas Waldmann \u003ctw@waldmann-edv.de\u003e\""
+    ],
+    [
+      0.000033,
+      "\r\ngpg:                 aka \"Thomas Waldmann \u003ctw-public@gmx.de\u003e\"\r\ngpg:                 aka \"Thomas Waldmann \u003ctwaldmann@thinkmo.de\u003e\"\r\n"
+    ],
+    [
+      0.000018,
+      "gpg:                 aka \"Thomas Waldmann \u003cthomas.j.waldmann@gmail.com\u003e\"\r\n"
+    ],
+    [
+      0.003077,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      2.000000,
+      "#"
+    ],
+    [
+      0.241501,
+      " "
+    ],
+    [
+      0.186571,
+      "e"
+    ],
+    [
+      0.214388,
+      "v"
+    ],
+    [
+      0.157101,
+      "e"
+    ],
+    [
+      0.042348,
+      "r"
+    ],
+    [
+      0.253261,
+      "y"
+    ],
+    [
+      0.254356,
+      "t"
+    ],
+    [
+      0.094622,
+      "h"
+    ],
+    [
+      0.213972,
+      "i"
+    ],
+    [
+      0.084853,
+      "n"
+    ],
+    [
+      0.084920,
+      "g"
+    ],
+    [
+      0.178519,
+      " "
+    ],
+    [
+      0.256151,
+      "o"
+    ],
+    [
+      0.217918,
+      "k"
+    ],
+    [
+      0.153899,
+      "!"
+    ],
+    [
+      0.246211,
+      "\r\n"
+    ],
+    [
+      0.000330,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      2.000000,
+      "#"
+    ],
+    [
+      0.288008,
+      " "
+    ],
+    [
+      0.232836,
+      "i"
+    ],
+    [
+      0.055326,
+      "n"
+    ],
+    [
+      0.142978,
+      "s"
+    ],
+    [
+      0.080599,
+      "t"
+    ],
+    [
+      0.139018,
+      "a"
+    ],
+    [
+      0.111052,
+      "l"
+    ],
+    [
+      0.132419,
+      "l"
+    ],
+    [
+      0.169037,
+      " "
+    ],
+    [
+      0.117036,
+      "t"
+    ],
+    [
+      0.092749,
+      "h"
+    ],
+    [
+      0.124768,
+      "e"
+    ],
+    [
+      0.088888,
+      " "
+    ],
+    [
+      0.184118,
+      "b"
+    ],
+    [
+      0.182336,
+      "i"
+    ],
+    [
+      0.075466,
+      "n"
+    ],
+    [
+      0.085516,
+      "a"
+    ],
+    [
+      0.060363,
+      "r"
+    ],
+    [
+      0.843225,
+      "y"
+    ],
+    [
+      0.209758,
+      " "
+    ],
+    [
+      0.168892,
+      "a"
+    ],
+    [
+      0.151126,
+      "s"
+    ],
+    [
+      0.127487,
+      " "
+    ],
+    [
+      0.300923,
+      "\""
+    ],
+    [
+      0.217060,
+      "b"
+    ],
+    [
+      0.221579,
+      "o"
+    ],
+    [
+      0.047775,
+      "r"
+    ],
+    [
+      0.107202,
+      "g"
+    ],
+    [
+      0.438939,
+      "\""
+    ],
+    [
+      0.253153,
+      ":"
+    ],
+    [
+      0.617823,
+      "\r\n"
+    ],
+    [
+      0.000326,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      0.816740,
+      "c"
+    ],
+    [
+      0.168734,
+      "p"
+    ],
+    [
+      0.230846,
+      " "
+    ],
+    [
+      0.299588,
+      "b"
+    ],
+    [
+      0.121082,
+      "o"
+    ],
+    [
+      0.214148,
+      "\u0007"
+    ],
+    [
+      0.000011,
+      "rg-linux64"
+    ],
+    [
+      0.331736,
+      " "
+    ],
+    [
+      0.812264,
+      "~"
+    ],
+    [
+      0.518926,
+      "/"
+    ],
+    [
+      0.233797,
+      "b"
+    ],
+    [
+      0.214141,
+      "i"
+    ],
+    [
+      0.098062,
+      "n"
+    ],
+    [
+      0.607725,
+      "/"
+    ],
+    [
+      0.566434,
+      "b"
+    ],
+    [
+      0.145886,
+      "o"
+    ],
+    [
+      0.113081,
+      "r"
+    ],
+    [
+      0.068870,
+      "g"
+    ],
+    [
+      0.851794,
+      "\r\n"
+    ],
+    [
+      0.012632,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      2.000000,
+      "#"
+    ],
+    [
+      0.269926,
+      " "
+    ],
+    [
+      0.208575,
+      "m"
+    ],
+    [
+      0.135192,
+      "a"
+    ],
+    [
+      0.119543,
+      "k"
+    ],
+    [
+      0.080873,
+      "e"
+    ],
+    [
+      0.156871,
+      " "
+    ],
+    [
+      0.197124,
+      "i"
+    ],
+    [
+      0.078784,
+      "t"
+    ],
+    [
+      0.142373,
+      " "
+    ],
+    [
+      0.189080,
+      "e"
+    ],
+    [
+      0.232597,
+      "x"
+    ],
+    [
+      0.170105,
+      "e"
+    ],
+    [
+      0.132039,
+      "c"
+    ],
+    [
+      0.230568,
+      "u"
+    ],
+    [
+      0.086573,
+      "t"
+    ],
+    [
+      0.255047,
+      "a"
+    ],
+    [
+      0.231478,
+      "b"
+    ],
+    [
+      0.283723,
+      "l"
+    ],
+    [
+      0.112987,
+      "e"
+    ],
+    [
+      0.518611,
+      ":"
+    ],
+    [
+      0.459423,
+      "\r\n"
+    ],
+    [
+      0.000822,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      0.353739,
+      "c"
+    ],
+    [
+      0.114161,
+      "h"
+    ],
+    [
+      0.268562,
+      "m"
+    ],
+    [
+      0.179085,
+      "o"
+    ],
+    [
+      0.145360,
+      "d"
+    ],
+    [
+      0.075599,
+      " "
+    ],
+    [
+      0.773964,
+      "+"
+    ],
+    [
+      0.113699,
+      "x"
+    ],
+    [
+      0.187579,
+      " "
+    ],
+    [
+      0.381391,
+      "~"
+    ],
+    [
+      0.512520,
+      "/"
+    ],
+    [
+      0.231090,
+      "b"
+    ],
+    [
+      0.197636,
+      "i"
+    ],
+    [
+      0.101238,
+      "n"
+    ],
+    [
+      0.341295,
+      "/"
+    ],
+    [
+      0.306047,
+      "b"
+    ],
+    [
+      0.106898,
+      "o"
+    ],
+    [
+      0.233773,
+      "rg "
+    ],
+    [
+      0.519336,
+      "\r\n"
+    ],
+    [
+      0.001408,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      2.000000,
+      "#"
+    ],
+    [
+      0.247104,
+      " "
+    ],
+    [
+      0.218717,
+      "i"
+    ],
+    [
+      0.067769,
+      "n"
+    ],
+    [
+      0.139583,
+      "s"
+    ],
+    [
+      0.092034,
+      "t"
+    ],
+    [
+      0.152729,
+      "a"
+    ],
+    [
+      0.083844,
+      "l"
+    ],
+    [
+      0.145806,
+      "l"
+    ],
+    [
+      0.120879,
+      "a"
+    ],
+    [
+      0.164967,
+      "t"
+    ],
+    [
+      0.065308,
+      "i"
+    ],
+    [
+      0.816983,
+      "o"
+    ],
+    [
+      0.231669,
+      "n"
+    ],
+    [
+      0.185168,
+      " "
+    ],
+    [
+      0.125214,
+      "d"
+    ],
+    [
+      0.112630,
+      "o"
+    ],
+    [
+      0.068650,
+      "n"
+    ],
+    [
+      0.108386,
+      "e"
+    ],
+    [
+      0.563031,
+      "!"
+    ],
+    [
+      2.000000,
+      "\r\n"
+    ],
+    [
+      0.000365,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      0.347093,
+      "#"
+    ],
+    [
+      0.262764,
+      " "
+    ],
+    [
+      0.191568,
+      "l"
+    ],
+    [
+      0.086614,
+      "e"
+    ],
+    [
+      0.110365,
+      "t"
+    ],
+    [
+      0.707057,
+      "'"
+    ],
+    [
+      0.220060,
+      "s"
+    ],
+    [
+      0.181690,
+      " "
+    ],
+    [
+      0.128039,
+      "c"
+    ],
+    [
+      0.176264,
+      "r"
+    ],
+    [
+      0.171208,
+      "e"
+    ],
+    [
+      0.199371,
+      "a"
+    ],
+    [
+      0.161622,
+      "t"
+    ],
+    [
+      0.145989,
+      "e"
+    ],
+    [
+      0.187920,
+      " "
+    ],
+    [
+      0.734653,
+      "a"
+    ],
+    [
+      0.185812,
+      " "
+    ],
+    [
+      0.270851,
+      "r"
+    ],
+    [
+      0.120000,
+      "e"
+    ],
+    [
+      0.161097,
+      "p"
+    ],
+    [
+      0.179813,
+      "o"
+    ],
+    [
+      0.170557,
+      "s"
+    ],
+    [
+      0.145457,
+      "i"
+    ],
+    [
+      0.165200,
+      "t"
+    ],
+    [
+      0.135578,
+      "o"
+    ],
+    [
+      0.130363,
+      "r"
+    ],
+    [
+      0.461631,
+      "y"
+    ],
+    [
+      0.303047,
+      ":"
+    ],
+    [
+      0.955198,
+      "\r\n"
+    ],
+    [
+      0.000300,
+      "\u001b]0;tw@tux: ~/borg/demo/download\u0007tw@tux:~/borg/demo/download$ "
+    ],
+    [
+      0.301237,
+      "c"
+    ],
+    [
+      0.084267,
+      "d"
+    ],
+    [
+      0.155241,
+      " "
+    ],
+    [
+      0.813751,
+      "."
+    ],
+    [
+      0.157147,
+      "."
+    ],
+    [
+      0.573720,
+      "\r\n"
+    ],
+    [
+      0.000508,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.225463,
+      "b"
+    ],
+    [
+      0.274841,
+      "o"
+    ],
+    [
+      0.125292,
+      "r"
+    ],
+    [
+      0.083313,
+      "g"
+    ],
+    [
+      0.088596,
+      " "
+    ],
+    [
+      0.231502,
+      "i"
+    ],
+    [
+      0.062726,
+      "n"
+    ],
+    [
+      0.144877,
+      "i"
+    ],
+    [
+      0.112508,
+      "t"
+    ],
+    [
+      0.313489,
+      " "
+    ],
+    [
+      0.298944,
+      "r"
+    ],
+    [
+      0.216556,
+      "e"
+    ],
+    [
+      0.180484,
+      "p"
+    ],
+    [
+      0.204141,
+      "o"
+    ],
+    [
+      0.682782,
+      "\r\n"
+    ],
+    [
+      0.352828,
+      "Initializing repository at \"repo\"\r\n"
+    ],
+    [
+      0.001407,
+      "Encryption NOT enabled.\r\nUse the \"--encryption=repokey|keyfile|passphrase\" to enable encryption."
+    ],
+    [
+      0.000009,
+      "\r\n"
+    ],
+    [
+      0.008492,
+      "Synchronizing chunks cache..."
+    ],
+    [
+      0.000009,
+      "\r\n"
+    ],
+    [
+      0.000030,
+      "Archives: 0, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 0."
+    ],
+    [
+      0.000004,
+      "\r\n"
+    ],
+    [
+      0.000024,
+      "Done."
+    ],
+    [
+      0.000004,
+      "\r\n"
+    ],
+    [
+      0.027827,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.988184,
+      "#"
+    ],
+    [
+      0.248844,
+      " "
+    ],
+    [
+      0.199486,
+      "l"
+    ],
+    [
+      0.104455,
+      "e"
+    ],
+    [
+      0.127960,
+      "t"
+    ],
+    [
+      0.484976,
+      "'"
+    ],
+    [
+      0.186103,
+      "s"
+    ],
+    [
+      0.151763,
+      " "
+    ],
+    [
+      0.177456,
+      "c"
+    ],
+    [
+      0.178972,
+      "r"
+    ],
+    [
+      0.183533,
+      "e"
+    ],
+    [
+      0.192725,
+      "a"
+    ],
+    [
+      0.146352,
+      "t"
+    ],
+    [
+      0.156199,
+      "e"
+    ],
+    [
+      0.232699,
+      " "
+    ],
+    [
+      0.513490,
+      "o"
+    ],
+    [
+      0.229828,
+      "u"
+    ],
+    [
+      0.104744,
+      "r"
+    ],
+    [
+      0.115068,
+      " "
+    ],
+    [
+      0.201439,
+      "f"
+    ],
+    [
+      0.333315,
+      "i"
+    ],
+    [
+      0.209070,
+      "r"
+    ],
+    [
+      0.259194,
+      "s"
+    ],
+    [
+      0.076346,
+      "t"
+    ],
+    [
+      0.125673,
+      " "
+    ],
+    [
+      0.198575,
+      "b"
+    ],
+    [
+      0.089009,
+      "a"
+    ],
+    [
+      0.238307,
+      "c"
+    ],
+    [
+      0.105568,
+      "k"
+    ],
+    [
+      0.254971,
+      "u"
+    ],
+    [
+      0.318094,
+      "p"
+    ],
+    [
+      0.690770,
+      ":"
+    ],
+    [
+      0.580155,
+      "\r\n"
+    ],
+    [
+      0.000308,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.603046,
+      "b"
+    ],
+    [
+      0.104492,
+      "o"
+    ],
+    [
+      0.148182,
+      "r"
+    ],
+    [
+      0.087024,
+      "g"
+    ],
+    [
+      0.176897,
+      " "
+    ],
+    [
+      0.183168,
+      "c"
+    ],
+    [
+      0.185325,
+      "r"
+    ],
+    [
+      0.183347,
+      "e"
+    ],
+    [
+      0.182868,
+      "a"
+    ],
+    [
+      0.170600,
+      "t"
+    ],
+    [
+      0.137005,
+      "e"
+    ],
+    [
+      0.164357,
+      " "
+    ],
+    [
+      0.427028,
+      "-"
+    ],
+    [
+      0.147791,
+      "-"
+    ],
+    [
+      0.440101,
+      "s"
+    ],
+    [
+      0.177193,
+      "t"
+    ],
+    [
+      0.203817,
+      "a"
+    ],
+    [
+      0.217150,
+      "t"
+    ],
+    [
+      0.229771,
+      "s"
+    ],
+    [
+      0.191220,
+      " "
+    ],
+    [
+      0.269939,
+      "-"
+    ],
+    [
+      0.145163,
+      "-"
+    ],
+    [
+      0.450053,
+      "p"
+    ],
+    [
+      0.165194,
+      "r"
+    ],
+    [
+      0.044264,
+      "o"
+    ],
+    [
+      0.204568,
+      "g"
+    ],
+    [
+      0.104759,
+      "r"
+    ],
+    [
+      0.213137,
+      "e"
+    ],
+    [
+      0.216596,
+      "s"
+    ],
+    [
+      0.163238,
+      "s"
+    ],
+    [
+      0.241084,
+      " "
+    ],
+    [
+      0.300727,
+      "-"
+    ],
+    [
+      0.149156,
+      "-"
+    ],
+    [
+      0.259608,
+      "c"
+    ],
+    [
+      0.120930,
+      "o"
+    ],
+    [
+      0.098838,
+      "m"
+    ],
+    [
+      0.234615,
+      "p"
+    ],
+    [
+      0.084600,
+      "r"
+    ],
+    [
+      0.166072,
+      "e"
+    ],
+    [
+      0.185576,
+      "s"
+    ],
+    [
+      0.159984,
+      "s"
+    ],
+    [
+      0.122793,
+      "i"
+    ],
+    [
+      0.180423,
+      "o"
+    ],
+    [
+      0.196311,
+      "n"
+    ],
+    [
+      0.181682,
+      " "
+    ],
+    [
+      0.242129,
+      "l"
+    ],
+    [
+      0.842020,
+      "z"
+    ],
+    [
+      0.707941,
+      "4"
+    ],
+    [
+      0.180354,
+      " "
+    ],
+    [
+      0.419080,
+      "r"
+    ],
+    [
+      0.189076,
+      "e"
+    ],
+    [
+      0.172527,
+      "p"
+    ],
+    [
+      0.154922,
+      "o"
+    ],
+    [
+      0.728059,
+      ":"
+    ],
+    [
+      0.147089,
+      ":"
+    ],
+    [
+      0.396117,
+      "b"
+    ],
+    [
+      0.090233,
+      "a"
+    ],
+    [
+      0.199537,
+      "c"
+    ],
+    [
+      0.084686,
+      "k"
+    ],
+    [
+      0.278049,
+      "u \r"
+    ],
+    [
+      0.268438,
+      "p"
+    ],
+    [
+      0.491592,
+      "1"
+    ],
+    [
+      0.508588,
+      " "
+    ],
+    [
+      0.174143,
+      "d"
+    ],
+    [
+      0.175430,
+      "a"
+    ],
+    [
+      0.166841,
+      "t"
+    ],
+    [
+      0.127029,
+      "a"
+    ],
+    [
+      0.380593,
+      "\r\n"
+    ],
+    [
+      0.557518,
+      "  2.68 MB O   1.25 MB C   1.25 MB D data/linux-4.1.8/Doc...ia/v4l/func-read.xml\r"
+    ],
+    [
+      0.200102,
+      "  5.37 MB O   2.46 MB C   2.46 MB D data/linux-4.1.8/Documentation/backlight   \r"
+    ],
+    [
+      0.200342,
+      "  6.99 MB O   3.36 MB C   3.36 MB D data/linux-4.1.8/Doc...rm_big_little_dt.txt\r"
+    ],
+    [
+      0.200137,
+      "  7.83 MB O   3.87 MB C   3.87 MB D data/linux-4.1.8/Doc...s/mtd/atmel-nand.txt\r"
+    ],
+    [
+      0.200271,
+      "  8.77 MB O   4.41 MB C   4.41 MB D data/linux-4.1.8/Doc...ngs/soc/fsl/qman.txt\r"
+    ],
+    [
+      0.200577,
+      "  9.99 MB O   5.12 MB C   5.12 MB D data/linux-4.1.8/Doc...ching/cachefiles.txt\r"
+    ],
+    [
+      0.200033,
+      " 12.11 MB O   6.34 MB C   6.33 MB D data/linux-4.1.8/Doc...infiniband/ipoib.txt\r"
+    ],
+    [
+      0.200272,
+      " 15.27 MB O   8.08 MB C   8.08 MB D data/linux-4.1.8/Doc.../networking/team.txt\r"
+    ],
+    [
+      0.200072,
+      " 18.22 MB O   9.72 MB C   9.71 MB D data/linux-4.1.8/Doc...tation/sysctl/vm.txt\r"
+    ],
+    [
+      0.202107,
+      " 21.05 MB O  11.22 MB C  11.21 MB D data/linux-4.1.8/MAINTAINERS               \r"
+    ],
+    [
+      0.200251,
+      " 23.04 MB O  12.20 MB C  12.20 MB D data/linux-4.1.8/arc...de/uapi/asm/unistd.h\r"
+    ],
+    [
+      0.200450,
+      " 25.45 MB O  13.17 MB C  13.17 MB D data/linux-4.1.8/arc.../boot/dts/imx23.dtsi\r"
+    ],
+    [
+      0.200093,
+      " 27.65 MB O  14.01 MB C  14.00 MB D data/linux-4.1.8/arc...omap3-overo-tobi.dts\r"
+    ],
+    [
+      0.200314,
+      " 30.26 MB O  14.89 MB C  14.89 MB D data/linux-4.1.8/arc...ot/dts/tps65910.dtsi\r"
+    ],
+    [
+      0.200003,
+      " 31.90 MB O  15.63 MB C  15.63 MB D data/linux-4.1.8/arc...include/asm/probes.h\r"
+    ],
+    [
+      0.200493,
+      " 34.66 MB O  16.84 MB C  16.83 MB D data/linux-4.1.8/arc...i/include/mach/psc.h\r"
+    ],
+    [
+      0.200675,
+      " 36.62 MB O  17.70 MB C  17.70 MB D data/linux-4.1.8/arc...mach-ixp4xx/common.c\r"
+    ],
+    [
+      0.200307,
+      " 38.40 MB O  18.54 MB C  18.53 MB D data/linux-4.1.8/arch/arm/mach-omap2/cm.h  \r"
+    ],
+    [
+      0.200254,
+      " 41.29 MB O  19.63 MB C  19.63 MB D data/linux-4.1.8/arch/arm/mach-pxa/idp.c   \r"
+    ],
+    [
+      0.219493,
+      " 43.57 MB O  20.67 MB C  20.66 MB D data/linux-4.1.8/arc...bile/clock-r8a7778.c\r"
+    ],
+    [
+      0.200451,
+      " 45.55 MB O  21.59 MB C  21.59 MB D data/linux-4.1.8/arc...m/plat-samsung/adc.c\r"
+    ],
+    [
+      0.200370,
+      " 47.50 MB O  22.51 MB C  22.51 MB D data/linux-4.1.8/arch/arm64/lib/memmove.S  \r"
+    ],
+    [
+      0.200686,
+      " 49.21 MB O  23.33 MB C  23.32 MB D data/linux-4.1.8/arc...ckfin/kernel/trace.c\r"
+    ],
+    [
+      0.200393,
+      " 53.22 MB O  24.51 MB C  24.50 MB D data/linux-4.1.8/arch/c6x/include/asm/soc.h\r"
+    ],
+    [
+      0.200371,
+      " 56.19 MB O  25.50 MB C  25.49 MB D data/linux-4.1.8/arc...op/iop_sw_cpu_defs.h\r"
+    ],
+    [
+      0.200450,
+      " 57.84 MB O  26.17 MB C  26.14 MB D data/linux-4.1.8/arc...include/asm/vm_mmu.h\r"
+    ],
+    [
+      0.200573,
+      " 60.21 MB O  27.27 MB C  27.25 MB D data/linux-4.1.8/arch/ia64/kernel/time.c   \r"
+    ],
+    [
+      0.200222,
+      " 62.31 MB O  28.18 MB C  28.15 MB D data/linux-4.1.8/arc.../coldfire/sltimers.c\r"
+    ],
+    [
+      0.200756,
+      " 67.09 MB O  29.98 MB C  29.90 MB D data/linux-4.1.8/arc...8k/include/asm/tlb.h\r"
+    ],
+    [
+      0.200716,
+      " 68.75 MB O  30.80 MB C  30.72 MB D data/linux-4.1.8/arc...ude/uapi/asm/fcntl.h\r"
+    ],
+    [
+      0.200734,
+      " 70.69 MB O  31.67 MB C  31.59 MB D data/linux-4.1.8/arc...figs/malta_defconfig\r"
+    ],
+    [
+      0.200198,
+      " 72.12 MB O  32.34 MB C  32.26 MB D data/linux-4.1.8/arc...de/asm/mc146818rtc.h\r"
+    ],
+    [
+      0.200446,
+      " 76.01 MB O  33.45 MB C  33.37 MB D data/linux-4.1.8/arch/mips/jazz/jazzdma.c  \r"
+    ],
+    [
+      0.200111,
+      " 78.19 MB O  34.46 MB C  34.38 MB D data/linux-4.1.8/arc...tlogic/common/time.c\r"
+    ],
+    [
+      0.200191,
+      " 79.84 MB O  35.30 MB C  35.21 MB D data/linux-4.1.8/arc...de/uapi/asm/msgbuf.h\r"
+    ],
+    [
+      0.200421,
+      " 81.35 MB O  36.07 MB C  35.99 MB D data/linux-4.1.8/arc...sc/include/asm/rtc.h\r"
+    ],
+    [
+      0.200090,
+      " 83.49 MB O  37.03 MB C  36.95 MB D data/linux-4.1.8/arc...fsl/qoriq-dma-1.dtsi\r"
+    ],
+    [
+      0.200331,
+      " 85.13 MB O  37.80 MB C  37.72 MB D data/linux-4.1.8/arc...pc836x_rdk_defconfig\r"
+    ],
+    [
+      0.200114,
+      " 87.04 MB O  38.71 MB C  38.63 MB D data/linux-4.1.8/arc...ude/uapi/asm/nvram.h\r"
+    ],
+    [
+      0.200280,
+      " 90.24 MB O  40.19 MB C  40.11 MB D data/linux-4.1.8/arc...pc/math-emu/mtfsfi.c\r"
+    ],
+    [
+      0.216796,
+      " 92.68 MB O  41.41 MB C  41.33 MB D data/linux-4.1.8/arc...rms/powermac/nvram.c\r"
+    ],
+    [
+      0.200198,
+      " 95.32 MB O  42.60 MB C  42.52 MB D data/linux-4.1.8/arc...nclude/asm/pgtable.h\r"
+    ],
+    [
+      0.200304,
+      " 97.31 MB O  43.50 MB C  43.42 MB D data/linux-4.1.8/arc...mach-dreamcast/irq.c\r"
+    ],
+    [
+      0.200328,
+      " 99.46 MB O  44.41 MB C  44.33 MB D data/linux-4.1.8/arc...artner-jet-setup.txt\r"
+    ],
+    [
+      0.200102,
+      "101.28 MB O  45.25 MB C  45.16 MB D data/linux-4.1.8/arc...rc/include/asm/ecc.h\r"
+    ],
+    [
+      0.200253,
+      "103.53 MB O  46.27 MB C  46.19 MB D data/linux-4.1.8/arc.../kernel/una_asm_64.S\r"
+    ],
+    [
+      0.200503,
+      "105.76 MB O  47.32 MB C  47.23 MB D data/linux-4.1.8/arch/tile/kernel/reboot.c \r"
+    ],
+    [
+      0.201177,
+      "107.64 MB O  48.27 MB C  48.18 MB D data/linux-4.1.8/arc...t/compressed/eboot.c\r"
+    ],
+    [
+      0.200192,
+      "109.82 MB O  49.22 MB C  49.13 MB D data/linux-4.1.8/arc...clude/asm/spinlock.h\r"
+    ],
+    [
+      0.200851,
+      "112.71 MB O  50.56 MB C  50.48 MB D data/linux-4.1.8/arch/x86/kernel/ptrace.c  \r"
+    ],
+    [
+      0.200195,
+      "115.71 MB O  51.96 MB C  51.87 MB D data/linux-4.1.8/arc...s/platform_emc1403.c\r"
+    ],
+    [
+      0.200306,
+      "117.28 MB O  52.79 MB C  52.70 MB D data/linux-4.1.8/arc...nclude/variant/tie.h\r"
+    ],
+    [
+      0.204475,
+      "122.68 MB O  55.35 MB C  55.26 MB D data/linux-4.1.8/fir...x-e1-6.2.9.0.fw.ihex\r"
+    ],
+    [
+      0.199974,
+      "127.39 MB O  58.15 MB C  57.97 MB D data/linux-4.1.8/fs/afs/fsclient.c         \r"
+    ],
+    [
+      0.201254,
+      "132.58 MB O  60.42 MB C  60.24 MB D data/linux-4.1.8/fs/cifs/cifssmb.c         \r"
+    ],
+    [
+      0.216710,
+      "136.76 MB O  62.28 MB C  62.10 MB D data/linux-4.1.8/fs/ext4/inline.c          \r"
+    ],
+    [
+      0.200891,
+      "140.78 MB O  64.15 MB C  63.97 MB D data/linux-4.1.8/fs/jbd2/commit.c          \r"
+    ],
+    [
+      0.199883,
+      "144.88 MB O  65.98 MB C  65.80 MB D data/linux-4.1.8/fs/nfs/objlayout          \r"
+    ],
+    [
+      0.201488,
+      "150.31 MB O  67.96 MB C  67.78 MB D data/linux-4.1.8/fs/...fy/dnotify/dnotify.c\r"
+    ],
+    [
+      0.205472,
+      "154.72 MB O  69.97 MB C  69.79 MB D data/linux-4.1.8/fs/quota/dquot.c          \r"
+    ],
+    [
+      0.200493,
+      "159.06 MB O  71.91 MB C  71.73 MB D data/linux-4.1.8/fs/...xfs/xfs_inode_fork.h\r"
+    ],
+    [
+      0.200000,
+      "161.54 MB O  73.09 MB C  72.91 MB D data/linux-4.1.8/inc.../crypto/public_key.h\r"
+    ],
+    [
+      0.205041,
+      "164.32 MB O  74.28 MB C  74.09 MB D data/linux-4.1.8/inc...inux/cgroup_subsys.h\r"
+    ],
+    [
+      0.200371,
+      "166.33 MB O  75.23 MB C  75.05 MB D data/linux-4.1.8/include/linux/if_team.h   \r"
+    ],
+    [
+      0.200340,
+      "168.82 MB O  76.24 MB C  76.06 MB D data/linux-4.1.8/inc.../mfd/pcf50633/gpio.h\r"
+    ],
+    [
+      0.200162,
+      "171.65 MB O  77.36 MB C  77.17 MB D data/linux-4.1.8/include/linux/phy.h       \r"
+    ],
+    [
+      0.200385,
+      "172.84 MB O  77.97 MB C  77.79 MB D data/linux-4.1.8/include/linux/scc.h       \r"
+    ],
+    [
+      0.200918,
+      "174.87 MB O  78.94 MB C  78.76 MB D data/linux-4.1.8/include/linux/wait.h      \r"
+    ],
+    [
+      0.200117,
+      "177.06 MB O  80.01 MB C  79.83 MB D data/linux-4.1.8/inc...er/nfnetlink_queue.h\r"
+    ],
+    [
+      0.200254,
+      "179.53 MB O  81.13 MB C  80.95 MB D data/linux-4.1.8/inc...e/events/intel-sst.h\r"
+    ],
+    [
+      0.200176,
+      "181.40 MB O  82.05 MB C  81.86 MB D data/linux-4.1.8/include/uapi/linux/mpls.h \r"
+    ],
+    [
+      0.200438,
+      "183.11 MB O  82.88 MB C  82.70 MB D data/linux-4.1.8/inc...api/scsi/fc/fc_els.h\r"
+    ],
+    [
+      0.200226,
+      "186.12 MB O  84.31 MB C  84.12 MB D data/linux-4.1.8/kernel/jump_label.c       \r"
+    ],
+    [
+      0.200138,
+      "190.76 MB O  86.46 MB C  86.28 MB D data/linux-4.1.8/lib/Kconfig.debug         \r"
+    ],
+    [
+      0.200958,
+      "194.21 MB O  87.82 MB C  87.64 MB D data/linux-4.1.8/mm/memblock.c             \r"
+    ],
+    [
+      0.200544,
+      "198.19 MB O  89.69 MB C  89.51 MB D data/linux-4.1.8/net/bluetooth/ecc.c       \r"
+    ],
+    [
+      0.200232,
+      "202.28 MB O  91.52 MB C  91.34 MB D data/linux-4.1.8/net/hsr/hsr_slave.c       \r"
+    ],
+    [
+      0.200153,
+      "206.23 MB O  93.40 MB C  93.22 MB D data/linux-4.1.8/net/ipx/af_ipx.c          \r"
+    ],
+    [
+      0.200526,
+      "210.30 MB O  95.08 MB C  94.89 MB D data/linux-4.1.8/net...ter/ipvs/ip_vs_ftp.c\r"
+    ],
+    [
+      0.200433,
+      "213.29 MB O  96.37 MB C  96.19 MB D data/linux-4.1.8/net/phonet/af_phonet.c    \r"
+    ],
+    [
+      0.200669,
+      "217.21 MB O  98.21 MB C  98.03 MB D data/linux-4.1.8/net.../svc_rdma_recvfrom.c\r"
+    ],
+    [
+      0.200014,
+      "220.20 MB O  99.53 MB C  99.35 MB D data/linux-4.1.8/scr...e/free/iounmap.cocci\r"
+    ],
+    [
+      0.200446,
+      "222.94 MB O 100.82 MB C 100.64 MB D data/linux-4.1.8/security/selinux/Makefile \r"
+    ],
+    [
+      0.214711,
+      "226.41 MB O 102.23 MB C 102.05 MB D data/linux-4.1.8/sou...seq/seq_midi_event.c\r"
+    ],
+    [
+      0.202631,
+      "228.96 MB O 103.31 MB C 103.13 MB D data/linux-4.1.8/sound/mips/ad1843.c       \r"
+    ],
+    [
+      0.200095,
+      "232.28 MB O 104.65 MB C 104.47 MB D data/linux-4.1.8/sound/pci/ctxfi/Makefile  \r"
+    ],
+    [
+      0.200726,
+      "236.33 MB O 106.24 MB C 106.06 MB D data/linux-4.1.8/sound/pci/nm256/Makefile  \r"
+    ],
+    [
+      0.199902,
+      "239.73 MB O 107.58 MB C 107.40 MB D data/linux-4.1.8/sou.../codecs/cs4271-spi.c\r"
+    ],
+    [
+      0.200592,
+      "244.29 MB O 109.08 MB C 108.90 MB D data/linux-4.1.8/sound/soc/codecs/wm8940.c \r"
+    ],
+    [
+      0.200357,
+      "247.98 MB O 110.35 MB C 110.17 MB D data/linux-4.1.8/sou...oc/omap/omap-mcpdm.c\r"
+    ],
+    [
+      0.200901,
+      "250.64 MB O 111.50 MB C 111.32 MB D data/linux-4.1.8/sound/usb/mixer_scarlett.c\r"
+    ],
+    [
+      0.200535,
+      "252.14 MB O 112.20 MB C 112.01 MB D data/linux-4.1.8/tools/perf/builtin-kvm.c  \r"
+    ],
+    [
+      0.200239,
+      "254.11 MB O 113.07 MB C 112.88 MB D data/linux-4.1.8/tools/perf/util/record.c  \r"
+    ],
+    [
+      0.200233,
+      "255.70 MB O 113.82 MB C 113.64 MB D data/linux-4.1.8/too...re/bin/configinit.sh\r"
+    ],
+    [
+      0.395702,
+      "                                                                               \r------------------------------------------------------------------------------\r\nArchive name: backup1\r\nArchive fingerprint: b3104802be9faa610f281619c69e4d3e672df2ce97528a35d83f15080d02ed86\r\nStart time: Sat Oct 24 22:27:24 2015\r\nEnd time: Sat Oct 24 22:27:43 2015\r\nDuration: 19.32 seconds\r\nNumber of files: 31557\r\n\r\n                       Original size      Compressed size    Deduplicated size\r\nThis archive:              257.06 MB            114.44 MB            114.26 MB\r\nAll archives:              257.06 MB            114.44 MB            114.26 MB\r\n\r\n                       Unique chunks         Total chunks\r\nChunk index:                   33731                34030\r\n------------------------------------------------------------------------------\r\n"
+    ],
+    [
+      0.046138,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      1.000000,
+      "#"
+    ],
+    [
+      0.564664,
+      " "
+    ],
+    [
+      0.313339,
+      "c"
+    ],
+    [
+      0.492152,
+      "h"
+    ],
+    [
+      0.479518,
+      "a"
+    ],
+    [
+      0.536708,
+      "n"
+    ],
+    [
+      0.134006,
+      "g"
+    ],
+    [
+      0.147326,
+      "e"
+    ],
+    [
+      0.068957,
+      " "
+    ],
+    [
+      0.179678,
+      "s"
+    ],
+    [
+      0.096249,
+      "o"
+    ],
+    [
+      0.081003,
+      "m"
+    ],
+    [
+      0.124342,
+      "e"
+    ],
+    [
+      0.117830,
+      " "
+    ],
+    [
+      0.138019,
+      "d"
+    ],
+    [
+      0.137898,
+      "a"
+    ],
+    [
+      0.199628,
+      "t"
+    ],
+    [
+      0.104935,
+      "a"
+    ],
+    [
+      0.150868,
+      " "
+    ],
+    [
+      0.144877,
+      "s"
+    ],
+    [
+      0.126816,
+      "l"
+    ],
+    [
+      0.178466,
+      "i"
+    ],
+    [
+      0.113395,
+      "g"
+    ],
+    [
+      0.101022,
+      "h"
+    ],
+    [
+      0.102395,
+      "t"
+    ],
+    [
+      0.311498,
+      "l"
+    ],
+    [
+      0.366608,
+      "y"
+    ],
+    [
+      0.657991,
+      ":"
+    ],
+    [
+      0.423140,
+      "\r\n"
+    ],
+    [
+      0.000708,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      2.000000,
+      "e"
+    ],
+    [
+      0.000021,
+      "c"
+    ],
+    [
+      0.000024,
+      "h"
+    ],
+    [
+      0.000029,
+      "o"
+    ],
+    [
+      0.000018,
+      " "
+    ],
+    [
+      0.000025,
+      "\""
+    ],
+    [
+      0.000025,
+      "s"
+    ],
+    [
+      0.000026,
+      "o"
+    ],
+    [
+      0.000070,
+      "me "
+    ],
+    [
+      0.000022,
+      "m"
+    ],
+    [
+      0.000028,
+      "o"
+    ],
+    [
+      0.000027,
+      "r"
+    ],
+    [
+      0.000026,
+      "e"
+    ],
+    [
+      0.000029,
+      " "
+    ],
+    [
+      0.000028,
+      "d"
+    ],
+    [
+      0.000028,
+      "a"
+    ],
+    [
+      0.000028,
+      "t"
+    ],
+    [
+      0.000026,
+      "a"
+    ],
+    [
+      0.000033,
+      "\""
+    ],
+    [
+      0.000028,
+      " "
+    ],
+    [
+      0.000059,
+      "\u003e"
+    ],
+    [
+      0.000045,
+      " "
+    ],
+    [
+      0.000020,
+      "d"
+    ],
+    [
+      0.000040,
+      "a"
+    ],
+    [
+      0.000035,
+      "t"
+    ],
+    [
+      0.000039,
+      "a"
+    ],
+    [
+      0.000034,
+      "/"
+    ],
+    [
+      0.000034,
+      "o"
+    ],
+    [
+      0.000034,
+      "n"
+    ],
+    [
+      0.000037,
+      "e"
+    ],
+    [
+      0.000036,
+      "_"
+    ],
+    [
+      0.000037,
+      "f"
+    ],
+    [
+      0.000037,
+      "i"
+    ],
+    [
+      0.000717,
+      "le_more\r\n"
+    ],
+    [
+      0.000181,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      2.000000,
+      "#"
+    ],
+    [
+      0.266289,
+      " "
+    ],
+    [
+      0.194686,
+      "n"
+    ],
+    [
+      0.157296,
+      "o"
+    ],
+    [
+      0.084026,
+      "w"
+    ],
+    [
+      0.092729,
+      " "
+    ],
+    [
+      0.148154,
+      "c"
+    ],
+    [
+      0.169136,
+      "r"
+    ],
+    [
+      0.214327,
+      "e"
+    ],
+    [
+      0.180678,
+      "a"
+    ],
+    [
+      0.161652,
+      "t"
+    ],
+    [
+      0.128260,
+      "e"
+    ],
+    [
+      0.158131,
+      " "
+    ],
+    [
+      0.118838,
+      "a"
+    ],
+    [
+      0.120885,
+      " "
+    ],
+    [
+      0.797511,
+      "s"
+    ],
+    [
+      0.200585,
+      "e"
+    ],
+    [
+      0.171811,
+      "c"
+    ],
+    [
+      0.106721,
+      "o"
+    ],
+    [
+      0.153298,
+      "n"
+    ],
+    [
+      0.052244,
+      "d"
+    ],
+    [
+      0.149675,
+      " "
+    ],
+    [
+      0.183517,
+      "b"
+    ],
+    [
+      0.076768,
+      "a"
+    ],
+    [
+      0.189428,
+      "c"
+    ],
+    [
+      0.088431,
+      "k"
+    ],
+    [
+      0.229617,
+      "u"
+    ],
+    [
+      0.272021,
+      "p"
+    ],
+    [
+      0.965855,
+      ":"
+    ],
+    [
+      0.674517,
+      "\r\n"
+    ],
+    [
+      0.000322,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.946131,
+      "b"
+    ],
+    [
+      0.111159,
+      "o"
+    ],
+    [
+      0.094622,
+      "r"
+    ],
+    [
+      0.085288,
+      "g"
+    ],
+    [
+      0.165429,
+      " "
+    ],
+    [
+      0.936087,
+      "c"
+    ],
+    [
+      0.192608,
+      "r"
+    ],
+    [
+      0.187511,
+      "e"
+    ],
+    [
+      0.173135,
+      "a"
+    ],
+    [
+      0.179441,
+      "t"
+    ],
+    [
+      0.125923,
+      "e"
+    ],
+    [
+      0.164920,
+      " "
+    ],
+    [
+      0.737259,
+      "-"
+    ],
+    [
+      0.185417,
+      "-"
+    ],
+    [
+      0.233405,
+      "s"
+    ],
+    [
+      0.152945,
+      "t"
+    ],
+    [
+      0.181548,
+      "a"
+    ],
+    [
+      0.330237,
+      "t"
+    ],
+    [
+      0.735524,
+      "s"
+    ],
+    [
+      0.179019,
+      " "
+    ],
+    [
+      0.245324,
+      "-"
+    ],
+    [
+      0.142362,
+      "-"
+    ],
+    [
+      0.233989,
+      "p"
+    ],
+    [
+      0.153782,
+      "r"
+    ],
+    [
+      0.064431,
+      "o"
+    ],
+    [
+      0.104827,
+      "g"
+    ],
+    [
+      0.090533,
+      "r"
+    ],
+    [
+      0.168129,
+      "e"
+    ],
+    [
+      0.206325,
+      "s"
+    ],
+    [
+      0.157551,
+      "s"
+    ],
+    [
+      0.383630,
+      " "
+    ],
+    [
+      0.759364,
+      "r"
+    ],
+    [
+      0.199262,
+      "e"
+    ],
+    [
+      0.139781,
+      "p"
+    ],
+    [
+      0.151367,
+      "o"
+    ],
+    [
+      0.720350,
+      ":"
+    ],
+    [
+      0.144801,
+      ":"
+    ],
+    [
+      0.532566,
+      "b"
+    ],
+    [
+      0.226514,
+      "a"
+    ],
+    [
+      0.209449,
+      "c"
+    ],
+    [
+      0.142062,
+      "k"
+    ],
+    [
+      0.300090,
+      "u"
+    ],
+    [
+      0.262794,
+      "p"
+    ],
+    [
+      0.218785,
+      "2"
+    ],
+    [
+      0.249599,
+      " "
+    ],
+    [
+      0.187125,
+      "d"
+    ],
+    [
+      0.157741,
+      "a"
+    ],
+    [
+      0.175739,
+      "t"
+    ],
+    [
+      0.139896,
+      "a"
+    ],
+    [
+      0.795560,
+      "\r\n"
+    ],
+    [
+      0.571808,
+      "  6.50 MB O   3.09 MB C       0 B D data/linux-4.1.8/Doc...ngs/arm/armadeus.txt\r"
+    ],
+    [
+      0.200103,
+      " 11.82 MB O   6.17 MB C       0 B D data/linux-4.1.8/Documentation/hwmon/w83795\r"
+    ],
+    [
+      0.200121,
+      " 27.38 MB O  13.89 MB C       0 B D data/linux-4.1.8/arc...ot/dts/nspire-cx.dts\r"
+    ],
+    [
+      0.200110,
+      " 39.92 MB O  19.04 MB C       0 B D data/linux-4.1.8/arc...omap2/opp2430_data.c\r"
+    ],
+    [
+      0.200088,
+      " 52.28 MB O  24.23 MB C       0 B D data/linux-4.1.8/arc...fin/mach-bf561/smp.c\r"
+    ],
+    [
+      0.200078,
+      " 67.02 MB O  29.94 MB C       0 B D data/linux-4.1.8/arc...8k/include/asm/pci.h\r"
+    ],
+    [
+      0.200116,
+      " 78.29 MB O  34.52 MB C       0 B D data/linux-4.1.8/arc...etlogic/xlr/wakeup.c\r"
+    ],
+    [
+      0.200081,
+      " 90.07 MB O  40.11 MB C       0 B D data/linux-4.1.8/arc...eature-fixups-test.S\r"
+    ],
+    [
+      0.200092,
+      "101.15 MB O  45.19 MB C       0 B D data/linux-4.1.8/arc...rc/crypto/sha1_asm.S\r"
+    ],
+    [
+      0.200078,
+      "115.05 MB O  51.63 MB C       0 B D data/linux-4.1.8/arc...6/mm/kasan_init_64.c\r"
+    ],
+    [
+      0.200062,
+      "147.39 MB O  66.98 MB C       0 B D data/linux-4.1.8/fs/nls/nls_cp864.c        \r"
+    ],
+    [
+      0.200117,
+      "169.16 MB O  76.38 MB C       0 B D data/linux-4.1.8/inc.../mfd/twl4030-audio.h\r"
+    ],
+    [
+      0.200074,
+      "181.43 MB O  82.06 MB C       0 B D data/linux-4.1.8/include/uapi/linux/mtio.h \r"
+    ],
+    [
+      0.200131,
+      "209.10 MB O  94.58 MB C       0 B D data/linux-4.1.8/net/mac80211/scan.c       \r"
+    ],
+    [
+      0.200079,
+      "234.87 MB O 105.68 MB C       0 B D data/linux-4.1.8/sou...i/hda/patch_si3054.c\r"
+    ],
+    [
+      0.200110,
+      "255.66 MB O 113.80 MB C       0 B D data/linux-4.1.8/too...ves/asm/asm-compat.h\r"
+    ],
+    [
+      0.201350,
+      "                                                                               \r------------------------------------------------------------------------------\r\nArchive name: backup2\r\nArchive fingerprint: 5737afe8ad2cda7667973b7f2e1d83f097ef3117b5753a38ba7664b616fbdc5a\r\nStart time: Sat Oct 24 22:28:24 2015\r\nEnd time: Sat Oct 24 22:28:27 2015\r\nDuration: 3.41 seconds\r\nNumber of files: 31557\r\n"
+    ],
+    [
+      0.001858,
+      "\r\n                       Original size      Compressed size    Deduplicated size\r\nThis archive:              257.06 MB            114.47 MB             45.19 kB\r\nAll archives:              514.12 MB            228.92 MB            114.31 MB\r\n\r\n                       Unique chunks         Total chunks\r\nChunk index:                   33733                68060\r\n------------------------------------------------------------------------------\r\n"
+    ],
+    [
+      0.033369,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      1.013315,
+      "#"
+    ],
+    [
+      0.482095,
+      " "
+    ],
+    [
+      0.319571,
+      "w"
+    ],
+    [
+      0.140740,
+      "o"
+    ],
+    [
+      0.090380,
+      "w"
+    ],
+    [
+      0.304400,
+      ","
+    ],
+    [
+      0.137310,
+      " "
+    ],
+    [
+      0.662280,
+      "t"
+    ],
+    [
+      0.162678,
+      "h"
+    ],
+    [
+      0.114083,
+      "a"
+    ],
+    [
+      0.077660,
+      "t"
+    ],
+    [
+      0.120839,
+      " "
+    ],
+    [
+      0.207626,
+      "w"
+    ],
+    [
+      0.195480,
+      "a"
+    ],
+    [
+      0.060188,
+      "s"
+    ],
+    [
+      0.149129,
+      " "
+    ],
+    [
+      0.094522,
+      "a"
+    ],
+    [
+      0.098801,
+      " "
+    ],
+    [
+      0.266235,
+      "l"
+    ],
+    [
+      0.184774,
+      "o"
+    ],
+    [
+      0.255040,
+      "t"
+    ],
+    [
+      0.170498,
+      " "
+    ],
+    [
+      0.201599,
+      "f"
+    ],
+    [
+      0.189774,
+      "a"
+    ],
+    [
+      0.229140,
+      "s"
+    ],
+    [
+      0.275243,
+      "t"
+    ],
+    [
+      0.177347,
+      "e"
+    ],
+    [
+      0.090806,
+      "r"
+    ],
+    [
+      0.204494,
+      "!"
+    ],
+    [
+      0.479851,
+      "\r\n"
+    ],
+    [
+      0.000316,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.734961,
+      "#"
+    ],
+    [
+      0.300000,
+      " "
+    ],
+    [
+      0.300000,
+      "n"
+    ],
+    [
+      0.199297,
+      "o"
+    ],
+    [
+      0.148047,
+      "t"
+    ],
+    [
+      0.071101,
+      "i"
+    ],
+    [
+      0.185554,
+      "c"
+    ],
+    [
+      0.395933,
+      "e"
+    ],
+    [
+      0.180285,
+      " "
+    ],
+    [
+      0.199321,
+      "t"
+    ],
+    [
+      0.094767,
+      "h"
+    ],
+    [
+      0.166966,
+      "a"
+    ],
+    [
+      0.102814,
+      "t"
+    ],
+    [
+      0.415016,
+      " "
+    ],
+    [
+      0.286089,
+      "\""
+    ],
+    [
+      0.795323,
+      "D"
+    ],
+    [
+      0.180152,
+      "e"
+    ],
+    [
+      0.311214,
+      "d"
+    ],
+    [
+      0.214812,
+      "u"
+    ],
+    [
+      0.251616,
+      "p"
+    ],
+    [
+      0.203533,
+      "l"
+    ],
+    [
+      0.187084,
+      "i"
+    ],
+    [
+      0.124066,
+      "c"
+    ],
+    [
+      0.158062,
+      "a"
+    ],
+    [
+      0.260540,
+      "t"
+    ],
+    [
+      0.136405,
+      "e"
+    ],
+    [
+      0.278039,
+      "d"
+    ],
+    [
+      0.323148,
+      " "
+    ],
+    [
+      0.172337,
+      "s"
+    ],
+    [
+      0.074541,
+      "i"
+    ],
+    [
+      0.269245,
+      "z"
+    ],
+    [
+      0.123599,
+      "e"
+    ],
+    [
+      0.533647,
+      "\""
+    ],
+    [
+      0.234738,
+      " "
+    ],
+    [
+      0.150720,
+      "f"
+    ],
+    [
+      0.144329,
+      "o"
+    ],
+    [
+      0.086533,
+      "r"
+    ],
+    [
+      0.159717,
+      " "
+    ],
+    [
+      0.274291,
+      "\""
+    ],
+    [
+      0.471163,
+      "T"
+    ],
+    [
+      0.162135,
+      "h"
+    ],
+    [
+      0.233501,
+      "i"
+    ],
+    [
+      0.134923,
+      "s"
+    ],
+    [
+      0.190779,
+      " "
+    ],
+    [
+      0.307322,
+      "a"
+    ],
+    [
+      0.153882,
+      "r"
+    ],
+    [
+      0.246471,
+      "c"
+    ],
+    [
+      0.110018,
+      "h"
+    ],
+    [
+      0.259798,
+      "i"
+    ],
+    [
+      0.132853,
+      "v"
+    ],
+    [
+      0.171373,
+      "e"
+    ],
+    [
+      0.560405,
+      "\""
+    ],
+    [
+      0.609162,
+      "!"
+    ],
+    [
+      0.559020,
+      "\r\n"
+    ],
+    [
+      0.000296,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.941369,
+      "#"
+    ],
+    [
+      0.254237,
+      " "
+    ],
+    [
+      0.176998,
+      "b"
+    ],
+    [
+      0.124943,
+      "o"
+    ],
+    [
+      0.097140,
+      "r"
+    ],
+    [
+      0.057513,
+      "g"
+    ],
+    [
+      0.232990,
+      " "
+    ],
+    [
+      0.549539,
+      "r"
+    ],
+    [
+      0.112992,
+      "e"
+    ],
+    [
+      0.240733,
+      "c"
+    ],
+    [
+      0.164146,
+      "o"
+    ],
+    [
+      0.209755,
+      "g"
+    ],
+    [
+      0.145638,
+      "n"
+    ],
+    [
+      0.151826,
+      "i"
+    ],
+    [
+      0.471625,
+      "z"
+    ],
+    [
+      0.759625,
+      "e"
+    ],
+    [
+      0.229566,
+      "d"
+    ],
+    [
+      0.254596,
+      " "
+    ],
+    [
+      0.209452,
+      "t"
+    ],
+    [
+      0.088606,
+      "h"
+    ],
+    [
+      0.155981,
+      "a"
+    ],
+    [
+      0.086797,
+      "t"
+    ],
+    [
+      0.098574,
+      " "
+    ],
+    [
+      0.243290,
+      "m"
+    ],
+    [
+      0.120288,
+      "o"
+    ],
+    [
+      0.092890,
+      "s"
+    ],
+    [
+      0.058823,
+      "t"
+    ],
+    [
+      0.125344,
+      " "
+    ],
+    [
+      0.211464,
+      "f"
+    ],
+    [
+      0.086483,
+      "i"
+    ],
+    [
+      0.213685,
+      "l"
+    ],
+    [
+      0.096764,
+      "e"
+    ],
+    [
+      0.176075,
+      "s"
+    ],
+    [
+      0.122962,
+      " "
+    ],
+    [
+      0.174342,
+      "d"
+    ],
+    [
+      0.103474,
+      "i"
+    ],
+    [
+      0.089744,
+      "d"
+    ],
+    [
+      0.181539,
+      " "
+    ],
+    [
+      0.461771,
+      "n"
+    ],
+    [
+      0.219395,
+      "o"
+    ],
+    [
+      0.095042,
+      "t"
+    ],
+    [
+      0.119662,
+      " "
+    ],
+    [
+      0.156060,
+      "c"
+    ],
+    [
+      0.116988,
+      "h"
+    ],
+    [
+      0.118775,
+      "a"
+    ],
+    [
+      0.126173,
+      "n"
+    ],
+    [
+      0.118518,
+      "g"
+    ],
+    [
+      0.109977,
+      "e"
+    ],
+    [
+      0.167095,
+      " "
+    ],
+    [
+      0.208137,
+      "a"
+    ],
+    [
+      0.155464,
+      "n"
+    ],
+    [
+      0.074939,
+      "d"
+    ],
+    [
+      0.616534,
+      "\r\n"
+    ],
+    [
+      0.000405,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.836535,
+      "#"
+    ],
+    [
+      0.248630,
+      " "
+    ],
+    [
+      0.211525,
+      "d"
+    ],
+    [
+      0.171252,
+      "e"
+    ],
+    [
+      0.244098,
+      "d"
+    ],
+    [
+      0.121718,
+      "u"
+    ],
+    [
+      0.219002,
+      "p"
+    ],
+    [
+      0.197839,
+      "l"
+    ],
+    [
+      0.161081,
+      "i"
+    ],
+    [
+      0.112763,
+      "c"
+    ],
+    [
+      0.154565,
+      "a"
+    ],
+    [
+      0.230427,
+      "t"
+    ],
+    [
+      0.180004,
+      "e"
+    ],
+    [
+      0.182279,
+      "d"
+    ],
+    [
+      0.201281,
+      " "
+    ],
+    [
+      0.202485,
+      "t"
+    ],
+    [
+      0.078397,
+      "h"
+    ],
+    [
+      0.178577,
+      "e"
+    ],
+    [
+      0.150264,
+      "m"
+    ],
+    [
+      0.482274,
+      "."
+    ],
+    [
+      0.300000,
+      "\r\n"
+    ],
+    [
+      0.000265,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.287124,
+      "#"
+    ],
+    [
+      0.314208,
+      " "
+    ],
+    [
+      0.219731,
+      "n"
+    ],
+    [
+      0.176210,
+      "o"
+    ],
+    [
+      0.108529,
+      "w"
+    ],
+    [
+      0.224056,
+      ","
+    ],
+    [
+      0.210976,
+      " "
+    ],
+    [
+      0.190508,
+      "l"
+    ],
+    [
+      0.098452,
+      "e"
+    ],
+    [
+      0.101431,
+      "t"
+    ],
+    [
+      0.855722,
+      "'"
+    ],
+    [
+      0.220403,
+      "s"
+    ],
+    [
+      0.229447,
+      " "
+    ],
+    [
+      0.134839,
+      "e"
+    ],
+    [
+      0.241915,
+      "x"
+    ],
+    [
+      0.217004,
+      "t"
+    ],
+    [
+      0.183774,
+      "r"
+    ],
+    [
+      0.231721,
+      "a"
+    ],
+    [
+      0.221361,
+      "c"
+    ],
+    [
+      0.436221,
+      "t"
+    ],
+    [
+      0.097256,
+      " "
+    ],
+    [
+      0.163933,
+      "a"
+    ],
+    [
+      0.099964,
+      " "
+    ],
+    [
+      0.216806,
+      "b"
+    ],
+    [
+      0.086493,
+      "a"
+    ],
+    [
+      0.211732,
+      "c"
+    ],
+    [
+      0.139016,
+      "k"
+    ],
+    [
+      0.379423,
+      "u"
+    ],
+    [
+      0.250049,
+      "p"
+    ],
+    [
+      0.717916,
+      ":"
+    ],
+    [
+      0.307136,
+      "\r\n"
+    ],
+    [
+      0.000301,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.967533,
+      "m"
+    ],
+    [
+      0.103707,
+      "v"
+    ],
+    [
+      0.407957,
+      " "
+    ],
+    [
+      0.177312,
+      "d"
+    ],
+    [
+      0.166158,
+      "a"
+    ],
+    [
+      0.242593,
+      "t"
+    ],
+    [
+      0.090471,
+      "a"
+    ],
+    [
+      0.699594,
+      " "
+    ],
+    [
+      0.273219,
+      "d"
+    ],
+    [
+      0.170371,
+      "a"
+    ],
+    [
+      0.169331,
+      "t"
+    ],
+    [
+      0.126739,
+      "a"
+    ],
+    [
+      0.288488,
+      "."
+    ],
+    [
+      0.305856,
+      "o"
+    ],
+    [
+      0.135252,
+      "r"
+    ],
+    [
+      0.152717,
+      "i"
+    ],
+    [
+      0.090343,
+      "g"
+    ],
+    [
+      0.312536,
+      "\r\n"
+    ],
+    [
+      0.002579,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.944876,
+      "b"
+    ],
+    [
+      0.149049,
+      "o"
+    ],
+    [
+      0.114834,
+      "r"
+    ],
+    [
+      0.074682,
+      "g"
+    ],
+    [
+      0.129000,
+      " "
+    ],
+    [
+      0.129618,
+      "e"
+    ],
+    [
+      0.261479,
+      "x"
+    ],
+    [
+      0.203937,
+      "t"
+    ],
+    [
+      0.196213,
+      "r"
+    ],
+    [
+      0.193561,
+      "a"
+    ],
+    [
+      0.215314,
+      "c"
+    ],
+    [
+      0.236817,
+      "t"
+    ],
+    [
+      0.188232,
+      " "
+    ],
+    [
+      0.177286,
+      "r"
+    ],
+    [
+      0.200598,
+      "e"
+    ],
+    [
+      0.105866,
+      "p"
+    ],
+    [
+      0.173864,
+      "o"
+    ],
+    [
+      0.388954,
+      ":"
+    ],
+    [
+      0.144865,
+      ":"
+    ],
+    [
+      0.347420,
+      "b"
+    ],
+    [
+      0.105814,
+      "a"
+    ],
+    [
+      0.198728,
+      "c"
+    ],
+    [
+      0.096349,
+      "k"
+    ],
+    [
+      0.261559,
+      "u"
+    ],
+    [
+      0.241998,
+      "p"
+    ],
+    [
+      0.240033,
+      "2"
+    ],
+    [
+      0.981903,
+      "\r\n"
+    ],
+    [
+      2.000000,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.937291,
+      "#"
+    ],
+    [
+      0.994897,
+      " "
+    ],
+    [
+      0.151752,
+      "c"
+    ],
+    [
+      0.096956,
+      "h"
+    ],
+    [
+      0.337975,
+      "e"
+    ],
+    [
+      0.207037,
+      "c"
+    ],
+    [
+      0.177028,
+      "k"
+    ],
+    [
+      0.740370,
+      " "
+    ],
+    [
+      0.330206,
+      "i"
+    ],
+    [
+      0.177976,
+      "f"
+    ],
+    [
+      0.218757,
+      " "
+    ],
+    [
+      0.329345,
+      "o"
+    ],
+    [
+      0.098735,
+      "r"
+    ],
+    [
+      0.098576,
+      "i"
+    ],
+    [
+      0.103157,
+      "g"
+    ],
+    [
+      0.107275,
+      "i"
+    ],
+    [
+      0.117332,
+      "n"
+    ],
+    [
+      0.194072,
+      "a"
+    ],
+    [
+      0.211456,
+      "l"
+    ],
+    [
+      0.197712,
+      " "
+    ],
+    [
+      0.189172,
+      "d"
+    ],
+    [
+      0.163930,
+      "a"
+    ],
+    [
+      0.188334,
+      "t"
+    ],
+    [
+      0.165129,
+      "a"
+    ],
+    [
+      0.220652,
+      " "
+    ],
+    [
+      0.224411,
+      "a"
+    ],
+    [
+      0.136137,
+      "n"
+    ],
+    [
+      0.155260,
+      "d"
+    ],
+    [
+      0.074238,
+      " "
+    ],
+    [
+      0.104154,
+      "r"
+    ],
+    [
+      0.690499,
+      "e"
+    ],
+    [
+      0.193678,
+      "s"
+    ],
+    [
+      0.165163,
+      "t"
+    ],
+    [
+      0.165594,
+      "o"
+    ],
+    [
+      0.111779,
+      "r"
+    ],
+    [
+      0.135625,
+      "e"
+    ],
+    [
+      0.202851,
+      "d"
+    ],
+    [
+      0.096040,
+      " "
+    ],
+    [
+      0.165090,
+      "d"
+    ],
+    [
+      0.155594,
+      "a"
+    ],
+    [
+      0.220606,
+      "t"
+    ],
+    [
+      0.163143,
+      "a"
+    ],
+    [
+      0.174099,
+      " "
+    ],
+    [
+      0.209780,
+      "d"
+    ],
+    [
+      0.166062,
+      "i"
+    ],
+    [
+      0.084688,
+      "f"
+    ],
+    [
+      0.140851,
+      "f"
+    ],
+    [
+      0.204458,
+      "e"
+    ],
+    [
+      0.088661,
+      "r"
+    ],
+    [
+      0.334162,
+      "s"
+    ],
+    [
+      0.904233,
+      ":"
+    ],
+    [
+      0.590489,
+      "\r\n"
+    ],
+    [
+      0.000283,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.503183,
+      "d"
+    ],
+    [
+      0.082614,
+      "i"
+    ],
+    [
+      0.216272,
+      "f"
+    ],
+    [
+      0.123813,
+      "f"
+    ],
+    [
+      0.183603,
+      " "
+    ],
+    [
+      0.302144,
+      "-"
+    ],
+    [
+      0.150946,
+      "r"
+    ],
+    [
+      0.152436,
+      " "
+    ],
+    [
+      2.000000,
+      "d"
+    ],
+    [
+      0.196047,
+      "a"
+    ],
+    [
+      0.206372,
+      "t"
+    ],
+    [
+      0.146051,
+      "a"
+    ],
+    [
+      0.326306,
+      "."
+    ],
+    [
+      0.363408,
+      "o"
+    ],
+    [
+      0.269988,
+      "rig/"
+    ],
+    [
+      0.776581,
+      " "
+    ],
+    [
+      0.137720,
+      "d"
+    ],
+    [
+      0.156080,
+      "a"
+    ],
+    [
+      0.242275,
+      "\u0007"
+    ],
+    [
+      0.000020,
+      "ta"
+    ],
+    [
+      0.872887,
+      "/"
+    ],
+    [
+      0.273993,
+      "\r\n"
+    ],
+    [
+      2.000000,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.488581,
+      "#"
+    ],
+    [
+      0.234021,
+      " "
+    ],
+    [
+      0.380938,
+      "n"
+    ],
+    [
+      0.240685,
+      "o"
+    ],
+    [
+      0.204436,
+      " "
+    ],
+    [
+      0.390794,
+      "o"
+    ],
+    [
+      0.225563,
+      "u"
+    ],
+    [
+      0.167295,
+      "t"
+    ],
+    [
+      0.140625,
+      "p"
+    ],
+    [
+      0.183668,
+      "u"
+    ],
+    [
+      0.106161,
+      "t"
+    ],
+    [
+      0.132063,
+      " "
+    ],
+    [
+      0.204757,
+      "m"
+    ],
+    [
+      0.082693,
+      "e"
+    ],
+    [
+      0.216428,
+      "a"
+    ],
+    [
+      0.121584,
+      "n"
+    ],
+    [
+      0.127398,
+      "s"
+    ],
+    [
+      0.264644,
+      " "
+    ],
+    [
+      0.201524,
+      "i"
+    ],
+    [
+      0.110738,
+      "t"
+    ],
+    [
+      0.120653,
+      " "
+    ],
+    [
+      0.311187,
+      "d"
+    ],
+    [
+      0.119826,
+      "o"
+    ],
+    [
+      0.082654,
+      "e"
+    ],
+    [
+      0.182518,
+      "s"
+    ],
+    [
+      0.096372,
+      " "
+    ],
+    [
+      0.192821,
+      "n"
+    ],
+    [
+      0.193829,
+      "o"
+    ],
+    [
+      0.065739,
+      "t"
+    ],
+    [
+      0.678808,
+      "."
+    ],
+    [
+      0.246797,
+      " "
+    ],
+    [
+      0.520369,
+      "f"
+    ],
+    [
+      0.058288,
+      "i"
+    ],
+    [
+      0.064783,
+      "n"
+    ],
+    [
+      0.104851,
+      "e"
+    ],
+    [
+      0.292910,
+      "."
+    ],
+    [
+      0.174086,
+      " "
+    ],
+    [
+      0.226556,
+      ":"
+    ],
+    [
+      0.249808,
+      ")"
+    ],
+    [
+      2.000000,
+      "\r\n"
+    ],
+    [
+      0.000261,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.477759,
+      "#"
+    ],
+    [
+      0.223699,
+      " "
+    ],
+    [
+      0.237979,
+      "l"
+    ],
+    [
+      0.152769,
+      "i"
+    ],
+    [
+      0.135150,
+      "s"
+    ],
+    [
+      0.068576,
+      "t"
+    ],
+    [
+      0.100516,
+      "i"
+    ],
+    [
+      0.078648,
+      "n"
+    ],
+    [
+      0.099435,
+      "g"
+    ],
+    [
+      0.157388,
+      " "
+    ],
+    [
+      0.115327,
+      "t"
+    ],
+    [
+      0.133738,
+      "h"
+    ],
+    [
+      0.135662,
+      "e"
+    ],
+    [
+      0.100677,
+      " "
+    ],
+    [
+      0.180392,
+      "r"
+    ],
+    [
+      0.190922,
+      "e"
+    ],
+    [
+      0.093920,
+      "p"
+    ],
+    [
+      0.173588,
+      "o"
+    ],
+    [
+      0.193023,
+      " "
+    ],
+    [
+      0.206907,
+      "c"
+    ],
+    [
+      0.106376,
+      "o"
+    ],
+    [
+      0.175291,
+      "n"
+    ],
+    [
+      0.080726,
+      "t"
+    ],
+    [
+      0.179258,
+      "e"
+    ],
+    [
+      0.101491,
+      "n"
+    ],
+    [
+      0.096807,
+      "t"
+    ],
+    [
+      0.211455,
+      "s"
+    ],
+    [
+      0.508210,
+      ":"
+    ],
+    [
+      0.373837,
+      "\r\n"
+    ],
+    [
+      0.000249,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.559782,
+      "b"
+    ],
+    [
+      0.116587,
+      "o"
+    ],
+    [
+      0.139513,
+      "r"
+    ],
+    [
+      0.072751,
+      "g"
+    ],
+    [
+      0.103968,
+      " "
+    ],
+    [
+      0.984928,
+      "l"
+    ],
+    [
+      0.173603,
+      "i"
+    ],
+    [
+      0.112444,
+      "s"
+    ],
+    [
+      0.066704,
+      "t"
+    ],
+    [
+      0.114771,
+      " "
+    ],
+    [
+      0.263745,
+      "r"
+    ],
+    [
+      0.113121,
+      "e"
+    ],
+    [
+      0.126283,
+      "p"
+    ],
+    [
+      0.187453,
+      "o"
+    ],
+    [
+      0.409044,
+      "\r\n"
+    ],
+    [
+      0.360675,
+      "backup1                              Sat Oct 24 22:27:43 2015"
+    ],
+    [
+      0.000011,
+      "\r\n"
+    ],
+    [
+      0.000006,
+      "backup2                              Sat Oct 24 22:28:27 2015"
+    ],
+    [
+      0.000005,
+      "\r\n"
+    ],
+    [
+      0.027766,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.637813,
+      "#"
+    ],
+    [
+      0.257629,
+      " "
+    ],
+    [
+      0.231710,
+      "l"
+    ],
+    [
+      0.213387,
+      "i"
+    ],
+    [
+      0.132149,
+      "s"
+    ],
+    [
+      0.244957,
+      "t"
+    ],
+    [
+      0.180264,
+      "i"
+    ],
+    [
+      0.082882,
+      "n"
+    ],
+    [
+      0.142810,
+      "g"
+    ],
+    [
+      0.134815,
+      " "
+    ],
+    [
+      0.167455,
+      "s"
+    ],
+    [
+      0.114155,
+      "o"
+    ],
+    [
+      0.106847,
+      "m"
+    ],
+    [
+      0.070629,
+      "e"
+    ],
+    [
+      0.507340,
+      " "
+    ],
+    [
+      0.234237,
+      "b"
+    ],
+    [
+      0.070181,
+      "a"
+    ],
+    [
+      0.220534,
+      "c"
+    ],
+    [
+      0.092316,
+      "k"
+    ],
+    [
+      0.257003,
+      "u"
+    ],
+    [
+      0.233598,
+      "p"
+    ],
+    [
+      0.201484,
+      " "
+    ],
+    [
+      0.124810,
+      "a"
+    ],
+    [
+      0.084732,
+      "r"
+    ],
+    [
+      0.249719,
+      "c"
+    ],
+    [
+      0.119605,
+      "h"
+    ],
+    [
+      0.203875,
+      "i"
+    ],
+    [
+      0.076269,
+      "v"
+    ],
+    [
+      0.174299,
+      "e"
+    ],
+    [
+      0.109711,
+      " "
+    ],
+    [
+      0.238294,
+      "c"
+    ],
+    [
+      0.102351,
+      "o"
+    ],
+    [
+      0.155761,
+      "n"
+    ],
+    [
+      0.060278,
+      "t"
+    ],
+    [
+      0.179564,
+      "e"
+    ],
+    [
+      0.112342,
+      "n"
+    ],
+    [
+      0.078100,
+      "t"
+    ],
+    [
+      0.190203,
+      "s"
+    ],
+    [
+      0.865560,
+      " "
+    ],
+    [
+      0.297799,
+      "("
+    ],
+    [
+      0.225741,
+      "s"
+    ],
+    [
+      0.080329,
+      "h"
+    ],
+    [
+      0.233668,
+      "o"
+    ],
+    [
+      0.127773,
+      "r"
+    ],
+    [
+      0.190065,
+      "t"
+    ],
+    [
+      0.187679,
+      "e"
+    ],
+    [
+      0.147219,
+      "n"
+    ],
+    [
+      0.064472,
+      "e"
+    ],
+    [
+      0.188512,
+      "d"
+    ],
+    [
+      0.459222,
+      ")"
+    ],
+    [
+      0.723165,
+      ":"
+    ],
+    [
+      0.645995,
+      "\r\n"
+    ],
+    [
+      0.000096,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.446688,
+      "b"
+    ],
+    [
+      0.145841,
+      "o"
+    ],
+    [
+      0.105605,
+      "r"
+    ],
+    [
+      0.088953,
+      "g"
+    ],
+    [
+      0.120803,
+      " "
+    ],
+    [
+      0.227780,
+      "l"
+    ],
+    [
+      0.175052,
+      "i"
+    ],
+    [
+      0.106579,
+      "s"
+    ],
+    [
+      0.058441,
+      "t"
+    ],
+    [
+      0.093196,
+      " "
+    ],
+    [
+      0.172940,
+      "r"
+    ],
+    [
+      0.134731,
+      "e"
+    ],
+    [
+      0.119062,
+      "p"
+    ],
+    [
+      0.183075,
+      "o"
+    ],
+    [
+      0.388321,
+      ":"
+    ],
+    [
+      0.140589,
+      ":"
+    ],
+    [
+      0.324109,
+      "b"
+    ],
+    [
+      0.058606,
+      "a"
+    ],
+    [
+      0.205450,
+      "c"
+    ],
+    [
+      0.105362,
+      "k"
+    ],
+    [
+      0.235009,
+      "u"
+    ],
+    [
+      0.243485,
+      "p"
+    ],
+    [
+      0.485432,
+      "2"
+    ],
+    [
+      0.148177,
+      " "
+    ],
+    [
+      0.632383,
+      "|"
+    ],
+    [
+      0.389914,
+      " "
+    ],
+    [
+      0.174128,
+      "t"
+    ],
+    [
+      0.201473,
+      "a"
+    ],
+    [
+      0.116517,
+      "i"
+    ],
+    [
+      0.225072,
+      "l"
+    ],
+    [
+      0.699624,
+      "\r\n"
+    ],
+    [
+      2.000000,
+      "-rw-rw-r-- tw     tw         5516 Jul 21 19:10 data/linux-4.1.8/virt/kvm/async_pf.c\r\n"
+    ],
+    [
+      0.000019,
+      "-rw-rw-r-- tw     tw         1120 Jul 21 19:10 data/linux-4.1.8/virt/kvm/async_pf.h\r\n-rw-rw-r-- tw     tw         4215 Jul 21 19:10 data/linux-4.1.8/virt/kvm/coalesced_mmio.c\r\n-rw-rw-r-- tw     tw          915 Jul 21 19:10 data/linux-4.1.8/virt/kvm/coalesced_mmio.h\r\n-rw-rw-r-- tw     tw        22879 Jul 21 19:10 data/linux-4.1.8/virt/kvm/eventfd.c\r\n-rw-rw-r-- tw     tw         5563 Jul 21 19:10 data/linux-4.1.8/virt/kvm/irqchip.c\r\n-rw-rw-r-- tw     tw        79385 Jul 21 19:10 data/linux-4.1.8/virt/kvm/kvm_main.c\r\n"
+    ],
+    [
+      0.000011,
+      "-rw-rw-r-- tw     tw         6132 Jul 21 19:10 data/linux-4.1.8/virt/kvm/vfio.c\r\n-rw-rw-r-- tw     tw          250 Jul 21 19:10 data/linux-4.1.8/virt/kvm/vfio.h\r\n"
+    ],
+    [
+      0.000009,
+      "-rw-rw-r-- tw     tw           15 Oct 24 22:28 data/one_file_more\r\n"
+    ],
+    [
+      0.000389,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      1.000000,
+      "#"
+    ],
+    [
+      0.337351,
+      " "
+    ],
+    [
+      0.147170,
+      "e"
+    ],
+    [
+      0.235736,
+      "a"
+    ],
+    [
+      0.251314,
+      "s"
+    ],
+    [
+      0.471185,
+      "y"
+    ],
+    [
+      0.277723,
+      ","
+    ],
+    [
+      0.204225,
+      " "
+    ],
+    [
+      0.182231,
+      "i"
+    ],
+    [
+      0.174424,
+      "s"
+    ],
+    [
+      0.074677,
+      "n"
+    ],
+    [
+      0.786274,
+      "'"
+    ],
+    [
+      0.264836,
+      "t"
+    ],
+    [
+      0.330352,
+      " "
+    ],
+    [
+      0.266876,
+      "i"
+    ],
+    [
+      0.112564,
+      "t"
+    ],
+    [
+      0.897299,
+      "?"
+    ],
+    [
+      0.623501,
+      " "
+    ],
+    [
+      0.656625,
+      "t"
+    ],
+    [
+      0.115934,
+      "h"
+    ],
+    [
+      0.625213,
+      "a"
+    ],
+    [
+      0.588409,
+      "t"
+    ],
+    [
+      0.160071,
+      " "
+    ],
+    [
+      0.830693,
+      "i"
+    ],
+    [
+      0.163118,
+      "s"
+    ],
+    [
+      0.075663,
+      " "
+    ],
+    [
+      0.186138,
+      "a"
+    ],
+    [
+      0.109916,
+      "l"
+    ],
+    [
+      0.137005,
+      "l"
+    ],
+    [
+      0.171009,
+      " "
+    ],
+    [
+      0.153348,
+      "y"
+    ],
+    [
+      0.132919,
+      "o"
+    ],
+    [
+      0.568100,
+      "u"
+    ],
+    [
+      0.211350,
+      " "
+    ],
+    [
+      0.195450,
+      "n"
+    ],
+    [
+      0.257974,
+      "e"
+    ],
+    [
+      0.185529,
+      "e"
+    ],
+    [
+      0.265130,
+      "d"
+    ],
+    [
+      0.129116,
+      " "
+    ],
+    [
+      0.169264,
+      "t"
+    ],
+    [
+      0.148964,
+      "o"
+    ],
+    [
+      0.437043,
+      " "
+    ],
+    [
+      0.431197,
+      "k"
+    ],
+    [
+      0.219557,
+      "n"
+    ],
+    [
+      0.257996,
+      "o"
+    ],
+    [
+      0.158826,
+      "w"
+    ],
+    [
+      0.406870,
+      " "
+    ],
+    [
+      0.659664,
+      "f"
+    ],
+    [
+      0.130963,
+      "o"
+    ],
+    [
+      0.125395,
+      "r"
+    ],
+    [
+      0.613713,
+      " "
+    ],
+    [
+      0.646957,
+      "b"
+    ],
+    [
+      0.154695,
+      "a"
+    ],
+    [
+      0.259741,
+      "s"
+    ],
+    [
+      0.156692,
+      "i"
+    ],
+    [
+      0.124345,
+      "c"
+    ],
+    [
+      0.513209,
+      "\r\n"
+    ],
+    [
+      0.000296,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.965828,
+      "#"
+    ],
+    [
+      0.232285,
+      " "
+    ],
+    [
+      0.266818,
+      "u"
+    ],
+    [
+      0.132723,
+      "s"
+    ],
+    [
+      0.216208,
+      "a"
+    ],
+    [
+      0.206362,
+      "g"
+    ],
+    [
+      0.142608,
+      "e"
+    ],
+    [
+      2.000000,
+      "."
+    ],
+    [
+      0.238868,
+      " "
+    ],
+    [
+      0.302986,
+      "i"
+    ],
+    [
+      0.196338,
+      "f"
+    ],
+    [
+      0.092936,
+      " "
+    ],
+    [
+      0.197594,
+      "y"
+    ],
+    [
+      0.122297,
+      "o"
+    ],
+    [
+      0.175360,
+      "u"
+    ],
+    [
+      0.145063,
+      " "
+    ],
+    [
+      0.313719,
+      "l"
+    ],
+    [
+      0.169678,
+      "i"
+    ],
+    [
+      0.185628,
+      "k"
+    ],
+    [
+      0.120660,
+      "e"
+    ],
+    [
+      0.078389,
+      " "
+    ],
+    [
+      0.648628,
+      "#"
+    ],
+    [
+      0.337514,
+      "b"
+    ],
+    [
+      0.108598,
+      "o"
+    ],
+    [
+      0.123792,
+      "r"
+    ],
+    [
+      0.136099,
+      "g"
+    ],
+    [
+      0.235539,
+      "b"
+    ],
+    [
+      0.091671,
+      "a"
+    ],
+    [
+      0.208697,
+      "c"
+    ],
+    [
+      0.100567,
+      "k"
+    ],
+    [
+      0.227477,
+      "u"
+    ],
+    [
+      0.236900,
+      "p"
+    ],
+    [
+      0.302154,
+      ","
+    ],
+    [
+      0.207291,
+      " "
+    ],
+    [
+      0.205656,
+      "s"
+    ],
+    [
+      0.123737,
+      "p"
+    ],
+    [
+      0.142016,
+      "r"
+    ],
+    [
+      0.197260,
+      "e"
+    ],
+    [
+      0.197471,
+      "a"
+    ],
+    [
+      0.104498,
+      "d"
+    ],
+    [
+      0.163267,
+      " "
+    ],
+    [
+      0.178420,
+      "t"
+    ],
+    [
+      0.091669,
+      "h"
+    ],
+    [
+      0.107735,
+      "e"
+    ],
+    [
+      0.102742,
+      " "
+    ],
+    [
+      0.211413,
+      "w"
+    ],
+    [
+      0.124959,
+      "o"
+    ],
+    [
+      0.105787,
+      "r"
+    ],
+    [
+      0.231403,
+      "d"
+    ],
+    [
+      0.299061,
+      "!"
+    ],
+    [
+      2.000000,
+      "\r\n"
+    ],
+    [
+      0.000307,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.304768,
+      "#"
+    ],
+    [
+      0.229433,
+      " "
+    ],
+    [
+      0.647220,
+      "t"
+    ],
+    [
+      0.070692,
+      "h"
+    ],
+    [
+      0.349432,
+      "a"
+    ],
+    [
+      0.112924,
+      "n"
+    ],
+    [
+      0.207031,
+      "k"
+    ],
+    [
+      0.567641,
+      "s"
+    ],
+    [
+      0.121708,
+      " "
+    ],
+    [
+      0.135723,
+      "f"
+    ],
+    [
+      0.139102,
+      "o"
+    ],
+    [
+      0.060453,
+      "r"
+    ],
+    [
+      0.152408,
+      " "
+    ],
+    [
+      0.116234,
+      "v"
+    ],
+    [
+      0.142885,
+      "i"
+    ],
+    [
+      0.106596,
+      "e"
+    ],
+    [
+      0.231115,
+      "w"
+    ],
+    [
+      0.416046,
+      "i"
+    ],
+    [
+      0.086563,
+      "n"
+    ],
+    [
+      0.144009,
+      "g"
+    ],
+    [
+      0.725139,
+      "!"
+    ],
+    [
+      0.299810,
+      "\r\n"
+    ],
+    [
+      0.000250,
+      "\u001b]0;tw@tux: ~/borg/demo\u0007tw@tux:~/borg/demo$ "
+    ],
+    [
+      0.710767,
+      "exit"
+    ],
+    [
+      0.000006,
+      "\r\n"
+    ]
+  ]
+}

docs/misc/asciinema/install_and_basics.txt

@@ -0,0 +1,51 @@
+# borgbackup - installation and basic usage
+
+# I have already downloaded the binary release from github:
+ls -l
+# binary file + GPG signature
+
+# verifying whether the binary is valid:
+gpg --verify borg-linux64.asc borg-linux64
+
+# install it as "borg":
+cp borg-linux64 ~/bin/borg
+
+# making it executable:
+chmod +x ~/bin/borg
+
+# yay, installation done! let's make backups!
+
+# creating a repository:
+borg init repo
+
+# creating our first backup with stuff from "data" directory:
+borg create --stats --progress --compression lz4 repo::backup1 data
+
+# changing the data slightly:
+echo "some more data" > data/one_file_more
+
+# creating another backup:
+borg create --stats --progress repo::backup2 data
+
+# that was much faster! it recognized/deduplicated unchanged files.
+# see the "Deduplicated size" column for "This archive"! :)
+
+# extracting a backup archive:
+mv data data.orig
+borg extract repo::backup2
+
+# checking if restored data differs from original data:
+diff -r data.orig data
+
+# no, it doesn't! :)
+
+# listing the repo contents:
+borg list repo
+
+# listing the backup2 archive contents (shortened):
+borg list repo::backup2 | tail
+
+# easy, isn't it?
+
+# if you like #borgbackup, spread the word!
+

docs/quickstart.rst

@@ -100,17 +100,17 @@ Backup compression
 Default is no compression, but we support different methods with high speed
 or high compression:
 
-If you have a quick repo storage and you want a little compression:
+If you have a quick repo storage and you want a little compression: ::
 
     $ borg create --compression lz4 /mnt/backup::repo ~
 
 If you have a medium fast repo storage and you want a bit more compression (N=0..9,
-0 means no compression, 9 means high compression):
+0 means no compression, 9 means high compression): ::
 
     $ borg create --compression zlib,N /mnt/backup::repo ~
 
 If you have a very slow repo storage and you want high compression (N=0..9, 0 means
-low compression, 9 means high compression):
+low compression, 9 means high compression): ::
 
     $ borg create --compression lzma,N /mnt/backup::repo ~
 
@@ -150,7 +150,11 @@ by providing the correct passphrase.
 For automated backups the passphrase can be specified using the
 `BORG_PASSPHRASE` environment variable.
 
-**The repository data is totally inaccessible without the key:**
+.. note:: Be careful about how you set that environment, see
+          :ref:`this note about password environments <password_env>`
+          for more information.
+
+.. important:: The repository data is totally inaccessible without the key:**
     Make a backup copy of the key file (``keyfile`` mode) or repo config
     file (``repokey`` mode) and keep it at a safe place, so you still have
     the key in case it gets corrupted or lost.

docs/usage.rst

@@ -22,19 +22,15 @@ Return codes
 
 ::
 
-    0      no error, normal termination
-    1      some error occurred (this can be a complete or a partial failure)
-    128+N  killed by signal N (e.g. 137 == kill -9)
+    0 = success (logged as INFO)
+    1 = warning (operation reached its normal end, but there were warnings -
+        you should check the log, logged as WARNING)
+    2 = error (like a fatal error, a local or remote exception, the operation
+        did not reach its normal end, logged as ERROR)
+    128+N = killed by signal N (e.g. 137 == kill -9)
 
+The return code is also logged at the indicated level as the last log entry.
 
-Note: we are aware that more distinct return codes might be useful, but it is
-not clear yet which return codes should be used for which precise conditions.
-
-See issue #61 for a discussion about that. Depending on the outcome of the
-discussion there, return codes may change in future (the only thing rather sure
-is that 0 will always mean some sort of success and "not 0" will always mean
-some sort of warning / error / failure - but the definition of success might
-change).
 
 Environment Variables
 ---------------------
@@ -60,6 +56,12 @@ Some "yes" sayers (if set, they automatically confirm that you really want to do
         For "Warning: The repository at location ... was previously located at ..."
     BORG_CHECK_I_KNOW_WHAT_I_AM_DOING
         For "Warning: 'check --repair' is an experimental feature that might result in data loss."
+    BORG_CYTHON_DISABLE
+        Disables the loading of Cython modules. This is currently
+        experimental and is used only to generate usage docs at build
+        time. It is unlikely to produce good results on a regular
+        run. The variable should be set to the name of the  calling class, and
+        should be unique across all of borg. It is currently only used by ``build_usage``.
 
 Directories:
     BORG_KEYS_DIR
@@ -128,6 +130,19 @@ Network:
 In case you are interested in more details, please read the internals documentation.
 
 
+Units
+-----
+
+To display quantities, |project_name| takes care of respecting the
+usual conventions of scale. Disk sizes are displayed in `decimal
+<https://en.wikipedia.org/wiki/Decimal>`_, using powers of ten (so
+``kB`` means 1000 bytes). For memory usage, `binary prefixes
+<https://en.wikipedia.org/wiki/Binary_prefix>`_ are used, and are
+indicated using the `IEC binary prefixes
+<https://en.wikipedia.org/wiki/IEC_80000-13#Prefixes_for_binary_multiples>`_,
+using powers of two (so ``KiB`` means 1024 bytes).
+
+
 .. include:: usage/init.rst.inc
 
 Examples
@@ -195,8 +210,9 @@ Examples
         --exclude '*.pyc'
 
     # Backup the root filesystem into an archive named "root-YYYY-MM-DD"
+    # use zlib compression (good, but slow) - default is no compression
     NAME="root-`date +%Y-%m-%d`"
-    $ borg create /mnt/backup::$NAME / --do-not-cross-mountpoints
+    $ borg create -C zlib,6 /mnt/backup::$NAME / --do-not-cross-mountpoints
 
     # Backup huge files with little chunk management overhead
     $ borg create --chunker-params 19,23,21,4095 /mnt/backup::VMs /srv/VMs
@@ -239,6 +255,8 @@ Note: currently, extract always writes into the current working directory ("."),
 
 .. include:: usage/check.rst.inc
 
+.. include:: usage/rename.rst.inc
+
 .. include:: usage/delete.rst.inc
 
 .. include:: usage/list.rst.inc
@@ -309,7 +327,7 @@ Examples
     Hostname: myhostname
     Username: root
     Time: Fri Aug  2 15:18:17 2013
-    Command line: /usr/bin/borg create --stats /mnt/backup::root-2013-08-02 / --do-not-cross-mountpoints
+    Command line: /usr/bin/borg create --stats -C zlib,6 /mnt/backup::root-2013-08-02 / --do-not-cross-mountpoints
     Number of files: 147429
     Original size: 5344169493 (4.98 GB)
     Compressed size: 1748189642 (1.63 GB)
@@ -362,65 +380,74 @@ Examples
     command="borg serve --restrict-to-path /mnt/backup" ssh-rsa AAAAB3[...]
 
 
+Miscellaneous Help
+------------------
+
+.. include:: usage/help.rst.inc
+
+
 Additional Notes
-================
+----------------
 
 Here are misc. notes about topics that are maybe not covered in enough detail in the usage section.
 
 --read-special
---------------
+~~~~~~~~~~~~~~
 
-The option --read-special is not intended for normal, filesystem-level (full or
+The option ``--read-special`` is not intended for normal, filesystem-level (full or
 partly-recursive) backups. You only give this option if you want to do something
-rather ... special - and if you have hand-picked some files that you want to treat
+rather ... special -- and if you have hand-picked some files that you want to treat
 that way.
 
-`borg create --read-special` will open all files without doing any special treatment
-according to the file type (the only exception here are directories: they will be
-recursed into). Just imagine what happens if you do `cat filename` - the content
-you will see there is what borg will backup for that filename.
+``borg create --read-special`` will open all files without doing any special
+treatment according to the file type (the only exception here are directories:
+they will be recursed into). Just imagine what happens if you do ``cat
+filename`` --- the content you will see there is what borg will backup for that
+filename.
 
 So, for example, symlinks will be followed, block device content will be read,
 named pipes / UNIX domain sockets will be read.
 
-You need to be careful with what you give as filename when using --read-special,
-e.g. if you give /dev/zero, your backup will never terminate.
+You need to be careful with what you give as filename when using ``--read-special``,
+e.g. if you give ``/dev/zero``, your backup will never terminate.
 
-The given files' metadata is saved as it would be saved without --read-special
-(e.g. its name, its size [might be 0], its mode, etc.) - but additionally, also
-the content read from it will be saved for it.
+The given files' metadata is saved as it would be saved without
+``--read-special`` (e.g. its name, its size [might be 0], its mode, etc.) - but
+additionally, also the content read from it will be saved for it.
 
-Restoring such files' content is currently only supported one at a time via --stdout
-option (and you have to redirect stdout to where ever it shall go, maybe directly
-into an existing device file of your choice or indirectly via dd).
+Restoring such files' content is currently only supported one at a time via
+``--stdout`` option (and you have to redirect stdout to where ever it shall go,
+maybe directly into an existing device file of your choice or indirectly via
+``dd``).
 
 Example
 ~~~~~~~
 
 Imagine you have made some snapshots of logical volumes (LVs) you want to backup.
 
-Note: For some scenarios, this is a good method to get "crash-like" consistency
-(I call it crash-like because it is the same as you would get if you just hit the
-reset button or your machine would abrubtly and completely crash).
-This is better than no consistency at all and a good method for some use cases,
-but likely not good enough if you have databases running.
+.. note::
+
+    For some scenarios, this is a good method to get "crash-like" consistency
+    (I call it crash-like because it is the same as you would get if you just
+    hit the reset button or your machine would abrubtly and completely crash).
+    This is better than no consistency at all and a good method for some use
+    cases, but likely not good enough if you have databases running.
 
 Then you create a backup archive of all these snapshots. The backup process will
 see a "frozen" state of the logical volumes, while the processes working in the
 original volumes continue changing the data stored there.
 
-You also add the output of `lvdisplay` to your backup, so you can see the LV sizes
-in case you ever need to recreate and restore them.
+You also add the output of ``lvdisplay`` to your backup, so you can see the LV
+sizes in case you ever need to recreate and restore them.
 
-After the backup has completed, you remove the snapshots again.
+After the backup has completed, you remove the snapshots again. ::
 
-::
     $ # create snapshots here
     $ lvdisplay > lvdisplay.txt
     $ borg create --read-special /mnt/backup::repo lvdisplay.txt /dev/vg0/*-snapshot
     $ # remove snapshots here
 
-Now, let's see how to restore some LVs from such a backup.
+Now, let's see how to restore some LVs from such a backup. ::
 
     $ borg extract /mnt/backup::repo lvdisplay.txt
     $ # create empty LVs with correct sizes here (look into lvdisplay.txt).

requirements.d/development.txt

@@ -2,4 +2,5 @@ tox
 mock
 pytest
 pytest-cov<2.0.0
+pytest-benchmark==3.0.0b1
 Cython

setup.cfg

@@ -2,7 +2,5 @@
 python_files = testsuite/*.py
 
 [flake8]
-ignore = E226,F403
-max-line-length = 250
-exclude = docs/conf.py,borg/_version.py,build,dist,.git,.idea,.cache
-max-complexity = 100
+max-line-length = 120
+exclude = build,dist,.git,.idea,.cache,.tox

setup.py

@@ -1,8 +1,15 @@
 # -*- encoding: utf-8 *-*
 import os
+import re
 import sys
 from glob import glob
 
+from distutils.command.build import build
+from distutils.core import Command
+from distutils.errors import DistutilsOptionError
+from distutils import log
+from setuptools.command.build_py import build_py
+
 min_python = (3, 2)
 my_python = sys.version_info
 
@@ -10,6 +17,9 @@ if my_python < min_python:
     print("Borg requires Python %d.%d or later" % min_python)
     sys.exit(1)
 
+# Are we building on ReadTheDocs?
+on_rtd = os.environ.get('READTHEDOCS')
+
 # msgpack pure python data corruption was fixed in 0.4.6.
 # Also, we might use some rather recent API features.
 install_requires=['msgpack-python>=0.4.6', ]
@@ -62,7 +72,7 @@ except ImportError:
     platform_freebsd_source = platform_freebsd_source.replace('.pyx', '.c')
     platform_darwin_source = platform_darwin_source.replace('.pyx', '.c')
     from distutils.command.build_ext import build_ext
-    if not all(os.path.exists(path) for path in [
+    if not on_rtd and not all(os.path.exists(path) for path in [
         compress_source, crypto_source, chunker_source, hashindex_source,
         platform_linux_source, platform_freebsd_source]):
         raise ImportError('The GIT version of Borg needs Cython. Install Cython or use a released version.')
@@ -101,31 +111,149 @@ library_dirs.append(os.path.join(ssl_prefix, 'lib'))
 
 possible_lz4_prefixes = ['/usr', '/usr/local', '/usr/local/opt/lz4', '/usr/local/lz4', '/usr/local/borg', '/opt/local']
 if os.environ.get('BORG_LZ4_PREFIX'):
-    possible_openssl_prefixes.insert(0, os.environ.get('BORG_LZ4_PREFIX'))
+    possible_lz4_prefixes.insert(0, os.environ.get('BORG_LZ4_PREFIX'))
 lz4_prefix = detect_lz4(possible_lz4_prefixes)
-if not lz4_prefix:
+if lz4_prefix:
+    include_dirs.append(os.path.join(lz4_prefix, 'include'))
+    library_dirs.append(os.path.join(lz4_prefix, 'lib'))
+elif not on_rtd:
     raise Exception('Unable to find LZ4 headers. (Looked here: {})'.format(', '.join(possible_lz4_prefixes)))
-include_dirs.append(os.path.join(lz4_prefix, 'include'))
-library_dirs.append(os.path.join(lz4_prefix, 'lib'))
 
 
 with open('README.rst', 'r') as fd:
     long_description = fd.read()
 
-cmdclass = {'build_ext': build_ext, 'sdist': Sdist}
+class build_usage(Command):
+    description = "generate usage for each command"
+
+    user_options = [
+        ('output=', 'O', 'output directory'),
+    ]
+    def initialize_options(self):
+        pass
+
+    def finalize_options(self):
+        pass
+
+    def run(self):
+        print('generating usage docs')
+        # allows us to build docs without the C modules fully loaded during help generation
+        if 'BORG_CYTHON_DISABLE' not in os.environ:
+            os.environ['BORG_CYTHON_DISABLE'] = self.__class__.__name__
+        from borg.archiver import Archiver
+        parser = Archiver().build_parser(prog='borg')
+        choices = {}
+        for action in parser._actions:
+            if action.choices is not None:
+                choices.update(action.choices)
+        print('found commands: %s' % list(choices.keys()))
+        if not os.path.exists('docs/usage'):
+            os.mkdir('docs/usage')
+        for command, parser in choices.items():
+            print('generating help for %s' % command)
+            with open('docs/usage/%s.rst.inc' % command, 'w') as doc:
+                if command == 'help':
+                    for topic in Archiver.helptext:
+                        params = {"topic": topic,
+                                  "underline": '~' * len('borg help ' + topic)}
+                        doc.write(".. _borg_{topic}:\n\n".format(**params))
+                        doc.write("borg help {topic}\n{underline}\n::\n\n".format(**params))
+                        doc.write(Archiver.helptext[topic])
+                else:
+                    params = {"command": command,
+                              "underline": '-' * len('borg ' + command)}
+                    doc.write(".. _borg_{command}:\n\n".format(**params))
+                    doc.write("borg {command}\n{underline}\n::\n\n".format(**params))
+                    epilog = parser.epilog
+                    parser.epilog = None
+                    doc.write(re.sub("^", "    ", parser.format_help(), flags=re.M))
+                    doc.write("\nDescription\n~~~~~~~~~~~\n")
+                    doc.write(epilog)
+        # return to regular Cython configuration, if we changed it
+        if os.environ.get('BORG_CYTHON_DISABLE') == self.__class__.__name__:
+            del os.environ['BORG_CYTHON_DISABLE']
+
+
+class build_api(Command):
+    description = "generate a basic api.rst file based on the modules available"
+
+    user_options = [
+        ('output=', 'O', 'output directory'),
+    ]
+    def initialize_options(self):
+        pass
+
+    def finalize_options(self):
+        pass
+
+    def run(self):
+        print("auto-generating API documentation")
+        with open("docs/api.rst", "w") as doc:
+            doc.write("""
+API Documentation
+=================
+""")
+            for mod in glob('borg/*.py') + glob('borg/*.pyx'):
+                print("examining module %s" % mod)
+                mod = mod.replace('.pyx', '').replace('.py', '').replace('/', '.')
+                if "._" not in mod:
+                    doc.write("""
+.. automodule:: %s
+    :members:
+    :undoc-members:
+""" % mod)
+
+# (function, predicate), see http://docs.python.org/2/distutils/apiref.html#distutils.cmd.Command.sub_commands
+# seems like this doesn't work on RTD, see below for build_py hack.
+build.sub_commands.append(('build_api', None))
+build.sub_commands.append(('build_usage', None))
+
+
+class build_py_custom(build_py):
+    """override build_py to also build our stuff
+
+    it is unclear why this is necessary, but in some environments
+    (Readthedocs.org, specifically), the above
+    ``build.sub_commands.append()`` doesn't seem to have an effect:
+    our custom build commands seem to be ignored when running
+    ``setup.py install``.
+
+    This class overrides the ``build_py`` target by forcing it to run
+    our custom steps as well.
+
+    See also the `bug report on RTD
+    <https://github.com/rtfd/readthedocs.org/issues/1740>`_.
+    """
+    def run(self):
+        super().run()
+        self.announce('calling custom build steps', level=log.INFO)
+        self.run_command('build_ext')
+        self.run_command('build_api')
+        self.run_command('build_usage')
+
+
+cmdclass = {
+    'build_ext': build_ext,
+    'build_api': build_api,
+    'build_usage': build_usage,
+    'build_py': build_py_custom,
+    'sdist': Sdist
+}
 
-ext_modules = [
+ext_modules = []
+if not on_rtd:
+    ext_modules += [
     Extension('borg.compress', [compress_source], libraries=['lz4'], include_dirs=include_dirs, library_dirs=library_dirs),
     Extension('borg.crypto', [crypto_source], libraries=['crypto'], include_dirs=include_dirs, library_dirs=library_dirs),
     Extension('borg.chunker', [chunker_source]),
     Extension('borg.hashindex', [hashindex_source])
 ]
-if sys.platform.startswith('linux'):
-    ext_modules.append(Extension('borg.platform_linux', [platform_linux_source], libraries=['acl']))
-elif sys.platform.startswith('freebsd'):
-    ext_modules.append(Extension('borg.platform_freebsd', [platform_freebsd_source]))
-elif sys.platform == 'darwin':
-    ext_modules.append(Extension('borg.platform_darwin', [platform_darwin_source]))
+    if sys.platform.startswith('linux'):
+        ext_modules.append(Extension('borg.platform_linux', [platform_linux_source], libraries=['acl']))
+    elif sys.platform.startswith('freebsd'):
+        ext_modules.append(Extension('borg.platform_freebsd', [platform_freebsd_source]))
+    elif sys.platform == 'darwin':
+        ext_modules.append(Extension('borg.platform_darwin', [platform_darwin_source]))
 
 setup(
     name='borgbackup',
@@ -134,7 +262,7 @@ setup(
     },
     author='The Borg Collective (see AUTHORS file)',
     author_email='borgbackup@librelist.com',
-    url='https://borgbackup.github.io/',
+    url='https://borgbackup.readthedocs.org/',
     description='Deduplicated, encrypted, authenticated and compressed backups',
     long_description=long_description,
     license='BSD',

tox.ini

@@ -11,6 +11,6 @@ changedir = {toxworkdir}
 deps =
      -rrequirements.d/development.txt
      attic
-commands = py.test --cov=borg --pyargs {posargs:borg.testsuite}
+commands = py.test --cov=borg --benchmark-skip --pyargs {posargs:borg.testsuite}
 # fakeroot -u needs some env vars:
 passenv = *