瀏覽代碼

Merge remote-tracking branch 'origin/master' into logging-refactor

Conflicts:
	borg/archive.py
	borg/archiver.py
	borg/cache.py
	borg/key.py
Antoine Beaupré 9 年之前
父節點
當前提交
bdbdbdde90
共有 27 個文件被更改,包括 937 次插入731 次删除
  1. 2 1
      .gitignore
  2. 6 2
      AUTHORS
  3. 0 475
      CHANGES.rst
  4. 1 0
      CHANGES.rst
  5. 24 14
      README.rst
  6. 3 2
      Vagrantfile
  7. 10 6
      borg/archive.py
  8. 41 36
      borg/archiver.py
  9. 4 2
      borg/cache.py
  10. 4 2
      borg/fuse.py
  11. 21 6
      borg/helpers.py
  12. 6 4
      borg/key.py
  13. 3 0
      borg/locking.py
  14. 29 15
      borg/remote.py
  15. 44 31
      borg/repository.py
  16. 19 1
      borg/testsuite/helpers.py
  17. 9 0
      borg/testsuite/repository.py
  18. 1 30
      docs/Makefile
  19. 2 2
      docs/_themes/local/sidebarusefullinks.html
  20. 11 0
      docs/authors.rst
  21. 510 3
      docs/changes.rst
  22. 40 44
      docs/development.rst
  23. 3 1
      docs/index.rst
  24. 3 35
      docs/installation.rst
  25. 0 7
      docs/intro.rst
  26. 8 0
      docs/usage.rst
  27. 133 12
      setup.py

+ 2 - 1
.gitignore

@@ -2,7 +2,7 @@ MANIFEST
 docs/_build
 build
 dist
-env
+borg-env
 .tox
 hashindex.c
 chunker.c
@@ -16,6 +16,7 @@ platform_linux.c
 *.pyo
 *.so
 docs/usage/*.inc
+docs/api.rst
 .idea/
 .cache/
 borg/_version.py

+ 6 - 2
AUTHORS

@@ -1,10 +1,14 @@
-Borg Developers / Contributors ("The Borg Collective")
-``````````````````````````````````````````````````````
+Borg Contributors ("The Borg Collective")
+=========================================
+
 - Thomas Waldmann <tw@waldmann-edv.de>
 - Antoine Beaupré
 - Radek Podgorny <radek@podgorny.cz>
 - Yuri D'Elia
 
+Attic authors
+-------------
+
 Borg is a fork of Attic. Attic is written and maintained
 by Jonas Borgström and various contributors:
 

+ 0 - 475
CHANGES.rst

@@ -1,475 +0,0 @@
-Borg Changelog
-==============
-
-Version 0.26.1
---------------
-
-This is a minor update, just docs and new pyinstaller binaries.
-
-- docs update about python and binary requirements
-- better docs for --read-special, fix #220
-- re-built the binaries, fix #218 and #213 (glibc version issue)
-- update web site about single-file pyinstaller binaries
-
-Note: if you did a python-based installation, there is no need to upgrade.
-
-
-Version 0.26.0
---------------
-
-New features:
-
-- Faster cache sync (do all in one pass, remove tar/compression stuff), #163
-- BORG_REPO env var to specify the default repo, #168
-- read special files as if they were regular files, #79
-- implement borg create --dry-run, attic issue #267
-- Normalize paths before pattern matching on OS X, #143
-- support OpenBSD and NetBSD (except xattrs/ACLs)
-- support / run tests on Python 3.5
-
-Bug fixes:
-
-- borg mount repo: use absolute path, attic #200, attic #137
-- chunker: use off_t to get 64bit on 32bit platform, #178
-- initialize chunker fd to -1, so it's not equal to STDIN_FILENO (0)
-- fix reaction to "no" answer at delete repo prompt, #182
-- setup.py: detect lz4.h header file location
-- to support python < 3.2.4, add less buggy argparse lib from 3.2.6 (#194)
-- fix for obtaining 'char *' from temporary Python value (old code causes
-  a compile error on Mint 17.2)
-- llfuse 0.41 install troubles on some platforms, require < 0.41
-  (UnicodeDecodeError exception due to non-ascii llfuse setup.py)
-- cython code: add some int types to get rid of unspecific python add /
-  subtract operations (avoid undefined symbol FPE_... error on some platforms)
-- fix verbose mode display of stdin backup
-- extract: warn if a include pattern never matched, fixes #209,
-  implement counters for Include/ExcludePatterns
-- archive names with slashes are invalid, attic issue #180
-- chunker: add a check whether the POSIX_FADV_DONTNEED constant is defined -
-  fixes building on OpenBSD.
-
-Other changes:
-
-- detect inconsistency / corruption / hash collision, #170
-- replace versioneer with setuptools_scm, #106
-- docs:
-
-  - pkg-config is needed for llfuse installation
-  - be more clear about pruning, attic issue #132
-- unit tests:
-
-  - xattr: ignore security.selinux attribute showing up
-  - ext3 seems to need a bit more space for a sparse file
-  - do not test lzma level 9 compression (avoid MemoryError)
-  - work around strange mtime granularity issue on netbsd, fixes #204
-  - ignore st_rdev if file is not a block/char device, fixes #203
-  - stay away from the setgid and sticky mode bits
-- use Vagrant to do easy cross-platform testing (#196), currently:
-
-  - Debian 7 "wheezy" 32bit, Debian 8 "jessie" 64bit
-  - Ubuntu 12.04 32bit, Ubuntu 14.04 64bit
-  - Centos 7 64bit
-  - FreeBSD 10.2 64bit
-  - OpenBSD 5.7 64bit
-  - NetBSD 6.1.5 64bit
-  - Darwin (OS X Yosemite)
-
-
-Version 0.25.0
---------------
-
-Compatibility notes:
-
-- lz4 compression library (liblz4) is a new requirement (#156)
-- the new compression code is very compatible: as long as you stay with zlib
-  compression, older borg releases will still be able to read data from a
-  repo/archive made with the new code (note: this is not the case for the
-  default "none" compression, use "zlib,0" if you want a "no compression" mode
-  that can be read by older borg). Also the new code is able to read repos and
-  archives made with older borg versions (for all zlib levels  0..9).
-
-Deprecations:
-
-- --compression N (with N being a number, as in 0.24) is deprecated.
-  We keep the --compression 0..9 for now to not break scripts, but it is
-  deprecated and will be removed later, so better fix your scripts now:
-  --compression 0 (as in 0.24) is the same as --compression zlib,0 (now).
-  BUT: if you do not want compression, you rather want --compression none
-  (which is the default).
-  --compression 1 (in 0.24) is the same as --compression zlib,1 (now)
-  --compression 9 (in 0.24) is the same as --compression zlib,9 (now)
-
-New features:
-
-- create --compression none (default, means: do not compress, just pass through
-  data "as is". this is more efficient than zlib level 0 as used in borg 0.24)
-- create --compression lz4 (super-fast, but not very high compression)
-- create --compression zlib,N (slower, higher compression, default for N is 6)
-- create --compression lzma,N (slowest, highest compression, default N is 6)
-- honor the nodump flag (UF_NODUMP) and do not backup such items
-- list --short just outputs a simple list of the files/directories in an archive
-
-Bug fixes:
-
-- fixed --chunker-params parameter order confusion / malfunction, fixes #154
-- close fds of segments we delete (during compaction)
-- close files which fell out the lrucache
-- fadvise DONTNEED now is only called for the byte range actually read, not for
-  the whole file, fixes #158.
-- fix issue with negative "all archives" size, fixes #165
-- restore_xattrs: ignore if setxattr fails with EACCES, fixes #162
-
-Other changes:
-
-- remove fakeroot requirement for tests, tests run faster without fakeroot
-  (test setup does not fail any more without fakeroot, so you can run with or
-  without fakeroot), fixes #151 and #91.
-- more tests for archiver
-- recover_segment(): don't assume we have an fd for segment
-- lrucache refactoring / cleanup, add dispose function, py.test tests
-- generalize hashindex code for any key length (less hardcoding)
-- lock roster: catch file not found in remove() method and ignore it
-- travis CI: use requirements file
-- improved docs:
-
-  - replace hack for llfuse with proper solution (install libfuse-dev)
-  - update docs about compression
-  - update development docs about fakeroot
-  - internals: add some words about lock files / locking system
-  - support: mention BountySource and for what it can be used
-  - theme: use a lighter green
-  - add pypi, wheel, dist package based install docs
-  - split install docs into system-specific preparations and generic instructions
-
-
-Version 0.24.0
---------------
-
-Incompatible changes (compared to 0.23):
-
-- borg now always issues --umask NNN option when invoking another borg via ssh
-  on the repository server. By that, it's making sure it uses the same umask
-  for remote repos as for local ones. Because of this, you must upgrade both
-  server and client(s) to 0.24.
-- the default umask is 077 now (if you do not specify via --umask) which might
-  be a different one as you used previously. The default umask avoids that
-  you accidentally give access permissions for group and/or others to files
-  created by borg (e.g. the repository).
-
-Deprecations:
-
-- "--encryption passphrase" mode is deprecated, see #85 and #97.
-  See the new "--encryption repokey" mode for a replacement.
-
-New features:
-
-- borg create --chunker-params ... to configure the chunker, fixes #16
-  (attic #302, attic #300, and somehow also #41).
-  This can be used to reduce memory usage caused by chunk management overhead,
-  so borg does not create a huge chunks index/repo index and eats all your RAM
-  if you back up lots of data in huge files (like VM disk images).
-  See docs/misc/create_chunker-params.txt for more information.
-- borg info now reports chunk counts in the chunk index.
-- borg create --compression 0..9 to select zlib compression level, fixes #66
-  (attic #295).
-- borg init --encryption repokey (to store the encryption key into the repo),
-  fixes #85
-- improve at-end error logging, always log exceptions and set exit_code=1
-- LoggedIO: better error checks / exceptions / exception handling
-- implement --remote-path to allow non-default-path borg locations, #125
-- implement --umask M and use 077 as default umask for better security, #117
-- borg check: give a named single archive to it, fixes #139
-- cache sync: show progress indication
-- cache sync: reimplement the chunk index merging in C
-
-Bug fixes:
-
-- fix segfault that happened for unreadable files (chunker: n needs to be a
-  signed size_t), #116
-- fix the repair mode, #144
-- repo delete: add destroy to allowed rpc methods, fixes issue #114
-- more compatible repository locking code (based on mkdir), maybe fixes #92
-  (attic #317, attic #201).
-- better Exception msg if no Borg is installed on the remote repo server, #56
-- create a RepositoryCache implementation that can cope with >2GiB,
-  fixes attic #326.
-- fix Traceback when running check --repair, attic #232
-- clarify help text, fixes #73.
-- add help string for --no-files-cache, fixes #140
-
-Other changes:
-
-- improved docs:
-
-  - added docs/misc directory for misc. writeups that won't be included
-    "as is" into the html docs.
-  - document environment variables and return codes (attic #324, attic #52)
-  - web site: add related projects, fix web site url, IRC #borgbackup
-  - Fedora/Fedora-based install instructions added to docs
-  - Cygwin-based install instructions added to docs
-  - updated AUTHORS
-  - add FAQ entries about redundancy / integrity
-  - clarify that borg extract uses the cwd as extraction target
-  - update internals doc about chunker params, memory usage and compression
-  - added docs about development
-  - add some words about resource usage in general
-  - document how to backup a raw disk
-  - add note about how to run borg from virtual env
-  - add solutions for (ll)fuse installation problems
-  - document what borg check does, fixes #138
-  - reorganize borgbackup.github.io sidebar, prev/next at top
-  - deduplicate and refactor the docs / README.rst
-
-- use borg-tmp as prefix for temporary files / directories
-- short prune options without "keep-" are deprecated, do not suggest them
-- improved tox configuration
-- remove usage of unittest.mock, always use mock from pypi
-- use entrypoints instead of scripts, for better use of the wheel format and
-  modern installs
-- add requirements.d/development.txt and modify tox.ini
-- use travis-ci for testing based on Linux and (new) OS X
-- use coverage.py, pytest-cov and codecov.io for test coverage support
-
-I forgot to list some stuff already implemented in 0.23.0, here they are:
-
-New features:
-
-- efficient archive list from manifest, meaning a big speedup for slow
-  repo connections and "list <repo>", "delete <repo>", "prune" (attic #242,
-  attic #167)
-- big speedup for chunks cache sync (esp. for slow repo connections), fixes #18
-- hashindex: improve error messages
-
-Other changes:
-
-- explicitly specify binary mode to open binary files
-- some easy micro optimizations
-
-
-Version 0.23.0
---------------
-
-Incompatible changes (compared to attic, fork related):
-
-- changed sw name and cli command to "borg", updated docs
-- package name (and name in urls) uses "borgbackup" to have less collisions
-- changed repo / cache internal magic strings from ATTIC* to BORG*,
-  changed cache location to .cache/borg/ - this means that it currently won't
-  accept attic repos (see issue #21 about improving that)
-
-Bug fixes:
-
-- avoid defect python-msgpack releases, fixes attic #171, fixes attic #185
-- fix traceback when trying to do unsupported passphrase change, fixes attic #189
-- datetime does not like the year 10.000, fixes attic #139
-- fix "info" all archives stats, fixes attic #183
-- fix parsing with missing microseconds, fixes attic #282
-- fix misleading hint the fuse ImportError handler gave, fixes attic #237
-- check unpacked data from RPC for tuple type and correct length, fixes attic #127
-- fix Repository._active_txn state when lock upgrade fails
-- give specific path to xattr.is_enabled(), disable symlink setattr call that
-  always fails
-- fix test setup for 32bit platforms, partial fix for attic #196
-- upgraded versioneer, PEP440 compliance, fixes attic #257
-
-New features:
-
-- less memory usage: add global option --no-cache-files
-- check --last N (only check the last N archives)
-- check: sort archives in reverse time order
-- rename repo::oldname newname (rename repository)
-- create -v output more informative
-- create --progress (backup progress indicator)
-- create --timestamp (utc string or reference file/dir)
-- create: if "-" is given as path, read binary from stdin
-- extract: if --stdout is given, write all extracted binary data to stdout
-- extract --sparse (simple sparse file support)
-- extra debug information for 'fread failed'
-- delete <repo> (deletes whole repo + local cache)
-- FUSE: reflect deduplication in allocated blocks
-- only allow whitelisted RPC calls in server mode
-- normalize source/exclude paths before matching
-- use posix_fadvise to not spoil the OS cache, fixes attic #252
-- toplevel error handler: show tracebacks for better error analysis
-- sigusr1 / sigint handler to print current file infos - attic PR #286
-- RPCError: include the exception args we get from remote
-
-Other changes:
-
-- source: misc. cleanups, pep8, style
-- docs and faq improvements, fixes, updates
-- cleanup crypto.pyx, make it easier to adapt to other AES modes
-- do os.fsync like recommended in the python docs
-- source: Let chunker optionally work with os-level file descriptor.
-- source: Linux: remove duplicate os.fsencode calls
-- source: refactor _open_rb code a bit, so it is more consistent / regular
-- source: refactor indicator (status) and item processing
-- source: use py.test for better testing, flake8 for code style checks
-- source: fix tox >=2.0 compatibility (test runner)
-- pypi package: add python version classifiers, add FreeBSD to platforms
-
-
-Attic Changelog
-===============
-
-Here you can see the full list of changes between each Attic release until Borg
-forked from Attic:
-
-Version 0.17
-------------
-
-(bugfix release, released on X)
-- Fix hashindex ARM memory alignment issue (#309)
-- Improve hashindex error messages (#298)
-
-Version 0.16
-------------
-
-(bugfix release, released on May 16, 2015)
-- Fix typo preventing the security confirmation prompt from working (#303)
-- Improve handling of systems with improperly configured file system encoding (#289)
-- Fix "All archives" output for attic info. (#183)
-- More user friendly error message when repository key file is not found (#236)
-- Fix parsing of iso 8601 timestamps with zero microseconds (#282)
-
-Version 0.15
-------------
-
-(bugfix release, released on Apr 15, 2015)
-- xattr: Be less strict about unknown/unsupported platforms (#239)
-- Reduce repository listing memory usage (#163).
-- Fix BrokenPipeError for remote repositories (#233)
-- Fix incorrect behavior with two character directory names (#265, #268)
-- Require approval before accessing relocated/moved repository (#271)
-- Require approval before accessing previously unknown unencrypted repositories (#271)
-- Fix issue with hash index files larger than 2GB.
-- Fix Python 3.2 compatibility issue with noatime open() (#164)
-- Include missing pyx files in dist files (#168)
-
-Version 0.14
-------------
-
-(feature release, released on Dec 17, 2014)
-- Added support for stripping leading path segments (#95)
-  "attic extract --strip-segments X"
-- Add workaround for old Linux systems without acl_extended_file_no_follow (#96)
-- Add MacPorts' path to the default openssl search path (#101)
-- HashIndex improvements, eliminates unnecessary IO on low memory systems.
-- Fix "Number of files" output for attic info. (#124)
-- limit create file permissions so files aren't read while restoring
-- Fix issue with empty xattr values (#106)
-
-Version 0.13
-------------
-
-(feature release, released on Jun 29, 2014)
-
-- Fix sporadic "Resource temporarily unavailable" when using remote repositories
-- Reduce file cache memory usage (#90)
-- Faster AES encryption (utilizing AES-NI when available)
-- Experimental Linux, OS X and FreeBSD ACL support (#66)
-- Added support for backup and restore of BSDFlags (OSX, FreeBSD) (#56)
-- Fix bug where xattrs on symlinks were not correctly restored
-- Added cachedir support. CACHEDIR.TAG compatible cache directories
-  can now be excluded using ``--exclude-caches`` (#74)
-- Fix crash on extreme mtime timestamps (year 2400+) (#81)
-- Fix Python 3.2 specific lockf issue (EDEADLK)
-
-Version 0.12
-------------
-
-(feature release, released on April 7, 2014)
-
-- Python 3.4 support (#62)
-- Various documentation improvements a new style
-- ``attic mount`` now supports mounting an entire repository not only
-  individual archives (#59)
-- Added option to restrict remote repository access to specific path(s):
-  ``attic serve --restrict-to-path X`` (#51)
-- Include "all archives" size information in "--stats" output. (#54)
-- Added ``--stats`` option to ``attic delete`` and ``attic prune``
-- Fixed bug where ``attic prune`` used UTC instead of the local time zone
-  when determining which archives to keep.
-- Switch to SI units (Power of 1000 instead 1024) when printing file sizes
-
-Version 0.11
-------------
-
-(feature release, released on March 7, 2014)
-
-- New "check" command for repository consistency checking (#24)
-- Documentation improvements
-- Fix exception during "attic create" with repeated files (#39)
-- New "--exclude-from" option for attic create/extract/verify.
-- Improved archive metadata deduplication.
-- "attic verify" has been deprecated. Use "attic extract --dry-run" instead.
-- "attic prune --hourly|daily|..." has been deprecated.
-  Use "attic prune --keep-hourly|daily|..." instead.
-- Ignore xattr errors during "extract" if not supported by the filesystem. (#46)
-
-Version 0.10
-------------
-
-(bugfix release, released on Jan 30, 2014)
-
-- Fix deadlock when extracting 0 sized files from remote repositories
-- "--exclude" wildcard patterns are now properly applied to the full path
-  not just the file name part (#5).
-- Make source code endianness agnostic (#1)
-
-Version 0.9
------------
-
-(feature release, released on Jan 23, 2014)
-
-- Remote repository speed and reliability improvements.
-- Fix sorting of segment names to ignore NFS left over files. (#17)
-- Fix incorrect display of time (#13)
-- Improved error handling / reporting. (#12)
-- Use fcntl() instead of flock() when locking repository/cache. (#15)
-- Let ssh figure out port/user if not specified so we don't override .ssh/config (#9)
-- Improved libcrypto path detection (#23).
-
-Version 0.8.1
--------------
-
-(bugfix release, released on Oct 4, 2013)
-
-- Fix segmentation fault issue.
-
-Version 0.8
------------
-
-(feature release, released on Oct 3, 2013)
-
-- Fix xattr issue when backing up sshfs filesystems (#4)
-- Fix issue with excessive index file size (#6)
-- Support access of read only repositories.
-- New syntax to enable repository encryption:
-    attic init --encryption="none|passphrase|keyfile".
-- Detect and abort if repository is older than the cache.
-
-
-Version 0.7
------------
-
-(feature release, released on Aug 5, 2013)
-
-- Ported to FreeBSD
-- Improved documentation
-- Experimental: Archives mountable as fuse filesystems.
-- The "user." prefix is no longer stripped from xattrs on Linux
-
-
-Version 0.6.1
--------------
-
-(bugfix release, released on July 19, 2013)
-
-- Fixed an issue where mtime was not always correctly restored.
-
-
-Version 0.6
------------
-
-First public release on July 9, 2013

+ 1 - 0
CHANGES.rst

@@ -0,0 +1 @@
+docs/changes.rst

+ 24 - 14
README.rst

@@ -9,8 +9,10 @@ since only changes are stored.
 The authenticated encryption technique makes it suitable for backups to not
 fully trusted targets.
 
-`Borg Installation docs <http://borgbackup.github.io/borgbackup/installation.html>`_
+See the `installation manual`_ or, if you have already
+downloaded Borg, ``docs/installation.rst`` to get started with Borg.
 
+.. _installation manual: http://borgbackup.github.io/borgbackup/installation.html
 
 Main features
 ~~~~~~~~~~~~~
@@ -63,16 +65,16 @@ Main features
     Backup archives are mountable as userspace filesystems for easy interactive
     backup examination and restores (e.g. by using a regular file manager).
 
-**Easy installation**
-    For Linux, Mac OS X and FreeBSD, we offer a single-file pyinstaller binary
-    that does not require installing anything - you can just run it.
+**Easy installation on multiple platforms**
+    We offer single-file binaries
+    that does not require installing anything - you can just run it on
+    the supported platforms:
 
-**Platforms Borg works on**
-  * Linux
-  * Mac OS X
-  * FreeBSD
-  * OpenBSD and NetBSD (for both: no xattrs/ACLs support yet)
-  * Cygwin (unsupported)
+    * Linux
+    * Mac OS X
+    * FreeBSD
+    * OpenBSD and NetBSD (no xattrs/ACLs support or binaries yet)
+    * Cygwin (not supported, no binaries yet)
 
 **Free and Open Source Software**
   * security and functionality can be audited independently
@@ -108,21 +110,29 @@ Notes
 -----
 
 Borg is a fork of `Attic <https://github.com/jborg/attic>`_ and maintained by
-"`The Borg Collective <https://github.com/borgbackup/borg/blob/master/AUTHORS>`_".
+"`The Borg collective`_".
+
+.. _The Borg collective: http://borgbackup.github.io/borgbackup/authors.html
 
 Read `issue #1 <https://github.com/borgbackup/borg/issues/1>`_ about the initial
 considerations regarding project goals and policy of the Borg project.
 
 BORG IS NOT COMPATIBLE WITH ORIGINAL ATTIC.
 EXPECT THAT WE WILL BREAK COMPATIBILITY REPEATEDLY WHEN MAJOR RELEASE NUMBER
-CHANGES (like when going from 0.x.y to 1.0.0). Please read CHANGES document.
+CHANGES (like when going from 0.x.y to 1.0.0). Please read the
+`changelog`_ (or ``CHANGES.rst`` in the source distribution) for more
+information.
+
+.. _changelog: https://borgbackup.github.io/borgbackup/changes.html
 
 NOT RELEASED DEVELOPMENT VERSIONS HAVE UNKNOWN COMPATIBILITY PROPERTIES.
 
 THIS IS SOFTWARE IN DEVELOPMENT, DECIDE YOURSELF WHETHER IT FITS YOUR NEEDS.
 
-For more information, please also see the
-`LICENSE  <https://github.com/borgbackup/borg/blob/master/LICENSE>`_.
+Borg is distributed under a 3-clause BSD license, see `the license`_
+for the complete license.
+
+.. _the license: https://borgbackup.github.io/borgbackup/authors.html#license
 
 |build| |coverage|
 

+ 3 - 2
Vagrantfile

@@ -62,6 +62,7 @@ def packages_darwin
     brew install lz4
     brew install fakeroot
     brew install git
+    brew install pkgconfig
     touch ~vagrant/.bash_profile ; chown vagrant ~vagrant/.bash_profile
   EOF
 end
@@ -204,7 +205,7 @@ def install_pyinstaller(boxname)
     . borg-env/bin/activate
     git clone https://github.com/pyinstaller/pyinstaller.git
     cd pyinstaller
-    git checkout develop
+    git checkout master
     pip install -e .
   EOF
 end
@@ -216,7 +217,7 @@ def install_pyinstaller_bootloader(boxname)
     . borg-env/bin/activate
     git clone https://github.com/pyinstaller/pyinstaller.git
     cd pyinstaller
-    git checkout python3
+    git checkout master
     # build bootloader, if it is not included
     cd bootloader
     python ./waf all

+ 10 - 6
borg/archive.py

@@ -10,7 +10,6 @@ logger = create_logger()
 from .key import key_factory
 from .remote import cache_if_remote
 
-import msgpack
 import os
 import socket
 import stat
@@ -18,11 +17,16 @@ import sys
 import time
 from io import BytesIO
 from . import xattr
-from .platform import acl_get, acl_set
-from .chunker import Chunker
-from .hashindex import ChunkIndex
-from .helpers import parse_timestamp, format_timedelta, Error, uid2user, user2uid, gid2group, group2gid, \
-    Manifest, Statistics, decode_dict, st_mtime_ns, make_path_safe, StableDict, int_to_bigint, bigint_to_int
+from .helpers import parse_timestamp, Error, uid2user, user2uid, gid2group, group2gid, format_timedelta, \
+    Manifest, Statistics, decode_dict, st_mtime_ns, make_path_safe, StableDict, int_to_bigint, bigint_to_int, have_cython
+if have_cython():
+    from .platform import acl_get, acl_set
+    from .chunker import Chunker
+    from .hashindex import ChunkIndex
+    import msgpack
+else:
+    import mock
+    msgpack = mock.Mock()
 
 ITEMS_BUFFER = 1024 * 1024
 

+ 41 - 36
borg/archiver.py

@@ -15,19 +15,20 @@ import textwrap
 import traceback
 
 from . import __version__
-from .archive import Archive, ArchiveChecker, CHUNKER_PARAMS
-from .compress import Compressor, COMPR_BUFFER
-from .logger import create_logger, setup_logging
-logger = create_logger()
-from .upgrader import AtticRepositoryUpgrader
-from .repository import Repository
-from .cache import Cache
-from .key import key_creator
 from .helpers import Error, location_validator, format_time, format_file_size, \
     format_file_mode, ExcludePattern, IncludePattern, exclude_path, adjust_patterns, to_localtime, timestamp, \
     get_cache_dir, get_keys_dir, format_timedelta, prune_within, prune_split, \
     Manifest, remove_surrogates, update_excludes, format_archive, check_extension_modules, Statistics, \
-    is_cachedir, bigint_to_int, ChunkerParams, CompressionSpec
+    is_cachedir, bigint_to_int, ChunkerParams, CompressionSpec, have_cython
+from .logger import create_logger, setup_logging
+logger = create_logger()
+if have_cython():
+    from .compress import Compressor, COMPR_BUFFER
+    from .upgrader import AtticRepositoryUpgrader
+    from .repository import Repository
+    from .cache import Cache
+    from .key import key_creator
+from .archive import Archive, ArchiveChecker, CHUNKER_PARAMS
 from .remote import RepositoryServer, RemoteRepository
 
 has_lchflags = hasattr(os, 'lchflags')
@@ -541,38 +542,18 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
                     print(warning)
         return args
 
-    def run(self, args=None):
-        check_extension_modules()
-        keys_dir = get_keys_dir()
-        if not os.path.exists(keys_dir):
-            os.makedirs(keys_dir)
-            os.chmod(keys_dir, stat.S_IRWXU)
-        cache_dir = get_cache_dir()
-        if not os.path.exists(cache_dir):
-            os.makedirs(cache_dir)
-            os.chmod(cache_dir, stat.S_IRWXU)
-            with open(os.path.join(cache_dir, 'CACHEDIR.TAG'), 'w') as fd:
-                fd.write(textwrap.dedent("""
-                    Signature: 8a477f597d28d172789f06886806bc55
-                    # This file is a cache directory tag created by Borg.
-                    # For information about cache directory tags, see:
-                    #       http://www.brynosaurus.com/cachedir/
-                    """).lstrip())
-        common_parser = argparse.ArgumentParser(add_help=False)
+    def build_parser(self, args=None, prog=None):
+        common_parser = argparse.ArgumentParser(add_help=False, prog=prog)
         common_parser.add_argument('-v', '--verbose', dest='verbose', action='count',
                                    help='verbose output, defaults to warnings only')
         common_parser.add_argument('--no-files-cache', dest='cache_files', action='store_false',
                                    help='do not load/update the file metadata cache used to detect unchanged files')
-        common_parser.add_argument('--umask', dest='umask', type=lambda s: int(s, 8), default=0o077, metavar='M',
-                                   help='set umask to M (local and remote, default: 0o077)')
-        common_parser.add_argument('--remote-path', dest='remote_path', default='borg', metavar='PATH',
-                                   help='set remote path to executable (default: "borg")')
+        common_parser.add_argument('--umask', dest='umask', type=lambda s: int(s, 8), default=RemoteRepository.umask, metavar='M',
+                                   help='set umask to M (local and remote, default: %(default)s)')
+        common_parser.add_argument('--remote-path', dest='remote_path', default=RemoteRepository.remote_path, metavar='PATH',
+                                   help='set remote path to executable (default: "%(default)s")')
 
-        # We can't use argparse for "serve" since we don't want it to show up in "Available commands"
-        if args:
-            args = self.preprocess_args(args)
-
-        parser = argparse.ArgumentParser(description='Borg %s - Deduplicated Backups' % __version__)
+        parser = argparse.ArgumentParser(prog=prog, description='Borg %s - Deduplicated Backups' % __version__)
         subparsers = parser.add_subparsers(title='Available commands')
 
         serve_epilog = textwrap.dedent("""
@@ -968,6 +949,30 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
         subparser.set_defaults(func=functools.partial(self.do_help, parser, subparsers.choices))
         subparser.add_argument('topic', metavar='TOPIC', type=str, nargs='?',
                                help='additional help on TOPIC')
+        return parser
+
+    def run(self, args=None):
+        check_extension_modules()
+        keys_dir = get_keys_dir()
+        if not os.path.exists(keys_dir):
+            os.makedirs(keys_dir)
+            os.chmod(keys_dir, stat.S_IRWXU)
+        cache_dir = get_cache_dir()
+        if not os.path.exists(cache_dir):
+            os.makedirs(cache_dir)
+            os.chmod(cache_dir, stat.S_IRWXU)
+            with open(os.path.join(cache_dir, 'CACHEDIR.TAG'), 'w') as fd:
+                fd.write(textwrap.dedent("""
+                    Signature: 8a477f597d28d172789f06886806bc55
+                    # This file is a cache directory tag created by Borg.
+                    # For information about cache directory tags, see:
+                    #       http://www.brynosaurus.com/cachedir/
+                    """).lstrip())
+
+        # We can't use argparse for "serve" since we don't want it to show up in "Available commands"
+        if args:
+            args = self.preprocess_args(args)
+        parser = self.build_parser(args)
 
         args = parser.parse_args(args or ['-h'])
         setup_logging(args)

+ 4 - 2
borg/cache.py

@@ -2,7 +2,6 @@ import configparser
 from .remote import cache_if_remote
 from collections import namedtuple
 import errno
-import msgpack
 import os
 import stat
 import sys
@@ -15,10 +14,13 @@ from .key import PlaintextKey
 from .logger import create_logger
 logger = create_logger()
 from .helpers import Error, get_cache_dir, decode_dict, st_mtime_ns, unhexlify, int_to_bigint, \
-    bigint_to_int, format_file_size
+    bigint_to_int, format_file_size, have_cython
 from .locking import UpgradableLock
 from .hashindex import ChunkIndex
 
+if have_cython():
+    import msgpack
+
 
 class Cache:
     """Client Side cache

+ 4 - 2
borg/fuse.py

@@ -2,15 +2,17 @@ from collections import defaultdict
 import errno
 import io
 import llfuse
-import msgpack
 import os
 import stat
 import tempfile
 import time
 from .archive import Archive
-from .helpers import daemonize
+from .helpers import daemonize, have_cython
 from .remote import cache_if_remote
 
+if have_cython():
+    import msgpack
+
 # Does this version of llfuse support ns precision?
 have_fuse_mtime_ns = hasattr(llfuse.EntryAttributes, 'st_mtime_ns')
 

+ 21 - 6
borg/helpers.py

@@ -16,11 +16,26 @@ from datetime import datetime, timezone, timedelta
 from fnmatch import translate
 from operator import attrgetter
 
-import msgpack
+def have_cython():
+    """allow for a way to disable Cython includes
 
-from . import hashindex
-from . import chunker
-from . import crypto
+    this is used during usage docs build, in setup.py. It is to avoid
+    loading the Cython libraries which are built, but sometimes not in
+    the search path (namely, during Tox runs).
+
+    we simply check an environment variable (``BORG_CYTHON_DISABLE``)
+    which, when set (to anything) will disable includes of Cython
+    libraries in key places to enable usage docs to be built.
+
+    :returns: True if Cython is available, False otherwise.
+    """
+    return not os.environ.get('BORG_CYTHON_DISABLE')
+
+if have_cython():
+    from . import hashindex
+    from . import chunker
+    from . import crypto
+    import msgpack
 
 
 class Error(Exception):
@@ -178,8 +193,8 @@ def get_keys_dir():
 
 def get_cache_dir():
     """Determine where to repository keys and cache"""
-    return os.environ.get('BORG_CACHE_DIR',
-                          os.path.join(os.path.expanduser('~'), '.cache', 'borg'))
+    xdg_cache = os.environ.get('XDG_CACHE_HOME', os.path.join(os.path.expanduser('~'), '.cache'))
+    return os.environ.get('BORG_CACHE_DIR', os.path.join(xdg_cache, 'borg'))
 
 
 def to_localtime(ts):

+ 6 - 4
borg/key.py

@@ -2,17 +2,19 @@ from binascii import hexlify, a2b_base64, b2a_base64
 import configparser
 import getpass
 import os
-import msgpack
 import textwrap
 import hmac
 from hashlib import sha256
 
-from .crypto import pbkdf2_sha256, get_random_bytes, AES, bytes_to_long, long_to_bytes, bytes_to_int, num_aes_blocks
-from .compress import Compressor, COMPR_BUFFER
-from .helpers import IntegrityError, get_keys_dir, Error
+from .helpers import IntegrityError, get_keys_dir, Error, have_cython
 from .logger import create_logger
 logger = create_logger()
 
+if have_cython():
+    from .crypto import pbkdf2_sha256, get_random_bytes, AES, bytes_to_long, long_to_bytes, bytes_to_int, num_aes_blocks
+    from .compress import Compressor, COMPR_BUFFER
+    import msgpack
+
 PREFIX = b'\0' * 8
 
 

+ 3 - 0
borg/locking.py

@@ -169,6 +169,9 @@ class LockRoster:
             if err.errno != errno.ENOENT:
                 raise
             data = {}
+        except ValueError:
+            # corrupt/empty roster file?
+            data = {}
         return data
 
     def save(self, data):

+ 29 - 15
borg/remote.py

@@ -1,8 +1,8 @@
 import errno
 import fcntl
-import msgpack
 import os
 import select
+import shlex
 from subprocess import Popen, PIPE
 import sys
 import tempfile
@@ -10,9 +10,12 @@ import traceback
 
 from . import __version__
 
-from .helpers import Error, IntegrityError
+from .helpers import Error, IntegrityError, have_cython
 from .repository import Repository
 
+if have_cython():
+    import msgpack
+
 BUFSIZE = 10 * 1024 * 1024
 
 
@@ -108,8 +111,9 @@ class RepositoryServer:  # pragma: no cover
 
 class RemoteRepository:
     extra_test_args = []
-    remote_path = None
-    umask = None
+    remote_path = 'borg'
+    # default umask, overriden by --umask, defaults to read/write only for owner
+    umask = 0o077
 
     class RPCError(Exception):
         def __init__(self, name):
@@ -125,19 +129,14 @@ class RemoteRepository:
         self.responses = {}
         self.unpacker = msgpack.Unpacker(use_list=False)
         self.p = None
-        # use local umask also for the remote process
-        umask = ['--umask', '%03o' % self.umask]
+        # XXX: ideally, the testsuite would subclass Repository and
+        # override ssh_cmd() instead of this crude hack, although
+        # __testsuite__ is not a valid domain name so this is pretty
+        # safe.
         if location.host == '__testsuite__':
-            args = [sys.executable, '-m', 'borg.archiver', 'serve'] + umask + self.extra_test_args
+            args = [sys.executable, '-m', 'borg.archiver', 'serve' ] + self.extra_test_args
         else:  # pragma: no cover
-            args = ['ssh']
-            if location.port:
-                args += ['-p', str(location.port)]
-            if location.user:
-                args.append('%s@%s' % (location.user, location.host))
-            else:
-                args.append('%s' % location.host)
-            args += [self.remote_path, 'serve'] + umask
+            args = self.ssh_cmd(location)
         self.p = Popen(args, bufsize=0, stdin=PIPE, stdout=PIPE)
         self.stdin_fd = self.p.stdin.fileno()
         self.stdout_fd = self.p.stdout.fileno()
@@ -160,6 +159,21 @@ class RemoteRepository:
     def __repr__(self):
         return '<%s %s>' % (self.__class__.__name__, self.location.canonical_path())
 
+    def umask_flag(self):
+        return ['--umask', '%03o' % self.umask]
+
+    def ssh_cmd(self, location):
+        args = shlex.split(os.environ.get('BORG_RSH', 'ssh'))
+        if location.port:
+            args += ['-p', str(location.port)]
+        if location.user:
+            args.append('%s@%s' % (location.user, location.host))
+        else:
+            args.append('%s' % location.host)
+        # use local umask also for the remote process
+        args += [self.remote_path, 'serve'] + self.umask_flag()
+        return args
+
     def call(self, cmd, *args, **kw):
         for resp in self.call_many(cmd, [args], **kw):
             return resp

+ 44 - 31
borg/repository.py

@@ -11,8 +11,9 @@ import struct
 import sys
 from zlib import crc32
 
-from .hashindex import NSIndex
-from .helpers import Error, IntegrityError, read_msgpack, write_msgpack, unhexlify
+from .helpers import Error, IntegrityError, read_msgpack, write_msgpack, unhexlify, have_cython
+if have_cython():
+    from .hashindex import NSIndex
 from .locking import UpgradableLock
 from .lrucache import LRUCache
 
@@ -304,7 +305,7 @@ class Repository:
             try:
                 objects = list(self.io.iter_objects(segment))
             except IntegrityError as err:
-                report_error('Error reading segment {}: {}'.format(segment, err))
+                report_error(str(err))
                 objects = []
                 if repair:
                     self.io.recover_segment(segment, filename)
@@ -533,30 +534,14 @@ class LoggedIO:
         fd = self.get_fd(segment)
         fd.seek(0)
         if fd.read(MAGIC_LEN) != MAGIC:
-            raise IntegrityError('Invalid segment magic')
+            raise IntegrityError('Invalid segment magic [segment {}, offset {}]'.format(segment, 0))
         offset = MAGIC_LEN
         header = fd.read(self.header_fmt.size)
         while header:
-            try:
-                crc, size, tag = self.header_fmt.unpack(header)
-            except struct.error as err:
-                raise IntegrityError('Invalid segment entry header [offset {}]: {}'.format(offset, err))
-            if size > MAX_OBJECT_SIZE or size < self.header_fmt.size:
-                raise IntegrityError('Invalid segment entry size [offset {}]'.format(offset))
-            length = size - self.header_fmt.size
-            rest = fd.read(length)
-            if len(rest) != length:
-                raise IntegrityError('Segment entry data short read [offset {}]: expected: {}, got {} bytes'.format(
-                                     offset, length, len(rest)))
-            if crc32(rest, crc32(memoryview(header)[4:])) & 0xffffffff != crc:
-                raise IntegrityError('Segment entry checksum mismatch [offset {}]'.format(offset))
-            if tag not in (TAG_PUT, TAG_DELETE, TAG_COMMIT):
-                raise IntegrityError('Invalid segment entry tag [offset {}]'.format(offset))
-            key = None
-            if tag in (TAG_PUT, TAG_DELETE):
-                key = rest[:32]
+            size, tag, key, data = self._read(fd, self.header_fmt, header, segment, offset,
+                                              (TAG_PUT, TAG_DELETE, TAG_COMMIT))
             if include_data:
-                yield tag, key, offset, rest[32:]
+                yield tag, key, offset, data
             else:
                 yield tag, key, offset
             offset += size
@@ -589,16 +574,44 @@ class LoggedIO:
         fd = self.get_fd(segment)
         fd.seek(offset)
         header = fd.read(self.put_header_fmt.size)
-        crc, size, tag, key = self.put_header_fmt.unpack(header)
-        if size > MAX_OBJECT_SIZE:
-            raise IntegrityError('Invalid segment object size')
-        data = fd.read(size - self.put_header_fmt.size)
-        if crc32(data, crc32(memoryview(header)[4:])) & 0xffffffff != crc:
-            raise IntegrityError('Segment checksum mismatch')
-        if tag != TAG_PUT or id != key:
-            raise IntegrityError('Invalid segment entry header')
+        size, tag, key, data = self._read(fd, self.put_header_fmt, header, segment, offset, (TAG_PUT, ))
+        if id != key:
+            raise IntegrityError('Invalid segment entry header, is not for wanted id [segment {}, offset {}]'.format(
+                segment, offset))
         return data
 
+    def _read(self, fd, fmt, header, segment, offset, acceptable_tags):
+        # some code shared by read() and iter_objects()
+        try:
+            hdr_tuple = fmt.unpack(header)
+        except struct.error as err:
+            raise IntegrityError('Invalid segment entry header [segment {}, offset {}]: {}'.format(
+                segment, offset, err))
+        if fmt is self.put_header_fmt:
+            crc, size, tag, key = hdr_tuple
+        elif fmt is self.header_fmt:
+            crc, size, tag = hdr_tuple
+            key = None
+        else:
+            raise TypeError("_read called with unsupported format")
+        if size > MAX_OBJECT_SIZE or size < fmt.size:
+            raise IntegrityError('Invalid segment entry size [segment {}, offset {}]'.format(
+                segment, offset))
+        length = size - fmt.size
+        data = fd.read(length)
+        if len(data) != length:
+            raise IntegrityError('Segment entry data short read [segment {}, offset {}]: expected {}, got {} bytes'.format(
+                segment, offset, length, len(data)))
+        if crc32(data, crc32(memoryview(header)[4:])) & 0xffffffff != crc:
+            raise IntegrityError('Segment entry checksum mismatch [segment {}, offset {}]'.format(
+                segment, offset))
+        if tag not in acceptable_tags:
+            raise IntegrityError('Invalid segment entry header, did not get acceptable tag [segment {}, offset {}]'.format(
+                segment, offset))
+        if key is None and tag in (TAG_PUT, TAG_DELETE):
+            key, data = data[:32], data[32:]
+        return size, tag, key, data
+
     def write_put(self, id, data):
         size = len(data) + self.put_header_fmt.size
         fd = self.get_write_fd()

+ 19 - 1
borg/testsuite/helpers.py

@@ -1,13 +1,14 @@
 import hashlib
 from time import mktime, strptime
 from datetime import datetime, timezone, timedelta
+import os
 
 import pytest
 import sys
 import msgpack
 
 from ..helpers import adjust_patterns, exclude_path, Location, format_timedelta, IncludePattern, ExcludePattern, make_path_safe, \
-    prune_within, prune_split, \
+    prune_within, prune_split, get_cache_dir, \
     StableDict, int_to_bigint, bigint_to_int, parse_timestamp, CompressionSpec, ChunkerParams
 from . import BaseTestCase
 
@@ -381,3 +382,20 @@ class TestParseTimestamp(BaseTestCase):
     def test(self):
         self.assert_equal(parse_timestamp('2015-04-19T20:25:00.226410'), datetime(2015, 4, 19, 20, 25, 0, 226410, timezone.utc))
         self.assert_equal(parse_timestamp('2015-04-19T20:25:00'), datetime(2015, 4, 19, 20, 25, 0, 0, timezone.utc))
+
+
+def test_get_cache_dir():
+    """test that get_cache_dir respects environement"""
+    # reset BORG_CACHE_DIR in order to test default
+    old_env = None
+    if os.environ.get('BORG_CACHE_DIR'):
+        old_env = os.environ['BORG_CACHE_DIR']
+        del(os.environ['BORG_CACHE_DIR'])
+    assert get_cache_dir() == os.path.join(os.path.expanduser('~'), '.cache', 'borg')
+    os.environ['XDG_CACHE_HOME'] = '/var/tmp/.cache'
+    assert get_cache_dir() == os.path.join('/var/tmp/.cache', 'borg')
+    os.environ['BORG_CACHE_DIR'] = '/var/tmp'
+    assert get_cache_dir() == '/var/tmp'
+    # reset old env
+    if old_env is not None:
+        os.environ['BORG_CACHE_DIR'] = old_env

+ 9 - 0
borg/testsuite/repository.py

@@ -325,6 +325,15 @@ class RemoteRepositoryTestCase(RepositoryTestCase):
     def test_invalid_rpc(self):
         self.assert_raises(InvalidRPCMethod, lambda: self.repository.call('__init__', None))
 
+    def test_ssh_cmd(self):
+        assert self.repository.umask is not None
+        assert self.repository.ssh_cmd(Location('example.com:foo')) == ['ssh', 'example.com', 'borg', 'serve'] + self.repository.umask_flag()
+        assert self.repository.ssh_cmd(Location('ssh://example.com/foo')) == ['ssh', 'example.com', 'borg', 'serve'] + self.repository.umask_flag()
+        assert self.repository.ssh_cmd(Location('ssh://user@example.com/foo')) == ['ssh', 'user@example.com', 'borg', 'serve'] + self.repository.umask_flag()
+        assert self.repository.ssh_cmd(Location('ssh://user@example.com:1234/foo')) == ['ssh', '-p', '1234', 'user@example.com', 'borg', 'serve'] + self.repository.umask_flag()
+        os.environ['BORG_RSH'] = 'ssh --foo'
+        assert self.repository.ssh_cmd(Location('example.com:foo')) == ['ssh', '--foo', 'example.com', 'borg', 'serve'] + self.repository.umask_flag()
+
 
 class RemoteRepositoryCheckTestCase(RepositoryCheckTestCase):
 

+ 1 - 30
docs/Makefile

@@ -36,7 +36,7 @@ help:
 clean:
 	-rm -rf $(BUILDDIR)/*
 
-html: usage api.rst
+html:
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
@@ -139,32 +139,3 @@ gh-io: html
 
 inotify: html
 	while inotifywait -r . --exclude usage.rst --exclude '_build/*' ; do make html ; done
-
-# generate list of targets
-usage: $(shell borg help | grep -A1 "Available commands:" | tail -1 | sed 's/[{} ]//g;s/,\|^/.rst.inc usage\//g;s/^.rst.inc//;s/usage\/help//')
-
-# generate help file based on usage
-usage/%.rst.inc: ../borg/archiver.py
-	@echo generating usage for $*
-	@printf ".. _borg_$*:\n\n" > $@
-	@printf "borg $*\n" >> $@
-	@echo -n borg $* | tr 'a-z- ' '-' >> $@
-	@printf "\n::\n\n" >> $@
-	@borg help $* --usage-only | sed -e 's/^/    /' >> $@
-	@printf "\nDescription\n~~~~~~~~~~~\n" >> $@
-	@borg help $* --epilog-only >> $@
-
-api.rst: Makefile
-	@echo "auto-generating API documentation"
-	@echo "Borg Backup API documentation" > $@
-	@echo "=============================" >> $@
-	@echo "" >> $@
-	@for mod in ../borg/*.pyx ../borg/*.py; do \
-		if echo "$$mod" | grep -q "/_"; then \
-			continue ; \
-		fi ; \
-		printf ".. automodule:: "; \
-		echo "$$mod" | sed "s!\.\./!!;s/\.pyx\?//;s!/!.!"; \
-		echo "    :members:"; \
-		echo "    :undoc-members:"; \
-	done >> $@

+ 2 - 2
docs/_themes/local/sidebarusefullinks.html

@@ -3,9 +3,9 @@
 
 <h3>Useful Links</h3>
 <ul>
-  <li><a href="https://borgbackup.github.io/borgbackup/">Main Web Site</a></li>
+  <li><a href="https://borgbackup.readthedocs.org/">Main Web Site</a></li>
+  <li><a href="https://github.com/borgbackup/borg/releases">Releases</a></li>
   <li><a href="https://pypi.python.org/pypi/borgbackup">PyPI packages</a></li>
-  <li><a href="https://github.com/borgbackup/borg/issues/147">Binary Packages</a></li>
   <li><a href="https://github.com/borgbackup/borg/blob/master/CHANGES.rst">Current ChangeLog</a></li>
   <li><a href="https://github.com/borgbackup/borg">GitHub</a></li>
   <li><a href="https://github.com/borgbackup/borg/issues">Issue Tracker</a></li>

+ 11 - 0
docs/authors.rst

@@ -0,0 +1,11 @@
+.. include:: global.rst.inc
+
+.. include:: ../AUTHORS
+
+License
+=======
+
+.. _license:
+
+.. include:: ../LICENSE
+   :literal:

+ 510 - 3
docs/changes.rst

@@ -1,4 +1,511 @@
-.. include:: global.rst.inc
-.. _changelog:
+Borg Changelog
+==============
 
-.. include:: ../CHANGES.rst
+Version 0.27.0
+--------------
+
+New features:
+
+- "borg upgrade" command - attic -> borg one time converter / migration, #21
+- temporary hack to avoid using lots of disk space for chunks.archive.d, #235:
+  To use it: rm -rf chunks.archive.d ; touch chunks.archive.d
+- respect XDG_CACHE_HOME, attic #181
+- add support for arbitrary SSH commands, attic #99
+- borg delete --cache-only REPO (only delete cache, not REPO), attic #123
+
+
+Bug fixes:
+
+- use Debian 7 (wheezy) to build pyinstaller borgbackup binaries, fixes slow
+  down observed when running the Centos6-built binary on Ubuntu, #222
+- do not crash on empty lock.roster, fixes #232
+- fix multiple issues with the cache config version check, #234
+- fix segment entry header size check, attic #352
+  plus other error handling improvements / code deduplication there.
+- always give segment and offset in repo IntegrityErrors
+
+
+Other changes:
+
+- stop producing binary wheels, remove docs about it, #147
+- docs:
+  - add warning about prune
+  - generate usage include files only as needed
+  - development docs: add Vagrant section
+  - update / improve / reformat FAQ
+  - hint to single-file pyinstaller binaries from README
+
+
+Version 0.26.1
+--------------
+
+This is a minor update, just docs and new pyinstaller binaries.
+
+- docs update about python and binary requirements
+- better docs for --read-special, fix #220
+- re-built the binaries, fix #218 and #213 (glibc version issue)
+- update web site about single-file pyinstaller binaries
+
+Note: if you did a python-based installation, there is no need to upgrade.
+
+
+Version 0.26.0
+--------------
+
+New features:
+
+- Faster cache sync (do all in one pass, remove tar/compression stuff), #163
+- BORG_REPO env var to specify the default repo, #168
+- read special files as if they were regular files, #79
+- implement borg create --dry-run, attic issue #267
+- Normalize paths before pattern matching on OS X, #143
+- support OpenBSD and NetBSD (except xattrs/ACLs)
+- support / run tests on Python 3.5
+
+Bug fixes:
+
+- borg mount repo: use absolute path, attic #200, attic #137
+- chunker: use off_t to get 64bit on 32bit platform, #178
+- initialize chunker fd to -1, so it's not equal to STDIN_FILENO (0)
+- fix reaction to "no" answer at delete repo prompt, #182
+- setup.py: detect lz4.h header file location
+- to support python < 3.2.4, add less buggy argparse lib from 3.2.6 (#194)
+- fix for obtaining ``char *`` from temporary Python value (old code causes
+  a compile error on Mint 17.2)
+- llfuse 0.41 install troubles on some platforms, require < 0.41
+  (UnicodeDecodeError exception due to non-ascii llfuse setup.py)
+- cython code: add some int types to get rid of unspecific python add /
+  subtract operations (avoid ``undefined symbol FPE_``... error on some platforms)
+- fix verbose mode display of stdin backup
+- extract: warn if a include pattern never matched, fixes #209,
+  implement counters for Include/ExcludePatterns
+- archive names with slashes are invalid, attic issue #180
+- chunker: add a check whether the POSIX_FADV_DONTNEED constant is defined -
+  fixes building on OpenBSD.
+
+Other changes:
+
+- detect inconsistency / corruption / hash collision, #170
+- replace versioneer with setuptools_scm, #106
+- docs:
+
+  - pkg-config is needed for llfuse installation
+  - be more clear about pruning, attic issue #132
+- unit tests:
+
+  - xattr: ignore security.selinux attribute showing up
+  - ext3 seems to need a bit more space for a sparse file
+  - do not test lzma level 9 compression (avoid MemoryError)
+  - work around strange mtime granularity issue on netbsd, fixes #204
+  - ignore st_rdev if file is not a block/char device, fixes #203
+  - stay away from the setgid and sticky mode bits
+- use Vagrant to do easy cross-platform testing (#196), currently:
+
+  - Debian 7 "wheezy" 32bit, Debian 8 "jessie" 64bit
+  - Ubuntu 12.04 32bit, Ubuntu 14.04 64bit
+  - Centos 7 64bit
+  - FreeBSD 10.2 64bit
+  - OpenBSD 5.7 64bit
+  - NetBSD 6.1.5 64bit
+  - Darwin (OS X Yosemite)
+
+
+Version 0.25.0
+--------------
+
+Compatibility notes:
+
+- lz4 compression library (liblz4) is a new requirement (#156)
+- the new compression code is very compatible: as long as you stay with zlib
+  compression, older borg releases will still be able to read data from a
+  repo/archive made with the new code (note: this is not the case for the
+  default "none" compression, use "zlib,0" if you want a "no compression" mode
+  that can be read by older borg). Also the new code is able to read repos and
+  archives made with older borg versions (for all zlib levels  0..9).
+
+Deprecations:
+
+- --compression N (with N being a number, as in 0.24) is deprecated.
+  We keep the --compression 0..9 for now to not break scripts, but it is
+  deprecated and will be removed later, so better fix your scripts now:
+  --compression 0 (as in 0.24) is the same as --compression zlib,0 (now).
+  BUT: if you do not want compression, you rather want --compression none
+  (which is the default).
+  --compression 1 (in 0.24) is the same as --compression zlib,1 (now)
+  --compression 9 (in 0.24) is the same as --compression zlib,9 (now)
+
+New features:
+
+- create --compression none (default, means: do not compress, just pass through
+  data "as is". this is more efficient than zlib level 0 as used in borg 0.24)
+- create --compression lz4 (super-fast, but not very high compression)
+- create --compression zlib,N (slower, higher compression, default for N is 6)
+- create --compression lzma,N (slowest, highest compression, default N is 6)
+- honor the nodump flag (UF_NODUMP) and do not backup such items
+- list --short just outputs a simple list of the files/directories in an archive
+
+Bug fixes:
+
+- fixed --chunker-params parameter order confusion / malfunction, fixes #154
+- close fds of segments we delete (during compaction)
+- close files which fell out the lrucache
+- fadvise DONTNEED now is only called for the byte range actually read, not for
+  the whole file, fixes #158.
+- fix issue with negative "all archives" size, fixes #165
+- restore_xattrs: ignore if setxattr fails with EACCES, fixes #162
+
+Other changes:
+
+- remove fakeroot requirement for tests, tests run faster without fakeroot
+  (test setup does not fail any more without fakeroot, so you can run with or
+  without fakeroot), fixes #151 and #91.
+- more tests for archiver
+- recover_segment(): don't assume we have an fd for segment
+- lrucache refactoring / cleanup, add dispose function, py.test tests
+- generalize hashindex code for any key length (less hardcoding)
+- lock roster: catch file not found in remove() method and ignore it
+- travis CI: use requirements file
+- improved docs:
+
+  - replace hack for llfuse with proper solution (install libfuse-dev)
+  - update docs about compression
+  - update development docs about fakeroot
+  - internals: add some words about lock files / locking system
+  - support: mention BountySource and for what it can be used
+  - theme: use a lighter green
+  - add pypi, wheel, dist package based install docs
+  - split install docs into system-specific preparations and generic instructions
+
+
+Version 0.24.0
+--------------
+
+Incompatible changes (compared to 0.23):
+
+- borg now always issues --umask NNN option when invoking another borg via ssh
+  on the repository server. By that, it's making sure it uses the same umask
+  for remote repos as for local ones. Because of this, you must upgrade both
+  server and client(s) to 0.24.
+- the default umask is 077 now (if you do not specify via --umask) which might
+  be a different one as you used previously. The default umask avoids that
+  you accidentally give access permissions for group and/or others to files
+  created by borg (e.g. the repository).
+
+Deprecations:
+
+- "--encryption passphrase" mode is deprecated, see #85 and #97.
+  See the new "--encryption repokey" mode for a replacement.
+
+New features:
+
+- borg create --chunker-params ... to configure the chunker, fixes #16
+  (attic #302, attic #300, and somehow also #41).
+  This can be used to reduce memory usage caused by chunk management overhead,
+  so borg does not create a huge chunks index/repo index and eats all your RAM
+  if you back up lots of data in huge files (like VM disk images).
+  See docs/misc/create_chunker-params.txt for more information.
+- borg info now reports chunk counts in the chunk index.
+- borg create --compression 0..9 to select zlib compression level, fixes #66
+  (attic #295).
+- borg init --encryption repokey (to store the encryption key into the repo),
+  fixes #85
+- improve at-end error logging, always log exceptions and set exit_code=1
+- LoggedIO: better error checks / exceptions / exception handling
+- implement --remote-path to allow non-default-path borg locations, #125
+- implement --umask M and use 077 as default umask for better security, #117
+- borg check: give a named single archive to it, fixes #139
+- cache sync: show progress indication
+- cache sync: reimplement the chunk index merging in C
+
+Bug fixes:
+
+- fix segfault that happened for unreadable files (chunker: n needs to be a
+  signed size_t), #116
+- fix the repair mode, #144
+- repo delete: add destroy to allowed rpc methods, fixes issue #114
+- more compatible repository locking code (based on mkdir), maybe fixes #92
+  (attic #317, attic #201).
+- better Exception msg if no Borg is installed on the remote repo server, #56
+- create a RepositoryCache implementation that can cope with >2GiB,
+  fixes attic #326.
+- fix Traceback when running check --repair, attic #232
+- clarify help text, fixes #73.
+- add help string for --no-files-cache, fixes #140
+
+Other changes:
+
+- improved docs:
+
+  - added docs/misc directory for misc. writeups that won't be included
+    "as is" into the html docs.
+  - document environment variables and return codes (attic #324, attic #52)
+  - web site: add related projects, fix web site url, IRC #borgbackup
+  - Fedora/Fedora-based install instructions added to docs
+  - Cygwin-based install instructions added to docs
+  - updated AUTHORS
+  - add FAQ entries about redundancy / integrity
+  - clarify that borg extract uses the cwd as extraction target
+  - update internals doc about chunker params, memory usage and compression
+  - added docs about development
+  - add some words about resource usage in general
+  - document how to backup a raw disk
+  - add note about how to run borg from virtual env
+  - add solutions for (ll)fuse installation problems
+  - document what borg check does, fixes #138
+  - reorganize borgbackup.github.io sidebar, prev/next at top
+  - deduplicate and refactor the docs / README.rst
+
+- use borg-tmp as prefix for temporary files / directories
+- short prune options without "keep-" are deprecated, do not suggest them
+- improved tox configuration
+- remove usage of unittest.mock, always use mock from pypi
+- use entrypoints instead of scripts, for better use of the wheel format and
+  modern installs
+- add requirements.d/development.txt and modify tox.ini
+- use travis-ci for testing based on Linux and (new) OS X
+- use coverage.py, pytest-cov and codecov.io for test coverage support
+
+I forgot to list some stuff already implemented in 0.23.0, here they are:
+
+New features:
+
+- efficient archive list from manifest, meaning a big speedup for slow
+  repo connections and "list <repo>", "delete <repo>", "prune" (attic #242,
+  attic #167)
+- big speedup for chunks cache sync (esp. for slow repo connections), fixes #18
+- hashindex: improve error messages
+
+Other changes:
+
+- explicitly specify binary mode to open binary files
+- some easy micro optimizations
+
+
+Version 0.23.0
+--------------
+
+Incompatible changes (compared to attic, fork related):
+
+- changed sw name and cli command to "borg", updated docs
+- package name (and name in urls) uses "borgbackup" to have less collisions
+- changed repo / cache internal magic strings from ATTIC* to BORG*,
+  changed cache location to .cache/borg/ - this means that it currently won't
+  accept attic repos (see issue #21 about improving that)
+
+Bug fixes:
+
+- avoid defect python-msgpack releases, fixes attic #171, fixes attic #185
+- fix traceback when trying to do unsupported passphrase change, fixes attic #189
+- datetime does not like the year 10.000, fixes attic #139
+- fix "info" all archives stats, fixes attic #183
+- fix parsing with missing microseconds, fixes attic #282
+- fix misleading hint the fuse ImportError handler gave, fixes attic #237
+- check unpacked data from RPC for tuple type and correct length, fixes attic #127
+- fix Repository._active_txn state when lock upgrade fails
+- give specific path to xattr.is_enabled(), disable symlink setattr call that
+  always fails
+- fix test setup for 32bit platforms, partial fix for attic #196
+- upgraded versioneer, PEP440 compliance, fixes attic #257
+
+New features:
+
+- less memory usage: add global option --no-cache-files
+- check --last N (only check the last N archives)
+- check: sort archives in reverse time order
+- rename repo::oldname newname (rename repository)
+- create -v output more informative
+- create --progress (backup progress indicator)
+- create --timestamp (utc string or reference file/dir)
+- create: if "-" is given as path, read binary from stdin
+- extract: if --stdout is given, write all extracted binary data to stdout
+- extract --sparse (simple sparse file support)
+- extra debug information for 'fread failed'
+- delete <repo> (deletes whole repo + local cache)
+- FUSE: reflect deduplication in allocated blocks
+- only allow whitelisted RPC calls in server mode
+- normalize source/exclude paths before matching
+- use posix_fadvise to not spoil the OS cache, fixes attic #252
+- toplevel error handler: show tracebacks for better error analysis
+- sigusr1 / sigint handler to print current file infos - attic PR #286
+- RPCError: include the exception args we get from remote
+
+Other changes:
+
+- source: misc. cleanups, pep8, style
+- docs and faq improvements, fixes, updates
+- cleanup crypto.pyx, make it easier to adapt to other AES modes
+- do os.fsync like recommended in the python docs
+- source: Let chunker optionally work with os-level file descriptor.
+- source: Linux: remove duplicate os.fsencode calls
+- source: refactor _open_rb code a bit, so it is more consistent / regular
+- source: refactor indicator (status) and item processing
+- source: use py.test for better testing, flake8 for code style checks
+- source: fix tox >=2.0 compatibility (test runner)
+- pypi package: add python version classifiers, add FreeBSD to platforms
+
+
+Attic Changelog
+===============
+
+Here you can see the full list of changes between each Attic release until Borg
+forked from Attic:
+
+Version 0.17
+------------
+
+(bugfix release, released on X)
+- Fix hashindex ARM memory alignment issue (#309)
+- Improve hashindex error messages (#298)
+
+Version 0.16
+------------
+
+(bugfix release, released on May 16, 2015)
+- Fix typo preventing the security confirmation prompt from working (#303)
+- Improve handling of systems with improperly configured file system encoding (#289)
+- Fix "All archives" output for attic info. (#183)
+- More user friendly error message when repository key file is not found (#236)
+- Fix parsing of iso 8601 timestamps with zero microseconds (#282)
+
+Version 0.15
+------------
+
+(bugfix release, released on Apr 15, 2015)
+- xattr: Be less strict about unknown/unsupported platforms (#239)
+- Reduce repository listing memory usage (#163).
+- Fix BrokenPipeError for remote repositories (#233)
+- Fix incorrect behavior with two character directory names (#265, #268)
+- Require approval before accessing relocated/moved repository (#271)
+- Require approval before accessing previously unknown unencrypted repositories (#271)
+- Fix issue with hash index files larger than 2GB.
+- Fix Python 3.2 compatibility issue with noatime open() (#164)
+- Include missing pyx files in dist files (#168)
+
+Version 0.14
+------------
+
+(feature release, released on Dec 17, 2014)
+
+- Added support for stripping leading path segments (#95)
+  "attic extract --strip-segments X"
+- Add workaround for old Linux systems without acl_extended_file_no_follow (#96)
+- Add MacPorts' path to the default openssl search path (#101)
+- HashIndex improvements, eliminates unnecessary IO on low memory systems.
+- Fix "Number of files" output for attic info. (#124)
+- limit create file permissions so files aren't read while restoring
+- Fix issue with empty xattr values (#106)
+
+Version 0.13
+------------
+
+(feature release, released on Jun 29, 2014)
+
+- Fix sporadic "Resource temporarily unavailable" when using remote repositories
+- Reduce file cache memory usage (#90)
+- Faster AES encryption (utilizing AES-NI when available)
+- Experimental Linux, OS X and FreeBSD ACL support (#66)
+- Added support for backup and restore of BSDFlags (OSX, FreeBSD) (#56)
+- Fix bug where xattrs on symlinks were not correctly restored
+- Added cachedir support. CACHEDIR.TAG compatible cache directories
+  can now be excluded using ``--exclude-caches`` (#74)
+- Fix crash on extreme mtime timestamps (year 2400+) (#81)
+- Fix Python 3.2 specific lockf issue (EDEADLK)
+
+Version 0.12
+------------
+
+(feature release, released on April 7, 2014)
+
+- Python 3.4 support (#62)
+- Various documentation improvements a new style
+- ``attic mount`` now supports mounting an entire repository not only
+  individual archives (#59)
+- Added option to restrict remote repository access to specific path(s):
+  ``attic serve --restrict-to-path X`` (#51)
+- Include "all archives" size information in "--stats" output. (#54)
+- Added ``--stats`` option to ``attic delete`` and ``attic prune``
+- Fixed bug where ``attic prune`` used UTC instead of the local time zone
+  when determining which archives to keep.
+- Switch to SI units (Power of 1000 instead 1024) when printing file sizes
+
+Version 0.11
+------------
+
+(feature release, released on March 7, 2014)
+
+- New "check" command for repository consistency checking (#24)
+- Documentation improvements
+- Fix exception during "attic create" with repeated files (#39)
+- New "--exclude-from" option for attic create/extract/verify.
+- Improved archive metadata deduplication.
+- "attic verify" has been deprecated. Use "attic extract --dry-run" instead.
+- "attic prune --hourly|daily|..." has been deprecated.
+  Use "attic prune --keep-hourly|daily|..." instead.
+- Ignore xattr errors during "extract" if not supported by the filesystem. (#46)
+
+Version 0.10
+------------
+
+(bugfix release, released on Jan 30, 2014)
+
+- Fix deadlock when extracting 0 sized files from remote repositories
+- "--exclude" wildcard patterns are now properly applied to the full path
+  not just the file name part (#5).
+- Make source code endianness agnostic (#1)
+
+Version 0.9
+-----------
+
+(feature release, released on Jan 23, 2014)
+
+- Remote repository speed and reliability improvements.
+- Fix sorting of segment names to ignore NFS left over files. (#17)
+- Fix incorrect display of time (#13)
+- Improved error handling / reporting. (#12)
+- Use fcntl() instead of flock() when locking repository/cache. (#15)
+- Let ssh figure out port/user if not specified so we don't override .ssh/config (#9)
+- Improved libcrypto path detection (#23).
+
+Version 0.8.1
+-------------
+
+(bugfix release, released on Oct 4, 2013)
+
+- Fix segmentation fault issue.
+
+Version 0.8
+-----------
+
+(feature release, released on Oct 3, 2013)
+
+- Fix xattr issue when backing up sshfs filesystems (#4)
+- Fix issue with excessive index file size (#6)
+- Support access of read only repositories.
+- New syntax to enable repository encryption:
+    attic init --encryption="none|passphrase|keyfile".
+- Detect and abort if repository is older than the cache.
+
+
+Version 0.7
+-----------
+
+(feature release, released on Aug 5, 2013)
+
+- Ported to FreeBSD
+- Improved documentation
+- Experimental: Archives mountable as fuse filesystems.
+- The "user." prefix is no longer stripped from xattrs on Linux
+
+
+Version 0.6.1
+-------------
+
+(bugfix release, released on July 19, 2013)
+
+- Fixed an issue where mtime was not always correctly restored.
+
+
+Version 0.6
+-----------
+
+First public release on July 9, 2013

+ 40 - 44
docs/development.rst

@@ -68,6 +68,11 @@ Now run::
 
 Then point a web browser at docs/_build/html/index.html.
 
+To update the web site, copy (and add, commit and push) the contents of the
+`_build` directory to the `borgbackup` directory in the web site's repository:
+https://github.com/borgbackup/borgbackup.github.io
+
+
 Using Vagrant
 -------------
 
@@ -91,63 +96,54 @@ Usage::
      vagrant scp OS:/vagrant/borg/borg/dist/borg .
 
 
-Creating a new release
-----------------------
-
-Checklist::
-
-- all issues for this milestone closed?
-- any low hanging fruit left on the issue tracker?
-- run tox on all supported platforms via vagrant, check for test fails.
-- is Travis CI happy also?
-- update CHANGES.rst (compare to git log). check version number of upcoming release.
-- check MANIFEST.in and setup.py - are they complete?
-- tag the release::
-
-  git tag -s -m "tagged release" 0.26.0
+Creating standalone binaries
+----------------------------
 
-- cd docs ; make html  # to update the usage include files
-- update website with the html
-- create a release on PyPi::
+Make sure you have everything built and installed (including llfuse and fuse).
+When using the Vagrant VMs, pyinstaller will already be installed.
 
-    python setup.py register sdist upload --identity="Thomas Waldmann" --sign
+With virtual env activated::
 
-- close release milestone.
-- announce on::
+  pip install pyinstaller>=3.0  # or git checkout master
+  pyinstaller -F -n borg-PLATFORM --hidden-import=logging.config borg/__main__.py
+  for file in dist/borg-*; do gpg --armor --detach-sign $file; done
 
-  - mailing list
-  - Twitter
-  - IRC channel (topic)
+If you encounter issues, see also our `Vagrantfile` for details.
 
-- create binary wheels and link them from issue tracker: https://github.com/borgbackup/borg/issues/147
-- create standalone binaries and link them from issue tracker: https://github.com/borgbackup/borg/issues/214
+.. note:: Standalone binaries built with pyinstaller are supposed to
+          work on same OS, same architecture (x86 32bit, amd64 64bit)
+          without external dependencies.
 
 
-Creating binary wheels
+Creating a new release
 ----------------------
 
-With virtual env activated::
-
-    pip install -U wheel
-    python setup.py bdist_wheel
-    ls -l dist/*.whl
-
-Note: Binary wheels are rather specific for the platform they get built on.
-      E.g. a wheel built for Ubuntu 14.04 64bit likely will not work on Centos7 64bit.
+Checklist:
 
+- make sure all issues for this milestone are closed or moved to the
+  next milestone
+- find and fix any low hanging fruit left on the issue tracker
+- run tox on all supported platforms via vagrant, check for test failures
+- check that Travis CI is also happy
+- update ``CHANGES.rst``, based on ``git log $PREVIOUS_RELEASE..``
+- check version number of upcoming release in ``CHANGES.rst``
+- verify that ``MANIFEST.in`` and ``setup.py`` are complete
+- tag the release::
 
-Creating standalone binaries
-----------------------------
+    git tag -s -m "tagged/signed release X.Y.Z" X.Y.Z
 
-Make sure you have everything built and installed (including llfuse and fuse).
+- build fresh docs and update the web site with them
+- create a release on PyPi::
 
-With virtual env activated::
+    python setup.py register sdist upload --identity="Thomas Waldmann" --sign
 
-  pip install pyinstaller==3.0.dev2  # or a later 3.x release or git checkout
-  pyinstaller -F -n borg-PLATFORM --hidden-import=logging.config borg/__main__.py
-  ls -l dist/*
+- close release milestone on Github
+- announce on::
 
-If you encounter issues, see also our `Vagrantfile` for details.
+ - `mailing list <mailto:borgbackup@librelist.org>`_
+ - Twitter (follow @ThomasJWaldmann for these tweets)
+ - `IRC channel <irc://irc.freenode.net/borgbackup>`_ (change ``/topic``
 
-Note: Standalone binaries built with pyinstaller are supposed to work on same OS,
-      same architecture (x86 32bit, amd64 64bit) without external dependencies.
+- create a Github release, include:
+  * standalone binaries (see above for how to create them)
+  * a link to ``CHANGES.rst``

+ 3 - 1
docs/index.rst

@@ -4,10 +4,11 @@
 Borg Documentation
 ==================
 
+.. include:: ../README.rst
+
 .. toctree::
    :maxdepth: 2
 
-   intro
    installation
    quickstart
    usage
@@ -16,4 +17,5 @@ Borg Documentation
    changes
    internals
    development
+   authors
    api

+ 3 - 35
docs/installation.rst

@@ -6,7 +6,7 @@ Installation
 
 |project_name| pyinstaller binary installation requires:
 
-* Linux: glibc >= 2.12 (ok for most supported Linux releases)
+* Linux: glibc >= 2.13 (ok for most supported Linux releases)
 * MacOS X: 10.10 (unknown whether it works for older releases)
 * FreeBSD: 10.2 (unknown whether it works for older releases)
 
@@ -29,11 +29,8 @@ Below, we describe different ways to install |project_name|.
   binary package (for your Linux/*BSD/OS X/... distribution).
 - **pyinstaller binary** - easy and fast, we provide a ready-to-use binary file
   that just works on the supported platforms
-- **wheel** - easy and fast, needs a platform specific borgbackup binary wheel,
-  which matches your platform [OS and CPU]).
 - **pypi** - installing a source package from pypi needs more installation steps
-  and will compile stuff - try this if there is no binary wheel that works for
-  you.
+  and will need a compiler, development headers, etc..
 - **git** - for developers and power users who want to have the latest code or
   use revision control (each release is tagged).
 
@@ -88,36 +85,7 @@ For some platforms we offer a ready-to-use standalone borg binary.
 
 It is supposed to work without requiring installation or preparations.
 
-Check https://github.com/borgbackup/borg/issues/214 for available binaries.
-
-
-Debian Jessie / Ubuntu 14.04 preparations (wheel)
--------------------------------------------------
-
-.. parsed-literal::
-
-    # Python stuff we need
-    apt-get install python3 python3-pip
-
-    # Libraries we need (fuse is optional)
-    apt-get install openssl libacl1 liblz4-1 fuse
-
-
-Installation (wheel)
---------------------
-
-This uses the latest binary wheel release.
-
-.. parsed-literal::
-
-    # Check https://github.com/borgbackup/borg/issues/147 for the correct
-    # platform-specific binary wheel, download and install it:
-
-    # system-wide installation, needs sudo/root permissions:
-    sudo pip install borgbackup.whl
-
-    # home directory installation, no sudo/root needed:
-    pip install --user borgbackup.whl
+Check https://github.com/borgbackup/borg/releases for available binaries.
 
 
 Debian Jessie / Ubuntu 14.04 preparations (git/pypi)

+ 0 - 7
docs/intro.rst

@@ -1,7 +0,0 @@
-.. include:: global.rst.inc
-.. _foreword:
-
-Introduction
-============
-
-.. include:: ../README.rst

+ 8 - 0
docs/usage.rst

@@ -48,6 +48,8 @@ General:
         can either leave it away or abbreviate as `::`, if a positional parameter is required.
     BORG_PASSPHRASE
         When set, use the value to answer the passphrase question for encrypted repositories.
+    BORG_RSH
+        When set, use this command instead of ``ssh``.
     TMPDIR
         where temporary files are stored (might need a lot of temporary space for some operations)
 
@@ -58,6 +60,12 @@ Some "yes" sayers (if set, they automatically confirm that you really want to do
         For "Warning: The repository at location ... was previously located at ..."
     BORG_CHECK_I_KNOW_WHAT_I_AM_DOING
         For "Warning: 'check --repair' is an experimental feature that might result in data loss."
+    BORG_CYTHON_DISABLE
+        Disables the loading of Cython modules. This is currently
+        experimental and is used only to generate usage docs at build
+        time. It is unlikely to produce good results on a regular
+        run. The variable should be set to the name of the  calling class, and
+        should be unique across all of borg. It is currently only used by ``build_usage``.
 
 Directories:
     BORG_KEYS_DIR

+ 133 - 12
setup.py

@@ -1,8 +1,15 @@
 # -*- encoding: utf-8 *-*
 import os
+import re
 import sys
 from glob import glob
 
+from distutils.command.build import build
+from distutils.core import Command
+from distutils.errors import DistutilsOptionError
+from distutils import log
+from setuptools.command.build_py import build_py
+
 min_python = (3, 2)
 my_python = sys.version_info
 
@@ -10,6 +17,9 @@ if my_python < min_python:
     print("Borg requires Python %d.%d or later" % min_python)
     sys.exit(1)
 
+# Are we building on ReadTheDocs?
+on_rtd = os.environ.get('READTHEDOCS')
+
 # msgpack pure python data corruption was fixed in 0.4.6.
 # Also, we might use some rather recent API features.
 install_requires=['msgpack-python>=0.4.6', ]
@@ -62,7 +72,7 @@ except ImportError:
     platform_freebsd_source = platform_freebsd_source.replace('.pyx', '.c')
     platform_darwin_source = platform_darwin_source.replace('.pyx', '.c')
     from distutils.command.build_ext import build_ext
-    if not all(os.path.exists(path) for path in [
+    if not on_rtd and not all(os.path.exists(path) for path in [
         compress_source, crypto_source, chunker_source, hashindex_source,
         platform_linux_source, platform_freebsd_source]):
         raise ImportError('The GIT version of Borg needs Cython. Install Cython or use a released version.')
@@ -103,29 +113,140 @@ possible_lz4_prefixes = ['/usr', '/usr/local', '/usr/local/opt/lz4', '/usr/local
 if os.environ.get('BORG_LZ4_PREFIX'):
     possible_openssl_prefixes.insert(0, os.environ.get('BORG_LZ4_PREFIX'))
 lz4_prefix = detect_lz4(possible_lz4_prefixes)
-if not lz4_prefix:
+if lz4_prefix:
+    include_dirs.append(os.path.join(lz4_prefix, 'include'))
+    library_dirs.append(os.path.join(lz4_prefix, 'lib'))
+elif not on_rtd:
     raise Exception('Unable to find LZ4 headers. (Looked here: {})'.format(', '.join(possible_lz4_prefixes)))
-include_dirs.append(os.path.join(lz4_prefix, 'include'))
-library_dirs.append(os.path.join(lz4_prefix, 'lib'))
 
 
 with open('README.rst', 'r') as fd:
     long_description = fd.read()
 
-cmdclass = {'build_ext': build_ext, 'sdist': Sdist}
+class build_usage(Command):
+    description = "generate usage for each command"
+
+    user_options = [
+        ('output=', 'O', 'output directory'),
+    ]
+    def initialize_options(self):
+        pass
+
+    def finalize_options(self):
+        pass
+
+    def run(self):
+        print('generating usage docs')
+        # allows us to build docs without the C modules fully loaded during help generation
+        if 'BORG_CYTHON_DISABLE' not in os.environ:
+            os.environ['BORG_CYTHON_DISABLE'] = self.__class__.__name__
+        from borg.archiver import Archiver
+        parser = Archiver().build_parser(prog='borg')
+        choices = {}
+        for action in parser._actions:
+            if action.choices is not None:
+                choices.update(action.choices)
+        print('found commands: %s' % list(choices.keys()))
+        if not os.path.exists('docs/usage'):
+            os.mkdir('docs/usage')
+        for command, parser in choices.items():
+            if command is 'help':
+                continue
+            with open('docs/usage/%s.rst.inc' % command, 'w') as doc:
+                print('generating help for %s' % command)
+                params = {"command": command,
+                          "underline": '-' * len('borg ' + command)}
+                doc.write(".. _borg_{command}:\n\n".format(**params))
+                doc.write("borg {command}\n{underline}\n::\n\n".format(**params))
+                epilog = parser.epilog
+                parser.epilog = None
+                doc.write(re.sub("^", "    ", parser.format_help(), flags=re.M))
+                doc.write("\nDescription\n~~~~~~~~~~~\n")
+                doc.write(epilog)
+        # return to regular Cython configuration, if we changed it
+        if os.environ.get('BORG_CYTHON_DISABLE') == self.__class__.__name__:
+            del os.environ['BORG_CYTHON_DISABLE']
+
+
+class build_api(Command):
+    description = "generate a basic api.rst file based on the modules available"
+
+    user_options = [
+        ('output=', 'O', 'output directory'),
+    ]
+    def initialize_options(self):
+        pass
+
+    def finalize_options(self):
+        pass
+
+    def run(self):
+        print("auto-generating API documentation")
+        with open("docs/api.rst", "w") as doc:
+            doc.write("""
+Borg Backup API documentation"
+=============================
+""")
+            for mod in glob('borg/*.py') + glob('borg/*.pyx'):
+                print("examining module %s" % mod)
+                if "/_" not in mod:
+                    doc.write("""
+.. automodule:: %s
+    :members:
+    :undoc-members:
+""" % mod)
+
+# (function, predicate), see http://docs.python.org/2/distutils/apiref.html#distutils.cmd.Command.sub_commands
+# seems like this doesn't work on RTD, see below for build_py hack.
+build.sub_commands.append(('build_api', None))
+build.sub_commands.append(('build_usage', None))
+
+
+class build_py_custom(build_py):
+    """override build_py to also build our stuff
+
+    it is unclear why this is necessary, but in some environments
+    (Readthedocs.org, specifically), the above
+    ``build.sub_commands.append()`` doesn't seem to have an effect:
+    our custom build commands seem to be ignored when running
+    ``setup.py install``.
+
+    This class overrides the ``build_py`` target by forcing it to run
+    our custom steps as well.
+
+    See also the `bug report on RTD
+    <https://github.com/rtfd/readthedocs.org/issues/1740>`_.
+    """
+    def run(self):
+        super().run()
+        self.announce('calling custom build steps', level=log.INFO)
+        self.run_command('build_ext')
+        self.run_command('build_api')
+        self.run_command('build_usage')
+
+
+cmdclass = {
+    'build_ext': build_ext,
+    'build_api': build_api,
+    'build_usage': build_usage,
+    'build_py': build_py_custom,
+    'sdist': Sdist
+}
 
-ext_modules = [
+ext_modules = []
+if not on_rtd:
+    ext_modules += [
     Extension('borg.compress', [compress_source], libraries=['lz4'], include_dirs=include_dirs, library_dirs=library_dirs),
     Extension('borg.crypto', [crypto_source], libraries=['crypto'], include_dirs=include_dirs, library_dirs=library_dirs),
     Extension('borg.chunker', [chunker_source]),
     Extension('borg.hashindex', [hashindex_source])
 ]
-if sys.platform.startswith('linux'):
-    ext_modules.append(Extension('borg.platform_linux', [platform_linux_source], libraries=['acl']))
-elif sys.platform.startswith('freebsd'):
-    ext_modules.append(Extension('borg.platform_freebsd', [platform_freebsd_source]))
-elif sys.platform == 'darwin':
-    ext_modules.append(Extension('borg.platform_darwin', [platform_darwin_source]))
+    if sys.platform.startswith('linux'):
+        ext_modules.append(Extension('borg.platform_linux', [platform_linux_source], libraries=['acl']))
+    elif sys.platform.startswith('freebsd'):
+        ext_modules.append(Extension('borg.platform_freebsd', [platform_freebsd_source]))
+    elif sys.platform == 'darwin':
+        ext_modules.append(Extension('borg.platform_darwin', [platform_darwin_source]))
 
 setup(
     name='borgbackup',