deduplicate and refactor the docs

README.rst (shown on github and also at the start of the html docs) shall
be like an elevator speech - convince readers in a very short time.
this is most important, everything else can come after we got the reader's interest.

include README into docs to avoid duplication.
also include CHANGES into docs.
add developer docs, move examples from tox.ini there
add separate support docs
remove glossary, most of what was there can be understood by an admin from context
move attic and compatibility note to the end

README.rst

@@ -1,52 +1,31 @@
-|build|
-
 What is Borg?
 -------------
-Borg is a deduplicating backup program. The main goal of Borg is to provide
-an efficient and secure way to backup data. The data deduplication
-technique used makes Borg suitable for daily backups since only changes
-are stored.
-
-Borg is a fork of `Attic <https://github.com/jborg/attic>`_ and maintained by "`The Borg Collective <https://github.com/borgbackup/borg/blob/master/AUTHORS>`_".
-
-BORG IS NOT COMPATIBLE WITH ORIGINAL ATTIC.
-EXPECT THAT WE WILL BREAK COMPATIBILITY REPEATEDLY WHEN MAJOR RELEASE NUMBER
-CHANGES (like when going from 0.x.y to 1.0.0). Please read CHANGES document.
-
-NOT RELEASED DEVELOPMENT VERSIONS HAVE UNKNOWN COMPATIBILITY PROPERTIES.
-
-THIS IS SOFTWARE IN DEVELOPMENT, DECIDE YOURSELF WHETHER IT FITS YOUR NEEDS.
+Borg is a deduplicating backup program.
+Optionally, it also supports compression and authenticated encryption.
 
-Read `issue #1 <https://github.com/borgbackup/borg/issues/1>`_ on the issue tracker, goals are being defined there.
-
-Please also see the `LICENSE  <https://github.com/borgbackup/borg/blob/master/LICENSE>`_ for more informations.
-
-Easy to use
-~~~~~~~~~~~
-Initialize backup repository and create a backup archive::
-
-    $ borg init /mnt/backup
-    $ borg create -v /mnt/backup::documents ~/Documents
-
-For a graphical frontend refer to our complementary project `BorgWeb <https://github.com/borgbackup/borgweb>`_.
+The main goal of Borg is to provide an efficient and secure way to backup data.
+The data deduplication technique used makes Borg suitable for daily backups
+since only changes are stored.
+The authenticated encryption technique makes it suitable for backups to not
+fully trusted targets.
 
 Main features
 ~~~~~~~~~~~~~
 Space efficient storage
-  Variable block size deduplication is used to reduce the number of bytes 
+  Variable block size deduplication is used to reduce the number of bytes
   stored by detecting redundant data. Each file is split into a number of
   variable length chunks and only chunks that have never been seen before are
-  compressed and added to the repository.
+  added to the repository.
 
   The content-defined chunking based deduplication is applied to remove
-  duplicate chunks within: 
+  duplicate chunks within:
 
   * the current backup data set (even inside single files / streams)
   * current and previous backups of same machine
   * all the chunks in the same repository, even if coming from other machines
 
   This advanced deduplication method does NOT depend on:
- 
+
   * file/directory names staying the same (so you can move your stuff around
     without killing the deduplication, even between machines sharing a repo)
   * complete files or time stamps staying the same (if a big file changes a
@@ -59,37 +38,84 @@ Optional data encryption
     All data can be protected using 256-bit AES encryption and data integrity
     and authenticity is verified using HMAC-SHA256.
 
+Optional compression
+    All data can be compressed (by zlib, level 0-9).
+
 Off-site backups
     Borg can store data on any remote host accessible over SSH.  This is
-    most efficient if Borg is also installed on the remote host.
+    most efficient if Borg is also installed on the remote host. If you can't
+    install Borg there, you can also use some network filesystem (sshfs, nfs,
+    ...), but it will be less efficient.
 
 Backups mountable as filesystems
     Backup archives are mountable as userspace filesystems for easy backup
     verification and restores.
 
-What do I need?
----------------
-Borg requires Python 3.2 or above to work.
-Borg also requires a sufficiently recent OpenSSL (>= 1.0.0).
-In order to mount archives as filesystems, llfuse is required.
+Platforms Borg works on
+  * Linux
+  * FreeBSD
+  * Mac OS X
+  * Cygwin (unsupported)
+
+
+Easy to use
+~~~~~~~~~~~
+Initialize a new backup repository and create a backup archive::
+
+    $ borg init /mnt/backup
+    $ borg create /mnt/backup::Monday ~/Documents
+
+Now doing another backup, just to show off the great deduplication::
+
+    $ borg create --stats /mnt/backup::Tuesday ~/Documents
+
+    Archive name: Tuesday
+    Archive fingerprint: 387a5e3f9b0e792e91ce87134b0f4bfe17677d9248cb5337f3fbf3a8e157942a
+    Start time: Tue Mar 25 12:00:10 2014
+    End time:   Tue Mar 25 12:00:10 2014
+    Duration: 0.08 seconds
+    Number of files: 358
+                           Original size      Compressed size    Deduplicated size
+    This archive:               57.16 MB             46.78 MB            151.67 kB
+    All archives:              114.02 MB             93.46 MB             44.81 MB
+
+For a graphical frontend refer to our complementary project
+`BorgWeb <https://github.com/borgbackup/borgweb>`_.
+
+
+How to proceed from here
+------------------------
+Everything about requirements, installation, getting a quick start, usage
+reference, FAQ, support info, internals and developer infos is in our
+documentation:
+
+See `our online documentation <https://borgbackup.github.io/>`_
+or alternatively read it in raw text form in the `docs/*.rst` files.
 
-How do I install it?
---------------------
-::
 
-  $ pip3 install borgbackup
+Notes
+-----
+
+Build status:
+|build|
+
+Borg is a fork of `Attic <https://github.com/jborg/attic>`_ and maintained by
+"`The Borg Collective <https://github.com/borgbackup/borg/blob/master/AUTHORS>`_".
+
+BORG IS NOT COMPATIBLE WITH ORIGINAL ATTIC.
+EXPECT THAT WE WILL BREAK COMPATIBILITY REPEATEDLY WHEN MAJOR RELEASE NUMBER
+CHANGES (like when going from 0.x.y to 1.0.0). Please read CHANGES document.
+
+NOT RELEASED DEVELOPMENT VERSIONS HAVE UNKNOWN COMPATIBILITY PROPERTIES.
+
+THIS IS SOFTWARE IN DEVELOPMENT, DECIDE YOURSELF WHETHER IT FITS YOUR NEEDS.
 
-Where are the docs?
--------------------
-Go to https://borgbackup.github.io/ for a prebuilt version of the documentation.
-You can also build it yourself from the docs folder.
+Read `issue #1 <https://github.com/borgbackup/borg/issues/1>`_ on the issue
+tracker, goals are being defined there.
 
-Where are the tests?
---------------------
-The tests are in the borg/testsuite package. To run the test suite use the
-following command::
+For more information, please also see the
+`LICENSE  <https://github.com/borgbackup/borg/blob/master/LICENSE>`_.
 
-  $ fakeroot -u tox  # you need to have tox and pytest installed
 
 .. |build| image:: https://travis-ci.org/borgbackup/borg.svg
         :alt: Build Status

docs/changes.rst

@@ -0,0 +1,4 @@
+.. include:: global.rst.inc
+.. _changelog:
+
+.. include:: ../CHANGES.rst

docs/development.rst

@@ -0,0 +1,67 @@
+.. include:: global.rst.inc
+.. _development:
+
+Development
+===========
+
+This chapter will get you started with |project_name|' development.
+
+|project_name| is written in Python (with a little bit of Cython and C for
+the performance critical parts).
+
+
+Building a development environment
+----------------------------------
+
+First, just install borg into a virtual env as described before.
+
+To install some additional packages needed for running the tests, activate your
+virtual env and run::
+
+  pip install -r requirements.d/development.txt
+
+
+Running the tests
+-----------------
+
+The tests are in the borg/testsuite package.
+
+To run them, you need to have fakeroot, tox and pytest installed.
+
+To run the test suite use the following command::
+
+  fakeroot -u tox  # run all tests
+
+Some more advanced examples::
+
+  # verify a changed tox.ini (run this after any change to tox.ini):
+  fakeroot -u tox --recreate
+
+  fakeroot -u tox -e py32  # run all tests, but only on python 3.2
+
+  fakeroot -u tox borg.testsuite.locking  # only run 1 test module
+
+  fakeroot -u tox borg.testsuite.locking -- -k '"not Timer"'  # exclude some tests
+
+  fakeroot -u tox borg.testsuite -- -v  # verbose py.test
+
+Important notes:
+
+- Without fakeroot -u some tests will fail.
+- When using -- to give options to py.test, you MUST also give borg.testsuite[.module].
+
+Building the docs with Sphinx
+-----------------------------
+
+The documentation (in reStructuredText format, .rst) is in docs/.
+
+To build the html version of it, you need to have sphinx installed::
+
+  pip3 install sphinx
+
+Now run::
+
+  cd docs/
+  make html
+
+Then point a web browser at docs/_build/html/index.html.

docs/foreword.rst

@@ -1,65 +0,0 @@
-.. include:: global.rst.inc
-.. _foreword:
-
-Foreword
-========
-
-|project_name| is a secure backup program for Linux, FreeBSD and Mac OS X. 
-|project_name| is designed for efficient data storage where only new or
-modified data is stored.
-
-Features
---------
-
-Space efficient storage
-    Variable block size `deduplication`_ is used to reduce the number of bytes 
-    stored by detecting redundant data. Each file is split into a number of
-    variable length chunks and only chunks that have never been seen before
-    are added to the repository (and optionally compressed).
-
-Optional data encryption
-    All data can be protected using 256-bit AES_ encryption and data integrity
-    and authenticity is verified using `HMAC-SHA256`_.
-
-Off-site backups
-    |project_name| can store data on any remote host accessible over SSH as
-    long as |project_name| is installed. If you don't have |project_name|
-    installed there, you can use some network filesytem (sshfs, nfs, ...)
-    to mount a filesystem located on your remote host and use it like it was
-    local (but that will be slower).
-
-Backups mountable as filesystems
-    Backup archives are :ref:`mountable <borg_mount>` as
-    `userspace filesystems`_ for easy backup verification and restores.
-
-
-Glossary
---------
-
-.. _deduplication_def:
-
-Deduplication
-    Deduplication is a technique for improving storage utilization by
-    eliminating redundant data. 
-
-.. _archive_def:
-
-Archive
-    An archive is a collection of files along with metadata that include file
-    permissions, directory structure and various file attributes.
-    Since each archive in a repository must have a unique name a good naming
-    convention is ``hostname-YYYY-MM-DD``.
-
-.. _repository_def:
-
-Repository
-    A repository is a filesystem directory storing data from zero or more
-    archives. The data in a repository is both deduplicated and 
-    optionally encrypted making it both efficient and safe. Repositories are
-    created using :ref:`borg_init` and the contents can be listed using
-    :ref:`borg_list`.
-
-Key file
-    When a repository is initialized a key file containing a password
-    protected encryption key is created. It is vital to keep this file safe
-    since the repository data is totally inaccessible without it.

docs/index.rst

@@ -1,81 +1,18 @@
 .. include:: global.rst.inc
 
-Welcome to Borg
-================
-|project_name| is a deduplicating backup program.
-Optionally, it also supports compression and authenticated encryption.
 
-The main goal of |project_name| is to provide an efficient and secure way
-to backup data. The data deduplication technique used makes |project_name|
-suitable for daily backups since only the changes are stored. The authenticated
-encryption makes it suitable for backups to not fully trusted targets.
-
-|project_name| is written in Python (with a little bit of Cython and C for
-the performance critical parts).
-
-
-Easy to use
------------
-Initialize a new backup :ref:`repository <repository_def>` and create your
-first backup :ref:`archive <archive_def>` in two lines::
-
-    $ borg init /mnt/backup
-    $ borg create /mnt/backup::Monday ~/Documents
-    $ borg create --stats /mnt/backup::Tuesday ~/Documents
-    Archive name: Tuesday
-    Archive fingerprint: 387a5e3f9b0e792e91ce87134b0f4bfe17677d9248cb5337f3fbf3a8e157942a
-    Start time: Tue Mar 25 12:00:10 2014
-    End time:   Tue Mar 25 12:00:10 2014
-    Duration: 0.08 seconds
-    Number of files: 358
-                           Original size      Compressed size    Deduplicated size
-    This archive:               57.16 MB             46.78 MB            151.67 kB
-    All archives:              114.02 MB             93.46 MB             44.81 MB
-
-See the :ref:`quickstart` chapter for a more detailed example.
-
-Easy installation
------------------
-You can use pip to install |project_name| quickly and easily::
-
-    $ pip3 install borgbackup
-
-Need more help with installing? See :ref:`installation`.
-
-User's Guide
-============
+Borg Documentation
+==================
 
 .. toctree::
    :maxdepth: 2
 
-   foreword
+   intro
    installation
    quickstart
    usage
    faq
+   support
+   changes
    internals
-
-Getting help
-============
-
-If you've found a bug or have a concrete feature request, please create a new
-ticket on the project's `issue tracker`_ (after checking whether someone else
-already has reported the same thing).
-
-For more general questions or discussions, IRC or mailing list are preferred.
-
-IRC
----
-Join us on channel #borgbackup on chat.freenode.net. As usual on IRC, just
-ask or tell directly and then patiently wait for replies. Stay connected.
-
-Mailing list
-------------
-
-There is a mailing list for Borg on librelist_ that you can use for feature
-requests and general discussions about Borg. A mailing list archive is
-available `here <http://librelist.com/browser/borgbackup/>`_.
-
-To subscribe to the list, send an email to borgbackup@librelist.com and reply
-to the confirmation mail. Likewise, to unsubscribe, send an email to 
-borgbackup-unsubscribe@librelist.com and reply to the confirmation mail.
+   development

docs/intro.rst

@@ -0,0 +1,7 @@
+.. include:: global.rst.inc
+.. _foreword:
+
+Introduction
+============
+
+.. include:: ../README.rst

docs/support.rst

@@ -0,0 +1,34 @@
+.. include:: global.rst.inc
+.. _support:
+
+Support
+=======
+
+Issue Tracker
+-------------
+
+If you've found a bug or have a concrete feature request, please create a new
+ticket on the project's `issue tracker`_ (after checking whether someone else
+already has reported the same thing).
+
+For more general questions or discussions, IRC or mailing list are preferred.
+
+IRC
+---
+Join us on channel #borgbackup on chat.freenode.net.
+
+As usual on IRC, just ask or tell directly and then patiently wait for replies.
+Stay connected.
+
+Mailing list
+------------
+
+There is a mailing list for Borg on librelist_ that you can use for feature
+requests and general discussions about Borg. A mailing list archive is
+available `here <http://librelist.com/browser/borgbackup/>`_.
+
+To subscribe to the list, send an email to borgbackup@librelist.com and reply
+to the confirmation mail.
+
+To unsubscribe, send an email to borgbackup-unsubscribe@librelist.com and reply
+to the confirmation mail.

tox.ini

@@ -1,16 +1,5 @@
 # tox configuration - if you change anything here, run this to verify:
 # fakeroot -u tox --recreate
-#
-# Invokation examples:
-# fakeroot -u tox  # run all tests
-# fakeroot -u tox -e py32  # run all tests, but only on python 3.2
-# fakeroot -u tox borg.testsuite.locking  # only run 1 test module
-# fakeroot -u tox borg.testsuite.locking -- -k '"not Timer"'  # exclude some tests
-# fakeroot -u tox borg.testsuite -- -v  # verbose py.test
-#
-# Important notes:
-# Without fakeroot -u some tests will fail.
-# When using -- to give options to py.test, you MUST also give borg.testsuite[.module].
 
 [tox]
 envlist = py32, py33, py34