Docs improvements

docs/Makefile

@@ -137,3 +137,5 @@ gh-pages: html
 	(cd $$GH_PAGES_CLONE && git add -A && git commit -m 'Updated gh-pages' && git push) && \
 	rm -rf $$GH_PAGES_CLONE
 
+inotify:
+	while inotifywait -r .; do make html ; done

docs/conf.py

@@ -91,7 +91,7 @@ pygments_style = 'sphinx'
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-html_theme = 'default'
+html_theme = 'nature'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the

docs/definitions.rst

@@ -0,0 +1,31 @@
+.. _definitions:
+.. include:: global.rst.inc
+
+Definitions
+===========
+
+.. _deduplication_def:
+
+Deduplication
+    Deduplication is a technique for improving storage utilization by eliminating
+    redundant data. 
+
+.. _archive_def:
+
+Archive
+    An archive is a collection of files along with metadata that include file
+    permissions, directory structure and various file attributes.
+    Since each archive in a repository must have a unique name a good naming
+    convention is ``hostname-YYYY-MM-DD``.
+
+.. _repository_def:
+
+Repository
+    A repository is a filesystem directory storing data from zero or more archives.
+    The data in a repository is both deduplicated and encrypted making it both 
+    efficient and safe.
+
+Key file
+    When a repository is initialized a key file containing a password protected
+    encryption key is created. It is vital to keep this file safe since the repository
+    data is totally inaccessible without it.

docs/faq.rst

@@ -0,0 +1,32 @@
+.. _faq:
+.. include:: global.rst.inc
+
+Frequently asked questions
+==========================
+
+Which platforms are supported?
+------------------------------
+
+Currently Linux and MacOS X are supported.
+
+Can I backup VM disk images?
+----------------------------
+
+Yes, the :ref:`deduplication <deduplication_def>` technique used by darc
+will make sure only the modified parts of the file is stored.
+
+Which file attributes are preserved?
+------------------------------------
+
+The following attributes are preserved:
+
+* Name
+* Contents
+* Time of last modification (nanosecond precision with Python >= 3.3)
+* User ID of owner
+* Group ID of owner
+* Unix Permission
+* Extended attributes (xattrs)
+
+.. Note::
+    POSIX Access Control Lists (ACL_) are not yet preserved.

docs/generalusage.rst

@@ -0,0 +1,91 @@
+.. include:: global.rst.inc
+.. _generalusage:
+
+General Usage
+=============
+
+The following examples showcases how to use |project_name| to backup some important files from a users home directory.
+
+Initialize a local :ref:`repository <repository_def>` to store backup :ref:`archives <archive_def>` in
+(See :ref:`encrypted_repos` and :ref:`remote_repos` for more details)::
+
+    $ darc init /somewhere/my-backup.darc
+
+Create an archive containing the ``~/src`` and ``~/Documents`` directories::
+
+    $ darc create -v /somwhere/my-backup.darc::first-backup ~/src ~/Documents
+
+Create another archive the next day. This backup will be a lot quicker since only new data is stored.
+The ``--stats`` option tells |project_name| to print statistics about the newly created archive such as
+the amount of unique data (not shared with other archives)::
+
+    $ darc create -v --stats /somwhere/my-backup.darc::second-backup ~/src ~/Documents
+
+List all archives in the repository::
+
+    $ darc list /somewhere/my-backup.darc
+
+List the files in the *first-backup* archive::
+
+    $ darc list /somewhere/my-backup.darc::first-backup
+
+Restore the *first-backup* archive::
+
+    $ darc extract -v /somwhere/my-backup.darc::first-backup
+
+Recover disk space by manually deleting the *first-backup* archive::
+
+    $ darc delete /somwhere/my-backup.darc::first-backup
+
+Use the ``prune`` subcommand to delete all archives except a given number of *daily*, *weekly*, 
+*monthly* and *yearly* archives::
+
+    $ darc prune /somwhere/my-backup.darc --daily=7 --weekly=2 --monthly=6
+
+
+.. _encrypted_repos:
+
+Repository encryption
+---------------------
+
+Repository encryption is enabled at repository encryption time::
+
+    $ darc init --passphrase | --key-file
+
+When repository encryption is enabled all data is encrypted using 256-bit
+AES_ encryption and the integrity and authenticity is verified using
+`HMAC-SHA256`_.
+
+|project_name| supports two different methods to derive the AES and HMAC keys.
+
+Passphrase based encryption
+    This method uses a user supplied passphrase to derive the keys using the
+    PBKDF2_ key derivation function. This method is convenient to use and
+    secure as long as a *strong* passphrase is used.
+
+Key file based encryption
+    This method generates random keys at repository initialization time 
+    that are stored in a password protected file in the ``~/.darc/keys/``
+    directory.
+    This method is secure and suitable for automated backups.
+
+    .. Note::
+        The repository data is totally inaccessible without the key file
+        so it must be kept **safe**.
+
+
+.. _remote_repos:
+
+Remote repositories
+-------------------
+
+|project_name| can initialize and access repositories on remote hosts as the host
+is accessible using SSH and |project_name| is installed.
+
+The following syntax is used to address remote repositories::
+
+  $ darc init user@hostname:repository.darc
+
+or::
+
+  $ darc init ssh://user@hostname:port/repository.darc

docs/global.rst.inc

@@ -0,0 +1,7 @@
+.. |project_name| replace:: ``darc``
+.. _deduplication: https://en.wikipedia.org/wiki/Data_deduplication
+.. _AES: https://en.wikipedia.org/wiki/Advanced_Encryption_Standard
+.. _HMAC-SHA256: http://en.wikipedia.org/wiki/HMAC
+.. _PBKDF2: https://en.wikipedia.org/wiki/PBKDF2
+.. _ACL: https://en.wikipedia.org/wiki/Access_control_list
+.. _github: https://github.com/jborg/darc

docs/index.rst

@@ -1,150 +1,58 @@
-Welcome to Darc's documentation!
-================================
+.. include:: global.rst.inc
 
-Darc is a Deduplicating ARChiver written in Python.
-The main goal of Darc is to provide an efficient and secure way to backup data.
+Darc
+====
+|project_name| is a Deduplicating ARChiver written in Python.
+The main goal of darc is to provide an efficient and secure way to backup data.
 
-Features
---------
+Main Features
+-------------
 Space efficient storage
-   Variable block size `deduplication <http://en.wikipedia.org/wiki/Data_deduplication>`_
-   is used to reduce the number of bytes stored by detecting redundant data.
-   Each file is split into a number of variable length chunks and only chunks
-   that have never been seen before are compressed and added to the repository.
+   Variable block size `deduplication`_ is used to reduce the number of bytes 
+   stored by detecting redundant data. Each file is split into a number of variable
+   length chunks and only chunks that have never been seen before are compressed
+   and added to the repository.
 
-Secure
-    All data is encrypted using `AES256 <http://en.wikipedia.org/wiki/Advanced_Encryption_Standard>`_
-    and the data integrity and authenticity is verified using
-    `HMAC-SHA256 <http://en.wikipedia.org/wiki/HMAC>`_.
+Optional data encryption
+    All data can be protected using 256-bit AES_ encryption and data integrity
+    and authenticity is verified using `HMAC-SHA256`_.
 
-Remote repositories
-    Darc can store data on remote hosts over SSH as long as Darc is installed on
-    the remote host. The following syntax is used to specify a remote repository::
+Off-site backups
+    |project_name| can store data on any remote host accessible over SSH as long as
+    |project_name| is installed.
 
-    $ darc list hostname:path
-    $ darc extract hostname:path::archive-name
-    $ darc extract username@hostname:path::archive-name
-
-
-Definitions
+Easy to use
 -----------
-Deduplication
-    Deduplication is a technique for improving storage utilization by eliminating
-    redundant data. 
-
-Archive
-    A Darc archive is a collection of files along with metadata that include file
-    permissions, directory structure and various file attributes.
-
-Repository
-    A Darc repository is a filesystem directory storing data from zero or more archives.
-    The data in a store is both deduplicated and encrypted making it both 
-    efficient and safe.
-
-Key file
-    When a Darc store is initialized a key file containing a password protected
-    encryption key is created. It is vital to keep this file safe since the store
-    data is totally inaccessible without it.
-
-
-Requirements
-------------
-* Python >= 3.2
-* msgpack-python
-
-
-Installation
-------------
-
-The following instructions will install Darc in ``/usr/local/darc/`` without interfering
-with the rest of the system.
-
-1. Initialize a new python environment::
-
-    $ virtualenv /usr/local/darc
-
-2. Extract the source code using GIT or a release tarball::
-
-    $ mkdir /usr/local/darc/src/
-    $ cd /usr/local/darc/src/
-    $ tar -xvzf darc-x.y.tar.gz
-    OR
-    $ git clone git://github.com/jborg/darc.git
+Initialize a new backup :ref:`repository <repository_def>` and create your first 
+backup :ref:`archive <archive_def>` in two lines::
 
-3. Install Darc::
+    $ darc init /usbdrive/my-backup.darc
+    $ darc create -v /usbdrive/my-backup.darc::documents ~/Documents
 
-    $ cd darc-x.y/
-    $ ../../bin/python setup.py install
+See the :ref:`generalusage` section for more detailed examples.
 
-4. Add /usr/local/darc/bin to $PATH
-
-
-Basic Usage
-===========
-
-Initializing a store
---------------------
-Before the first archive can be created a store needs to be initialized::
-
-    $ darc init /data/my-backup.darc
-    Initializing repository at "/data/my-backup.darc"
-    Key file password (Leave blank for no password): *****
-    Key file password again: *****
-    Key file "/home/YOU/.darc/keys/data_my_backup_darc" created.
-    Remember that this file (and password) is needed to access your data. Keep it safe!
-
-
-Archive creation
-----------------
-The following command will create a new archive called ``backup-2011-09-10`` containing
-all files in ``~/Documents`` and ``~/src``::
-
-    $ darc create -v /data/my-backup.darc::backup-2011-09-10 ~/Documents ~/src
-
-Extract an archive
-------------------
-The following command will extract the archive ``backup-2011-09-10``::
-
-    $ darc extract -v /data/my-backup.darc::backup-2011-09-10
-
-Delete an archive
+Easy installation
 -----------------
-The following command will delete archive ``backup-2011-09-10``::
-
-    $ darc delete /data/my-backup.darc::backup-2011-09-10
-
-List store contents
--------------------
-The following command will list the names of all archives in the store::
-
-    $ darc list /data/my-backup.darc
-    backup-2011-09-09
-    backup-2011-09-10
-    ...
-
-List archive contents
----------------------
-The following command will list the contents of the ``backup-2011-09-10`` archive::
+You can use pip to install |project_name| quickly and easily::
 
-    $ darc list /data/my-backup.darc::backup-2011-09-10
-    -rw-r--r-- YOU    users       280 May 14  2010 home/YOU/Documents/something.txt
-    -rw-r--r-- YOU    users       280 May 14  2010 home/YOU/Documents/something-else.pdf
-    ...
+    $ pip install darc
 
-Prune old archives
-------------------
-When performing automatic backups it is important to periodically prune old backup
-archives to stop the store from growing too big.
+Python >= 3.2 is required.
 
-The following command will prune old archives and only keep the
-seven latest end of day archives and the five latest end of week archives::
+User's Guide
+============
 
-    $ darc prune --daily=7 --weekly=5 /data/my-backup.darc
+.. toctree::
+   :maxdepth: 2
 
+   generalusage
+   faq
+   definitions
 
-Indices and tables
-==================
+Contribute
+==========
 
-* :ref:`genindex`
-* :ref:`search`
+Found a bug? Have any ideas to improve |project_name|?
+Head over to |project_name|'s github_ page and create an issue or a pull request.
 
+You can also ask the author a question directly by `email <mailto:jonas@borgstrom.se>`_.

docs/make.bat

@@ -1,170 +0,0 @@
-@ECHO OFF
-
-REM Command file for Sphinx documentation
-
-if "%SPHINXBUILD%" == "" (
-	set SPHINXBUILD=sphinx-build
-)
-set BUILDDIR=_build
-set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% .
-if NOT "%PAPER%" == "" (
-	set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS%
-)
-
-if "%1" == "" goto help
-
-if "%1" == "help" (
-	:help
-	echo.Please use `make ^<target^>` where ^<target^> is one of
-	echo.  html       to make standalone HTML files
-	echo.  dirhtml    to make HTML files named index.html in directories
-	echo.  singlehtml to make a single large HTML file
-	echo.  pickle     to make pickle files
-	echo.  json       to make JSON files
-	echo.  htmlhelp   to make HTML files and a HTML help project
-	echo.  qthelp     to make HTML files and a qthelp project
-	echo.  devhelp    to make HTML files and a Devhelp project
-	echo.  epub       to make an epub
-	echo.  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter
-	echo.  text       to make text files
-	echo.  man        to make manual pages
-	echo.  changes    to make an overview over all changed/added/deprecated items
-	echo.  linkcheck  to check all external links for integrity
-	echo.  doctest    to run all doctests embedded in the documentation if enabled
-	goto end
-)
-
-if "%1" == "clean" (
-	for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i
-	del /q /s %BUILDDIR%\*
-	goto end
-)
-
-if "%1" == "html" (
-	%SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The HTML pages are in %BUILDDIR%/html.
-	goto end
-)
-
-if "%1" == "dirhtml" (
-	%SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml.
-	goto end
-)
-
-if "%1" == "singlehtml" (
-	%SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml.
-	goto end
-)
-
-if "%1" == "pickle" (
-	%SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished; now you can process the pickle files.
-	goto end
-)
-
-if "%1" == "json" (
-	%SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished; now you can process the JSON files.
-	goto end
-)
-
-if "%1" == "htmlhelp" (
-	%SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished; now you can run HTML Help Workshop with the ^
-.hhp project file in %BUILDDIR%/htmlhelp.
-	goto end
-)
-
-if "%1" == "qthelp" (
-	%SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished; now you can run "qcollectiongenerator" with the ^
-.qhcp project file in %BUILDDIR%/qthelp, like this:
-	echo.^> qcollectiongenerator %BUILDDIR%\qthelp\Darc.qhcp
-	echo.To view the help file:
-	echo.^> assistant -collectionFile %BUILDDIR%\qthelp\Darc.ghc
-	goto end
-)
-
-if "%1" == "devhelp" (
-	%SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished.
-	goto end
-)
-
-if "%1" == "epub" (
-	%SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The epub file is in %BUILDDIR%/epub.
-	goto end
-)
-
-if "%1" == "latex" (
-	%SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished; the LaTeX files are in %BUILDDIR%/latex.
-	goto end
-)
-
-if "%1" == "text" (
-	%SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The text files are in %BUILDDIR%/text.
-	goto end
-)
-
-if "%1" == "man" (
-	%SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The manual pages are in %BUILDDIR%/man.
-	goto end
-)
-
-if "%1" == "changes" (
-	%SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.The overview file is in %BUILDDIR%/changes.
-	goto end
-)
-
-if "%1" == "linkcheck" (
-	%SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Link check complete; look for any errors in the above output ^
-or in %BUILDDIR%/linkcheck/output.txt.
-	goto end
-)
-
-if "%1" == "doctest" (
-	%SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Testing of doctests in the sources finished, look at the ^
-results in %BUILDDIR%/doctest/output.txt.
-	goto end
-)
-
-:end