development.rst 6.3 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212
  1. .. include:: global.rst.inc
  2. .. _development:
  3. Development
  4. ===========
  5. This chapter will get you started with |project_name| development.
  6. |project_name| is written in Python (with a little bit of Cython and C for
  7. the performance critical parts).
  8. Style guide
  9. -----------
  10. We generally follow `pep8
  11. <https://www.python.org/dev/peps/pep-0008/>`_, with 120 columns
  12. instead of 79. We do *not* use form-feed (``^L``) characters to
  13. separate sections either. Compliance is tested automatically when
  14. you run the tests.
  15. Output and Logging
  16. ------------------
  17. When writing logger calls, always use correct log level (debug only for
  18. debugging, info for informative messages, warning for warnings, error for
  19. errors, critical for critical errors/states).
  20. When directly talking to the user (e.g. Y/N questions), do not use logging,
  21. but directly output to stderr (not: stdout, it could be connected to a pipe).
  22. To control the amount and kinds of messages output to stderr or emitted at
  23. info level, use flags like ``--stats`` or ``--list``.
  24. Building a development environment
  25. ----------------------------------
  26. First, just install borg into a virtual env as described before.
  27. To install some additional packages needed for running the tests, activate your
  28. virtual env and run::
  29. pip install -r requirements.d/development.txt
  30. Building on Windows
  31. +++++++++++++++++++
  32. Download and install MSYS from https://msys2.github.io/
  33. Use `Mingw64-w64 64bit Shell`::
  34. pacman -S mingw-w64-x86_64-python3 git mingw-w64-x86_64-lz4 mingw-w64-x86_64-python3-pip \
  35. mingw-w64-x86_64-cython mingw-w64-x86_64-gcc mingw-w64-x86_64-ntldd-git
  36. Use git to get the source and checkout `windows` branch then::
  37. pip3 install -r requirements.d/development.txt
  38. pip3 install -e .
  39. Running the tests
  40. -----------------
  41. The tests are in the borg/testsuite package.
  42. To run all the tests, you need to have fakeroot installed. If you do not have
  43. fakeroot, you still will be able to run most tests, just leave away the
  44. `fakeroot -u` from the given command lines.
  45. To run the test suite use the following command::
  46. fakeroot -u tox # run all tests
  47. Some more advanced examples::
  48. # verify a changed tox.ini (run this after any change to tox.ini):
  49. fakeroot -u tox --recreate
  50. fakeroot -u tox -e py34 # run all tests, but only on python 3.4
  51. fakeroot -u tox borg.testsuite.locking # only run 1 test module
  52. fakeroot -u tox borg.testsuite.locking -- -k '"not Timer"' # exclude some tests
  53. fakeroot -u tox borg.testsuite -- -v # verbose py.test
  54. Important notes:
  55. - When using ``--`` to give options to py.test, you MUST also give ``borg.testsuite[.module]``.
  56. As tox doesn't run on Windows you have to manually run command::
  57. py.test --cov=borg --cov-config=.coveragerc --benchmark-skip --pyargs borg/testsuite
  58. Regenerate usage files
  59. ----------------------
  60. Usage and API documentation is currently committed directly to git,
  61. although those files are generated automatically from the source
  62. tree.
  63. When a new module is added, the ``docs/api.rst`` file needs to be
  64. regenerated::
  65. ./setup.py build_api
  66. When a command is added, a commandline flag changed, added or removed,
  67. the usage docs need to be rebuilt as well::
  68. ./setup.py build_usage
  69. Building the docs with Sphinx
  70. -----------------------------
  71. The documentation (in reStructuredText format, .rst) is in docs/.
  72. To build the html version of it, you need to have sphinx installed::
  73. pip3 install sphinx # important: this will install sphinx with Python 3
  74. Now run::
  75. cd docs/
  76. make html
  77. Then point a web browser at docs/_build/html/index.html.
  78. The website is updated automatically through Github web hooks on the
  79. main repository.
  80. Using Vagrant
  81. -------------
  82. We use Vagrant for the automated creation of testing environments and borgbackup
  83. standalone binaries for various platforms.
  84. For better security, there is no automatic sync in the VM to host direction.
  85. The plugin `vagrant-scp` is useful to copy stuff from the VMs to the host.
  86. Usage::
  87. # To create and provision the VM:
  88. vagrant up OS
  89. # To create an ssh session to the VM:
  90. vagrant ssh OS
  91. # To execute a command via ssh in the VM:
  92. vagrant ssh OS -c "command args"
  93. # To shut down the VM:
  94. vagrant halt OS
  95. # To shut down and destroy the VM:
  96. vagrant destroy OS
  97. # To copy files from the VM (in this case, the generated binary):
  98. vagrant scp OS:/vagrant/borg/borg.exe .
  99. Creating standalone binaries
  100. ----------------------------
  101. Make sure you have everything built and installed (including llfuse and fuse).
  102. When using the Vagrant VMs, pyinstaller will already be installed.
  103. With virtual env activated::
  104. pip install pyinstaller # or git checkout master
  105. pyinstaller -F -n borg-PLATFORM borg/__main__.py
  106. for file in dist/borg-*; do gpg --armor --detach-sign $file; done
  107. If you encounter issues, see also our `Vagrantfile` for details.
  108. .. note:: Standalone binaries built with pyinstaller are supposed to
  109. work on same OS, same architecture (x86 32bit, amd64 64bit)
  110. without external dependencies.
  111. On Windows use `python buildwin32.py` to build standalone executable in `win32exe` directory
  112. with all necessary files to run.
  113. Creating a new release
  114. ----------------------
  115. Checklist:
  116. - make sure all issues for this milestone are closed or moved to the
  117. next milestone
  118. - find and fix any low hanging fruit left on the issue tracker
  119. - check that Travis CI is happy
  120. - update ``CHANGES.rst``, based on ``git log $PREVIOUS_RELEASE..``
  121. - check version number of upcoming release in ``CHANGES.rst``
  122. - verify that ``MANIFEST.in`` and ``setup.py`` are complete
  123. - ``python setup.py build_api ; python setup.py build_usage`` and commit
  124. - tag the release::
  125. git tag -s -m "tagged/signed release X.Y.Z" X.Y.Z
  126. - run tox and/or binary builds on all supported platforms via vagrant,
  127. check for test failures
  128. - create a release on PyPi::
  129. python setup.py register sdist upload --identity="Thomas Waldmann" --sign
  130. - close release milestone on Github
  131. - announce on:
  132. - Mailing list
  133. - Twitter (follow @ThomasJWaldmann for these tweets)
  134. - IRC channel (change ``/topic``)
  135. - create a Github release, include:
  136. * standalone binaries (see above for how to create them)
  137. + for OS X, document the OS X Fuse version in the README of the binaries.
  138. OS X FUSE uses a kernel extension that needs to be compatible with the
  139. code contained in the binary.
  140. * a link to ``CHANGES.rst``