faq.rst 32 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733
  1. .. include:: global.rst.inc
  2. .. highlight:: none
  3. .. _faq:
  4. Frequently asked questions
  5. ==========================
  6. Usage & Limitations
  7. ###################
  8. Can I backup VM disk images?
  9. ----------------------------
  10. Yes, the `deduplication`_ technique used by
  11. |project_name| makes sure only the modified parts of the file are stored.
  12. Also, we have optional simple sparse file support for extract.
  13. If you use non-snapshotting backup tools like Borg to back up virtual machines,
  14. then the VMs should be turned off for the duration of the backup. Backing up live VMs can (and will)
  15. result in corrupted or inconsistent backup contents: a VM image is just a regular file to
  16. Borg with the same issues as regular files when it comes to concurrent reading and writing from
  17. the same file.
  18. For backing up live VMs use file system snapshots on the VM host, which establishes
  19. crash-consistency for the VM images. This means that with most file systems
  20. (that are journaling) the FS will always be fine in the backup (but may need a
  21. journal replay to become accessible).
  22. Usually this does not mean that file *contents* on the VM are consistent, since file
  23. contents are normally not journaled. Notable exceptions are ext4 in data=journal mode,
  24. ZFS and btrfs (unless nodatacow is used).
  25. Applications designed with crash-consistency in mind (most relational databases
  26. like PostgreSQL, SQLite etc. but also for example Borg repositories) should always
  27. be able to recover to a consistent state from a backup created with
  28. crash-consistent snapshots (even on ext4 with data=writeback or XFS).
  29. Hypervisor snapshots capturing most of the VM's state can also be used for backups
  30. and can be a better alternative to pure file system based snapshots of the VM's disk,
  31. since no state is lost. Depending on the application this can be the easiest and most
  32. reliable way to create application-consistent backups.
  33. Other applications may require a lot of work to reach application-consistency:
  34. It's a broad and complex issue that cannot be explained in entirety here.
  35. Borg doesn't intend to address these issues due to their huge complexity
  36. and platform/software dependency. Combining Borg with the mechanisms provided
  37. by the platform (snapshots, hypervisor features) will be the best approach
  38. to start tackling them.
  39. Can I backup from multiple servers into a single repository?
  40. ------------------------------------------------------------
  41. Yes, but in order for the deduplication used by |project_name| to work, it
  42. needs to keep a local cache containing checksums of all file
  43. chunks already stored in the repository. This cache is stored in
  44. ``~/.cache/borg/``. If |project_name| detects that a repository has been
  45. modified since the local cache was updated it will need to rebuild
  46. the cache. This rebuild can be quite time consuming.
  47. So, yes it's possible. But it will be most efficient if a single
  48. repository is only modified from one place. Also keep in mind that
  49. |project_name| will keep an exclusive lock on the repository while creating
  50. or deleting archives, which may make *simultaneous* backups fail.
  51. Can I copy or synchronize my repo to another location?
  52. ------------------------------------------------------
  53. Yes, you could just copy all the files. Make sure you do that while no
  54. backup is running (use `borg with-lock ...`). So what you get here is this:
  55. - client machine ---borg create---> repo1
  56. - repo1 ---copy---> repo2
  57. There is no special borg command to do the copying, just use cp or rsync if
  58. you want to do that.
  59. But think about whether that is really what you want. If something goes
  60. wrong in repo1, you will have the same issue in repo2 after the copy.
  61. If you want to have 2 independent backups, it is better to do it like this:
  62. - client machine ---borg create---> repo1
  63. - client machine ---borg create---> repo2
  64. Which file types, attributes, etc. are *not* preserved?
  65. -------------------------------------------------------
  66. * UNIX domain sockets (because it does not make sense - they are
  67. meaningless without the running process that created them and the process
  68. needs to recreate them in any case). So, don't panic if your backup
  69. misses a UDS!
  70. * The precise on-disk (or rather: not-on-disk) representation of the holes
  71. in a sparse file.
  72. Archive creation has no special support for sparse files, holes are
  73. backed up as (deduplicated and compressed) runs of zero bytes.
  74. Archive extraction has optional support to extract all-zero chunks as
  75. holes in a sparse file.
  76. * Some filesystem specific attributes, like btrfs NOCOW, see :ref:`platforms`.
  77. Are there other known limitations?
  78. ----------------------------------
  79. - A single archive can only reference a limited volume of file/dir metadata,
  80. usually corresponding to tens or hundreds of millions of files/dirs.
  81. When trying to go beyond that limit, you will get a fatal IntegrityError
  82. exception telling that the (archive) object is too big.
  83. An easy workaround is to create multiple archives with less items each.
  84. See also the :ref:`archive_limitation` and :issue:`1452`.
  85. :ref:`borg_info` shows how large (relative to the maximum size) existing
  86. archives are.
  87. .. _checkpoints_parts:
  88. If a backup stops mid-way, does the already-backed-up data stay there?
  89. ----------------------------------------------------------------------
  90. Yes, |project_name| supports resuming backups.
  91. During a backup a special checkpoint archive named ``<archive-name>.checkpoint``
  92. is saved every checkpoint interval (the default value for this is 30
  93. minutes) containing all the data backed-up until that point.
  94. This checkpoint archive is a valid archive,
  95. but it is only a partial backup (not all files that you wanted to backup are
  96. contained in it). Having it in the repo until a successful, full backup is
  97. completed is useful because it references all the transmitted chunks up
  98. to the checkpoint. This means that in case of an interruption, you only need to
  99. retransfer the data since the last checkpoint.
  100. If a backup was interrupted, you do not need to do any special considerations,
  101. just invoke ``borg create`` as you always do. You may use the same archive name
  102. as in previous attempt or a different one (e.g. if you always include the current
  103. datetime), it does not matter.
  104. |project_name| always does full single-pass backups, so it will start again
  105. from the beginning - but it will be much faster, because some of the data was
  106. already stored into the repo (and is still referenced by the checkpoint
  107. archive), so it does not need to get transmitted and stored again.
  108. Once your backup has finished successfully, you can delete all
  109. ``<archive-name>.checkpoint`` archives. If you run ``borg prune``, it will
  110. also care for deleting unneeded checkpoints.
  111. Note: the checkpointing mechanism creates hidden, partial files in an archive,
  112. so that checkpoints even work while a big file is being processed.
  113. They are named ``<filename>.borg_part_<N>`` and all operations usually ignore
  114. these files, but you can make them considered by giving the option
  115. ``--consider-part-files``. You usually only need that option if you are
  116. really desperate (e.g. if you have no completed backup of that file and you'ld
  117. rather get a partial file extracted than nothing). You do **not** want to give
  118. that option under any normal circumstances.
  119. How can I backup huge file(s) over a unstable connection?
  120. ---------------------------------------------------------
  121. This is not a problem anymore.
  122. For more details, see :ref:`checkpoints_parts`.
  123. How can I restore huge file(s) over an unstable connection?
  124. -----------------------------------------------------------
  125. If you cannot manage to extract the whole big file in one go, you can extract
  126. all the part files and manually concatenate them together.
  127. For more details, see :ref:`checkpoints_parts`.
  128. Can |project_name| add redundancy to the backup data to deal with hardware malfunction?
  129. ---------------------------------------------------------------------------------------
  130. No, it can't. While that at first sounds like a good idea to defend against
  131. some defect HDD sectors or SSD flash blocks, dealing with this in a
  132. reliable way needs a lot of low-level storage layout information and
  133. control which we do not have (and also can't get, even if we wanted).
  134. So, if you need that, consider RAID or a filesystem that offers redundant
  135. storage or just make backups to different locations / different hardware.
  136. See also :issue:`225`.
  137. Can |project_name| verify data integrity of a backup archive?
  138. -------------------------------------------------------------
  139. Yes, if you want to detect accidental data damage (like bit rot), use the
  140. ``check`` operation. It will notice corruption using CRCs and hashes.
  141. If you want to be able to detect malicious tampering also, use an encrypted
  142. repo. It will then be able to check using CRCs and HMACs.
  143. Can I use Borg on SMR hard drives?
  144. ----------------------------------
  145. SMR (shingled magnetic recording) hard drives are very different from
  146. regular hard drives. Applications have to behave in certain ways or
  147. performance will be heavily degraded.
  148. Borg 1.1 ships with default settings suitable for SMR drives,
  149. and has been successfully tested on *Seagate Archive v2* drives
  150. using the ext4 file system.
  151. Some Linux kernel versions between 3.19 and 4.5 had various bugs
  152. handling device-managed SMR drives, leading to IO errors, unresponsive
  153. drives and unreliable operation in general.
  154. For more details, refer to :issue:`2252`.
  155. .. _faq-integrityerror:
  156. I get an IntegrityError or similar - what now?
  157. ----------------------------------------------
  158. A single error does not necessarily indicate bad hardware or a Borg
  159. bug. All hardware exhibits a bit error rate (BER). Hard drives are typically
  160. specified as exhibiting less than one error every 12 to 120 TB
  161. (one bit error in 10e14 to 10e15 bits). The specification is often called
  162. *unrecoverable read error rate* (URE rate).
  163. Apart from these very rare errors there are two main causes of errors:
  164. (i) Defective hardware: described below.
  165. (ii) Bugs in software (Borg, operating system, libraries):
  166. Ensure software is up to date.
  167. Check whether the issue is caused by any fixed bugs described in :ref:`important_notes`.
  168. .. rubric:: Finding defective hardware
  169. .. note::
  170. Hardware diagnostics are operating system dependent and do not
  171. apply universally. The commands shown apply for popular Unix-like
  172. systems. Refer to your operating system's manual.
  173. Checking hard drives
  174. Find the drive containing the repository and use *findmnt*, *mount* or *lsblk*
  175. to learn the device path (typically */dev/...*) of the drive.
  176. Then, smartmontools can retrieve self-diagnostics of the drive in question::
  177. # smartctl -a /dev/sdSomething
  178. The *Offline_Uncorrectable*, *Current_Pending_Sector* and *Reported_Uncorrect*
  179. attributes indicate data corruption. A high *UDMA_CRC_Error_Count* usually
  180. indicates a bad cable.
  181. I/O errors logged by the system (refer to the system journal or
  182. dmesg) can point to issues as well. I/O errors only affecting the
  183. file system easily go unnoticed, since they are not reported to
  184. applications (e.g. Borg), while these errors can still corrupt data.
  185. Drives can corrupt some sectors in one event, while remaining
  186. reliable otherwise. Conversely, drives can fail completely with no
  187. advance warning. If in doubt, copy all data from the drive in
  188. question to another drive -- just in case it fails completely.
  189. If any of these are suspicious, a self-test is recommended::
  190. # smartctl -t long /dev/sdSomething
  191. Running ``fsck`` if not done already might yield further insights.
  192. Checking memory
  193. Intermittent issues, such as ``borg check`` finding errors
  194. inconsistently between runs, are frequently caused by bad memory.
  195. Run memtest86+ (or an equivalent memory tester) to verify that
  196. the memory subsystem is operating correctly.
  197. Checking processors
  198. Processors rarely cause errors. If they do, they are usually overclocked
  199. or otherwise operated outside their specifications. We do not recommend to
  200. operate hardware outside its specifications for productive use.
  201. Tools to verify correct processor operation include Prime95 (mprime), linpack,
  202. and the `Intel Processor Diagnostic Tool
  203. <https://downloadcenter.intel.com/download/19792/Intel-Processor-Diagnostic-Tool>`_
  204. (applies only to Intel processors).
  205. .. rubric:: Repairing a damaged repository
  206. With any defective hardware found and replaced, the damage done to the repository
  207. needs to be ascertained and fixed.
  208. :ref:`borg_check` provides diagnostics and ``--repair`` options for repositories with
  209. issues. We recommend to first run without ``--repair`` to assess the situation.
  210. If the found issues and proposed repairs seem right, re-run "check" with ``--repair`` enabled.
  211. Security
  212. ########
  213. How can I specify the encryption passphrase programmatically?
  214. -------------------------------------------------------------
  215. The encryption passphrase or a command to retrieve the passphrase can be
  216. specified programmatically using the `BORG_PASSPHRASE` or `BORG_PASSCOMMAND`
  217. environment variables. This is convenient when setting up automated encrypted
  218. backups. Another option is to use key file based encryption with a blank passphrase.
  219. See :ref:`encrypted_repos` for more details.
  220. .. _password_env:
  221. .. note:: Be careful how you set the environment; using the ``env``
  222. command, a ``system()`` call or using inline shell scripts
  223. (e.g. ``BORG_PASSPHRASE=hunter12 borg ...``)
  224. might expose the credentials in the process list directly
  225. and they will be readable to all users on a system. Using
  226. ``export`` in a shell script file should be safe, however, as
  227. the environment of a process is `accessible only to that
  228. user
  229. <https://security.stackexchange.com/questions/14000/environment-variable-accessibility-in-linux/14009#14009>`_.
  230. When backing up to remote encrypted repos, is encryption done locally?
  231. ----------------------------------------------------------------------
  232. Yes, file and directory metadata and data is locally encrypted, before
  233. leaving the local machine. We do not mean the transport layer encryption
  234. by that, but the data/metadata itself. Transport layer encryption (e.g.
  235. when ssh is used as a transport) applies additionally.
  236. When backing up to remote servers, do I have to trust the remote server?
  237. ------------------------------------------------------------------------
  238. Yes and No.
  239. No, as far as data confidentiality is concerned - if you use encryption,
  240. all your files/dirs data and metadata are stored in their encrypted form
  241. into the repository.
  242. Yes, as an attacker with access to the remote server could delete (or
  243. otherwise make unavailable) all your backups.
  244. How can I protect against a hacked backup client?
  245. -------------------------------------------------
  246. Assume you backup your backup client machine C to the backup server S and
  247. C gets hacked. In a simple push setup, the attacker could then use borg on
  248. C to delete all backups residing on S.
  249. These are your options to protect against that:
  250. - Do not allow to permanently delete data from the repo, see :ref:`append_only_mode`.
  251. - Use a pull-mode setup using ``ssh -R``, see :issue:`900`.
  252. - Mount C's filesystem on another machine and then create a backup of it.
  253. - Do not give C filesystem-level access to S.
  254. How can I protect against a hacked backup server?
  255. -------------------------------------------------
  256. Just in case you got the impression that pull-mode backups are way more safe
  257. than push-mode, you also need to consider the case that your backup server S
  258. gets hacked. In case S has access to a lot of clients C, that might bring you
  259. into even bigger trouble than a hacked backup client in the previous FAQ entry.
  260. These are your options to protect against that:
  261. - Use the standard push-mode setup (see also previous FAQ entry).
  262. - Mount (the repo part of) S's filesystem on C.
  263. - Do not give S file-system level access to C.
  264. - Have your backup server at a well protected place (maybe not reachable from
  265. the internet), configure it safely, apply security updates, monitor it, ...
  266. How can I protect against theft, sabotage, lightning, fire, ...?
  267. ----------------------------------------------------------------
  268. In general: if your only backup medium is nearby the backupped machine and
  269. always connected, you can easily get into trouble: they likely share the same
  270. fate if something goes really wrong.
  271. Thus:
  272. - have multiple backup media
  273. - have media disconnected from network, power, computer
  274. - have media at another place
  275. - have a relatively recent backup on your media
  276. How do I report a security issue with Borg?
  277. -------------------------------------------
  278. Send a private email to the :ref:`security contact <security-contact>`
  279. if you think you have discovered a security issue.
  280. Please disclose security issues responsibly.
  281. Common issues
  282. #############
  283. Why do I get "connection closed by remote" after a while?
  284. ---------------------------------------------------------
  285. When doing a backup to a remote server (using a ssh: repo URL), it sometimes
  286. stops after a while (some minutes, hours, ... - not immediately) with
  287. "connection closed by remote" error message. Why?
  288. That's a good question and we are trying to find a good answer in :issue:`636`.
  289. Why am I seeing idle borg serve processes on the repo server?
  290. -------------------------------------------------------------
  291. Maybe the ssh connection between client and server broke down and that was not
  292. yet noticed on the server. Try these settings:
  293. ::
  294. # /etc/ssh/sshd_config on borg repo server - kill connection to client
  295. # after ClientAliveCountMax * ClientAliveInterval seconds with no response
  296. ClientAliveInterval 20
  297. ClientAliveCountMax 3
  298. If you have multiple borg create ... ; borg create ... commands in a already
  299. serialized way in a single script, you need to give them ``--lock-wait N`` (with N
  300. being a bit more than the time the server needs to terminate broken down
  301. connections and release the lock).
  302. .. _disable_archive_chunks:
  303. The borg cache eats way too much disk space, what can I do?
  304. -----------------------------------------------------------
  305. There is a temporary (but maybe long lived) hack to avoid using lots of disk
  306. space for chunks.archive.d (see :issue:`235` for details):
  307. ::
  308. # this assumes you are working with the same user as the backup.
  309. # you can get the REPOID from the "config" file inside the repository.
  310. cd ~/.cache/borg/<REPOID>
  311. rm -rf chunks.archive.d ; touch chunks.archive.d
  312. This deletes all the cached archive chunk indexes and replaces the directory
  313. that kept them with a file, so borg won't be able to store anything "in" there
  314. in future.
  315. This has some pros and cons, though:
  316. - much less disk space needs for ~/.cache/borg.
  317. - chunk cache resyncs will be slower as it will have to transfer chunk usage
  318. metadata for all archives from the repository (which might be slow if your
  319. repo connection is slow) and it will also have to build the hashtables from
  320. that data.
  321. chunk cache resyncs happen e.g. if your repo was written to by another
  322. machine (if you share same backup repo between multiple machines) or if
  323. your local chunks cache was lost somehow.
  324. The long term plan to improve this is called "borgception", see :issue:`474`.
  325. Can I backup my root partition (/) with Borg?
  326. ---------------------------------------------
  327. Backing up your entire root partition works just fine, but remember to
  328. exclude directories that make no sense to backup, such as /dev, /proc,
  329. /sys, /tmp and /run, and to use ``--one-file-system`` if you only want to
  330. backup the root partition (and not any mounted devices e.g.).
  331. If it crashes with a UnicodeError, what can I do?
  332. -------------------------------------------------
  333. Check if your encoding is set correctly. For most POSIX-like systems, try::
  334. export LANG=en_US.UTF-8 # or similar, important is correct charset
  335. I can't extract non-ascii filenames by giving them on the commandline!?
  336. -----------------------------------------------------------------------
  337. This might be due to different ways to represent some characters in unicode
  338. or due to other non-ascii encoding issues.
  339. If you run into that, try this:
  340. - avoid the non-ascii characters on the commandline by e.g. extracting
  341. the parent directory (or even everything)
  342. - mount the repo using FUSE and use some file manager
  343. .. _a_status_oddity:
  344. I am seeing 'A' (added) status for an unchanged file!?
  345. ------------------------------------------------------
  346. The files cache is used to determine whether |project_name| already
  347. "knows" / has backed up a file and if so, to skip the file from
  348. chunking. It does intentionally *not* contain files that have a modification
  349. time (mtime) same as the newest mtime in the created archive.
  350. So, if you see an 'A' status for unchanged file(s), they are likely the files
  351. with the most recent mtime in that archive.
  352. This is expected: it is to avoid data loss with files that are backed up from
  353. a snapshot and that are immediately changed after the snapshot (but within
  354. mtime granularity time, so the mtime would not change). Without the code that
  355. removes these files from the files cache, the change that happened right after
  356. the snapshot would not be contained in the next backup as |project_name| would
  357. think the file is unchanged.
  358. This does not affect deduplication, the file will be chunked, but as the chunks
  359. will often be the same and already stored in the repo (except in the above
  360. mentioned rare condition), it will just re-use them as usual and not store new
  361. data chunks.
  362. If you want to avoid unnecessary chunking, just create or touch a small or
  363. empty file in your backup source file set (so that one has the latest mtime,
  364. not your 50GB VM disk image) and, if you do snapshots, do the snapshot after
  365. that.
  366. Since only the files cache is used in the display of files status,
  367. those files are reported as being added when, really, chunks are
  368. already used.
  369. .. _always_chunking:
  370. It always chunks all my files, even unchanged ones!
  371. ---------------------------------------------------
  372. |project_name| maintains a files cache where it remembers the mtime, size and
  373. inode of files. When |project_name| does a new backup and starts processing a
  374. file, it first looks whether the file has changed (compared to the values
  375. stored in the files cache). If the values are the same, the file is assumed
  376. unchanged and thus its contents won't get chunked (again).
  377. |project_name| can't keep an infinite history of files of course, thus entries
  378. in the files cache have a "maximum time to live" which is set via the
  379. environment variable BORG_FILES_CACHE_TTL (and defaults to 20).
  380. Every time you do a backup (on the same machine, using the same user), the
  381. cache entries' ttl values of files that were not "seen" are incremented by 1
  382. and if they reach BORG_FILES_CACHE_TTL, the entry is removed from the cache.
  383. So, for example, if you do daily backups of 26 different data sets A, B,
  384. C, ..., Z on one machine (using the default TTL), the files from A will be
  385. already forgotten when you repeat the same backups on the next day and it
  386. will be slow because it would chunk all the files each time. If you set
  387. BORG_FILES_CACHE_TTL to at least 26 (or maybe even a small multiple of that),
  388. it would be much faster.
  389. Another possible reason is that files don't always have the same path, for
  390. example if you mount a filesystem without stable mount points for each backup or if you are running the backup from a filesystem snapshot whose name is not stable.
  391. If the directory where you mount a filesystem is different every time,
  392. |project_name| assume they are different files.
  393. Is there a way to limit bandwidth with |project_name|?
  394. ------------------------------------------------------
  395. To limit upload (i.e. :ref:`borg_create`) bandwidth, use the
  396. ``--remote-ratelimit`` option.
  397. There is no built-in way to limit *download*
  398. (i.e. :ref:`borg_extract`) bandwidth, but limiting download bandwidth
  399. can be accomplished with pipeviewer_:
  400. Create a wrapper script: /usr/local/bin/pv-wrapper ::
  401. #!/bin/bash
  402. ## -q, --quiet do not output any transfer information at all
  403. ## -L, --rate-limit RATE limit transfer to RATE bytes per second
  404. export RATE=307200
  405. pv -q -L $RATE | "$@"
  406. Add BORG_RSH environment variable to use pipeviewer wrapper script with ssh. ::
  407. export BORG_RSH='/usr/local/bin/pv-wrapper ssh'
  408. Now |project_name| will be bandwidth limited. Nice thing about pv is that you can change rate-limit on the fly: ::
  409. pv -R $(pidof pv) -L 102400
  410. .. _pipeviewer: http://www.ivarch.com/programs/pv.shtml
  411. I am having troubles with some network/FUSE/special filesystem, why?
  412. --------------------------------------------------------------------
  413. |project_name| is doing nothing special in the filesystem, it only uses very
  414. common and compatible operations (even the locking is just "mkdir").
  415. So, if you are encountering issues like slowness, corruption or malfunction
  416. when using a specific filesystem, please try if you can reproduce the issues
  417. with a local (non-network) and proven filesystem (like ext4 on Linux).
  418. If you can't reproduce the issue then, you maybe have found an issue within
  419. the filesystem code you used (not with |project_name|). For this case, it is
  420. recommended that you talk to the developers / support of the network fs and
  421. maybe open an issue in their issue tracker. Do not file an issue in the
  422. |project_name| issue tracker.
  423. If you can reproduce the issue with the proven filesystem, please file an
  424. issue in the |project_name| issue tracker about that.
  425. Why does running 'borg check --repair' warn about data loss?
  426. ------------------------------------------------------------
  427. Repair usually works for recovering data in a corrupted archive. However,
  428. it's impossible to predict all modes of corruption. In some very rare
  429. instances, such as malfunctioning storage hardware, additional repo
  430. corruption may occur. If you can't afford to lose the repo, it's strongly
  431. recommended that you perform repair on a copy of the repo.
  432. In other words, the warning is there to emphasize that |project_name|:
  433. - Will perform automated routines that modify your backup repository
  434. - Might not actually fix the problem you are experiencing
  435. - Might, in very rare cases, further corrupt your repository
  436. In the case of malfunctioning hardware, such as a drive or USB hub
  437. corrupting data when read or written, it's best to diagnose and fix the
  438. cause of the initial corruption before attempting to repair the repo. If
  439. the corruption is caused by a one time event such as a power outage,
  440. running `borg check --repair` will fix most problems.
  441. Miscellaneous
  442. #############
  443. Requirements for the borg single-file binary, esp. (g)libc?
  444. -----------------------------------------------------------
  445. We try to build the binary on old, but still supported systems - to keep the
  446. minimum requirement for the (g)libc low. The (g)libc can't be bundled into
  447. the binary as it needs to fit your kernel and OS, but Python and all other
  448. required libraries will be bundled into the binary.
  449. If your system fulfills the minimum (g)libc requirement (see the README that
  450. is released with the binary), there should be no problem. If you are slightly
  451. below the required version, maybe just try. Due to the dynamic loading (or not
  452. loading) of some shared libraries, it might still work depending on what
  453. libraries are actually loaded and used.
  454. In the borg git repository, there is scripts/glibc_check.py that can determine
  455. (based on the symbols' versions they want to link to) whether a set of given
  456. (Linux) binaries works with a given glibc version.
  457. Why was Borg forked from Attic?
  458. -------------------------------
  459. Borg was created in May 2015 in response to the difficulty of getting new
  460. code or larger changes incorporated into Attic and establishing a bigger
  461. developer community / more open development.
  462. More details can be found in `ticket 217
  463. <https://github.com/jborg/attic/issues/217>`_ that led to the fork.
  464. Borg intends to be:
  465. * simple:
  466. * as simple as possible, but no simpler
  467. * do the right thing by default, but offer options
  468. * open:
  469. * welcome feature requests
  470. * accept pull requests of good quality and coding style
  471. * give feedback on PRs that can't be accepted "as is"
  472. * discuss openly, don't work in the dark
  473. * changing:
  474. * Borg is not compatible with Attic
  475. * do not break compatibility accidentally, without a good reason
  476. or without warning. allow compatibility breaking for other cases.
  477. * if major version number changes, it may have incompatible changes
  478. Migrating from Attic
  479. ####################
  480. What are the differences between Attic and Borg?
  481. ------------------------------------------------
  482. Borg is a fork of `Attic`_ and maintained by "`The Borg collective`_".
  483. .. _Attic: https://github.com/jborg/attic
  484. .. _The Borg collective: https://borgbackup.readthedocs.org/en/latest/authors.html
  485. Here's a (incomplete) list of some major changes:
  486. * lots of attic issues fixed (see `issue #5 <https://github.com/borgbackup/borg/issues/5>`_),
  487. including critical data corruption bugs and security issues.
  488. * more open, faster paced development (see `issue #1 <https://github.com/borgbackup/borg/issues/1>`_)
  489. * less chunk management overhead (less memory and disk usage for chunks index)
  490. * faster remote cache resync (useful when backing up multiple machines into same repo)
  491. * compression: no, lz4, zlib or lzma compression, adjustable compression levels
  492. * repokey replaces problematic passphrase mode (you can't change the passphrase nor the pbkdf2 iteration count in "passphrase" mode)
  493. * simple sparse file support, great for virtual machine disk files
  494. * can read special files (e.g. block devices) or from stdin, write to stdout
  495. * mkdir-based locking is more compatible than attic's posix locking
  496. * uses fadvise to not spoil / blow up the fs cache
  497. * better error messages / exception handling
  498. * better logging, screen output, progress indication
  499. * tested on misc. Linux systems, 32 and 64bit, FreeBSD, OpenBSD, NetBSD, Mac OS X
  500. Please read the :ref:`changelog` (or ``docs/changes.rst`` in the source distribution) for more
  501. information.
  502. Borg is not compatible with original Attic (but there is a one-way conversion).
  503. How do I migrate from Attic to Borg?
  504. ------------------------------------
  505. Use :ref:`borg_upgrade`. This is a one-way process that cannot be reversed.
  506. There are some caveats:
  507. - The upgrade can only be performed on local repositories.
  508. It cannot be performed on remote repositories.
  509. - If the repository is in "keyfile" encryption mode, the keyfile must
  510. exist locally or it must be manually moved after performing the upgrade:
  511. 1. Locate the repository ID, contained in the ``config`` file in the repository.
  512. 2. Locate the attic key file at ``~/.attic/keys/``. The correct key for the
  513. repository starts with the line ``ATTIC_KEY <repository id>``.
  514. 3. Copy the attic key file to ``~/.config/borg/keys/``
  515. 4. Change the first line from ``ATTIC_KEY ...`` to ``BORG_KEY ...``.
  516. 5. Verify that the repository is now accessible (e.g. ``borg list <repository>``).
  517. - Attic and Borg use different :ref:`"chunker params" <chunker-params>`.
  518. This means that data added by Borg won't deduplicate with the existing data
  519. stored by Attic. The effect is lessened if the files cache is used with Borg.
  520. - Repositories in "passphrase" mode *must* be migrated to "repokey" mode using
  521. :ref:`borg_key_migrate-to-repokey`. Borg does not support the "passphrase" mode
  522. any other way.
  523. Why is my backup bigger than with attic?
  524. ----------------------------------------
  525. Attic was rather unflexible when it comes to compression, it always
  526. compressed using zlib level 6 (no way to switch compression off or
  527. adjust the level or algorithm).
  528. The default in Borg is lz4, which is fast enough to not use significant CPU time
  529. in most cases, but can only achieve modest compression. It still compresses
  530. easily compressed data fairly well.
  531. zlib compression with all levels (1-9) as well as LZMA (1-6) are available
  532. as well, for cases where they are worth it.
  533. Which choice is the best option depends on a number of factors, like
  534. bandwidth to the repository, how well the data compresses, available CPU
  535. power and so on.