.. include:: ../global.rst.inc .. highlight:: json .. _json_output: All about JSON: How to develop frontends ======================================== Borg does not have a public API on the Python level. That does not keep you from writing :code:`import borg`, but does mean that there are no release-to-release guarantees on what you might find in that package, not even for point releases (1.1.x), and there is no documentation beyond the code and the internals documents. Borg does on the other hand provide an API on a command-line level. In other words, a frontend should to (for example) create a backup archive just invoke :ref:`borg_create`. Logging ------- Especially for graphical frontends it is important to be able to convey and reformat progress information in meaningful ways. The ``--log-json`` option turns the stderr stream of Borg into a stream of JSON lines, where each line is a JSON object. The *type* key of the object determines its other contents. Since JSON can only encode text, any string representing a file system path may miss non-text parts. The following types are in use. Progress information is governed by the usual rules for progress information, it is not produced unless ``--progress`` is specified. archive_progress Output during operations creating archives (:ref:`borg_create` and :ref:`borg_recreate`). The following keys exist, each represents the current progress. original_size Original size of data processed so far (before compression and deduplication) compressed_size Compressed size deduplicated_size Deduplicated size nfiles Number of (regular) files processed so far path Current path progress_message A message-based progress information with no concrete progress information, just a message saying what is currently being worked on. operation unique, opaque integer ID of the operation :ref:`msgid ` Message ID of the operation (may be *none*) finished boolean indicating whether the operation has finished, only the last object for an *operation* can have this property set to *true*. message current progress message (may be empty/absent) progress_percent Absolute progress information with defined end/total and current value. operation unique, opaque integer ID of the operation :ref:`msgid ` Message ID of the operation (may be *none*) finished boolean indicating whether the operation has finished, only the last object for an *operation* can have this property set to *true*. message A formatted progress message, this will include the percentage and perhaps other information current Current value (always less-or-equal to *total*) info Array that describes the current item, may be *none*, contents depend on *msgid* total Total value file_status This is only output by :ref:`borg_create` and :ref:`borg_recreate` if ``--list`` is specified. The usual rules for the file listing applies, including the ``--filter`` option. status Single-character status as for regular list output path Path of the file system object log_message Any regular log output invokes this type. Regular log options and filtering applies to these as well. created Unix timestamp (float) levelname Upper-case log level name (also called severity). Defined levels are: DEBUG, INFO, WARNING, ERROR, CRITICAL name Name of the emitting entity message Formatted log message :ref:`msgid ` Message ID, may be *none* or absent .. rubric:: Examples (reformatted, each object would be on exactly one line) :ref:`borg_extract` progress:: {"message": "100.0% Extracting: src/borgbackup.egg-info/entry_points.txt", "current": 13000228, "total": 13004993, "info": ["src/borgbackup.egg-info/entry_points.txt"], "operation": 1, "msgid": "extract", "type": "progress_percent", "finished": false} {"message": "100.0% Extracting: src/borgbackup.egg-info/SOURCES.txt", "current": 13004993, "total": 13004993, "info": ["src/borgbackup.egg-info/SOURCES.txt"], "operation": 1, "msgid": "extract", "type": "progress_percent", "finished": false} {"operation": 1, "msgid": "extract", "type": "progress_percent", "finished": true} :ref:`borg_create` file listing with progress:: {"original_size": 0, "compressed_size": 0, "deduplicated_size": 0, "nfiles": 0, "type": "archive_progress", "path": "src"} {"type": "file_status", "status": "U", "path": "src/borgbackup.egg-info/entry_points.txt"} {"type": "file_status", "status": "U", "path": "src/borgbackup.egg-info/SOURCES.txt"} {"type": "file_status", "status": "d", "path": "src/borgbackup.egg-info"} {"type": "file_status", "status": "d", "path": "src"} {"original_size": 13176040, "compressed_size": 11386863, "deduplicated_size": 503, "nfiles": 277, "type": "archive_progress", "path": ""} Internal transaction progress:: {"message": "Saving files cache", "operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": false} {"message": "Saving cache config", "operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": false} {"message": "Saving chunks cache", "operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": false} {"operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": true} A debug log message:: {"message": "35 self tests completed in 0.08 seconds", "type": "log_message", "created": 1488278449.5575905, "levelname": "DEBUG", "name": "borg.archiver"} Standard output --------------- *stdout* is different and more command-dependent. Commands like :ref:`borg_info`, :ref:`borg_create` and :ref:`borg_list` implement a ``--json`` option which turns their regular output into a single JSON object. Dates are formatted according to ISO-8601 with the strftime format string '%a, %Y-%m-%d %H:%M:%S', e.g. *Sat, 2016-02-25 23:50:06*. The root object at least contains a *repository* key with an object containing: id The ID of the repository, normally 64 hex characters location Canonicalized repository path, thus this may be different from what is specified on the command line last_modified Date when the repository was last modified by the Borg client The *encryption* key, if present, contains: mode Textual encryption mode name (same as :ref:`borg_init` ``--encryption`` names) keyfile Path to the local key file used for access. Depending on *mode* this key may be absent. The *cache* key, if present, contains: path Path to the local repository cache stats Object containing cache stats: total_chunks Number of chunks total_unique_chunks Number of unique chunks total_size Total uncompressed size of all chunks multiplied with their reference counts total_csize Total compressed and encrypted size of all chunks multiplied with their reference counts unique_size Uncompressed size of all chunks unique_csize Compressed and encrypted size of all chunks Example *borg info* output:: { "cache": { "path": "/home/user/.cache/borg/0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23", "stats": { "total_chunks": 511533, "total_csize": 17948017540, "total_size": 22635749792, "total_unique_chunks": 54892, "unique_csize": 1920405405, "unique_size": 2449675468 } }, "encryption": { "mode": "repokey" }, "repository": { "id": "0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23", "last_modified": "Mon, 2017-02-27 21:21:58", "location": "/home/user/testrepo" }, "security_dir": "/home/user/.config/borg/security/0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23" } .. rubric:: Archive formats :ref:`borg_info` uses an extended format for archives, which is more expensive to retrieve, while :ref:`borg_list` uses a simpler format that is faster to retrieve. Either return archives in an array under the *archives* key, while :ref:`borg_create` returns a single archive object under the *archive* key. Both formats contain a *name* key with the archive name, the *id* key with the hexadecimal archive ID, and the *start* key with the start timestamp. *borg info* and *borg create* further have: end End timestamp duration Duration in seconds between start and end in seconds (float) stats Archive statistics (freshly calculated, this is what makes "info" more expensive) original_size Size of files and metadata before compression compressed_size Size after compression deduplicated_size Deduplicated size (against the current repository, not when the archive was created) nfiles Number of regular files in the archive limits Object describing the utilization of Borg limits max_archive_size Float between 0 and 1 describing how large this archive is relative to the maximum size allowed by Borg command_line Array of strings of the command line that created the archive The note about paths from above applies here as well. :ref:`borg_info` further has: hostname Hostname of the creating host username Name of the creating user comment Archive comment, if any Example of a simple archive listing (``borg list --last 1 --json``):: { "archives": [ { "id": "80cd07219ad725b3c5f665c1dcf119435c4dee1647a560ecac30f8d40221a46a", "name": "host-system-backup-2017-02-27", "start": "Mon, 2017-02-27 21:21:52" } ], "encryption": { "mode": "repokey" }, "repository": { "id": "0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23", "last_modified": "Mon, 2017-02-27 21:21:58", "location": "/home/user/repository" } } The same archive with more information (``borg info --last 1 --json``):: { "archives": [ { "command_line": [ "/home/user/.local/bin/borg", "create", "/home/user/repository", "..." ], "comment": "", "duration": 5.641542, "end": "Mon, 2017-02-27 21:21:58", "hostname": "host", "id": "80cd07219ad725b3c5f665c1dcf119435c4dee1647a560ecac30f8d40221a46a", "limits": { "max_archive_size": 0.0001330855110409714 }, "name": "host-system-backup-2017-02-27", "start": "Mon, 2017-02-27 21:21:52", "stats": { "compressed_size": 1880961894, "deduplicated_size": 2791, "nfiles": 53669, "original_size": 2400471280 }, "username": "user" } ], "cache": { "path": "/home/user/.cache/borg/0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23", "stats": { "total_chunks": 511533, "total_csize": 17948017540, "total_size": 22635749792, "total_unique_chunks": 54892, "unique_csize": 1920405405, "unique_size": 2449675468 } }, "encryption": { "mode": "repokey" }, "repository": { "id": "0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23", "last_modified": "Mon, 2017-02-27 21:21:58", "location": "/home/user/repository" } } .. rubric:: File listings Listing the contents of an archive can produce *a lot* of JSON. Each item (file, directory, ...) is described by one object in the *items* array of the :ref:`borg_list` output. Refer to the *borg list* documentation for the available keys and their meaning. Example (excerpt):: { "encryption": { "mode": "repokey" }, "repository": { "id": "0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23", "last_modified": "Mon, 2017-02-27 21:21:58", "location": "/home/user/repository" }, "items": [ { "type": "d", "mode": "drwxr-xr-x", "user": "user", "group": "user", "uid": 1000, "gid": 1000, "path": "linux", "healthy": true, "source": "", "linktarget": "", "flags": null, "isomtime": "Sat, 2016-05-07 19:46:01", "size": 0 } ] } .. _msgid: Message IDs ----------- Message IDs are strings that essentially give a log message or operation a name, without actually using the full text, since texts change more frequently. Message IDs are unambiguous and reduce the need to parse log messages. Assigned message IDs are: .. See scripts/errorlist.py; this is slightly edited. Errors Archive.AlreadyExists Archive {} already exists Archive.DoesNotExist Archive {} does not exist Archive.IncompatibleFilesystemEncodingError Failed to encode filename "{}" into file system encoding "{}". Consider configuring the LANG environment variable. Cache.CacheInitAbortedError Cache initialization aborted Cache.EncryptionMethodMismatch Repository encryption method changed since last access, refusing to continue Cache.RepositoryAccessAborted Repository access aborted Cache.RepositoryIDNotUnique Cache is newer than repository - do you have multiple, independently updated repos with same ID? Cache.RepositoryReplay Cache is newer than repository - this is either an attack or unsafe (multiple repos with same ID) Buffer.MemoryLimitExceeded Requested buffer size {} is above the limit of {}. ExtensionModuleError The Borg binary extension modules do not seem to be properly installed IntegrityError Data integrity error: {} NoManifestError Repository has no manifest. PlaceholderError Formatting Error: "{}".format({}): {}({}) KeyfileInvalidError Invalid key file for repository {} found in {}. KeyfileMismatchError Mismatch between repository {} and key file {}. KeyfileNotFoundError No key file for repository {} found in {}. PassphraseWrong passphrase supplied in BORG_PASSPHRASE is incorrect PasswordRetriesExceeded exceeded the maximum password retries RepoKeyNotFoundError No key entry found in the config of repository {}. UnsupportedManifestError Unsupported manifest envelope. A newer version is required to access this repository. UnsupportedPayloadError Unsupported payload type {}. A newer version is required to access this repository. NotABorgKeyFile This file is not a borg key backup, aborting. RepoIdMismatch This key backup seems to be for a different backup repository, aborting. UnencryptedRepo Keymanagement not available for unencrypted repositories. UnknownKeyType Keytype {0} is unknown. LockError Failed to acquire the lock {}. LockErrorT Failed to acquire the lock {}. ConnectionClosed Connection closed by remote host InvalidRPCMethod RPC method {} is not valid PathNotAllowed Repository path not allowed RemoteRepository.RPCServerOutdated Borg server is too old for {}. Required version {} UnexpectedRPCDataFormatFromClient Borg {}: Got unexpected RPC data format from client. UnexpectedRPCDataFormatFromServer Got unexpected RPC data format from server: {} Repository.AlreadyExists Repository {} already exists. Repository.CheckNeeded Inconsistency detected. Please run "borg check {}". Repository.DoesNotExist Repository {} does not exist. Repository.InsufficientFreeSpaceError Insufficient free space to complete transaction (required: {}, available: {}). Repository.InvalidRepository {} is not a valid repository. Check repo config. Repository.ObjectNotFound Object with key {} not found in repository {}. Operations - cache.begin_transaction - cache.commit - cache.sync *info* is one string element, the name of the archive currently synced. - repository.compact_segments - repository.replay_segments - repository.check_segments - check.verify_data - extract *info* is one string element, the name of the path currently extracted. - extract.permissions - archive.delete