File manager - Edit - /home/newsbmcs.com/public_html/static/img/logo/_sources.tar
Back
020_installation.rst.txt 0000644 00000021201 15030441404 0011165 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ############ Installation ############ Packages ******** Note that if at any point the package you’re trying to use is outdated, you always have the option to use an official binary from the restic project. These are up to date binaries, built in a reproducible and verifiable way, that you can download and run without having to do additional installation work. Please see the :ref:`official_binaries` section below for various downloads. Official binaries can be updated in place by using the ``restic self-update`` command. Alpine Linux ============ On `Alpine Linux <https://www.alpinelinux.org>`__ you can install the ``restic`` package from the official community repos, e.g. using ``apk``: .. code-block:: console $ apk add restic Arch Linux ========== On `Arch Linux <https://www.archlinux.org/>`__, there is a package called ``restic`` installed from the official community repos, e.g. with ``pacman -S``: .. code-block:: console $ pacman -S restic Debian ====== On Debian, there's a package called ``restic`` which can be installed from the official repos, e.g. with ``apt-get``: .. code-block:: console $ apt-get install restic Fedora ====== restic can be installed using ``dnf``: .. code-block:: console $ dnf install restic If you used restic from copr previously, remove the copr repo as follows to avoid any conflicts: .. code-block:: console $ dnf copr remove copart/restic macOS ===== If you are using macOS, you can install restic using the `homebrew <https://brew.sh/>`__ package manager: .. code-block:: console $ brew install restic You may also install it using `MacPorts <https://www.macports.org/>`__: .. code-block:: console $ sudo port install restic Nix & NixOS =========== If you are using `Nix <https://nixos.org/nix/>`__ or `NixOS <https://nixos.org/>`__ there is a package available named ``restic``. It can be installed using ``nix-env``: .. code-block:: console $ nix-env --install restic OpenBSD ======= On OpenBSD 6.3 and greater, you can install restic using ``pkg_add``: .. code-block:: console # pkg_add restic FreeBSD ======= On FreeBSD (11 and probably later versions), you can install restic using ``pkg install``: .. code-block:: console # pkg install restic openSUSE ======== On openSUSE (leap 15.0 and greater, and tumbleweed), you can install restic using the ``zypper`` package manager: .. code-block:: console # zypper install restic RHEL & CentOS ============= restic can be installed via copr repository, for RHEL7/CentOS you can try the following: .. code-block:: console $ yum install yum-plugin-copr $ yum copr enable copart/restic $ yum install restic If that doesn't work, you can try adding the repository directly, for CentOS6 use: .. code-block:: console $ yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-6/copart-restic-epel-6.repo For CentOS7 use: .. code-block:: console $ yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-7/copart-restic-epel-7.repo Solus ===== restic can be installed from the official repo of Solus via the ``eopkg`` package manager: .. code-block:: console $ eopkg install restic Windows ======= restic can be installed using `Scoop <https://scoop.sh/>`__: .. code-block:: console scoop install restic Using this installation method, ``restic.exe`` will automatically be available in the ``PATH``. It can be called from cmd.exe or PowerShell by typing ``restic``. .. _official_binaries: Official Binaries ***************** Stable Releases =============== You can download the latest stable release versions of restic from the `restic release page <https://github.com/restic/restic/releases/latest>`__. These builds are considered stable and releases are made regularly in a controlled manner. There's both pre-compiled binaries for different platforms as well as the source code available for download. Just download and run the one matching your system. The official binaries can be updated in place using the ``restic self-update`` command (needs restic 0.9.3 or later): .. code-block:: console $ restic version restic 0.9.3 compiled with go1.11.2 on linux/amd64 $ restic self-update find latest release of restic at GitHub latest version is 0.9.4 download file SHA256SUMS download SHA256SUMS download file SHA256SUMS download SHA256SUMS.asc GPG signature verification succeeded download restic_0.9.4_linux_amd64.bz2 downloaded restic_0.9.4_linux_amd64.bz2 saved 12115904 bytes in ./restic successfully updated restic to version 0.9.4 $ restic version restic 0.9.4 compiled with go1.12.1 on linux/amd64 The ``self-update`` command uses the GPG signature on the files uploaded to GitHub to verify their authenticity. No external programs are necessary. .. note:: Please be aware that the user executing the ``restic self-update`` command must have the permission to replace the restic binary. If you want to save the downloaded restic binary into a different file, pass the file name via the option ``--output``. Unstable Builds =============== Another option is to use the latest builds for the master branch, available on the `restic beta download site <https://beta.restic.net/?sort=time&order=desc>`__. These too are pre-compiled and ready to run, and a new version is built every time a push is made to the master branch. Windows ======= On Windows, put the `restic.exe` binary into `%SystemRoot%\\System32` to use restic in scripts without the need for absolute paths to the binary. This requires administrator rights. Docker Container **************** We're maintaining a bare docker container with just a few files and the restic binary, you can get it with `docker pull` like this: .. code-block:: console $ docker pull restic/restic .. note:: | Another docker container which offers more configuration options is | available as a contribution (Thank you!). You can find it at | https://github.com/Lobaro/restic-backup-docker From Source *********** restic is written in the Go programming language and you need at least Go version 1.13. Building restic may also work with older versions of Go, but that's not supported. See the `Getting started <https://golang.org/doc/install>`__ guide of the Go project for instructions how to install Go. In order to build restic from source, execute the following steps: .. code-block:: console $ git clone https://github.com/restic/restic [...] $ cd restic $ go run build.go You can easily cross-compile restic for all supported platforms, just supply the target OS and platform via the command-line options like this (for Windows and FreeBSD respectively): .. code-block:: console $ go run build.go --goos windows --goarch amd64 $ go run build.go --goos freebsd --goarch 386 $ go run build.go --goos linux --goarch arm --goarm 6 The resulting binary is statically linked and does not require any libraries. At the moment, the only tested compiler for restic is the official Go compiler. Building restic with gccgo may work, but is not supported. Autocompletion ************** Restic can write out man pages and bash/fish/zsh compatible autocompletion scripts: .. code-block:: console $ ./restic generate --help The "generate" command writes automatically generated files (like the man pages and the auto-completion files for bash, fish and zsh). Usage: restic generate [flags] [command] Flags: --bash-completion file write bash completion file --fish-completion file write fish completion file -h, --help help for generate --man directory write man pages to directory --zsh-completion file write zsh completion file Example for using sudo to write a bash completion script directly to the system-wide location: .. code-block:: console $ sudo ./restic generate --bash-completion /etc/bash_completion.d/restic writing bash completion file to /etc/bash_completion.d/restic .. note:: The path for the ``--bash-completion`` option may vary depending on the operating system used, e.g. ``/usr/share/bash-completion/completions/restic`` in Debian and derivatives. Please look up the correct path in the appropriate documentation. 045_working_with_repos.rst.txt 0000644 00000027033 15030441404 0012427 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ######################### Working with repositories ######################### Listing all snapshots ===================== Now, you can list all the snapshots stored in the repository: .. code-block:: console $ restic -r /srv/restic-repo snapshots enter password for repository: ID Date Host Tags Directory ---------------------------------------------------------------------- 40dc1520 2015-05-08 21:38:30 kasimir /home/user/work 79766175 2015-05-08 21:40:19 kasimir /home/user/work bdbd3439 2015-05-08 21:45:17 luigi /home/art 590c8fc8 2015-05-08 21:47:38 kazik /srv 9f0bc19e 2015-05-08 21:46:11 luigi /srv You can filter the listing by directory path: .. code-block:: console $ restic -r /srv/restic-repo snapshots --path="/srv" enter password for repository: ID Date Host Tags Directory ---------------------------------------------------------------------- 590c8fc8 2015-05-08 21:47:38 kazik /srv 9f0bc19e 2015-05-08 21:46:11 luigi /srv Or filter by host: .. code-block:: console $ restic -r /srv/restic-repo snapshots --host luigi enter password for repository: ID Date Host Tags Directory ---------------------------------------------------------------------- bdbd3439 2015-05-08 21:45:17 luigi /home/art 9f0bc19e 2015-05-08 21:46:11 luigi /srv Combining filters is also possible. Furthermore you can group the output by the same filters (host, paths, tags): .. code-block:: console $ restic -r /srv/restic-repo snapshots --group-by host enter password for repository: snapshots for (host [kasimir]) ID Date Host Tags Directory ---------------------------------------------------------------------- 40dc1520 2015-05-08 21:38:30 kasimir /home/user/work 79766175 2015-05-08 21:40:19 kasimir /home/user/work 2 snapshots snapshots for (host [luigi]) ID Date Host Tags Directory ---------------------------------------------------------------------- bdbd3439 2015-05-08 21:45:17 luigi /home/art 9f0bc19e 2015-05-08 21:46:11 luigi /srv 2 snapshots snapshots for (host [kazik]) ID Date Host Tags Directory ---------------------------------------------------------------------- 590c8fc8 2015-05-08 21:47:38 kazik /srv 1 snapshots Copying snapshots between repositories ====================================== In case you want to transfer snapshots between two repositories, for example from a local to a remote repository, you can use the ``copy`` command: .. code-block:: console $ restic -r /srv/restic-repo copy --repo2 /srv/restic-repo-copy repository d6504c63 opened successfully, password is correct repository 3dd0878c opened successfully, password is correct snapshot 410b18a2 of [/home/user/work] at 2020-06-09 23:15:57.305305 +0200 CEST) copy started, this may take a while... snapshot 7a746a07 saved snapshot 4e5d5487 of [/home/user/work] at 2020-05-01 22:44:07.012113 +0200 CEST) skipping snapshot 4e5d5487, was already copied to snapshot 50eb62b7 The example command copies all snapshots from the source repository ``/srv/restic-repo`` to the destination repository ``/srv/restic-repo-copy``. Snapshots which have previously been copied between repositories will be skipped by later copy runs. .. important:: This process will have to both download (read) and upload (write) the entire snapshot(s) due to the different encryption keys used in the source and destination repository. This *may incur higher bandwidth usage and costs* than expected during normal backup runs. .. important:: The copying process does not re-chunk files, which may break deduplication between the files copied and files already stored in the destination repository. This means that copied files, which existed in both the source and destination repository, *may occupy up to twice their space* in the destination repository. See below for how to avoid this. The destination repository is specified with ``--repo2`` or can be read from a file specified via ``--repository-file2``. Both of these options can also set as environment variables ``$RESTIC_REPOSITORY2`` or ``$RESTIC_REPOSITORY_FILE2`` respectively. For the destination repository the password can be read from a file ``--password-file2`` or from a command ``--password-command2``. Alternatively the environment variables ``$RESTIC_PASSWORD_COMMAND2`` and ``$RESTIC_PASSWORD_FILE2`` can be used. It is also possible to directly pass the password via ``$RESTIC_PASSWORD2``. The key which should be used for decryption can be selected by passing its ID via the flag ``--key-hint2`` or the environment variable ``$RESTIC_KEY_HINT2``. .. note:: In case the source and destination repository use the same backend, the configuration options and environment variables used to configure the backend may apply to both repositories – for example it might not be possible to specify different accounts for the source and destination repository. You can avoid this limitation by using the rclone backend along with remotes which are configured in rclone. Filtering snapshots to copy --------------------------- The list of snapshots to copy can be filtered by host, path in the backup and / or a comma-separated tag list: .. code-block:: console $ restic -r /srv/restic-repo copy --repo2 /srv/restic-repo-copy --host luigi --path /srv --tag foo,bar It is also possible to explicitly specify the list of snapshots to copy, in which case only these instead of all snapshots will be copied: .. code-block:: console $ restic -r /srv/restic-repo copy --repo2 /srv/restic-repo-copy 410b18a2 4e5d5487 latest Ensuring deduplication for copied snapshots ------------------------------------------- Even though the copy command can transfer snapshots between arbitrary repositories, deduplication between snapshots from the source and destination repository may not work. To ensure proper deduplication, both repositories have to use the same parameters for splitting large files into smaller chunks, which requires additional setup steps. With the same parameters restic will for both repositories split identical files into identical chunks and therefore deduplication also works for snapshots copied between these repositories. The chunker parameters are generated once when creating a new (destination) repository. That is for a copy destination repository we have to instruct restic to initialize it using the same chunker parameters as the source repository: .. code-block:: console $ restic -r /srv/restic-repo-copy init --repo2 /srv/restic-repo --copy-chunker-params Note that it is not possible to change the chunker parameters of an existing repository. Checking integrity and consistency ================================== Imagine your repository is saved on a server that has a faulty hard drive, or even worse, attackers get privileged access and modify the files in your repository with the intention to make you restore malicious data: .. code-block:: console $ echo "boom" > /srv/restic-repo/index/de30f3231ca2e6a59af4aa84216dfe2ef7339c549dc11b09b84000997b139628 Trying to restore a snapshot which has been modified as shown above will yield an error: .. code-block:: console $ restic -r /srv/restic-repo --no-cache restore c23e491f --target /tmp/restore-work ... Fatal: unable to load index de30f323: load <index/de30f3231c>: invalid data returned In order to detect these things before they become a problem, it's a good idea to regularly use the ``check`` command to test whether your repository is healthy and consistent, and that your precious backup data is unharmed. There are two types of checks that can be performed: - Structural consistency and integrity, e.g. snapshots, trees and pack files (default) - Integrity of the actual data that you backed up (enabled with flags, see below) To verify the structure of the repository, issue the ``check`` command. If the repository is damaged like in the example above, ``check`` will detect this and yield the same error as when you tried to restore: .. code-block:: console $ restic -r /srv/restic-repo check ... load indexes error: error loading index de30f323: load <index/de30f3231c>: invalid data returned Fatal: LoadIndex returned errors If the repository structure is intact, restic will show that no errors were found: .. code-block:: console $ restic -r /src/restic-repo check ... load indexes check all packs check snapshots, trees and blobs no errors were found By default, the ``check`` command does not verify that the actual pack files on disk in the repository are unmodified, because doing so requires reading a copy of every pack file in the repository. To tell restic to also verify the integrity of the pack files in the repository, use the ``--read-data`` flag: .. code-block:: console $ restic -r /srv/restic-repo check --read-data ... load indexes check all packs check snapshots, trees and blobs read all data [0:00] 100.00% 3 / 3 items duration: 0:00 no errors were found .. note:: Since ``--read-data`` has to download all pack files in the repository, beware that it might incur higher bandwidth costs than usual and also that it takes more time than the default ``check``. Alternatively, use the ``--read-data-subset`` parameter to check only a subset of the repository pack files at a time. It supports two ways to select a subset. One selects a specific range of pack files, the other selects a random percentage of pack files. Use ``--read-data-subset=n/t`` to check only a subset of the repository pack files at a time. The parameter takes two values, ``n`` and ``t``. When the check command runs, all pack files in the repository are logically divided in ``t`` (roughly equal) groups, and only files that belong to group number ``n`` are checked. For example, the following commands check all repository pack files over 5 separate invocations: .. code-block:: console $ restic -r /srv/restic-repo check --read-data-subset=1/5 $ restic -r /srv/restic-repo check --read-data-subset=2/5 $ restic -r /srv/restic-repo check --read-data-subset=3/5 $ restic -r /srv/restic-repo check --read-data-subset=4/5 $ restic -r /srv/restic-repo check --read-data-subset=5/5 Use ``--read-data-subset=n%`` to check a randomly choosen subset of the repository pack files. It takes one parameter, ``n``, the percentage of pack files to check as an integer or floating point number. This will not guarantee to cover all available pack files after sufficient runs, but it is easy to automate checking a small subset of data after each backup. For a floating point value the following command may be used: .. code-block:: console $ restic -r /srv/restic-repo check --read-data-subset=2.5% When checking bigger subsets you most likely specify the percentage as an integer: .. code-block:: console $ restic -r /srv/restic-repo check --read-data-subset=10% design.rst.txt 0000644 00000065327 15030441404 0007375 0 ustar 00 Terminology =========== This section introduces terminology used in this document. *Repository*: All data produced during a backup is sent to and stored in a repository in a structured form, for example in a file system hierarchy with several subdirectories. A repository implementation must be able to fulfill a number of operations, e.g. list the contents. *Blob*: A Blob combines a number of data bytes with identifying information like the SHA-256 hash of the data and its length. *Pack*: A Pack combines one or more Blobs, e.g. in a single file. *Snapshot*: A Snapshot stands for the state of a file or directory that has been backed up at some point in time. The state here means the content and meta data like the name and modification time for the file or the directory and its contents. *Storage ID*: A storage ID is the SHA-256 hash of the content stored in the repository. This ID is required in order to load the file from the repository. Repository Format ================= All data is stored in a restic repository. A repository is able to store data of several different types, which can later be requested based on an ID. This so-called "storage ID" is the SHA-256 hash of the content of a file. All files in a repository are only written once and never modified afterwards. This allows accessing and even writing to the repository with multiple clients in parallel. Only the ``prune`` operation removes data from the repository. Repositories consist of several directories and a top-level file called ``config``. For all other files stored in the repository, the name for the file is the lower case hexadecimal representation of the storage ID, which is the SHA-256 hash of the file's contents. This allows for easy verification of files for accidental modifications, like disk read errors, by simply running the program ``sha256sum`` on the file and comparing its output to the file name. If the prefix of a filename is unique amongst all the other files in the same directory, the prefix may be used instead of the complete filename. Apart from the files stored within the ``keys`` directory, all files are encrypted with AES-256 in counter mode (CTR). The integrity of the encrypted data is secured by a Poly1305-AES message authentication code (sometimes also referred to as a "signature"). In the first 16 bytes of each encrypted file the initialisation vector (IV) is stored. It is followed by the encrypted data and completed by the 16 byte MAC. The format is: ``IV || CIPHERTEXT || MAC``. The complete encryption overhead is 32 bytes. For each file, a new random IV is selected. The file ``config`` is encrypted this way and contains a JSON document like the following: .. code:: json { "version": 1, "id": "5956a3f67a6230d4a92cefb29529f10196c7d92582ec305fd71ff6d331d6271b", "chunker_polynomial": "25b468838dcb75" } After decryption, restic first checks that the version field contains a version number that it understands, otherwise it aborts. At the moment, the version is expected to be 1. The field ``id`` holds a unique ID which consists of 32 random bytes, encoded in hexadecimal. This uniquely identifies the repository, regardless if it is accessed via SFTP or locally. The field ``chunker_polynomial`` contains a parameter that is used for splitting large files into smaller chunks (see below). Repository Layout ----------------- The ``local`` and ``sftp`` backends are implemented using files and directories stored in a file system. The directory layout is the same for both backend types. The basic layout of a repository stored in a ``local`` or ``sftp`` backend is shown here: :: /tmp/restic-repo ├── config ├── data │ ├── 21 │ │ └── 2159dd48f8a24f33c307b750592773f8b71ff8d11452132a7b2e2a6a01611be1 │ ├── 32 │ │ └── 32ea976bc30771cebad8285cd99120ac8786f9ffd42141d452458089985043a5 │ ├── 59 │ │ └── 59fe4bcde59bd6222eba87795e35a90d82cd2f138a27b6835032b7b58173a426 │ ├── 73 │ │ └── 73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c │ [...] ├── index │ ├── c38f5fb68307c6a3e3aa945d556e325dc38f5fb68307c6a3e3aa945d556e325d │ └── ca171b1b7394d90d330b265d90f506f9984043b342525f019788f97e745c71fd ├── keys │ └── b02de829beeb3c01a63e6b25cbd421a98fef144f03b9a02e46eff9e2ca3f0bd7 ├── locks ├── snapshots │ └── 22a5af1bdc6e616f8a29579458c49627e01b32210d09adb288d1ecda7c5711ec └── tmp A local repository can be initialized with the ``restic init`` command, e.g.: .. code-block:: console $ restic -r /tmp/restic-repo init The local and sftp backends will auto-detect and accept all layouts described in the following sections, so that remote repositories mounted locally e.g. via fuse can be accessed. The layout auto-detection can be overridden by specifying the option ``-o local.layout=default``, valid values are ``default`` and ``s3legacy``. The option for the sftp backend is named ``sftp.layout``, for the s3 backend ``s3.layout``. S3 Legacy Layout ---------------- Unfortunately during development the AWS S3 backend uses slightly different paths (directory names use singular instead of plural for ``key``, ``lock``, and ``snapshot`` files), and the pack files are stored directly below the ``data`` directory. The S3 Legacy repository layout looks like this: :: /config /data ├── 2159dd48f8a24f33c307b750592773f8b71ff8d11452132a7b2e2a6a01611be1 ├── 32ea976bc30771cebad8285cd99120ac8786f9ffd42141d452458089985043a5 ├── 59fe4bcde59bd6222eba87795e35a90d82cd2f138a27b6835032b7b58173a426 ├── 73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c [...] /index ├── c38f5fb68307c6a3e3aa945d556e325dc38f5fb68307c6a3e3aa945d556e325d └── ca171b1b7394d90d330b265d90f506f9984043b342525f019788f97e745c71fd /key └── b02de829beeb3c01a63e6b25cbd421a98fef144f03b9a02e46eff9e2ca3f0bd7 /lock /snapshot └── 22a5af1bdc6e616f8a29579458c49627e01b32210d09adb288d1ecda7c5711ec The S3 backend understands and accepts both forms, new backends are always created with the default layout for compatibility reasons. Pack Format =========== All files in the repository except Key and Pack files just contain raw data, stored as ``IV || Ciphertext || MAC``. Pack files may contain one or more Blobs of data. A Pack's structure is as follows: :: EncryptedBlob1 || ... || EncryptedBlobN || EncryptedHeader || Header_Length At the end of the Pack file is a header, which describes the content. The header is encrypted and authenticated. ``Header_Length`` is the length of the encrypted header encoded as a four byte integer in little-endian encoding. Placing the header at the end of a file allows writing the blobs in a continuous stream as soon as they are read during the backup phase. This reduces code complexity and avoids having to re-write a file once the pack is complete and the content and length of the header is known. All the blobs (``EncryptedBlob1``, ``EncryptedBlobN`` etc.) are authenticated and encrypted independently. This enables repository reorganisation without having to touch the encrypted Blobs. In addition it also allows efficient indexing, for only the header needs to be read in order to find out which Blobs are contained in the Pack. Since the header is authenticated, authenticity of the header can be checked without having to read the complete Pack. After decryption, a Pack's header consists of the following elements: :: Type_Blob1 || Length(EncryptedBlob1) || Hash(Plaintext_Blob1) || [...] Type_BlobN || Length(EncryptedBlobN) || Hash(Plaintext_Blobn) || This is enough to calculate the offsets for all the Blobs in the Pack. Length is the length of a Blob as a four byte integer in little-endian format. The type field is a one byte field and labels the content of a blob according to the following table: +--------+-----------+ | Type | Meaning | +========+===========+ | 0 | data | +--------+-----------+ | 1 | tree | +--------+-----------+ All other types are invalid, more types may be added in the future. For reconstructing the index or parsing a pack without an index, first the last four bytes must be read in order to find the length of the header. Afterwards, the header can be read and parsed, which yields all plaintext hashes, types, offsets and lengths of all included blobs. Indexing ======== Index files contain information about Data and Tree Blobs and the Packs they are contained in and store this information in the repository. When the local cached index is not accessible any more, the index files can be downloaded and used to reconstruct the index. The files are encrypted and authenticated like Data and Tree Blobs, so the outer structure is ``IV || Ciphertext || MAC`` again. The plaintext consists of a JSON document like the following: .. code:: json { "supersedes": [ "ed54ae36197f4745ebc4b54d10e0f623eaaaedd03013eb7ae90df881b7781452" ], "packs": [ { "id": "73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c", "blobs": [ { "id": "3ec79977ef0cf5de7b08cd12b874cd0f62bbaf7f07f3497a5b1bbcc8cb39b1ce", "type": "data", "offset": 0, "length": 25 },{ "id": "9ccb846e60d90d4eb915848add7aa7ea1e4bbabfc60e573db9f7bfb2789afbae", "type": "tree", "offset": 38, "length": 100 }, { "id": "d3dc577b4ffd38cc4b32122cabf8655a0223ed22edfd93b353dc0c3f2b0fdf66", "type": "data", "offset": 150, "length": 123 } ] }, [...] ] } This JSON document lists Packs and the blobs contained therein. In this example, the Pack ``73d04e61`` contains two data Blobs and one Tree blob, the plaintext hashes are listed afterwards. The field ``supersedes`` lists the storage IDs of index files that have been replaced with the current index file. This happens when index files are repacked, for example when old snapshots are removed and Packs are recombined. There may be an arbitrary number of index files, containing information on non-disjoint sets of Packs. The number of packs described in a single file is chosen so that the file size is kept below 8 MiB. Keys, Encryption and MAC ======================== All data stored by restic in the repository is encrypted with AES-256 in counter mode and authenticated using Poly1305-AES. For encrypting new data first 16 bytes are read from a cryptographically secure pseudorandom number generator as a random nonce. This is used both as the IV for counter mode and the nonce for Poly1305. This operation needs three keys: A 32 byte for AES-256 for encryption, a 16 byte AES key and a 16 byte key for Poly1305. For details see the original paper `The Poly1305-AES message-authentication code <https://cr.yp.to/mac/poly1305-20050329.pdf>`__ by Dan Bernstein. The data is then encrypted with AES-256 and afterwards a message authentication code (MAC) is computed over the ciphertext, everything is then stored as IV \|\| CIPHERTEXT \|\| MAC. The directory ``keys`` contains key files. These are simple JSON documents which contain all data that is needed to derive the repository's master encryption and message authentication keys from a user's password. The JSON document from the repository can be pretty-printed for example by using the Python module ``json`` (shortened to increase readability): :: $ python -mjson.tool /tmp/restic-repo/keys/b02de82* { "hostname": "kasimir", "username": "fd0" "kdf": "scrypt", "N": 65536, "r": 8, "p": 1, "created": "2015-01-02T18:10:13.48307196+01:00", "data": "tGwYeKoM0C4j4/9DFrVEmMGAldvEn/+iKC3te/QE/6ox/V4qz58FUOgMa0Bb1cIJ6asrypCx/Ti/pRXCPHLDkIJbNYd2ybC+fLhFIJVLCvkMS+trdywsUkglUbTbi+7+Ldsul5jpAj9vTZ25ajDc+4FKtWEcCWL5ICAOoTAxnPgT+Lh8ByGQBH6KbdWabqamLzTRWxePFoYuxa7yXgmj9A==", "salt": "uW4fEI1+IOzj7ED9mVor+yTSJFd68DGlGOeLgJELYsTU5ikhG/83/+jGd4KKAaQdSrsfzrdOhAMftTSih5Ux6w==", } When the repository is opened by restic, the user is prompted for the repository password. This is then used with ``scrypt``, a key derivation function (KDF), and the supplied parameters (``N``, ``r``, ``p`` and ``salt``) to derive 64 key bytes. The first 32 bytes are used as the encryption key (for AES-256) and the last 32 bytes are used as the message authentication key (for Poly1305-AES). These last 32 bytes are divided into a 16 byte AES key ``k`` followed by 16 bytes of secret key ``r``. The key ``r`` is then masked for use with Poly1305 (see the paper for details). Those keys are used to authenticate and decrypt the bytes contained in the JSON field ``data`` with AES-256 and Poly1305-AES as if they were any other blob (after removing the Base64 encoding). If the password is incorrect or the key file has been tampered with, the computed MAC will not match the last 16 bytes of the data, and restic exits with an error. Otherwise, the data yields a JSON document which contains the master encryption and message authentication keys for this repository (encoded in Base64). The command ``restic cat masterkey`` can be used as follows to decrypt and pretty-print the master key: .. code-block:: console $ restic -r /tmp/restic-repo cat masterkey { "mac": { "k": "evFWd9wWlndL9jc501268g==", "r": "E9eEDnSJZgqwTOkDtOp+Dw==" }, "encrypt": "UQCqa0lKZ94PygPxMRqkePTZnHRYh1k1pX2k2lM2v3Q=", } All data in the repository is encrypted and authenticated with these master keys. For encryption, the AES-256 algorithm in Counter mode is used. For message authentication, Poly1305-AES is used as described above. A repository can have several different passwords, with a key file for each. This way, the password can be changed without having to re-encrypt all data. Snapshots ========= A snapshot represents a directory with all files and sub-directories at a given point in time. For each backup that is made, a new snapshot is created. A snapshot is a JSON document that is stored in an encrypted file below the directory ``snapshots`` in the repository. The filename is the storage ID. This string is unique and used within restic to uniquely identify a snapshot. The command ``restic cat snapshot`` can be used as follows to decrypt and pretty-print the contents of a snapshot file: .. code-block:: console $ restic -r /tmp/restic-repo cat snapshot 251c2e58 enter password for repository: { "time": "2015-01-02T18:10:50.895208559+01:00", "tree": "2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf", "dir": "/tmp/testdata", "hostname": "kasimir", "username": "fd0", "uid": 1000, "gid": 100, "tags": [ "NL" ] } Here it can be seen that this snapshot represents the contents of the directory ``/tmp/testdata``. The most important field is ``tree``. When the meta data (e.g. the tags) of a snapshot change, the snapshot needs to be re-encrypted and saved. This will change the storage ID, so in order to relate these seemingly different snapshots, a field ``original`` is introduced which contains the ID of the original snapshot, e.g. after adding the tag ``DE`` to the snapshot above it becomes: .. code-block:: console $ restic -r /tmp/restic-repo cat snapshot 22a5af1b enter password for repository: { "time": "2015-01-02T18:10:50.895208559+01:00", "tree": "2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf", "dir": "/tmp/testdata", "hostname": "kasimir", "username": "fd0", "uid": 1000, "gid": 100, "tags": [ "NL", "DE" ], "original": "251c2e5841355f743f9d4ffd3260bee765acee40a6229857e32b60446991b837" } Once introduced, the ``original`` field is not modified when the snapshot's meta data is changed again. All content within a restic repository is referenced according to its SHA-256 hash. Before saving, each file is split into variable sized Blobs of data. The SHA-256 hashes of all Blobs are saved in an ordered list which then represents the content of the file. In order to relate these plaintext hashes to the actual location within a Pack file , an index is used. If the index is not available, the header of all data Blobs can be read. Trees and Data ============== A snapshot references a tree by the SHA-256 hash of the JSON string representation of its contents. Trees and data are saved in pack files in a subdirectory of the directory ``data``. The command ``restic cat blob`` can be used to inspect the tree referenced above (piping the output of the command to ``jq .`` so that the JSON is indented): .. code-block:: console $ restic -r /tmp/restic-repo cat blob 2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf | jq . enter password for repository: { "nodes": [ { "name": "testdata", "type": "dir", "mode": 493, "mtime": "2014-12-22T14:47:59.912418701+01:00", "atime": "2014-12-06T17:49:21.748468803+01:00", "ctime": "2014-12-22T14:47:59.912418701+01:00", "uid": 1000, "gid": 100, "user": "fd0", "inode": 409704562, "content": null, "subtree": "b26e315b0988ddcd1cee64c351d13a100fedbc9fdbb144a67d1b765ab280b4dc" } ] } A tree contains a list of entries (in the field ``nodes``) which contain meta data like a name and timestamps. When the entry references a directory, the field ``subtree`` contains the plain text ID of another tree object. When the command ``restic cat blob`` is used, the plaintext ID is needed to print a tree. The tree referenced above can be dumped as follows: .. code-block:: console $ restic -r /tmp/restic-repo cat blob b26e315b0988ddcd1cee64c351d13a100fedbc9fdbb144a67d1b765ab280b4dc enter password for repository: { "nodes": [ { "name": "testfile", "type": "file", "mode": 420, "mtime": "2014-12-06T17:50:23.34513538+01:00", "atime": "2014-12-06T17:50:23.338468713+01:00", "ctime": "2014-12-06T17:50:23.34513538+01:00", "uid": 1000, "gid": 100, "user": "fd0", "inode": 416863351, "size": 1234, "links": 1, "content": [ "50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d" ] }, [...] ] } This tree contains a file entry. This time, the ``subtree`` field is not present and the ``content`` field contains a list with one plain text SHA-256 hash. The command ``restic cat blob`` can also be used to extract and decrypt data given a plaintext ID, e.g. for the data mentioned above: .. code-block:: console $ restic -r /tmp/restic-repo cat blob 50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d | sha256sum enter password for repository: 50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d - As can be seen from the output of the program ``sha256sum``, the hash matches the plaintext hash from the map included in the tree above, so the correct data has been returned. Locks ===== The restic repository structure is designed in a way that allows parallel access of multiple instance of restic and even parallel writes. However, there are some functions that work more efficient or even require exclusive access of the repository. In order to implement these functions, restic processes are required to create a lock on the repository before doing anything. Locks come in two types: Exclusive and non-exclusive locks. At most one process can have an exclusive lock on the repository, and during that time there must not be any other locks (exclusive and non-exclusive). There may be multiple non-exclusive locks in parallel. A lock is a file in the subdir ``locks`` whose filename is the storage ID of the contents. It is encrypted and authenticated the same way as other files in the repository and contains the following JSON structure: .. code:: json { "time": "2015-06-27T12:18:51.759239612+02:00", "exclusive": false, "hostname": "kasimir", "username": "fd0", "pid": 13607, "uid": 1000, "gid": 100 } The field ``exclusive`` defines the type of lock. When a new lock is to be created, restic checks all locks in the repository. When a lock is found, it is tested if the lock is stale, which is the case for locks with timestamps older than 30 minutes. If the lock was created on the same machine, even for younger locks it is tested whether the process is still alive by sending a signal to it. If that fails, restic assumes that the process is dead and considers the lock to be stale. When a new lock is to be created and no other conflicting locks are detected, restic creates a new lock, waits, and checks if other locks appeared in the repository. Depending on the type of the other locks and the lock to be created, restic either continues or fails. Backups and Deduplication ========================= For creating a backup, restic scans the source directory for all files, sub-directories and other entries. The data from each file is split into variable length Blobs cut at offsets defined by a sliding window of 64 bytes. The implementation uses Rabin Fingerprints for implementing this Content Defined Chunking (CDC). An irreducible polynomial is selected at random and saved in the file ``config`` when a repository is initialized, so that watermark attacks are much harder. Files smaller than 512 KiB are not split, Blobs are of 512 KiB to 8 MiB in size. The implementation aims for 1 MiB Blob size on average. For modified files, only modified Blobs have to be saved in a subsequent backup. This even works if bytes are inserted or removed at arbitrary positions within the file. Threat Model ============ The design goals for restic include being able to securely store backups in a location that is not completely trusted (e.g., a shared system where others can potentially access the files) or even modify or delete them in the case of the system administrator. General assumptions: - The host system a backup is created on is trusted. This is the most basic requirement, and it is essential for creating trustworthy backups. - The user uses an authentic copy of restic. - The user does not share the repository password with an attacker. - The restic backup program is not designed to protect against attackers deleting files at the storage location. There is nothing that can be done about this. If this needs to be guaranteed, get a secure location without any access from third parties. - The whole repository is re-encrypted if a key is leaked. With the current key management design, it is impossible to securely revoke a leaked key without re-encrypting the whole repository. - Advances in cryptography attacks against the cryptographic primitives used by restic (i.e, AES-256-CTR-Poly1305-AES and SHA-256) have not occurred. Such advances could render the confidentiality or integrity protections provided by restic useless. - Sufficient advances in computing have not occurred to make bruteforce attacks against restic's cryptographic protections feasible. The restic backup program guarantees the following: - Unencrypted content of stored files and metadata cannot be accessed without a password for the repository. Everything except the metadata included for informational purposes in the key files is encrypted and authenticated. The cache is also encrypted to prevent metadata leaks. - Modifications to data stored in the repository (due to bad RAM, broken harddisk, etc.) can be detected. - Data that has been tampered will not be decrypted. With the aforementioned assumptions and guarantees in mind, the following are examples of things an adversary could achieve in various circumstances. An adversary with read access to your backup storage location could: - Attempt a brute force password guessing attack against a copy of the repository (even more reason to use long, 30+ character passwords). - Infer which packs probably contain trees via file access patterns. - Infer the size of backups by using creation timestamps of repository objects. An adversary with network access could: - Attempt to DoS the server storing the backup repository or the network connection between client and server. - Determine from where you create your backups (i.e., the location where the requests originate). - Determine where you store your backups (i.e., which provider/target system). - Infer the size of backups by using creation timestamps of repository objects. The following are examples of the implications associated with violating some of the aforementioned assumptions. An adversary who compromises (via malware, physical access, etc.) the host system making backups could: - Render the entire backup process untrustworthy (e.g., intercept password, copy files, manipulate data). - Create snapshots (containing garbage data) which cover all modified files and wait until a trusted host has used forget often enough to forget all correct snapshots. - Create a garbage snapshot for every existing snapshot with a slightly different timestamp and wait until forget has run, thereby removing all correct snapshots at once. An adversary with write access to your files at the storage location could: - Delete or manipulate your backups, thereby impairing your ability to restore files from the compromised storage location. - Determine which files belong to what snapshot (e.g., based on the timestamps of the stored files). When only these files are deleted, the particular snapshot vanishes and all snapshots depending on data that has been added in the snapshot cannot be restored completely. Restic is not designed to detect this attack. An adversary who compromises a host system with append-only access to the backup repository could: - Render new backups untrustworthy *after* the host has been compromised (due to having complete control over new backups). An attacker cannot delete or manipulate old backups. As such, restoring old snapshots created *before* a host compromise remains possible. *Note: It is **not** recommended to ever run forget automatically for an append-only backup to which a potentially compromised host has access because an attacker using fake snapshots could cause forget to remove correct snapshots.* An adversary who has a leaked key for a repository which has not been re-encrypted could: - Decrypt existing and future backup data. If multiple hosts backup into the same repository, an attacker will get access to the backup data of every host. 010_introduction.rst.txt 0000644 00000001123 15030441404 0011205 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ############ Introduction ############ Restic is a fast and secure backup program. In the following sections, we will present typical workflows, starting with installing, preparing a new repository, and making the first backup. 080_examples.rst.txt 0000644 00000031232 15030441404 0010315 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ######## Examples ######## ******************************** Setting up restic with Amazon S3 ******************************** Preface ======= This tutorial will show you how to use restic with AWS S3. It will show you how to navigate the AWS web interface, create an S3 bucket, create a user with access to only this bucket, and finally how to connect restic to this bucket. Prerequisites ============= You should already have a ``restic`` binary available on your system that you can run. Furthermore, you should also have an account with `AWS <https://aws.amazon.com/>`__. You will likely need to provide credit card details for billing purposes, even if you use their `free-tier <https://aws.amazon.com/free/>`__. Logging into AWS ================ Point your browser to https://console.aws.amazon.com and log in using your AWS account. You will be presented with the AWS homepage: .. image:: images/aws_s3/01_aws_start.png :alt: AWS Homepage By using the "Services" button in the upper left corder, a menu of all services provided by AWS can be opened: .. image:: images/aws_s3/02_aws_menu.png :alt: AWS Services Menu For this tutorial, the Simple Storage Service (S3), as well as Identity and Access Management (IAM) are relevant. Creating the bucket =================== First, a bucket to store your backups in must be created. Using the "Services" menu, navigate to S3. In case you already have some S3 buckets, you will see a list of them here: .. image:: images/aws_s3/03_buckets_list_before.png :alt: List of S3 Buckets Click the "Create bucket" button and choose a name and region for your new bucket. For the purpose of this tutorial, the bucket will be named ``restic-demo`` and reside in Frankfurt. Because the bucket name space is shared among all AWS users, the name ``restic-demo`` may not be available to you. Be creative and choose a unique bucket name. .. image:: images/aws_s3/04_bucket_create_start.png :alt: Create a Bucket It is not necessary to configure any special properties or permissions of the bucket just yet. Therefore, just finish the wizard without making any further changes: .. image:: images/aws_s3/05_bucket_create_review.png :alt: Review Bucket Creation The newly created ``restic-demo`` bucket will now appear on the list of S3 buckets: .. image:: images/aws_s3/06_buckets_list_after.png :alt: List With New Bucket Creating a user =============== Use the "Services" menu of the AWS web interface to navigate to IAM. This will bring you to the IAM homepage. To create a new user, click on the "Users" menu entry on the left: .. image:: images/aws_s3/07_iam_start.png :alt: IAM Home Page In case you already have set-up users with IAM before, you will see a list of them here. Use the "Add user" button at the top to create a new user: .. image:: images/aws_s3/08_user_list.png :alt: IAM User List For this tutorial, the new user will be named ``restic-demo-user``. Feel free to choose your own name that best fits your needs. This user will only ever access AWS through the ``restic`` program and not through the web interface. Therefore, "Programmatic access" is selected for "Access type": .. image:: images/aws_s3/09_user_name.png :alt: Choose User Name and Access Type During the next step, permissions can be assigned to the new user. To use this user with restic, it only needs access to the ``restic-demo`` bucket. Select "Attach existing policies directly", which will bring up a list of pre-defined policies below. Afterwards, click the "Create policy" button to create a custom policy: .. image:: images/aws_s3/10_user_pre_policy.png :alt: Assign a Policy A new browser window or tab will open with the policy wizard. In Amazon IAM, policies are defined as JSON documents. For this tutorial, the "Visual editor" will be used to generate a policy: .. image:: images/aws_s3/11_policy_start.png :alt: Create a New Policy For restic to work, two permission statements must be created using the visual policy editor. The first statement is set up as follows: .. code:: Service: S3 Allow Actions: DeleteObject, GetObject, PutObject Resources: arn:aws:s3:::restic-demo/* This statement allows restic to create, read and delete objects inside the S3 bucket named ``restic-demo``. Adjust the bucket's name to the name of the bucket you created earlier. Next, add a second statement using the "Add additional permissions" button: .. code:: Service: S3 Allow Actions: ListBucket, GetBucketLocation Resource: arn:aws:s3:::restic-demo Again, substitute ``restic-demo`` with the actual name of your bucket. Note that, unlike before, there is no ``/*`` after the bucket name. This statement allows restic to list the objects stored in the ``restic-demo`` bucket and to query the bucket's region. Continue to the next step by clicking the "Review policy" button and enter a name and description for this policy. For this tutorial, the policy will be named ``restic-demo-policy``. Click "Create policy" to finish the process: .. image:: images/aws_s3/13_policy_review.png :alt: Policy Review Go back to the browser window or tab where you were previously creating the new user. Click the button labeled "Refresh" above the list of policies to make sure the newly created policy is available to you. Afterwards, use the search function to search for the ``restic-demo-policy``. Select this policy using the checkbox on the left. Then, continue to the next step. .. image:: images/aws_s3/14_user_attach_policy.png :alt: Attach Policy to User The next page will present an overview of the user account that is about to be created. If everything looks good, click "Create user" to complete the process: .. image:: images/aws_s3/15_user_review.png :alt: User Creation Review After the user has been created, its access credentials will be displayed. They consist of the "Access key ID" (think user name), and the "Secret access key" (think password). Copy these down to a safe place. .. image:: images/aws_s3/16_user_created.png :alt: User Credentials You have now completed the configuration in AWS. Feel free to close your web browser now. Initializing the restic repository ================================== Open a terminal and make sure you have the ``restic`` binary ready. First, choose a password to encrypt your backups with. In this tutorial, ``apg`` is used for this purpose: .. code-block:: console $ apg -a 1 -m 32 -n 1 -M NCL I9n7G7G0ZpDWA3GOcJbIuwQCGvGUBkU5 Note this password somewhere safe along with your AWS credentials. Next, the configuration of restic will be placed into environment variables. This will include sensitive information, such as your AWS secret and repository password. Therefore, make sure the next commands **do not** end up in your shell's history file. Adjust the contents of the environment variables to fit your bucket's name and your user's API credentials. .. code-block:: console $ unset HISTFILE $ export RESTIC_REPOSITORY="s3:https://s3.amazonaws.com/restic-demo" $ export AWS_ACCESS_KEY_ID="AKIAJAJSLTZCAZ4SRI5Q" $ export AWS_SECRET_ACCESS_KEY="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" $ export RESTIC_PASSWORD="I9n7G7G0ZpDWA3GOcJbIuwQCGvGUBkU5" After the environment is set up, restic may be called to initialize the repository: .. code-block:: console $ ./restic init created restic backend b5c661a86a at s3:https://s3.amazonaws.com/restic-demo Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. restic is now ready to be used with AWS S3. Try to create a backup: .. code-block:: console $ dd if=/dev/urandom bs=1M count=10 of=test.bin 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0,0891322 s, 118 MB/s $ ./restic backup test.bin scan [/home/philip/restic-demo/test.bin] scanned 0 directories, 1 files in 0:00 [0:04] 100.00% 2.500 MiB/s 10.000 MiB / 10.000 MiB 1 / 1 items ... ETA 0:00 duration: 0:04, 2.47MiB/s snapshot 10fdbace saved $ ./restic snapshots ID Date Host Tags Directory ---------------------------------------------------------------------- 10fdbace 2017-03-26 16:41:50 blackbox /home/philip/restic-demo/test.bin A snapshot was created and stored in the S3 bucket. By default backups to AWS S3 will use the ``STANDARD`` storage class. Available storage classes include ``STANDARD``, ``STANDARD_IA``, ``ONEZONE_IA``, ``INTELLIGENT_TIERING``, and ``REDUCED_REDUNDANCY``. A different storage class could have been specified in the above command by using ``-o`` or ``--option``: .. code-block:: console $ ./restic backup -o s3.storage-class=REDUCED_REDUNDANCY test.bin This snapshot may now be restored: .. code-block:: console $ mkdir restore $ ./restic restore 10fdbace --target restore restoring <Snapshot 10fdbace of [/home/philip/restic-demo/test.bin] at 2017-03-26 16:41:50.201418102 +0200 CEST by philip@blackbox> to restore $ ls restore/ test.bin The snapshot was successfully restored. This concludes the tutorial. ***************************************************** Backing up your system without running restic as root ***************************************************** Motivation ========== Creating a complete backup of a machine requires a privileged process that is able to read all files. On UNIX-like systems this is traditionally the ``root`` user. Processes running as root have superpower. They cannot only read all files but do also have the power to modify the system in any possible way. With great power comes great responsibility. If a process running as root malfunctions, is exploited, or simply configured in a wrong way it can cause any possible damage to the system. This means you only want to run programs as root that you trust completely. And even if you trust a program, it is good and common practice to run it with the least possible privileges. Capabilities on Linux ===================== Fortunately, Linux has functionality to divide root's power into single separate *capabilities*. You can remove these from a process running as root to restrict it. And you can add capabilities to a process running as a normal user, which is what we are going to do. Full backup without root ======================== To be able to completely backup a system, restic has to read all the files. Luckily Linux knows a capability that allows precisely this. We can assign this single capability to restic and then run it as an unprivileged user. First we create a new user called ``restic`` that is going to create the backups: .. code-block:: console root@a3e580b6369d:/# useradd -m restic Then we download and install the restic binary into the user's home directory (please adjust the URL to refer to the latest restic version). .. code-block:: console root@a3e580b6369d:/# mkdir ~restic/bin root@a3e580b6369d:/# curl -L https://github.com/restic/restic/releases/download/v0.9.6/restic_0.9.6_linux_amd64.bz2 | bunzip2 > ~restic/bin/restic Before we assign any special capability to the restic binary we restrict its permissions so that only root and the newly created restic user can execute it. Otherwise another - possibly untrusted - user could misuse the privileged restic binary to circumvent file access controls. .. code-block:: console root@a3e580b6369d:/# chown root:restic ~restic/bin/restic root@a3e580b6369d:/# chmod 750 ~restic/bin/restic Finally we can use ``setcap`` to add an extended attribute to the restic binary. On every execution the system will read the extended attribute, interpret it and assign capabilities accordingly. .. code-block:: console root@a3e580b6369d:/# setcap cap_dac_read_search=+ep ~restic/bin/restic .. important:: The capabilities of the ``setcap`` command only applies to this specific copy of the restic binary. If you run ``restic self-update`` or in any other way replace or update the binary, the capabilities you added above will not be in effect for the new binary. To mitigate this, simply run the ``setcap`` command again, to make sure that the new binary has the same and intended capabilities. From now on the user ``restic`` can run restic to backup the whole system. .. code-block:: console root@a3e580b6369d:/# sudo -u restic /home/restic/bin/restic --exclude={/dev,/media,/mnt,/proc,/run,/sys,/tmp,/var/tmp} -r /tmp backup / 100_references.rst.txt 0000644 00000000763 15030441404 0010616 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ########## References ########## ****** Design ****** .. include:: design.rst .. include:: cache.rst .. include:: REST_backend.rst 075_scripting.rst.txt 0000644 00000002622 15030441404 0010506 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ######################### Scripting ######################### This is a list of how certain tasks may be accomplished when you use restic via scripts. Check if a repository is already initialized ******************************************** You may find a need to check if a repository is already initialized, perhaps to prevent your script from initializing a repository multiple times. The command ``snapshots`` may be used for this purpose: .. code-block:: console $ restic -r /srv/restic-repo snapshots Fatal: unable to open config file: Stat: stat /srv/restic-repo/config: no such file or directory Is there a repository at the following location? /srv/restic-repo If a repository does not exist, restic will return a non-zero exit code and print an error message. Note that restic will also return a non-zero exit code if a different error is encountered (e.g.: incorrect password to ``snapshots``) and it may print a different error message. If there are no errors, restic will return a zero exit code and print all the snapshots. 030_preparing_a_new_repo.rst.txt 0000644 00000063670 15030441404 0012672 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ########################## Preparing a new repository ########################## The place where your backups will be saved is called a "repository". This chapter explains how to create ("init") such a repository. The repository can be stored locally, or on some remote server or service. We'll first cover using a local repository; the remaining sections of this chapter cover all the other options. You can skip to the next chapter once you've read the relevant section here. For automated backups, restic accepts the repository location in the environment variable ``RESTIC_REPOSITORY``. Restic can also read the repository location from a file specified via the ``--repository-file`` option or the environment variable ``RESTIC_REPOSITORY_FILE``. For the password, several options exist: * Setting the environment variable ``RESTIC_PASSWORD`` * Specifying the path to a file with the password via the option ``--password-file`` or the environment variable ``RESTIC_PASSWORD_FILE`` * Configuring a program to be called when the password is needed via the option ``--password-command`` or the environment variable ``RESTIC_PASSWORD_COMMAND`` Local ***** In order to create a repository at ``/srv/restic-repo``, run the following command and enter the same password twice: .. code-block:: console $ restic init --repo /srv/restic-repo enter password for new repository: enter password again: created restic repository 085b3c76b9 at /srv/restic-repo Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. .. warning:: Remembering your password is important! If you lose it, you won't be able to access data stored in the repository. .. warning:: On Linux, storing the backup repository on a CIFS (SMB) share is not recommended due to compatibility issues. Either use another backend or set the environment variable `GODEBUG` to `asyncpreemptoff=1`. Refer to GitHub issue `#2659 <https://github.com/restic/restic/issues/2659>`_ for further explanations. SFTP **** In order to backup data via SFTP, you must first set up a server with SSH and let it know your public key. Passwordless login is really important since restic fails to connect to the repository if the server prompts for credentials. Once the server is configured, the setup of the SFTP repository can simply be achieved by changing the URL scheme in the ``init`` command: .. code-block:: console $ restic -r sftp:user@host:/srv/restic-repo init enter password for new repository: enter password again: created restic repository f1c6108821 at sftp:user@host:/srv/restic-repo Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. You can also specify a relative (read: no slash (``/``) character at the beginning) directory, in this case the dir is relative to the remote user's home directory. Also, if the SFTP server is enforcing domain-confined users, you can specify the user this way: ``user@domain@host``. .. note:: Please be aware that sftp servers do not expand the tilde character (``~``) normally used as an alias for a user's home directory. If you want to specify a path relative to the user's home directory, pass a relative path to the sftp backend. If you need to specify a port number or IPv6 address, you'll need to use URL syntax. E.g., the repository ``/srv/restic-repo`` on ``[::1]`` (localhost) at port 2222 with username ``user`` can be specified as :: sftp://user@[::1]:2222//srv/restic-repo Note the double slash: the first slash separates the connection settings from the path, while the second is the start of the path. To specify a relative path, use one slash. Alternatively, you can create an entry in the ``ssh`` configuration file, usually located in your home directory at ``~/.ssh/config`` or in ``/etc/ssh/ssh_config``: :: Host foo User bar Port 2222 Then use the specified host name ``foo`` normally (you don't need to specify the user name in this case): :: $ restic -r sftp:foo:/srv/restic-repo init You can also add an entry with a special host name which does not exist, just for use with restic, and use the ``Hostname`` option to set the real host name: :: Host restic-backup-host Hostname foo User bar Port 2222 Then use it in the backend specification: :: $ restic -r sftp:restic-backup-host:/srv/restic-repo init Last, if you'd like to use an entirely different program to create the SFTP connection, you can specify the command to be run with the option ``-o sftp.command="foobar"``. .. note:: Please be aware that sftp servers close connections when no data is received by the client. This can happen when restic is processing huge amounts of unchanged data. To avoid this issue add the following lines to the client’s .ssh/config file: :: ServerAliveInterval 60 ServerAliveCountMax 240 REST Server *********** In order to backup data to the remote server via HTTP or HTTPS protocol, you must first set up a remote `REST server <https://github.com/restic/rest-server>`__ instance. Once the server is configured, accessing it is achieved by changing the URL scheme like this: .. code-block:: console $ restic -r rest:http://host:8000/ Depending on your REST server setup, you can use HTTPS protocol, password protection, multiple repositories or any combination of those features. The TCP/IP port is also configurable. Here are some more examples: .. code-block:: console $ restic -r rest:https://host:8000/ $ restic -r rest:https://user:pass@host:8000/ $ restic -r rest:https://user:pass@host:8000/my_backup_repo/ If you use TLS, restic will use the system's CA certificates to verify the server certificate. When the verification fails, restic refuses to proceed and exits with an error. If you have your own self-signed certificate, or a custom CA certificate should be used for verification, you can pass restic the certificate filename via the ``--cacert`` option. It will then verify that the server's certificate is contained in the file passed to this option, or signed by a CA certificate in the file. In this case, the system CA certificates are not considered at all. REST server uses exactly the same directory structure as local backend, so you should be able to access it both locally and via HTTP, even simultaneously. Amazon S3 ********* Restic can backup data to any Amazon S3 bucket. However, in this case, changing the URL scheme is not enough since Amazon uses special security credentials to sign HTTP requests. By consequence, you must first setup the following environment variables with the credentials you obtained while creating the bucket. .. code-block:: console $ export AWS_ACCESS_KEY_ID=<MY_ACCESS_KEY> $ export AWS_SECRET_ACCESS_KEY=<MY_SECRET_ACCESS_KEY> You can then easily initialize a repository that uses your Amazon S3 as a backend. If the bucket does not exist it will be created in the default location: .. code-block:: console $ restic -r s3:s3.amazonaws.com/bucket_name init enter password for new repository: enter password again: created restic repository eefee03bbd at s3:s3.amazonaws.com/bucket_name Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. If needed, you can manually specify the region to use by either setting the environment variable ``AWS_DEFAULT_REGION`` or calling restic with an option parameter like ``-o s3.region="us-east-1"``. If the region is not specified, the default region is used. Afterwards, the S3 server (at least for AWS, ``s3.amazonaws.com``) will redirect restic to the correct endpoint. Until version 0.8.0, restic used a default prefix of ``restic``, so the files in the bucket were placed in a directory named ``restic``. If you want to access a repository created with an older version of restic, specify the path after the bucket name like this: .. code-block:: console $ restic -r s3:s3.amazonaws.com/bucket_name/restic [...] For an S3-compatible server that is not Amazon (like Minio, see below), or is only available via HTTP, you can specify the URL to the server like this: ``s3:http://server:port/bucket_name``. .. note:: restic expects `path-style URLs <https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro>`__ like for example ``s3.us-west-2.amazonaws.com/bucket_name``. Virtual-hosted–style URLs like ``bucket_name.s3.us-west-2.amazonaws.com``, where the bucket name is part of the hostname are not supported. These must be converted to path-style URLs instead, for example ``s3.us-west-2.amazonaws.com/bucket_name``. .. note:: Certain S3-compatible servers do not properly implement the ``ListObjectsV2`` API, most notably Ceph versions before v14.2.5. On these backends, as a temporary workaround, you can provide the ``-o s3.list-objects-v1=true`` option to use the older ``ListObjects`` API instead. This option may be removed in future versions of restic. Minio Server ************ `Minio <https://www.minio.io>`__ is an Open Source Object Storage, written in Go and compatible with AWS S3 API. - Download and Install `Minio Server <https://minio.io/downloads/#minio-server>`__. - You can also refer to https://docs.minio.io for step by step guidance on installation and getting started on Minio Client and Minio Server. You must first setup the following environment variables with the credentials of your Minio Server. .. code-block:: console $ export AWS_ACCESS_KEY_ID=<YOUR-MINIO-ACCESS-KEY-ID> $ export AWS_SECRET_ACCESS_KEY= <YOUR-MINIO-SECRET-ACCESS-KEY> Now you can easily initialize restic to use Minio server as a backend with this command. .. code-block:: console $ ./restic -r s3:http://localhost:9000/restic init enter password for new repository: enter password again: created restic repository 6ad29560f5 at s3:http://localhost:9000/restic1 Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. Wasabi ************ `Wasabi <https://wasabi.com>`__ is a low cost AWS S3 conformant object storage provider. Due to it's S3 conformance, Wasabi can be used as a storage provider for a restic repository. - Create a Wasabi bucket using the `Wasabi Console <https://console.wasabisys.com>`__. - Determine the correct Wasabi service URL for your bucket `here <https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-regions->`__. You must first setup the following environment variables with the credentials of your Wasabi account. .. code-block:: console $ export AWS_ACCESS_KEY_ID=<YOUR-WASABI-ACCESS-KEY-ID> $ export AWS_SECRET_ACCESS_KEY=<YOUR-WASABI-SECRET-ACCESS-KEY> Now you can easily initialize restic to use Wasabi as a backend with this command. .. code-block:: console $ ./restic -r s3:https://<WASABI-SERVICE-URL>/<WASABI-BUCKET-NAME> init enter password for new repository: enter password again: created restic repository xxxxxxxxxx at s3:https://<WASABI-SERVICE-URL>/<WASABI-BUCKET-NAME> Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. Alibaba Cloud (Aliyun) Object Storage System (OSS) ************************************************** `Alibaba OSS <https://www.alibabacloud.com/product/oss/>`__ is an encrypted, secure, cost-effective, and easy-to-use object storage service that enables you to store, back up, and archive large amounts of data in the cloud. Alibaba OSS is S3 compatible so it can be used as a storage provider for a restic repository with a couple of extra parameters. - Determine the correct `Alibaba OSS region endpoint <https://www.alibabacloud.com/help/doc-detail/31837.htm>`__ - this will be something like ``oss-eu-west-1.aliyuncs.com`` - You'll need the region name too - this will be something like ``oss-eu-west-1`` You must first setup the following environment variables with the credentials of your Alibaba OSS account. .. code-block:: console $ export AWS_ACCESS_KEY_ID=<YOUR-OSS-ACCESS-KEY-ID> $ export AWS_SECRET_ACCESS_KEY=<YOUR-OSS-SECRET-ACCESS-KEY> Now you can easily initialize restic to use Alibaba OSS as a backend with this command. .. code-block:: console $ ./restic -o s3.bucket-lookup=dns -o s3.region=<OSS-REGION> -r s3:https://<OSS-ENDPOINT>/<OSS-BUCKET-NAME> init enter password for new backend: enter password again: created restic backend xxxxxxxxxx at s3:https://<OSS-ENDPOINT>/<OSS-BUCKET-NAME> Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. For example with an actual endpoint: .. code-block:: console $ restic -o s3.bucket-lookup=dns -o s3.region=oss-eu-west-1 -r s3:https://oss-eu-west-1.aliyuncs.com/bucketname init OpenStack Swift *************** Restic can backup data to an OpenStack Swift container. Because Swift supports various authentication methods, credentials are passed through environment variables. In order to help integration with existing OpenStack installations, the naming convention of those variables follows the official Python Swift client: .. code-block:: console # For keystone v1 authentication $ export ST_AUTH=<MY_AUTH_URL> $ export ST_USER=<MY_USER_NAME> $ export ST_KEY=<MY_USER_PASSWORD> # For keystone v2 authentication (some variables are optional) $ export OS_AUTH_URL=<MY_AUTH_URL> $ export OS_REGION_NAME=<MY_REGION_NAME> $ export OS_USERNAME=<MY_USERNAME> $ export OS_PASSWORD=<MY_PASSWORD> $ export OS_TENANT_ID=<MY_TENANT_ID> $ export OS_TENANT_NAME=<MY_TENANT_NAME> # For keystone v3 authentication (some variables are optional) $ export OS_AUTH_URL=<MY_AUTH_URL> $ export OS_REGION_NAME=<MY_REGION_NAME> $ export OS_USERNAME=<MY_USERNAME> $ export OS_USER_ID=<MY_USER_ID> $ export OS_PASSWORD=<MY_PASSWORD> $ export OS_USER_DOMAIN_NAME=<MY_DOMAIN_NAME> $ export OS_USER_DOMAIN_ID=<MY_DOMAIN_ID> $ export OS_PROJECT_NAME=<MY_PROJECT_NAME> $ export OS_PROJECT_DOMAIN_NAME=<MY_PROJECT_DOMAIN_NAME> $ export OS_PROJECT_DOMAIN_ID=<MY_PROJECT_DOMAIN_ID> $ export OS_TRUST_ID=<MY_TRUST_ID> # For keystone v3 application credential authentication (application credential id) $ export OS_AUTH_URL=<MY_AUTH_URL> $ export OS_APPLICATION_CREDENTIAL_ID=<MY_APPLICATION_CREDENTIAL_ID> $ export OS_APPLICATION_CREDENTIAL_SECRET=<MY_APPLICATION_CREDENTIAL_SECRET> # For keystone v3 application credential authentication (application credential name) $ export OS_AUTH_URL=<MY_AUTH_URL> $ export OS_USERNAME=<MY_USERNAME> $ export OS_USER_DOMAIN_NAME=<MY_DOMAIN_NAME> $ export OS_APPLICATION_CREDENTIAL_NAME=<MY_APPLICATION_CREDENTIAL_NAME> $ export OS_APPLICATION_CREDENTIAL_SECRET=<MY_APPLICATION_CREDENTIAL_SECRET> # For authentication based on tokens $ export OS_STORAGE_URL=<MY_STORAGE_URL> $ export OS_AUTH_TOKEN=<MY_AUTH_TOKEN> Restic should be compatible with an `OpenStack RC file <https://docs.openstack.org/user-guide/common/cli-set-environment-variables-using-openstack-rc.html>`__ in most cases. Once environment variables are set up, a new repository can be created. The name of the Swift container and optional path can be specified. If the container does not exist, it will be created automatically: .. code-block:: console $ restic -r swift:container_name:/path init # path is optional enter password for new repository: enter password again: created restic repository eefee03bbd at swift:container_name:/path Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. The policy of the new container created by restic can be changed using environment variable: .. code-block:: console $ export SWIFT_DEFAULT_CONTAINER_POLICY=<MY_CONTAINER_POLICY> Backblaze B2 ************ Restic can backup data to any Backblaze B2 bucket. You need to first setup the following environment variables with the credentials you can find in the dashboard on the "Buckets" page when signed into your B2 account: .. code-block:: console $ export B2_ACCOUNT_ID=<MY_APPLICATION_KEY_ID> $ export B2_ACCOUNT_KEY=<MY_APPLICATION_KEY> .. note:: As of version 0.9.2, restic supports both master and non-master `application keys <https://www.backblaze.com/b2/docs/application_keys.html>`__. If using a non-master application key, ensure that it is created with at least **read and write** access to the B2 bucket. On earlier versions of restic, a master application key is required. You can then initialize a repository stored at Backblaze B2. If the bucket does not exist yet and the credentials you passed to restic have the privilege to create buckets, it will be created automatically: .. code-block:: console $ restic -r b2:bucketname:path/to/repo init enter password for new repository: enter password again: created restic repository eefee03bbd at b2:bucketname:path/to/repo Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. Note that the bucket name must be unique across all of B2. The number of concurrent connections to the B2 service can be set with the ``-o b2.connections=10`` switch. By default, at most five parallel connections are established. Microsoft Azure Blob Storage **************************** You can also store backups on Microsoft Azure Blob Storage. Export the Azure account name and key as follows: .. code-block:: console $ export AZURE_ACCOUNT_NAME=<ACCOUNT_NAME> $ export AZURE_ACCOUNT_KEY=<SECRET_KEY> Afterwards you can initialize a repository in a container called ``foo`` in the root path like this: .. code-block:: console $ restic -r azure:foo:/ init enter password for new repository: enter password again: created restic repository a934bac191 at azure:foo:/ [...] The number of concurrent connections to the Azure Blob Storage service can be set with the ``-o azure.connections=10`` switch. By default, at most five parallel connections are established. Google Cloud Storage ******************** Restic supports Google Cloud Storage as a backend and connects via a `service account`_. For normal restic operation, the service account must have the ``storage.objects.{create,delete,get,list}`` permissions for the bucket. These are included in the "Storage Object Admin" role. ``restic init`` can create the repository bucket. Doing so requires the ``storage.buckets.create`` permission ("Storage Admin" role). If the bucket already exists, that permission is unnecessary. To use the Google Cloud Storage backend, first `create a service account key`_ and download the JSON credentials file. Second, find the Google Project ID that you can see in the Google Cloud Platform console at the "Storage/Settings" menu. Export the path to the JSON key file and the project ID as follows: .. code-block:: console $ export GOOGLE_PROJECT_ID=123123123123 $ export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gs-secret-restic-key.json Restic uses Google's client library to generate `default authentication material`_, which means if you're running in Google Container Engine or are otherwise located on an instance with default service accounts then these should work out of the box. Alternatively, you can specify an existing access token directly: .. code-block:: console $ export GOOGLE_ACCESS_TOKEN=ya29.a0AfH6SMC78... If ``GOOGLE_ACCESS_TOKEN`` is set all other authentication mechanisms are disabled. The access token must have at least the ``https://www.googleapis.com/auth/devstorage.read_write`` scope. Keep in mind that access tokens are short-lived (usually one hour), so they are not suitable if creating a backup takes longer than that, for instance. Once authenticated, you can use the ``gs:`` backend type to create a new repository in the bucket ``foo`` at the root path: .. code-block:: console $ restic -r gs:foo:/ init enter password for new repository: enter password again: created restic repository bde47d6254 at gs:foo2/ [...] The number of concurrent connections to the GCS service can be set with the ``-o gs.connections=10`` switch. By default, at most five parallel connections are established. .. _service account: https://cloud.google.com/storage/docs/authentication#service_accounts .. _create a service account key: https://cloud.google.com/storage/docs/authentication#generating-a-private-key .. _default authentication material: https://developers.google.com/identity/protocols/application-default-credentials Other Services via rclone ************************* The program `rclone`_ can be used to access many other different services and store data there. First, you need to install and `configure`_ rclone. The general backend specification format is ``rclone:<remote>:<path>``, the ``<remote>:<path>`` component will be directly passed to rclone. When you configure a remote named ``foo``, you can then call restic as follows to initiate a new repository in the path ``bar`` in the repo: .. code-block:: console $ restic -r rclone:foo:bar init Restic takes care of starting and stopping rclone. As a more concrete example, suppose you have configured a remote named ``b2prod`` for Backblaze B2 with rclone, with a bucket called ``yggdrasil``. You can then use rclone to list files in the bucket like this: .. code-block:: console $ rclone ls b2prod:yggdrasil In order to create a new repository in the root directory of the bucket, call restic like this: .. code-block:: console $ restic -r rclone:b2prod:yggdrasil init If you want to use the path ``foo/bar/baz`` in the bucket instead, pass this to restic: .. code-block:: console $ restic -r rclone:b2prod:yggdrasil/foo/bar/baz init Listing the files of an empty repository directly with rclone should return a listing similar to the following: .. code-block:: console $ rclone ls b2prod:yggdrasil/foo/bar/baz 155 bar/baz/config 448 bar/baz/keys/4bf9c78049de689d73a56ed0546f83b8416795295cda12ec7fb9465af3900b44 Rclone can be `configured with environment variables`_, so for instance configuring a bandwidth limit for rclone can be achieved by setting the ``RCLONE_BWLIMIT`` environment variable: .. code-block:: console $ export RCLONE_BWLIMIT=1M For debugging rclone, you can set the environment variable ``RCLONE_VERBOSE=2``. The rclone backend has two additional options: * ``-o rclone.program`` specifies the path to rclone, the default value is just ``rclone`` * ``-o rclone.args`` allows setting the arguments passed to rclone, by default this is ``serve restic --stdio --b2-hard-delete`` The reason for the ``--b2-hard-delete`` parameters can be found in the corresponding GitHub `issue #1657`_. In order to start rclone, restic will build a list of arguments by joining the following lists (in this order): ``rclone.program``, ``rclone.args`` and as the last parameter the value that follows the ``rclone:`` prefix of the repository specification. So, calling restic like this .. code-block:: console $ restic -o rclone.program="/path/to/rclone" \ -o rclone.args="serve restic --stdio --bwlimit 1M --b2-hard-delete --verbose" \ -r rclone:b2:foo/bar runs rclone as follows: .. code-block:: console $ /path/to/rclone serve restic --stdio --bwlimit 1M --b2-hard-delete --verbose b2:foo/bar Manually setting ``rclone.program`` also allows running a remote instance of rclone e.g. via SSH on a server, for example: .. code-block:: console $ restic -o rclone.program="ssh user@remotehost rclone" -r rclone:b2:foo/bar With these options, restic works with local files. It uses rclone and credentials stored on ``remotehost`` to communicate with B2. All data (except credentials) is encrypted/decrypted locally, then sent/received via ``remotehost`` to/from B2. A more advanced version of this setup forbids specific hosts from removing files in a repository. See the `blog post by Simon Ruderich <https://ruderich.org/simon/notes/append-only-backups-with-restic-and-rclone>`_ for details. The rclone command may also be hard-coded in the SSH configuration or the user's public key, in this case it may be sufficient to just start the SSH connection (and it's irrelevant what's passed after ``rclone:`` in the repository specification): .. code-block:: console $ restic -o rclone.program="ssh user@host" -r rclone:x .. _rclone: https://rclone.org/ .. _configure: https://rclone.org/docs/ .. _configured with environment variables: https://rclone.org/docs/#environment-variables .. _issue #1657: https://github.com/restic/restic/pull/1657#issuecomment-377707486 Password prompt on Windows ************************** At the moment, restic only supports the default Windows console interaction. If you use emulation environments like `MSYS2 <https://msys2.github.io/>`__ or `Cygwin <https://www.cygwin.com/>`__, which use terminals like ``Mintty`` or ``rxvt``, you may get a password error. You can workaround this by using a special tool called ``winpty`` (look `here <https://github.com/msys2/msys2/wiki/Porting>`__ and `here <https://github.com/rprichard/winpty>`__ for detail information). On MSYS2, you can install ``winpty`` as follows: .. code-block:: console $ pacman -S winpty $ winpty restic -r /srv/restic-repo init index.rst.txt 0000644 00000000550 15030441404 0007216 0 ustar 00 Restic Documentation ==================== .. toctree:: :maxdepth: 2 010_introduction 020_installation 030_preparing_a_new_repo 040_backup 045_working_with_repos 050_restore 060_forget 070_encryption 075_scripting 080_examples 090_participating 100_references 110_talks faq manual_rest developer_information REST_backend.rst.txt 0000644 00000010020 15030441404 0010324 0 ustar 00 ************ REST Backend ************ Restic can interact with HTTP Backend that respects the following REST API. The following values are valid for ``{type}``: * ``data`` * ``keys`` * ``locks`` * ``snapshots`` * ``index`` * ``config`` The API version is selected via the ``Accept`` HTTP header in the request. The following values are defined: * ``application/vnd.x.restic.rest.v1`` or empty: Select API version 1 * ``application/vnd.x.restic.rest.v2``: Select API version 2 The server will respond with the value of the highest version it supports in the ``Content-Type`` HTTP response header for the HTTP requests which should return JSON. Any different value for this header means API version 1. The placeholder ``{path}`` in this document is a path to the repository, so that multiple different repositories can be accessed. The default path is ``/``. The path must end with a slash. POST {path}?create=true ======================= This request is used to initially create a new repository. The server responds with "200 OK" if the repository structure was created successfully or already exists, otherwise an error is returned. DELETE {path} ============= Deletes the repository on the server side. The server responds with "200 OK" if the repository was successfully removed. If this function is not implemented the server returns "501 Not Implemented", if this it is denied by the server it returns "403 Forbidden". HEAD {path}/config ================== Returns "200 OK" if the repository has a configuration, an HTTP error otherwise. GET {path}/config ================= Returns the content of the configuration file if the repository has a configuration, an HTTP error otherwise. Response format: binary/octet-stream POST {path}/config ================== Returns "200 OK" if the configuration of the request body has been saved, an HTTP error otherwise. GET {path}/{type}/ ================== API version 1 ------------- Returns a JSON array containing the names of all the blobs stored for a given type, example: .. code:: json [ "245bc4c430d393f74fbe7b13325e30dbde9fb0745e50caad57c446c93d20096b", "85b420239efa1132c41cea0065452a40ebc20c6f8e0b132a5b2f5848360973ec", "8e2006bb5931a520f3c7009fe278d1ebb87eb72c3ff92a50c30e90f1b8cf3e60", "e75c8c407ea31ba399ab4109f28dd18c4c68303d8d86cc275432820c42ce3649" ] API version 2 ------------- Returns a JSON array containing an object for each file of the given type. The objects have two keys: ``name`` for the file name, and ``size`` for the size in bytes. .. code:: json [ { "name": "245bc4c430d393f74fbe7b13325e30dbde9fb0745e50caad57c446c93d20096b", "size": 2341058 }, { "name": "85b420239efa1132c41cea0065452a40ebc20c6f8e0b132a5b2f5848360973ec", "size": 2908900 }, { "name": "8e2006bb5931a520f3c7009fe278d1ebb87eb72c3ff92a50c30e90f1b8cf3e60", "size": 3030712 }, { "name": "e75c8c407ea31ba399ab4109f28dd18c4c68303d8d86cc275432820c42ce3649", "size": 2804 } ] HEAD {path}/{type}/{name} ========================= Returns "200 OK" if the blob with the given name and type is stored in the repository, "404 not found" otherwise. If the blob exists, the HTTP header ``Content-Length`` is set to the file size. GET {path}/{type}/{name} ======================== Returns the content of the blob with the given name and type if it is stored in the repository, "404 not found" otherwise. If the request specifies a partial read with a Range header field, then the status code of the response is 206 instead of 200 and the response only contains the specified range. Response format: binary/octet-stream POST {path}/{type}/{name} ========================= Saves the content of the request body as a blob with the given name and type, an HTTP error otherwise. Request format: binary/octet-stream DELETE {path}/{type}/{name} =========================== Returns "200 OK" if the blob with the given name and type has been deleted from the repository, an HTTP error otherwise. faq.rst.txt 0000644 00000021351 15030441404 0006660 0 ustar 00 FAQ === This is the list of Frequently Asked Questions for restic. Will restic resume an interrupted backup? ----------------------------------------- Yes, restic will resume interrupted backups when they are re-run. When backing up, restic periodically writes index files to keep a record of the uploaded data. Even if there's no snapshot created in the end (due to the backup being interrupted), these indexes are stored in the repository for the data that has been uploaded so far. Next time restic runs, it is then able to find the uploaded data through these indexes, and thereby reference it again without having to upload it a second time. This effectively makes it continue from where it saved the last index, which should be up to a few minutes ago. It does not matter if the backup was interrupted by the user or if it was due to unforeseen circumstances such as connectivity issues, power loss, etc. Simply re-run the backup again and restic should only upload what it needs to in order to complete the interrupted backup and create a snapshot. Note however that during the initial backup run and any re-tries, until there has been a first snapshot created for the backup set (list of files and directories to be backed up), restic will need to re-scan the files on disk as there is no parent snapshot to compare the filesystem with to determine which files have changed. This process should however be far quicker than the uploading, and it's normal to see restic scan the files again when re-running the backup. ``restic check`` reports packs that aren't referenced in any index, is my repository broken? -------------------------------------------------------------------------------------------- When ``restic check`` reports that there are pack files in the repository that are not referenced in any index, that's (in contrast to what restic reports at the moment) not a source for concern. The output looks like this: :: $ restic check Create exclusive lock for repository Load indexes Check all packs pack 819a9a52e4f51230afa89aefbf90df37fb70996337ae57e6f7a822959206a85e: not referenced in any index pack de299e69fb075354a3775b6b045d152387201f1cdc229c31d1caa34c3b340141: not referenced in any index Check snapshots, trees and blobs Fatal: repository contains errors The message means that there is more data stored in the repo than strictly necessary. With high probability this is duplicate data. In order to clean it up, the command ``restic prune`` can be used. The cause of this bug is not yet known. I ran a ``restic`` command but it is not working as intended, what do I do now? ------------------------------------------------------------------------------- If you are running a restic command and it is not working as you hoped it would, there is an easy way of checking how your shell interpreted the command you are trying to run. Here is an example of a mistake in a backup command that results in the command not working as expected. A user wants to run the following ``restic backup`` command :: $ restic backup --exclude "~/documents" ~ .. important:: This command contains an intentional user error described in this paragraph. This command will result in a complete backup of the current logged in user's home directory and it won't exclude the folder ``~/documents/`` - which is not what the user wanted to achieve. The problem is how the path to ``~/documents`` is passed to restic. In order to spot an issue like this, you can make use of the following ruby command preceding your restic command. :: $ ruby -e 'puts ARGV.inspect' restic backup --exclude "~/documents" ~ ["restic", "backup", "--exclude", "~/documents", "/home/john"] As you can see, the command outputs every argument you have passed to the shell. This is what restic sees when you run your command. The error here is that the tilde ``~`` in ``"~/documents"`` didn't get expanded as it is quoted. :: $ echo ~/documents /home/john/documents $ echo "~/documents" ~/document $ echo "$HOME/documents" /home/john/documents Restic handles globbing and expansion in the following ways: - Globbing is only expanded for lines read via ``--files-from`` - Environment variables are not expanded in the file read via ``--files-from`` - ``*`` is expanded for paths read via ``--files-from`` - e.g. For backup targets given to restic as arguments on the shell, neither glob expansion nor shell variable replacement is done. If restic is called as ``restic backup '*' '$HOME'``, it will try to backup the literal file(s)/dir(s) ``*`` and ``$HOME`` - Double-asterisk ``**`` only works in exclude patterns as this is a custom extension built into restic; the shell must not expand it How can I specify encryption passwords automatically? ----------------------------------------------------- When you run ``restic backup``, you need to enter the passphrase on the console. This is not very convenient for automated backups, so you can also provide the password through the ``--password-file`` option, or one of the environment variables ``RESTIC_PASSWORD`` or ``RESTIC_PASSWORD_FILE``. A discussion is in progress over implementing unattended backups happens in :issue:`533`. .. important:: Be careful how you set the environment; using the env command, a `system()` call or using inline shell scripts (e.g. `RESTIC_PASSWORD=password restic ...`) might expose the credentials in the process list directly and they will be readable to all users on a system. Using export in a shell script file should be safe, however, as the environment of a process is `accessible only to that user`_. Please make sure that the permissions on the files where the password is eventually stored are safe (e.g. `0600` and owned by root). .. _accessible only to that user: https://security.stackexchange.com/questions/14000/environment-variable-accessibility-in-linux/14009#14009 How to prioritize restic's IO and CPU time ------------------------------------------ If you'd like to change the **IO priority** of restic, run it in the following way :: $ ionice -c2 -n0 ./restic -r /media/your/backup/ backup /home This runs ``restic`` in the so-called best *effort class* (``-c2``), with the highest possible priority (``-n0``). Take a look at the `ionice manpage`_ to learn about the other classes. .. _ionice manpage: https://linux.die.net/man/1/ionice To change the **CPU scheduling priority** to a higher-than-standard value, use would run: :: $ nice --10 ./restic -r /media/your/backup/ backup /home Again, the `nice manpage`_ has more information. .. _nice manpage: https://linux.die.net/man/1/nice You can also **combine IO and CPU scheduling priority**: :: $ ionice -c2 nice -n19 ./restic -r /media/gour/backup/ backup /home This example puts restic in the IO class 2 (best effort) and tells the CPU scheduling algorithm to give it the least favorable niceness (19). The above example makes sure that the system the backup runs on is not slowed down, which is particularly useful for servers. Creating new repo on a Synology NAS via sftp fails -------------------------------------------------- For using restic with a Synology NAS via sftp, please make sure that the specified path is absolute, it must start with a slash (``/``). Sometimes creating a new restic repository on a Synology NAS via sftp fails with an error similar to the following: :: $ restic init -r sftp:user@nas:/volume1/restic-repo init create backend at sftp:user@nas:/volume1/restic-repo/ failed: mkdirAll(/volume1/restic-repo/index): unable to create directories: [...] Although you can log into the NAS via SSH and see that the directory structure is there. The reason for this behavior is that apparently Synology NAS expose a different directory structure via sftp, so the path that needs to be specified is different than the directory structure on the device and maybe even as exposed via other protocols. Try removing the ``/volume1`` prefix in your paths. If this does not work, use sftp and ls to explore the SFTP file system hierarchy on your NAS. The following may work: :: $ restic init -r sftp:user@nas:/restic-repo init Why does restic perform so poorly on Windows? --------------------------------------------- In some cases the real-time protection of antivirus software can interfere with restic's operations. If you are experiencing bad performance you can try to temporarily disable your antivirus software to find out if it is the cause for your performance problems. If you are certain that the antivirus software is the cause for this and you want to gain maximum performance, you have to add the restic binary to an exclusions list within the antivirus software. 050_restore.rst.txt 0000644 00000012742 15030441404 0010164 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ##################### Restoring from backup ##################### Restoring from a snapshot ========================= Restoring a snapshot is as easy as it sounds, just use the following command to restore the contents of the latest snapshot to ``/tmp/restore-work``: .. code-block:: console $ restic -r /srv/restic-repo restore 79766175 --target /tmp/restore-work enter password for repository: restoring <Snapshot of [/home/user/work] at 2015-05-08 21:40:19.884408621 +0200 CEST> to /tmp/restore-work Use the word ``latest`` to restore the last backup. You can also combine ``latest`` with the ``--host`` and ``--path`` filters to choose the last backup for a specific host, path or both. .. code-block:: console $ restic -r /srv/restic-repo restore latest --target /tmp/restore-art --path "/home/art" --host luigi enter password for repository: restoring <Snapshot of [/home/art] at 2015-05-08 21:45:17.884408621 +0200 CEST> to /tmp/restore-art Use ``--exclude`` and ``--include`` to restrict the restore to a subset of files in the snapshot. For example, to restore a single file: .. code-block:: console $ restic -r /srv/restic-repo restore 79766175 --target /tmp/restore-work --include /work/foo enter password for repository: restoring <Snapshot of [/home/user/work] at 2015-05-08 21:40:19.884408621 +0200 CEST> to /tmp/restore-work This will restore the file ``foo`` to ``/tmp/restore-work/work/foo``. You can use the command ``restic ls latest`` or ``restic find foo`` to find the path to the file within the snapshot. This path you can then pass to ``--include`` in verbatim to only restore the single file or directory. There are case insensitive variants of ``--exclude`` and ``--include`` called ``--iexclude`` and ``--iinclude``. These options will behave the same way but ignore the casing of paths. Restore using mount =================== Browsing your backup as a regular file system is also very easy. First, create a mount point such as ``/mnt/restic`` and then use the following command to serve the repository with FUSE: .. code-block:: console $ mkdir /mnt/restic $ restic -r /srv/restic-repo mount /mnt/restic enter password for repository: Now serving /srv/restic-repo at /mnt/restic When finished, quit with Ctrl-c or umount the mountpoint. Mounting repositories via FUSE is only possible on Linux, macOS and FreeBSD. On Linux, the ``fuse`` kernel module needs to be loaded and the ``fusermount`` command needs to be in the ``PATH``. On macOS, you need `FUSE for macOS <https://osxfuse.github.io/>`__. On FreeBSD, you may need to install FUSE and load the kernel module (``kldload fuse``). Restic supports storage and preservation of hard links. However, since hard links exist in the scope of a filesystem by definition, restoring hard links from a fuse mount should be done by a program that preserves hard links. A program that does so is ``rsync``, used with the option --hard-links. Printing files to stdout ======================== Sometimes it's helpful to print files to stdout so that other programs can read the data directly. This can be achieved by using the `dump` command, like this: .. code-block:: console $ restic -r /srv/restic-repo dump latest production.sql | mysql If you have saved multiple different things into the same repo, the ``latest`` snapshot may not be the right one. For example, consider the following snapshots in a repo: .. code-block:: console $ restic -r /srv/restic-repo snapshots ID Date Host Tags Directory ---------------------------------------------------------------------- 562bfc5e 2018-07-14 20:18:01 mopped /home/user/file1 bbacb625 2018-07-14 20:18:07 mopped /home/other/work e922c858 2018-07-14 20:18:10 mopped /home/other/work 098db9d5 2018-07-14 20:18:13 mopped /production.sql b62f46ec 2018-07-14 20:18:16 mopped /home/user/file1 1541acae 2018-07-14 20:18:18 mopped /home/other/work ---------------------------------------------------------------------- Here, restic would resolve ``latest`` to the snapshot ``1541acae``, which does not contain the file we'd like to print at all (``production.sql``). In this case, you can pass restic the snapshot ID of the snapshot you like to restore: .. code-block:: console $ restic -r /srv/restic-repo dump 098db9d5 production.sql | mysql Or you can pass restic a path that should be used for selecting the latest snapshot. The path must match the patch printed in the "Directory" column, e.g.: .. code-block:: console $ restic -r /srv/restic-repo dump --path /production.sql latest production.sql | mysql It is also possible to ``dump`` the contents of a whole folder structure to stdout. To retain the information about the files and folders Restic will output the contents in the tar (default) or zip format: .. code-block:: console $ restic -r /srv/restic-repo dump latest /home/other/work > restore.tar .. code-block:: console $ restic -r /srv/restic-repo dump -a zip latest /home/other/work > restore.zip 040_backup.rst.txt 0000644 00000061277 15030441404 0007754 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ########## Backing up ########## Now we're ready to backup some data. The contents of a directory at a specific point in time is called a "snapshot" in restic. Run the following command and enter the repository password you chose above again: .. code-block:: console $ restic -r /srv/restic-repo --verbose backup ~/work open repository enter password for repository: password is correct lock repository load index files start scan start backup scan finished in 1.837s processed 1.720 GiB in 0:12 Files: 5307 new, 0 changed, 0 unmodified Dirs: 1867 new, 0 changed, 0 unmodified Added: 1.200 GiB snapshot 40dc1520 saved As you can see, restic created a backup of the directory and was pretty fast! The specific snapshot just created is identified by a sequence of hexadecimal characters, ``40dc1520`` in this case. You can see that restic tells us it processed 1.720 GiB of data, this is the size of the files and directories in ``~/work`` on the local file system. It also tells us that only 1.200 GiB was added to the repository. This means that some of the data was duplicate and restic was able to efficiently reduce it. If you don't pass the ``--verbose`` option, restic will print less data. You'll still get a nice live status display. Be aware that the live status shows the processed files and not the transferred data. Transferred volume might be lower (due to de-duplication) or higher. On Windows, the ``--use-fs-snapshot`` option will use Windows' Volume Shadow Copy Service (VSS) when creating backups. Restic will transparently create a VSS snapshot for each volume that contains files to backup. Files are read from the VSS snapshot instead of the regular filesystem. This allows to backup files that are exclusively locked by another process during the backup. By default VSS ignores Outlook OST files. This is not a restriction of restic but the default Windows VSS configuration. The files not to snapshot are configured in the Windows registry under the following key: .. code-block:: console HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\BackupRestore\FilesNotToSnapshot For more details refer the official Windows documentation e.g. the article ``Registry Keys and Values for Backup and Restore``. If you run the backup command again, restic will create another snapshot of your data, but this time it's even faster and no new data was added to the repository (since all data is already there). This is de-duplication at work! .. code-block:: console $ restic -r /srv/restic-repo --verbose backup ~/work open repository enter password for repository: password is correct lock repository load index files using parent snapshot d875ae93 start scan start backup scan finished in 1.881s processed 1.720 GiB in 0:03 Files: 0 new, 0 changed, 5307 unmodified Dirs: 0 new, 0 changed, 1867 unmodified Added: 0 B snapshot 79766175 saved You can even backup individual files in the same repository (not passing ``--verbose`` means less output): .. code-block:: console $ restic -r /srv/restic-repo backup ~/work.txt enter password for repository: password is correct snapshot 249d0210 saved If you're interested in what restic does, pass ``--verbose`` twice (or ``--verbose=2``) to display detailed information about each file and directory restic encounters: .. code-block:: console $ echo 'more data foo bar' >> ~/work.txt $ restic -r /srv/restic-repo --verbose --verbose backup ~/work.txt open repository enter password for repository: password is correct lock repository load index files using parent snapshot f3f8d56b start scan start backup scan finished in 2.115s modified /home/user/work.txt, saved in 0.007s (22 B added) modified /home/user/, saved in 0.008s (0 B added, 378 B metadata) modified /home/, saved in 0.009s (0 B added, 375 B metadata) processed 22 B in 0:02 Files: 0 new, 1 changed, 0 unmodified Dirs: 0 new, 2 changed, 0 unmodified Data Blobs: 1 new Tree Blobs: 3 new Added: 1.116 KiB snapshot 8dc503fc saved In fact several hosts may use the same repository to backup directories and files leading to a greater de-duplication. Now is a good time to run ``restic check`` to verify that all data is properly stored in the repository. You should run this command regularly to make sure the internal structure of the repository is free of errors. File change detection ********************* When restic encounters a file that has already been backed up, whether in the current backup or a previous one, it makes sure the file's contents are only stored once in the repository. To do so, it normally has to scan the entire contents of every file. Because this can be very expensive, restic also uses a change detection rule based on file metadata to determine whether a file is likely unchanged since a previous backup. If it is, the file is not scanned again. Change detection is only performed for regular files (not special files, symlinks or directories) that have the exact same path as they did in a previous backup of the same location. If a file or one of its containing directories was renamed, it is considered a different file and its entire contents will be scanned again. Metadata changes (permissions, ownership, etc.) are always included in the backup, even if file contents are considered unchanged. On **Unix** (including Linux and Mac), given that a file lives at the same location as a file in a previous backup, the following file metadata attributes have to match for its contents to be presumed unchanged: * Modification timestamp (mtime). * Metadata change timestamp (ctime). * File size. * Inode number (internal number used to reference a file in a filesystem). The reason for requiring both mtime and ctime to match is that Unix programs can freely change mtime (and some do). In such cases, a ctime change may be the only hint that a file did change. The following ``restic backup`` command line flags modify the change detection rules: * ``--force``: turn off change detection and rescan all files. * ``--ignore-ctime``: require mtime to match, but allow ctime to differ. * ``--ignore-inode``: require mtime to match, but allow inode number and ctime to differ. The option ``--ignore-inode`` exists to support FUSE-based filesystems and pCloud, which do not assign stable inodes to files. Note that the device id of the containing mount point is never taken into account. Device numbers are not stable for removable devices and ZFS snapshots. If you want to force a re-scan in such a case, you can change the mountpoint. On **Windows**, a file is considered unchanged when its path, size and modification time match, and only ``--force`` has any effect. The other options are recognized but ignored. Excluding Files *************** You can exclude folders and files by specifying exclude patterns, currently the exclude options are: - ``--exclude`` Specified one or more times to exclude one or more items - ``--iexclude`` Same as ``--exclude`` but ignores the case of paths - ``--exclude-caches`` Specified once to exclude folders containing a special file - ``--exclude-file`` Specified one or more times to exclude items listed in a given file - ``--iexclude-file`` Same as ``exclude-file`` but ignores cases like in ``--iexclude`` - ``--exclude-if-present foo`` Specified one or more times to exclude a folder's content if it contains a file called ``foo`` (optionally having a given header, no wildcards for the file name supported) - ``--exclude-larger-than size`` Specified once to excludes files larger than the given size Please see ``restic help backup`` for more specific information about each exclude option. Let's say we have a file called ``excludes.txt`` with the following content: :: # exclude go-files *.go # exclude foo/x/y/z/bar foo/x/bar foo/bar foo/**/bar It can be used like this: .. code-block:: console $ restic -r /srv/restic-repo backup ~/work --exclude="*.c" --exclude-file=excludes.txt This instructs restic to exclude files matching the following criteria: * All files matching ``*.c`` (parameter ``--exclude``) * All files matching ``*.go`` (second line in ``excludes.txt``) * All files and sub-directories named ``bar`` which reside somewhere below a directory called ``foo`` (fourth line in ``excludes.txt``) Patterns use `filepath.Glob <https://golang.org/pkg/path/filepath/#Glob>`__ internally, see `filepath.Match <https://golang.org/pkg/path/filepath/#Match>`__ for syntax. Patterns are tested against the full path of a file/dir to be saved, even if restic is passed a relative path to save. Empty lines and lines starting with a ``#`` are ignored. Environment variables in exclude files are expanded with `os.ExpandEnv <https://golang.org/pkg/os/#ExpandEnv>`__, so ``/home/$USER/foo`` will be expanded to ``/home/bob/foo`` for the user ``bob``. To get a literal dollar sign, write ``$$`` to the file - this has to be done even when there's no matching environment variable for the word following a single ``$``. Note that tilde (``~``) is not expanded, instead use the ``$HOME`` or equivalent environment variable (depending on your operating system). Patterns need to match on complete path components. For example, the pattern ``foo``: * matches ``/dir1/foo/dir2/file`` and ``/dir/foo`` * does not match ``/dir/foobar`` or ``barfoo`` A trailing ``/`` is ignored, a leading ``/`` anchors the pattern at the root directory. This means, ``/bin`` matches ``/bin/bash`` but does not match ``/usr/bin/restic``. Regular wildcards cannot be used to match over the directory separator ``/``, e.g. ``b*ash`` matches ``/bin/bash`` but does not match ``/bin/ash``. For this, the special wildcard ``**`` can be used to match arbitrary sub-directories: The pattern ``foo/**/bar`` matches: * ``/dir1/foo/dir2/bar/file`` * ``/foo/bar/file`` * ``/tmp/foo/bar`` Spaces in patterns listed in an exclude file can be specified verbatim. That is, in order to exclude a file named ``foo bar star.txt``, put that just as it reads on one line in the exclude file. Please note that beginning and trailing spaces are trimmed - in order to match these, use e.g. a ``*`` at the beginning or end of the filename. Spaces in patterns listed in the other exclude options (e.g. ``--exclude`` on the command line) are specified in different ways depending on the operating system and/or shell. Restic itself does not need any escaping, but your shell may need some escaping in order to pass the name/pattern as a single argument to restic. On most Unixy shells, you can either quote or use backslashes. For example: * ``--exclude='foo bar star/foo.txt'`` * ``--exclude="foo bar star/foo.txt"`` * ``--exclude=foo\ bar\ star/foo.txt`` By specifying the option ``--one-file-system`` you can instruct restic to only backup files from the file systems the initially specified files or directories reside on. In other words, it will prevent restic from crossing filesystem boundaries and subvolumes when performing a backup. For example, if you backup ``/`` with this option and you have external media mounted under ``/media/usb`` then restic will not back up ``/media/usb`` at all because this is a different filesystem than ``/``. Virtual filesystems such as ``/proc`` are also considered different and thereby excluded when using ``--one-file-system``: .. code-block:: console $ restic -r /srv/restic-repo backup --one-file-system / Please note that this does not prevent you from specifying multiple filesystems on the command line, e.g: .. code-block:: console $ restic -r /srv/restic-repo backup --one-file-system / /media/usb will back up both the ``/`` and ``/media/usb`` filesystems, but will not include other filesystems like ``/sys`` and ``/proc``. .. note:: ``--one-file-system`` is currently unsupported on Windows, and will cause the backup to immediately fail with an error. Files larger than a given size can be excluded using the `--exclude-larger-than` option: .. code-block:: console $ restic -r /srv/restic-repo backup ~/work --exclude-larger-than 1M This excludes files in ``~/work`` which are larger than 1 MB from the backup. The default unit for the size value is bytes, so e.g. ``--exclude-larger-than 2048`` would exclude files larger than 2048 bytes (2 kilobytes). To specify other units, suffix the size value with one of ``k``/``K`` for kilobytes, ``m``/``M`` for megabytes, ``g``/``G`` for gigabytes and ``t``/``T`` for terabytes (e.g. ``1k``, ``10K``, ``20m``, ``20M``, ``30g``, ``30G``, ``2t`` or ``2T``). Including Files *************** The options ``--files-from``, ``--files-from-verbatim`` and ``--files-from-raw`` allow you to give restic a file containing lists of file patterns or paths to be backed up. This is useful e.g. when you want to back up files from many different locations, or when you use some other software to generate the list of files to back up. The argument passed to ``--files-from`` must be the name of a text file that contains one *pattern* per line. The file must be encoded as UTF-8, or UTF-16 with a byte-order mark. Leading and trailing whitespace is removed from the patterns. Empty lines and lines starting with a ``#`` are ignored and each pattern is expanded when read, such that special characters in it are expanded using the Go function `filepath.Glob <https://golang.org/pkg/path/filepath/#Glob>`__ - please see its documentation for the syntax you can use in the patterns. The argument passed to ``--files-from-verbatim`` must be the name of a text file that contains one *path* per line, e.g. as generated by GNU ``find`` with the ``-print`` flag. Unlike ``--files-from``, ``--files-from-verbatim`` does not expand any special characters in the list of paths, does not strip off any whitespace and does not ignore lines starting with a ``#``. This option simply reads and uses each line as-is, although empty lines are still ignored. Use this option when you want to backup a list of filenames containing the special characters that would otherwise be expanded when using ``--files-from``. The ``--files-from-raw`` option is a variant of ``--files-from-verbatim`` that requires each line in the file to be terminated by an ASCII NUL character (the ``\0`` zero byte) instead of a newline, so that it can even handle file paths containing newlines in their name or are not encoded as UTF-8 (except on Windows, where the listed filenames must still be encoded in UTF-8. This option is the safest choice when generating the list of filenames from a script (e.g. GNU ``find`` with the ``-print0`` flag). All three options interpret the argument ``-`` as standard input and will read the list of files/patterns from there instead of a text file. In all cases, paths may be absolute or relative to ``restic backup``'s working directory. For example, maybe you want to backup files which have a name that matches a certain regular expression pattern (uses GNU ``find``): .. code-block:: console $ find /tmp/some_folder -regex PATTERN -print0 > /tmp/files_to_backup You can then use restic to backup the filtered files: .. code-block:: console $ restic -r /srv/restic-repo backup --files-from-raw /tmp/files_to_backup You can combine all three options with each other and with the normal file arguments: .. code-block:: console $ restic backup --files-from /tmp/files_to_backup /tmp/some_additional_file $ restic backup --files-from /tmp/glob-pattern --files-from-raw /tmp/generated-list /tmp/some_additional_file Comparing Snapshots ******************* Restic has a `diff` command which shows the difference between two snapshots and displays a small statistic, just pass the command two snapshot IDs: .. code-block:: console $ restic -r /srv/restic-repo diff 5845b002 2ab627a6 password is correct comparing snapshot ea657ce5 to 2ab627a6: C /restic/cmd_diff.go + /restic/foo C /restic/restic Files: 0 new, 0 removed, 2 changed Dirs: 1 new, 0 removed Others: 0 new, 0 removed Data Blobs: 14 new, 15 removed Tree Blobs: 2 new, 1 removed Added: 16.403 MiB Removed: 16.402 MiB Backing up special items and metadata ************************************* **Symlinks** are archived as symlinks, ``restic`` does not follow them. When you restore, you get the same symlink again, with the same link target and the same timestamps. If there is a **bind-mount** below a directory that is to be saved, restic descends into it. **Device files** are saved and restored as device files. This means that e.g. ``/dev/sda`` is archived as a block device file and restored as such. This also means that the content of the corresponding disk is not read, at least not from the device file. By default, restic does not save the access time (atime) for any files or other items, since it is not possible to reliably disable updating the access time by restic itself. This means that for each new backup a lot of metadata is written, and the next backup needs to write new metadata again. If you really want to save the access time for files and directories, you can pass the ``--with-atime`` option to the ``backup`` command. Reading data from stdin *********************** Sometimes it can be nice to directly save the output of a program, e.g. ``mysqldump`` so that the SQL can later be restored. Restic supports this mode of operation, just supply the option ``--stdin`` to the ``backup`` command like this: .. code-block:: console $ set -o pipefail $ mysqldump [...] | restic -r /srv/restic-repo backup --stdin This creates a new snapshot of the output of ``mysqldump``. You can then use e.g. the fuse mounting option (see below) to mount the repository and read the file. By default, the file name ``stdin`` is used, a different name can be specified with ``--stdin-filename``, e.g. like this: .. code-block:: console $ mysqldump [...] | restic -r /srv/restic-repo backup --stdin --stdin-filename production.sql The option ``pipefail`` is highly recommended so that a non-zero exit code from one of the programs in the pipe (e.g. ``mysqldump`` here) makes the whole chain return a non-zero exit code. Refer to the `Use the Unofficial Bash Strict Mode <http://redsymbol.net/articles/unofficial-bash-strict-mode/>`__ for more details on this. Tags for backup *************** Snapshots can have one or more tags, short strings which add identifying information. Just specify the tags for a snapshot one by one with ``--tag``: .. code-block:: console $ restic -r /srv/restic-repo backup --tag projectX --tag foo --tag bar ~/work [...] The tags can later be used to keep (or forget) snapshots with the ``forget`` command. The command ``tag`` can be used to modify tags on an existing snapshot. Space requirements ****************** Restic currently assumes that your backup repository has sufficient space for the backup operation you are about to perform. This is a realistic assumption for many cloud providers, but may not be true when backing up to local disks. Should you run out of space during the middle of a backup, there will be some additional data in the repository, but the snapshot will never be created as it would only be written at the very (successful) end of the backup operation. Previous snapshots will still be there and will still work. Environment Variables ********************* In addition to command-line options, restic supports passing various options in environment variables. The following lists these environment variables: .. code-block:: console RESTIC_REPOSITORY_FILE Name of file containing the repository location (replaces --repository-file) RESTIC_REPOSITORY Location of repository (replaces -r) RESTIC_PASSWORD_FILE Location of password file (replaces --password-file) RESTIC_PASSWORD The actual password for the repository RESTIC_PASSWORD_COMMAND Command printing the password for the repository to stdout RESTIC_KEY_HINT ID of key to try decrypting first, before other keys RESTIC_CACHE_DIR Location of the cache directory RESTIC_PROGRESS_FPS Frames per second by which the progress bar is updated TMPDIR Location for temporary files AWS_ACCESS_KEY_ID Amazon S3 access key ID AWS_SECRET_ACCESS_KEY Amazon S3 secret access key AWS_DEFAULT_REGION Amazon S3 default region ST_AUTH Auth URL for keystone v1 authentication ST_USER Username for keystone v1 authentication ST_KEY Password for keystone v1 authentication OS_AUTH_URL Auth URL for keystone authentication OS_REGION_NAME Region name for keystone authentication OS_USERNAME Username for keystone authentication OS_USER_ID User ID for keystone v3 authentication OS_PASSWORD Password for keystone authentication OS_TENANT_ID Tenant ID for keystone v2 authentication OS_TENANT_NAME Tenant name for keystone v2 authentication OS_USER_DOMAIN_NAME User domain name for keystone authentication OS_USER_DOMAIN_ID User domain ID for keystone v3 authentication OS_PROJECT_NAME Project name for keystone authentication OS_PROJECT_DOMAIN_NAME Project domain name for keystone authentication OS_PROJECT_DOMAIN_ID Project domain ID for keystone v3 authentication OS_TRUST_ID Trust ID for keystone v3 authentication OS_APPLICATION_CREDENTIAL_ID Application Credential ID (keystone v3) OS_APPLICATION_CREDENTIAL_NAME Application Credential Name (keystone v3) OS_APPLICATION_CREDENTIAL_SECRET Application Credential Secret (keystone v3) OS_STORAGE_URL Storage URL for token authentication OS_AUTH_TOKEN Auth token for token authentication B2_ACCOUNT_ID Account ID or applicationKeyId for Backblaze B2 B2_ACCOUNT_KEY Account Key or applicationKey for Backblaze B2 AZURE_ACCOUNT_NAME Account name for Azure AZURE_ACCOUNT_KEY Account key for Azure GOOGLE_PROJECT_ID Project ID for Google Cloud Storage GOOGLE_APPLICATION_CREDENTIALS Application Credentials for Google Cloud Storage (e.g. $HOME/.config/gs-secret-restic-key.json) RCLONE_BWLIMIT rclone bandwidth limit See :ref:`caching` for the rules concerning cache locations when ``RESTIC_CACHE_DIR`` is not set. The external programs that restic may execute include ``rclone`` (for rclone backends) and ``ssh`` (for the SFTP backend). These may respond to further environment variables and configuration files; see their respective manuals. Exit status codes ***************** Restic returns one of the following exit status codes after the backup command is run: * 0 when the backup was successful (snapshot with all source files created) * 1 when there was a fatal error (no snapshot created) * 3 when some source files could not be read (incomplete snapshot with remaining files created) Fatal errors occur for example when restic is unable to write to the backup destination, when there are network connectivity issues preventing successful communication, or when an invalid password or command line argument is provided. When restic returns this exit status code, one should not expect a snapshot to have been created. Source file read errors occur when restic fails to read one or more files or directories that it was asked to back up, e.g. due to permission problems. Restic displays the number of source file read errors that occurred while running the backup. If there are errors of this type, restic will still try to complete the backup run with all the other files, and create a snapshot that then contains all but the unreadable files. One can use these exit status codes in scripts and other automation tools, to make them aware of the outcome of the backup run. To manually inspect the exit code in e.g. Linux, run ``echo $?``. 060_forget.rst.txt 0000644 00000042627 15030441404 0007775 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ######################### Removing backup snapshots ######################### All backup space is finite, so restic allows removing old snapshots. This can be done either manually (by specifying a snapshot ID to remove) or by using a policy that describes which snapshots to forget. For all remove operations, two commands need to be called in sequence: ``forget`` to remove a snapshot and ``prune`` to actually remove the data that was referenced by the snapshot from the repository. This can be automated with the ``--prune`` option of the ``forget`` command, which runs ``prune`` automatically if snapshots have been removed. Pruning snapshots can be a time-consuming process, depending on the amount of snapshots and data to process. During a prune operation, the repository is locked and backups cannot be completed. Please plan your pruning so that there's time to complete it and it doesn't interfere with regular backup runs. It is advisable to run ``restic check`` after pruning, to make sure you are alerted, should the internal data structures of the repository be damaged. Remove a single snapshot ************************ The command ``snapshots`` can be used to list all snapshots in a repository like this: .. code-block:: console $ restic -r /srv/restic-repo snapshots enter password for repository: ID Date Host Tags Directory ---------------------------------------------------------------------- 40dc1520 2015-05-08 21:38:30 kasimir /home/user/work 79766175 2015-05-08 21:40:19 kasimir /home/user/work bdbd3439 2015-05-08 21:45:17 luigi /home/art 590c8fc8 2015-05-08 21:47:38 kazik /srv 9f0bc19e 2015-05-08 21:46:11 luigi /srv In order to remove the snapshot of ``/home/art``, use the ``forget`` command and specify the snapshot ID on the command line: .. code-block:: console $ restic -r /srv/restic-repo forget bdbd3439 enter password for repository: removed snapshot bdbd3439 Afterwards this snapshot is removed: .. code-block:: console $ restic -r /srv/restic-repo snapshots enter password for repository: ID Date Host Tags Directory ---------------------------------------------------------------------- 40dc1520 2015-05-08 21:38:30 kasimir /home/user/work 79766175 2015-05-08 21:40:19 kasimir /home/user/work 590c8fc8 2015-05-08 21:47:38 kazik /srv 9f0bc19e 2015-05-08 21:46:11 luigi /srv But the data that was referenced by files in this snapshot is still stored in the repository. To cleanup unreferenced data, the ``prune`` command must be run: .. code-block:: console $ restic -r /srv/restic-repo prune enter password for repository: repository 33002c5e opened successfully, password is correct loading all snapshots... loading indexes... finding data that is still in use for 4 snapshots [0:00] 100.00% 4 / 4 snapshots searching used packs... collecting packs for deletion and repacking [0:00] 100.00% 5 / 5 packs processed to repack: 69 blobs / 1.078 MiB this removes 67 blobs / 1.047 MiB to delete: 7 blobs / 25.726 KiB total prune: 74 blobs / 1.072 MiB remaining: 16 blobs / 38.003 KiB unused size after prune: 0 B (0.00% of remaining size) repacking packs [0:00] 100.00% 2 / 2 packs repacked rebuilding index [0:00] 100.00% 3 / 3 packs processed deleting obsolete index files [0:00] 100.00% 3 / 3 files deleted removing 3 old packs [0:00] 100.00% 3 / 3 files deleted done Afterwards the repository is smaller. You can automate this two-step process by using the ``--prune`` switch to ``forget``: .. code-block:: console $ restic forget --keep-last 1 --prune snapshots for host mopped, directories /home/user/work: keep 1 snapshots: ID Date Host Tags Directory ---------------------------------------------------------------------- 4bba301e 2017-02-21 10:49:18 mopped /home/user/work remove 1 snapshots: ID Date Host Tags Directory ---------------------------------------------------------------------- 8c02b94b 2017-02-21 10:48:33 mopped /home/user/work 1 snapshots have been removed, running prune loading all snapshots... loading indexes... finding data that is still in use for 1 snapshots [0:00] 100.00% 1 / 1 snapshots searching used packs... collecting packs for deletion and repacking [0:00] 100.00% 5 / 5 packs processed to repack: 69 blobs / 1.078 MiB this removes 67 blobs / 1.047 MiB to delete: 7 blobs / 25.726 KiB total prune: 74 blobs / 1.072 MiB remaining: 16 blobs / 38.003 KiB unused size after prune: 0 B (0.00% of remaining size) repacking packs [0:00] 100.00% 2 / 2 packs repacked rebuilding index [0:00] 100.00% 3 / 3 packs processed deleting obsolete index files [0:00] 100.00% 3 / 3 files deleted removing 3 old packs [0:00] 100.00% 3 / 3 files deleted done Removing snapshots according to a policy **************************************** Removing snapshots manually is tedious and error-prone, therefore restic allows specifying which snapshots should be removed automatically according to a policy. You can specify how many hourly, daily, weekly, monthly and yearly snapshots to keep, any other snapshots are removed. The most important command-line parameter here is ``--dry-run`` which instructs restic to not remove anything but print which snapshots would be removed. When ``forget`` is run with a policy, restic loads the list of all snapshots, then groups these by host name and list of directories. The grouping options can be set with ``--group-by``, to only group snapshots by paths and tags use ``--group-by paths,tags``. The policy is then applied to each group of snapshots separately. This is a safety feature. The ``forget`` command accepts the following parameters: - ``--keep-last n`` never delete the ``n`` last (most recent) snapshots - ``--keep-hourly n`` for the last ``n`` hours in which a snapshot was made, keep only the last snapshot for each hour. - ``--keep-daily n`` for the last ``n`` days which have one or more snapshots, only keep the last one for that day. - ``--keep-weekly n`` for the last ``n`` weeks which have one or more snapshots, only keep the last one for that week. - ``--keep-monthly n`` for the last ``n`` months which have one or more snapshots, only keep the last one for that month. - ``--keep-yearly n`` for the last ``n`` years which have one or more snapshots, only keep the last one for that year. - ``--keep-tag`` keep all snapshots which have all tags specified by this option (can be specified multiple times). - ``--keep-within duration`` keep all snapshots which have been made within the duration of the latest snapshot. ``duration`` needs to be a number of years, months, days, and hours, e.g. ``2y5m7d3h`` will keep all snapshots made in the two years, five months, seven days, and three hours before the latest snapshot. - ``--keep-within-hourly duration`` keep all hourly snapshots made within specified duration of the latest snapshot. The duration is specified in the same way as for ``--keep-within`` and the method for determining hourly snapshots is the same as for ``--keep-hourly``. - ``--keep-within-daily duration`` keep all daily snapshots made within specified duration of the latest snapshot. - ``--keep-within-weekly duration`` keep all weekly snapshots made within specified duration of the latest snapshot. - ``--keep-within-monthly duration`` keep all monthly snapshots made within specified duration of the latest snapshot. - ``--keep-within-yearly duration`` keep all yearly snapshots made within specified duration of the latest snapshot. .. note:: All calendar related ``--keep-*`` options work on the natural time boundaries and not relative to when you run the ``forget`` command. Weeks are Monday 00:00 -> Sunday 23:59, days 00:00 to 23:59, hours :00 to :59, etc. .. note:: Specifying ``--keep-tag ''`` will match untagged snapshots only. Multiple policies will be ORed together so as to be as inclusive as possible for keeping snapshots. Additionally, you can restrict removing snapshots to those which have a particular hostname with the ``--host`` parameter, or tags with the ``--tag`` option. When multiple tags are specified, only the snapshots which have all the tags are considered. For example, the following command removes all but the latest snapshot of all snapshots that have the tag ``foo``: .. code-block:: console $ restic forget --tag foo --keep-last 1 This command removes all but the last snapshot of all snapshots that have either the ``foo`` or ``bar`` tag set: .. code-block:: console $ restic forget --tag foo --tag bar --keep-last 1 To only keep the last snapshot of all snapshots with both the tag ``foo`` and ``bar`` set use: .. code-block:: console $ restic forget --tag foo,bar --keep-last 1 To ensure only untagged snapshots are considered, specify the empty string '' as the tag. .. code-block:: console $ restic forget --tag '' --keep-last 1 All the ``--keep-*`` options above only count hours/days/weeks/months/years which have a snapshot, so those without a snapshot are ignored. For safety reasons, restic refuses to act on an "empty" policy. For example, if one were to specify ``--keep-last 0`` to forget *all* snapshots in the repository, restic will respond that no snapshots will be removed. To delete all snapshots, use ``--keep-last 1`` and then finally remove the last snapshot ID manually (by passing the ID to ``forget``). All snapshots are evaluated against all matching ``--keep-*`` counts. A single snapshot on 2017-09-30 (Sat) will count as a daily, weekly and monthly. Let's explain this with an example: Suppose you have only made a backup on each Sunday for 12 weeks: .. code-block:: console $ restic snapshots repository f00c6e2a opened successfully, password is correct ID Time Host Tags Paths --------------------------------------------------------------- 0a1f9759 2019-09-01 11:00:00 mopped /home/user/work 46cfe4d5 2019-09-08 11:00:00 mopped /home/user/work f6b1f037 2019-09-15 11:00:00 mopped /home/user/work eb430a5d 2019-09-22 11:00:00 mopped /home/user/work 8cf1cb9a 2019-09-29 11:00:00 mopped /home/user/work 5d33b116 2019-10-06 11:00:00 mopped /home/user/work b9553125 2019-10-13 11:00:00 mopped /home/user/work e1a7b58b 2019-10-20 11:00:00 mopped /home/user/work 8f8018c0 2019-10-27 11:00:00 mopped /home/user/work 59403279 2019-11-03 11:00:00 mopped /home/user/work dfee9fb4 2019-11-10 11:00:00 mopped /home/user/work e1ae2f40 2019-11-17 11:00:00 mopped /home/user/work --------------------------------------------------------------- 12 snapshots Then ``forget --keep-daily 4`` will keep the last four snapshots for the last four Sundays, but remove the rest: .. code-block:: console $ restic forget --keep-daily 4 --dry-run repository f00c6e2a opened successfully, password is correct Applying Policy: keep the last 4 daily snapshots keep 4 snapshots: ID Time Host Tags Reasons Paths ------------------------------------------------------------------------------- 8f8018c0 2019-10-27 11:00:00 mopped daily snapshot /home/user/work 59403279 2019-11-03 11:00:00 mopped daily snapshot /home/user/work dfee9fb4 2019-11-10 11:00:00 mopped daily snapshot /home/user/work e1ae2f40 2019-11-17 11:00:00 mopped daily snapshot /home/user/work ------------------------------------------------------------------------------- 4 snapshots remove 8 snapshots: ID Time Host Tags Paths --------------------------------------------------------------- 0a1f9759 2019-09-01 11:00:00 mopped /home/user/work 46cfe4d5 2019-09-08 11:00:00 mopped /home/user/work f6b1f037 2019-09-15 11:00:00 mopped /home/user/work eb430a5d 2019-09-22 11:00:00 mopped /home/user/work 8cf1cb9a 2019-09-29 11:00:00 mopped /home/user/work 5d33b116 2019-10-06 11:00:00 mopped /home/user/work b9553125 2019-10-13 11:00:00 mopped /home/user/work e1a7b58b 2019-10-20 11:00:00 mopped /home/user/work --------------------------------------------------------------- 8 snapshots The result of the ``forget --keep-daily`` operation does not depend on when it is run, it will only count the days for which a snapshot exists. This is a safety feature: it prevents restic from removing snapshots when no new ones are created. Otherwise, running ``forget --keep-daily 4`` on a Friday (without any snapshot Monday to Thursday) would remove all snapshots! Another example: Suppose you make daily backups for 100 years. Then ``forget --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 75`` will keep the most recent 7 daily snapshots, then 4 (remember, 7 dailies already include a week!) last-day-of-the-weeks and 11 or 12 last-day-of-the-months (11 or 12 depends if the 5 weeklies cross a month). And finally 75 last-day-of-the-year snapshots. All other snapshots are removed. You might want to maintain the same policy as for the example above, but have irregular backups. For example, the 7 snapshots specified with ``--keep-daily 7`` might be spread over a longer period. If what you want is to keep daily snapshots for a week, weekly for a month, monthly for a year and yearly for 75 years, you could specify: ``forget --keep-daily-within 7d --keep-weekly-within 1m --keep-monthly-within 1y --keep-yearly-within 75y`` (Note that `1w` is not a recognized duration, so you will have to specify `7d` instead) Customize pruning ***************** To understand the custom options, we first explain how the pruning process works: 1. All snapshots and directories within snapshots are scanned to determine which data is still in use. 2. For all files in the repository, restic finds out if the file is fully used, partly used or completely unused. 3. Completely unused files are marked for deletion. Fully used files are kept. A partially used file is either kept or marked for repacking depending on user options. Note that for repacking, restic must download the file from the repository storage and re-upload the needed data in the repository. This can be very time-consuming for remote repositories. 4. After deciding what to do, ``prune`` will actually perform the repack, modify the index according to the changes and delete the obsolete files. The ``prune`` command accepts the following options: - ``--max-unused limit`` allow unused data up to the specified limit within the repository. This allows restic to keep partly used files instead of repacking them. The limit can be specified in several ways: * As an absolute size (e.g. ``200M``). If you want to minimize the space used by your repository, pass ``0`` to this option. * As a size relative to the total repo size (e.g. ``10%``). This means that after prune, at most ``10%`` of the total data stored in the repo may be unused data. If the repo after prune has as size of 500MB, then at most 50MB may be unused. * If the string ``unlimited`` is passed, there is no limit for partly unused files. This means that as long as some data is still used within a file stored in the repo, restic will just leave it there. Use this if you want to minimize the time and bandwidth used by the ``prune`` operation. Note that metadata will still be repacked. Restic tries to repack as little data as possible while still ensuring this limit for unused data. The default value is 5%. - ``--max-repack-size size`` if set limits the total size of files to repack. As ``prune`` first stores all repacked files and deletes the obsolete files at the end, this option might be handy if you expect many files to be repacked and fear to run low on storage. - ``--repack-cacheable-only`` if set to true only files which contain metadata and would be stored in the cache are repacked. Other pack files are not repacked if this option is set. This allows a very fast repacking using only cached data. It can, however, imply that the unused data in your repository exceeds the value given by ``--max-unused``. The default value is false. - ``--dry-run`` only show what ``prune`` would do. - ``--verbose`` increased verbosity shows additional statistics for ``prune``. 110_talks.rst.txt 0000644 00000002325 15030441404 0007610 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ##### Talks ##### The following talks will be or have been given about restic: - 2016-01-31: Lightning Talk at the Go Devroom at FOSDEM 2016, Brussels, Belgium - 2016-01-29: `restic - Backups mal richtig <https://media.ccc.de/v/c4.openchaos.2016.01.restic>`__: Public lecture in German at `CCC Cologne e.V. <https://koeln.ccc.de>`__ in Cologne, Germany - 2015-08-23: `A Solution to the Backup Inconvenience <https://programm.froscon.de/2015/events/1515.html>`__: Lecture at `FROSCON 2015 <https://www.froscon.de>`__ in Bonn, Germany - 2015-02-01: `Lightning Talk at FOSDEM 2015 <https://www.youtube.com/watch?v=oM-MfeflUZ8&t=11m40s>`__: A short introduction (with slightly outdated command line) - 2015-01-27: `Talk about restic at CCC Aachen <https://videoag.fsmpi.rwth-aachen.de/?view=player&lectureid=4442#content>`__ (in German) developer_information.rst.txt 0000644 00000011270 15030441404 0012502 0 ustar 00 Developer Information ##################### Reproducible Builds ******************* This section describes how to reproduce the official released binaries for restic for version 0.10.0 and later. For restic versions down to 0.9.3 please refer to the documentation for the respective version. The binary produced depends on the following things: * The source code for the release * The exact version of the official `Go compiler <https://golang.org>`__ used to produce the binaries (running ``restic version`` will print this) * The architecture and operating system the Go compiler runs on (Linux, ``amd64``) * The path where the source code is extracted to (``/restic``) * The path to the Go compiler (``/usr/local/go``) * The build tags (for official binaries, it's the tag ``selfupdate``) * The environment variables (mostly ``$GOOS``, ``$GOARCH``, ``$CGO_ENABLED``) In addition, The compressed ZIP files for Windows depends on the modification timestamp and filename of the binary contained in it. In order to reproduce the exact same ZIP file every time, we update the timestamp of the file ``VERSION`` in the source code archive and set the timezone to Europe/Berlin. In the following example, we'll use the file ``restic-0.10.0.tar.gz`` and Go 1.15.2 to reproduce the released binaries. 1. Determine the Go compiler version used to build the released binaries, then download and extract the Go compiler into ``/usr/local/go``: .. code:: $ restic version restic 0.10.0 compiled with go1.15.2 on linux/amd64 $ cd /usr/local $ curl -L https://dl.google.com/go/go1.15.2.linux-amd64.tar.gz | tar xz 2. Extract the restic source code into ``/restic`` .. code:: $ mkdir /restic $ cd /restic $ TZ=Europe/Berlin curl -L https://github.com/restic/restic/releases/download/v0.10.0/restic-0.10.0.tar.gz | tar xz --strip-components=1 3. Build the binaries for Windows and Linux: .. code:: $ export PATH=/usr/local/go/bin:$PATH $ go version go version go1.15.2 linux/amd64 $ GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags "-s -w" -tags selfupdate -o restic_linux_amd64 ./cmd/restic $ bzip2 restic_linux_amd64 $ GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go build -ldflags "-s -w" -tags selfupdate -o restic_0.10.0_windows_amd64.exe ./cmd/restic $ touch --reference VERSION restic_0.10.0_windows_amd64.exe $ TZ=Europe/Berlin zip -q -X restic_0.10.0_windows_amd64.zip restic_0.10.0_windows_amd64.exe Building the Official Binaries ****************************** The released binaries for restic are built using a Docker container. You can find it on `Docker Hub <https://hub.docker.com/r/restic/builder>`__ as ``restic/builder``, the ``Dockerfile`` and instructions on how to build the container can be found in the `GitHub repository <https://github.com/restic/builder>`__ The container serves the following goals: * Have a very controlled environment which is independent from the local system * Make it easy to have the correct version of the Go compiler at the right path * Make it easy to pass in the source code to build at a well-defined path The following steps are necessary to build the binaries: 1. Either build the container (see the instructions in the `repository's README <https://github.com/restic/builder>`__). Alternatively, download the container from the hub: .. code:: docker pull restic/builder 2. Extract the source code somewhere: .. code:: tar xvzf restic-0.10.0.tar.gz 3. Create a directory to place the resulting binaries in: .. code:: mkdir output 3. Mount the source code and the output directory in the container and run the default command, which starts ``helpers/build-release-binaries/main.go``: .. code:: docker run --rm \ --volume "$PWD/restic-0.10.0:/restic" \ --volume "$PWD/output:/output" \ restic/builder \ go run helpers/build-release-binaries/main.go --version 0.10.0 4. If anything goes wrong, you can enable debug output like this: .. code:: docker run --rm \ --volume "$PWD/restic-0.10.0:/restic" \ --volume "$PWD/output:/output" \ restic/builder \ go run helpers/build-release-binaries/main.go --version 0.10.0 --verbose Prepare a New Release ********************* Publishing a new release of restic requires many different steps. We've automated this in the Go program ``helpers/prepare-release/main.go`` which also includes checking that e.g. the changelog is correctly generated. The only required argument is the new version number (in `Semantic Versioning <https://semver.org/>`__ format ``MAJOR.MINOR.PATCH``): .. code:: go run helpers/prepare-release/main.go 0.10.0 Checks can be skipped on demand via flags, please see ``--help`` for details. 070_encryption.rst.txt 0000644 00000003506 15030441404 0010673 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ########## Encryption ########## *"The design might not be perfect, but it’s good. Encryption is a first-class feature, the implementation looks sane and I guess the deduplication trade-off is worth it. So… I’m going to use restic for my personal backups.*" `Filippo Valsorda`_ .. _Filippo Valsorda: https://blog.filippo.io/restic-cryptography/ ********************** Manage repository keys ********************** The ``key`` command allows you to set multiple access keys or passwords per repository. In fact, you can use the ``list``, ``add``, ``remove``, and ``passwd`` (changes a password) sub-commands to manage these keys very precisely: .. code-block:: console $ restic -r /srv/restic-repo key list enter password for repository: ID User Host Created ---------------------------------------------------------------------- *eb78040b username kasimir 2015-08-12 13:29:57 $ restic -r /srv/restic-repo key add enter password for repository: enter password for new key: enter password again: saved new key as <Key of username@kasimir, created on 2015-08-12 13:35:05.316831933 +0200 CEST> $ restic -r /srv/restic-repo key list enter password for repository: ID User Host Created ---------------------------------------------------------------------- 5c657874 username kasimir 2015-08-12 13:35:05 *eb78040b username kasimir 2015-08-12 13:29:57 manual_rest.rst.txt 0000644 00000046633 15030441404 0010435 0 ustar 00 Manual ====== Usage help ---------- Usage help is available: .. code-block:: console $ ./restic --help restic is a backup program which allows saving multiple revisions of files and directories in an encrypted repository stored on different backends. Usage: restic [command] Available Commands: backup Create a new backup of files and/or directories cache Operate on local cache directories cat Print internal objects to stdout check Check the repository for errors copy Copy snapshots from one repository to another diff Show differences between two snapshots dump Print a backed-up file to stdout find Find a file, a directory or restic IDs forget Remove snapshots from the repository generate Generate manual pages and auto-completion files (bash, fish, zsh) help Help about any command init Initialize a new repository key Manage keys (passwords) list List objects in the repository ls List files in a snapshot migrate Apply migrations mount Mount the repository prune Remove unneeded data from the repository rebuild-index Build a new index recover Recover data from the repository restore Extract the data from a snapshot self-update Update the restic binary snapshots List all snapshots stats Scan the repository and show basic statistics tag Modify tags on snapshots unlock Remove locks other processes created version Print version information Flags: --cacert file file to load root certificates from (default: use system certificates) --cache-dir directory set the cache directory. (default: use system default cache directory) --cleanup-cache auto remove old cache directories -h, --help help for restic --json set output mode to JSON for commands that support it --key-hint key key ID of key to try decrypting first (default: $RESTIC_KEY_HINT) --limit-download int limits downloads to a maximum rate in KiB/s. (default: unlimited) --limit-upload int limits uploads to a maximum rate in KiB/s. (default: unlimited) --no-cache do not use a local cache --no-lock do not lock the repository, this allows some operations on read-only repositories -o, --option key=value set extended option (key=value, can be specified multiple times) --password-command command shell command to obtain the repository password from (default: $RESTIC_PASSWORD_COMMAND) -p, --password-file file file to read the repository password from (default: $RESTIC_PASSWORD_FILE) -q, --quiet do not output comprehensive progress report -r, --repo repository repository to backup to or restore from (default: $RESTIC_REPOSITORY) --repository-file file file to read the repository location from (default: $RESTIC_REPOSITORY_FILE) --tls-client-cert file path to a file containing PEM encoded TLS client certificate and private key -v, --verbose n be verbose (specify multiple times or a level using --verbose=n, max level/times is 3) Use "restic [command] --help" for more information about a command. Similar to programs such as ``git``, restic has a number of sub-commands. You can see these commands in the listing above. Each sub-command may have own command-line options, and there is a help option for each command which lists them, e.g. for the ``backup`` command: .. code-block:: console $ ./restic backup --help The "backup" command creates a new snapshot and saves the files and directories given as the arguments. EXIT STATUS =========== Exit status is 0 if the command was successful. Exit status is 1 if there was a fatal error (no snapshot created). Exit status is 3 if some source data could not be read (incomplete snapshot created). Usage: restic backup [flags] FILE/DIR [FILE/DIR] ... Flags: -e, --exclude pattern exclude a pattern (can be specified multiple times) --exclude-caches excludes cache directories that are marked with a CACHEDIR.TAG file. See https://bford.info/cachedir/ for the Cache Directory Tagging Standard --exclude-file file read exclude patterns from a file (can be specified multiple times) --exclude-if-present filename[:header] takes filename[:header], exclude contents of directories containing filename (except filename itself) if header of that file is as provided (can be specified multiple times) --exclude-larger-than size max size of the files to be backed up (allowed suffixes: k/K, m/M, g/G, t/T) --files-from file read the files to backup from file (can be combined with file args; can be specified multiple times) --files-from-raw file read the files to backup from file (can be combined with file args; can be specified multiple times) --files-from-verbatim file read the files to backup from file (can be combined with file args; can be specified multiple times) -f, --force force re-reading the target files/directories (overrides the "parent" flag) -h, --help help for backup -H, --host hostname set the hostname for the snapshot manually. To prevent an expensive rescan use the "parent" flag --iexclude pattern same as --exclude pattern but ignores the casing of filenames --iexclude-file file same as --exclude-file but ignores casing of filenames in patterns --ignore-inode ignore inode number changes when checking for modified files -x, --one-file-system exclude other file systems, don't cross filesystem boundaries and subvolumes --parent snapshot use this parent snapshot (default: last snapshot in the repo that has the same target files/directories) --stdin read backup from stdin --stdin-filename filename filename to use when reading from stdin (default "stdin") --tag tags add tags for the new snapshot in the format `tag[,tag,...]` (can be specified multiple times) (default []) --time time time of the backup (ex. '2012-11-01 22:08:41') (default: now) --use-fs-snapshot use filesystem snapshot where possible (currently only Windows VSS) --with-atime store the atime for all files and directories Global Flags: --cacert file file to load root certificates from (default: use system certificates) --cache-dir directory set the cache directory. (default: use system default cache directory) --cleanup-cache auto remove old cache directories --json set output mode to JSON for commands that support it --key-hint key key ID of key to try decrypting first (default: $RESTIC_KEY_HINT) --limit-download int limits downloads to a maximum rate in KiB/s. (default: unlimited) --limit-upload int limits uploads to a maximum rate in KiB/s. (default: unlimited) --no-cache do not use a local cache --no-lock do not lock the repository, this allows some operations on read-only repositories -o, --option key=value set extended option (key=value, can be specified multiple times) --password-command command shell command to obtain the repository password from (default: $RESTIC_PASSWORD_COMMAND) -p, --password-file file file to read the repository password from (default: $RESTIC_PASSWORD_FILE) -q, --quiet do not output comprehensive progress report -r, --repo repository repository to backup to or restore from (default: $RESTIC_REPOSITORY) --repository-file file file to read the repository location from (default: $RESTIC_REPOSITORY_FILE) --tls-client-cert file path to a file containing PEM encoded TLS client certificate and private key -v, --verbose n be verbose (specify multiple times or a level using --verbose=n, max level/times is 3) Subcommands that support showing progress information such as ``backup``, ``check`` and ``prune`` will do so unless the quiet flag ``-q`` or ``--quiet`` is set. When running from a non-interactive console progress reporting is disabled by default to not fill your logs. For interactive and non-interactive consoles the environment variable ``RESTIC_PROGRESS_FPS`` can be used to control the frequency of progress reporting. Use for example ``0.016666`` to only update the progress once per minute. Additionally, on Unix systems if ``restic`` receives a SIGUSR1 signal the current progress will be written to the standard output so you can check up on the status at will. Setting the `RESTIC_PROGRESS_FPS` environment variable or sending a `SIGUSR1` signal prints a status report even when `--quiet` was specified. Manage tags ----------- Managing tags on snapshots is done with the ``tag`` command. The existing set of tags can be replaced completely, tags can be added or removed. The result is directly visible in the ``snapshots`` command. Let's say we want to tag snapshot ``590c8fc8`` with the tags ``NL`` and ``CH`` and remove all other tags that may be present, the following command does that: .. code-block:: console $ restic -r /srv/restic-repo tag --set NL --set CH 590c8fc8 create exclusive lock for repository modified tags on 1 snapshots Note the snapshot ID has changed, so between each change we need to look up the new ID of the snapshot. But there is an even better way - the ``tag`` command accepts a filter using the ``--tag`` option, so we can filter snapshots based on the tag we just added. This way we can add and remove tags incrementally: .. code-block:: console $ restic -r /srv/restic-repo tag --tag NL --remove CH create exclusive lock for repository modified tags on 1 snapshots $ restic -r /srv/restic-repo tag --tag NL --add UK create exclusive lock for repository modified tags on 1 snapshots $ restic -r /srv/restic-repo tag --tag NL --remove NL create exclusive lock for repository modified tags on 1 snapshots $ restic -r /srv/restic-repo tag --tag NL --add SOMETHING no snapshots were modified To operate on untagged snapshots only, specify the empty string ``''`` as the filter value to ``--tag``. The following command will add the tag ``OTHER`` to all untagged snapshots: .. code-block:: console $ restic -r /srv/restic-repo tag --tag '' --add OTHER Under the hood -------------- Browse repository objects ~~~~~~~~~~~~~~~~~~~~~~~~~ Internally, a repository stores data of several different types described in the `design documentation <https://github.com/restic/restic/blob/master/doc/design.rst>`__. You can ``list`` objects such as blobs, packs, index, snapshots, keys or locks with the following command: .. code-block:: console $ restic -r /srv/restic-repo list snapshots d369ccc7d126594950bf74f0a348d5d98d9e99f3215082eb69bf02dc9b3e464c The ``find`` command searches for a given `pattern <https://golang.org/pkg/path/filepath/#Match>`__ in the repository. .. code-block:: console $ restic -r backup find test.txt debug log file restic.log debug enabled enter password for repository: found 1 matching entries in snapshot 196bc5760c909a7681647949e80e5448e276521489558525680acf1bd428af36 -rw-r--r-- 501 20 5 2015-08-26 14:09:57 +0200 CEST path/to/test.txt The ``cat`` command allows you to display the JSON representation of the objects or their raw content. .. code-block:: console $ restic -r /srv/restic-repo cat snapshot d369ccc7d126594950bf74f0a348d5d98d9e99f3215082eb69bf02dc9b3e464c enter password for repository: { "time": "2015-08-12T12:52:44.091448856+02:00", "tree": "05cec17e8d3349f402576d02576a2971fc0d9f9776ce2f441c7010849c4ff5af", "paths": [ "/home/user/work" ], "hostname": "kasimir", "username": "username", "uid": 501, "gid": 20 } Metadata handling ~~~~~~~~~~~~~~~~~ Restic saves and restores most default attributes, including extended attributes like ACLs. Sparse files are not handled in a special way yet, and aren't restored. The following metadata is handled by restic: - Name - Type - Mode - ModTime - AccessTime - ChangeTime - UID - GID - User - Group - Inode - Size - Links - LinkTarget - Device - Content - Subtree - ExtendedAttributes Getting information about repository data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Use the ``stats`` command to count up stats about the data in the repository. There are different counting modes available using the ``--mode`` flag, depending on what you want to calculate. The default is the restore size, or the size required to restore the files: - ``restore-size`` (default) counts the size of the restored files. - ``files-by-contents`` counts the total size of unique files as given by their contents. This can be useful since a file is considered unique only if it has unique contents. Keep in mind that a small change to a large file (even when the file name/path hasn't changed) will cause them to look like different files, thus essentially causing the whole size of the file to be counted twice. - ``raw-data`` counts the size of the blobs in the repository, regardless of how many files reference them. This tells you how much restic has reduced all your original data down to (either for a single snapshot or across all your backups), and compared to the size given by the restore-size mode, can tell you how much deduplication is helping you. - ``blobs-per-file`` is kind of a mix between files-by-contents and raw-data modes; it is useful for knowing how much value your backup is providing you in terms of unique data stored by file. Like files-by-contents, it is resilient to file renames/moves. Unlike files-by-contents, it does not balloon to high values when large files have small edits, as long as the file path stayed the same. Unlike raw-data, this mode DOES consider how many files point to each blob such that the more files a blob is referenced by, the more it counts toward the size. For example, to calculate how much space would be required to restore the latest snapshot (from any host that made it): .. code-block:: console $ restic stats latest password is correct Total File Count: 10538 Total Size: 37.824 GiB If multiple hosts are backing up to the repository, the latest snapshot may not be the one you want. You can specify the latest snapshot from only a specific host by using the ``--host`` flag: .. code-block:: console $ restic stats --host myserver latest password is correct Total File Count: 21766 Total Size: 481.783 GiB There we see that it would take 482 GiB of disk space to restore the latest snapshot from "myserver". In case you have multiple backups running from the same host so can also use ``--tag`` and ``--path`` to be more specific about which snapshots you are looking for. But how much space does that snapshot take on disk? In other words, how much has restic's deduplication helped? We can check: .. code-block:: console $ restic stats --host myserver --mode raw-data latest password is correct Total Blob Count: 340847 Total Size: 458.663 GiB Comparing this size to the previous command, we see that restic has saved about 23 GiB of space with deduplication. Which mode you use depends on your exact use case. Some modes are more useful across all snapshots, while others make more sense on just a single snapshot, depending on what you're trying to calculate. Scripting --------- Restic supports the output of some commands in JSON format, the JSON data can then be processed by other programs (e.g. `jq <https://stedolan.github.io/jq/>`__). The following example lists all snapshots as JSON and uses ``jq`` to pretty-print the result: .. code-block:: console $ restic -r /srv/restic-repo snapshots --json | jq . [ { "time": "2017-03-11T09:57:43.26630619+01:00", "tree": "bf25241679533df554fc0fd0ae6dbb9dcf1859a13f2bc9dd4543c354eff6c464", "paths": [ "/home/work/doc" ], "hostname": "kasimir", "username": "fd0", "uid": 1000, "gid": 100, "id": "bbeed6d28159aa384d1ccc6fa0b540644b1b9599b162d2972acda86b1b80f89e" }, { "time": "2017-03-11T09:58:57.541446938+01:00", "tree": "7f8c95d3420baaac28dc51609796ae0e0ecfb4862b609a9f38ffaf7ae2d758da", "paths": [ "/home/user/shared" ], "hostname": "kasimir", "username": "fd0", "uid": 1000, "gid": 100, "id": "b157d91c16f0ba56801ece3a708dfc53791fe2a97e827090d6ed9a69a6ebdca0" } ] .. _temporary_files: Temporary files --------------- During some operations (e.g. ``backup`` and ``prune``) restic uses temporary files to store data. These files will, by default, be saved to the system's temporary directory, on Linux this is usually located in ``/tmp/``. The environment variable ``TMPDIR`` can be used to specify a different directory, e.g. to use the directory ``/var/tmp/restic-tmp`` instead of the default, set the environment variable like this: .. code-block:: console $ export TMPDIR=/var/tmp/restic-tmp $ restic -r /srv/restic-repo backup ~/work .. _caching: Caching ------- Restic keeps a cache with some files from the repository on the local machine. This allows faster operations, since meta data does not need to be loaded from a remote repository. The cache is automatically created, usually in an OS-specific cache folder: * Linux/other: ``$XDG_CACHE_HOME/restic``, or ``~/.cache/restic`` if ``XDG_CACHE_HOME`` is not set * macOS: ``~/Library/Caches/restic`` * Windows: ``%LOCALAPPDATA%/restic`` If the relevant environment variables are not set, restic exits with an error message. The command line parameter ``--cache-dir`` or the environment variable ``$RESTIC_CACHE_DIR`` can be used to override the default cache location. The parameter ``--no-cache`` disables the cache entirely. In this case, all data is loaded from the repo. The cache is ephemeral: When a file cannot be read from the cache, it is loaded from the repository. Within the cache directory, there's a sub directory for each repository the cache was used with. Restic updates the timestamps of a repo directory each time it is used, so by looking at the timestamps of the sub directories of the cache directory it can decide which sub directories are old and probably not needed any more. You can either remove these directories manually, or run a restic command with the ``--cleanup-cache`` flag. cache.rst.txt 0000644 00000001652 15030441404 0007156 0 ustar 00 *********** Local Cache *********** In order to speed up certain operations, restic manages a local cache of data. The location of the cache directory depends on the operating system and the environment; see :ref:`caching`. Each repository has its own cache sub-directory, consisting of the repository ID which is chosen at ``init``. All cache directories for different repos are independent of each other. Snapshots, Data and Indexes =========================== Snapshot, Data and Index files are cached in the sub-directories ``snapshots``, ``data`` and ``index``, as read from the repository. Expiry ====== Whenever a cache directory for a repo is used, that directory's modification timestamp is updated to the current time. By looking at the modification timestamps of the repo cache directories it is easy to decide which directories are old and haven't been used in a long time. Those are probably stale and can be removed. 090_participating.rst.txt 0000644 00000011673 15030441404 0011345 0 ustar 00 .. Normally, there are no heading levels assigned to certain characters as the structure is determined from the succession of headings. However, this convention is used in Python’s Style Guide for documenting which you may follow: # with overline, for parts * for chapters = for sections - for subsections ^ for subsubsections " for paragraphs ############# Participating ############# ********* Debugging ********* The program can be built with debug support like this: .. code-block:: console $ go run build.go -tags debug Afterwards, extensive debug messages are written to the file in environment variable ``DEBUG_LOG``, e.g.: .. code-block:: console $ DEBUG_LOG=/tmp/restic-debug.log restic backup ~/work If you suspect that there is a bug, you can have a look at the debug log. Please be aware that the debug log might contain sensitive information such as file and directory names. The debug log will always contain all log messages restic generates. You can also instruct restic to print some or all debug messages to stderr. These can also be limited to e.g. a list of source files or a list of patterns for function names. The patterns are globbing patterns (see the documentation for `path.Glob <https://golang.org/pkg/path/#Glob>`__), multiple patterns are separated by commas. Patterns are case sensitive. Printing all log messages to the console can be achieved by setting the file filter to ``*``: .. code-block:: console $ DEBUG_FILES=* restic check If you want restic to just print all debug log messages from the files ``main.go`` and ``lock.go``, set the environment variable ``DEBUG_FILES`` like this: .. code-block:: console $ DEBUG_FILES=main.go,lock.go restic check The following command line instructs restic to only print debug statements originating in functions that match the pattern ``*unlock*`` (case sensitive): .. code-block:: console $ DEBUG_FUNCS=*unlock* restic check ************ Contributing ************ Contributions are welcome! Please **open an issue first** (or add a comment to an existing issue) if you plan to work on any code or add a new feature. This way, duplicate work is prevented and we can discuss your ideas and design first. More information and a description of the development environment can be found in `CONTRIBUTING.md <https://github.com/restic/restic/blob/master/CONTRIBUTING.md>`__. A document describing the design of restic and the data structures stored on the back end is contained in `Design <https://restic.readthedocs.io/en/latest/design.html>`__. If you'd like to start contributing to restic, but don't know exactly what do to, have a look at this great article by Dave Cheney: `Suggestions for contributing to an Open Source project <https://dave.cheney.net/2016/03/12/suggestions-for-contributing-to-an-open-source-project>`__ A few issues have been tagged with the label ``help wanted``, you can start looking at those: https://github.com/restic/restic/labels/help%20wanted ******** Security ******** **Important**: If you discover something that you believe to be a possible critical security problem, please do *not* open a GitHub issue but send an email directly to alexander@bumpern.de. If possible, please encrypt your email using the following PGP key (`0x91A6868BD3F7A907 <https://pgp.mit.edu/pks/lookup?op=get&search=0xCF8F18F2844575973F79D4E191A6868BD3F7A907>`__): :: pub 4096R/91A6868BD3F7A907 2014-11-01 Key fingerprint = CF8F 18F2 8445 7597 3F79 D4E1 91A6 868B D3F7 A907 uid Alexander Neumann <alexander@bumpern.de> sub 4096R/D5FC2ACF4043FDF1 2014-11-01 ************* Compatibility ************* Backward compatibility for backups is important so that our users are always able to restore saved data. Therefore restic follows `Semantic Versioning <https://semver.org>`__ to clearly define which versions are compatible. The repository and data structures contained therein are considered the "Public API" in the sense of Semantic Versioning. This goes for all released versions of restic, this may not be the case for the master branch. We guarantee backward compatibility of all repositories within one major version; as long as we do not increment the major version, data can be read and restored. We strive to be fully backward compatible to all prior versions. ********************** Building documentation ********************** The restic documentation is built with `Sphinx <https://www.sphinx-doc.org>`__, therefore building it locally requires a recent Python version and requirements listed in ``doc/requirements.txt``. This example will guide you through the process using `virtualenv <https://virtualenv.pypa.io>`__: :: $ virtualenv venv # create virtual python environment $ source venv/bin/activate # activate the virtual environment $ cd doc $ pip install -r requirements.txt # install dependencies $ make html # build html documentation $ # open _build/html/index.html with your favorite browser
| ver. 1.4 |
Github
|
.
| PHP 8.2.28 | Generation time: 0.02 |
proxy
|
phpinfo
|
Settings