Note: a mistake in creating or restoring a backup is easily made. Make sure to read the documentation carefully. Any feedback on this page is welcome. Please post your remarks to the Nextcloud forum
Note 2: documentation is always behind the times compared to the actual functionality it describes. Therefore, it is advised to inspect the most recent NCP shell scripts to see whether functionality has changed for a specific tool. We provide a link to these scripts for every subsection describing a tool.
This article will elaborate on the following topics:
- How to preserve my data?
- Difference between snapshots and backups
- Creating a Snapshot
- Creating a Backup
- How to restore my data?
- Restoring a snapshot
- Restoring a (full) backup
How to preserve my data?
As long as your Nextcloud instance is up and running, it is easy to forget what could possibly cause data loss. However, it is not unimaginable that one of your hard drives sooner or later gets corrupt (i.e. unusable), that your SD card or eMMC-module (operating system) suddenly won’t boot after an update, or even worse: that a house fire destroys both.
Gladly, NCP offers several ways to preserve copies of your data and configuration in order to restore them if needed. These tools support you in following a specific backup strategy. To determine a strategy, please read this article of NachoParker. In the documentation we will mainly focus on the tools that are available to perform backup-operations. There are roughly two methods to backup your data: snapshots and backups.
Difference between snapshots and backups
A snapshot is only available for BTRFS file systems. BTRFS consists of one or more subvolumes, which appear like a “normal” directory in many ways. In the case of NextcloudPi, the relevant subvolume is the data directory (nc-datadir). Creating a snapshot will capture the current state of the subvolume holding your data (let’s say: A) and share the content of A with another subvolume: the snapshot (let’s say: B). After performing a snapshot-operation, changes in subvolume A will not be visible to subvolume B until the next snapshot operation. A virtue of BTRFS snapshots is that they are created nearly instant, and that they are incremental, thus taking up little space.
Snapshots, however, do not capture your Nextcloud configuration on the SD card or eMMC module. Therefore, there is also the option to create a backup of your database + Nextcloud configuration. A backup operation copies these files to a location of choice (preferably another disk). In case of a failure in the Nextcloud configuration (e.g. corrupt database, broken SD-card) a backup could be used to restore the configuration. In addition to backing up the configuration, it is also possible to include a copy of your data-directory (nc-datadir). However, take into account that this requires your Nextcloud to be in maintainance-mode, thus being unaccessible during the backup process. Moreover, this could take up a long time! Creating backups is described in more detail in the section “Creating a Backup”.
Creating a Snapshot
Note: only available for BTRFS file systems
The starting point for creating a snapshot is either the NCP web panel or the NCP terminal user interface (TUI). The NCP web panel is available in your browser at https://[nextcloudpi address]:4443. The NCP TUI is accessible via ssh. Check this article for more information on accessing the desired UI. For this tutorial, the NCP web panel will be used in the examples.
There are three actions available for creating and migrating snapshots:
This function can be used for manually creating snapshots. As a user, you are able to provide a maximum number of snapshots to keep in parallel. For example, if the limit is set to 4 and a 5th snapshot is created, the 1st snapshot is removed. Setting the limit to 0 disables the snapshot limit. The limit merely applies to the manually created snapshots with
nc-snapshot, and is independent of the automatic snapshots (
In contrast to
nc-snapshot-auto automatically creates backups for you, as the name suggests. Every hour, a snapshot is created in the folder
ncp-snapshots next to your data directory. Snapshots are stored in an incremental manner on an hourly, daily, weekly and montly basis. The limits per time unit are as follows:
|Time unit||Maximum number of backups|
This means that you could delete a file right now and still be able to restore it one year later, using the oldest backup available (given that your current instance is already creating snapshots for over a year). However, for the attentive reader it is already clear that the snapshots are stored on the same disk as the data. Thus, in case of a hard drive failure, don’t we still lose all our data and our snapshots? Yes, that is the case. Therefore, it is important to sync your snapshots with another hard drive, or even with a device located outside your home. This will be discussed in the next section.
Since keeping snapshots on a single disk is insufficient in case of a hard drive failure, NCP contains a tool to copy these snapshots to another device:
nc-snapshot-sync. This tool uses the underlying
btrfs-snp-script, which is explained here. It comes pre-installed with NCP, thus you don’t need to install it yourself. Once
nc-snapshot-sync is enabled, the tool will run automatically. How often it will run, depends on the selected sync periodicity. It will synchronise the snapshots from the “Snapshot directory” with the chosen “Desination”.
There are roughly two ways to configure
nc-snapshot-sync: (1) copying snapshots to a disk that is connected to the same NCP-device, and (2) copying snapshots to a remote disk. Option (1) requires a second data storage device (e.g. a hard drive or SSD) to be connected to your NCP instance. Make sure it is properly mounted and formatted as BTRFS file system, otherwise it won’t be possible to synchronise the snapshots. Once you created a mount point (e.g.
/media/mybackup) and formatted the disk as BTRFS, it is possible to set the value for Destination (e.g.
/media/mybackup/ncp-snapshots). Make sure to press the “apply” button. Congratulations: your snapshots will be synced automatically from now on. Note: it will probably not run immediately. One way to confirm whether the setup is correct, is to check after 24 hours whether the snapshots synced correctly. In contrast to creating a snapshot, synching snapshots may take a while to complete, especially the first time.
Option (2), copying snapshots to a remote disk, requires additional configuration.. Schematically it works as follows:
At first, it requires the configuration of an additional device other than your Nextcloud-device, with a hard drive or SSD attached formatted with the BTRFS file system. Your device running NCP (sender) should be able to login automatically over SSH with the user “root”, onto the backup device (receiver) with a custom user. In this example, the user on the receiving device is called “btrfs”. Thus, at first we create a user “btrfs” on the destination device.
sudo adduser btrfs
On your sending machine, generate a public key for the user “root”
sudo -u root ssh-keygen
Note: it is not necessary to overwrite existing keys
Then, give the user “root” on the sender passwordless access to the user “btrfs” at the receiving machine. Run the following command on the sending machine:
sudo -u root ssh-copy-id btrfs@[ip-address-of-receiving-machine]
To validate whether auto-login works as expected, you may want to make a login attempt over SSH. Finally, give the user “btrfs” at the receiving machine the permission to run btrfs-commands.
- Create a file “
/etc/sudoers.d/90_btrfs-sync” with the following contents:
btrfs ALL=(root:nobody) NOPASSWD:NOEXEC: /bin/btrfs
And that’s it for the additional configuration. Now you can set the “Destination” field in the NCP web panel. It has the following structure: “user@ip:/path/to/backup”. In this case, our user is “btrfs”, and ip is equal to the IP-address of your receiving device. /path/to/backup is the path where the snapshots are located at your receiving machine (for example: “/media/mybackup/ncp-snapshots”).
Beside the ability to create snapshots and synchronise them with other machines at block level, it is also possible to perform backup-operations at file level. The next section will elaborate on creating a backup of your files, scheduling data-less backups and how to use rsync to copy files to another disk or remote device.
Creating a backup
In this section we will dive into the subject of automated backups (with or without data). We will see how to use rsync to preserve a copy of your files on another drive or even a remote device. An advantage of creating full backups over snapshots, is that your disk(s) don’t require the BTRFS file system. Backups are not a competitor of snapshots, but rather a supplement; It allows us to create a backup of our Nextcloud-configuration, including the Nextcloud-database, something that nc-snapshot doesn’t cover for us. We will discuss the following NCP-tools:
nc-backup operates on the file level, which means you don’t need specific file systems to be able to use it. Basically, it creates a backup of your Nextcloud-database and the entire Nextcloud-directory (not your nc-datadir, but the folder that’s responsible for the actual Nextcloud-functionality). These files will be copied to a “Destination directory” of the user’s choice, e.g.
/media/backupdrive/ncp-backups. An important choice to make is whether to include your data in the backup. Including your data means that the entire
nc-datadir is copied to another location. Copying all your files over to another location could take a long time if there is a serious amount of data in the
nc-datadir. How long it will take also depends on the speed of your device.
nc-backup requires your Nextcloud to be in maintenance mode, meaning it can’t be reached by any of the users until the backup-operation is completed. Therefore, before checking the option “Include data”, it is advised to benchmark the speed of the backup-operation on your specific device.
Compressing your data could save bandwidth. However, if bandwith is not an issue, it is recommended to disable compression for performance reasons. At last, the number of backups to keep is the maximum amount of backups that are preserved at the same time. Thus, creating a 5th backup with a limit of 4 will delete the first (oldest) backup. Setting the backup limit to “0” allows an unlimited number of backups.
nc-backup manually every time can be pretty exhaustive. Therefore,
nc-backup-auto schedules backup operations for you. It contains the same options as
nc-backup, such as the option to provide a “Destination directory”, whether to include your data, whether to compress your data and how many backups to keep at most.
The “Backup periodicity” can be set to determine how often a backup operation should be scheduled. Setting this value to “2” will create a backup today, the day after tomorrow, three days after tomorrow, etcetera. Thus, the lower the value, the more often a backup will be created. For determining the backup periodicity in combination with the maximum number of backups to keep, you could consider the volatility of your database and the amount of space available. Please note: backups are not synchronised with
nc-snapshot. If you wish to copy your backups to a remote device, you could consider a custom
btrfs-sync (non-NCP) implementation.
Rsync is a tool for synchronising files to (remote) devices. A difference between
btrfs-sync is that
rsync compares the deltas of individual files, whereas
btrfs-sync compares changes between source and destination at block level. Typically, rsync transfers the deltas of changed files over SSH. By default,
nc-rsync will keep the source and destination identical by means of the “–delete” option in the nc-rsync.sh script. Thus, deleted files on the source will also delete the same files on the destination device.
nc-rsync can be used, SSH-autologin should be enabled. In order to do so, please follow the steps mentioned in section nc-snapshot-sync.
Once SSH-autologin is enabled, provide the right “Destination” in the following format: “user@ip:/path/to/destination”. Here, “user” is the user at the remote device, “ip” is the IP-address of your device and “/path/to/destination” is the path of the mounted hard drive or SSD on the destination device where the files should be transfered to. The “SSH port number” is usually “22”, but can be set to another value, if necessary.
Instead of performing manual
rsync operations, NCP allows us to automate this process. The “Destination” and “SSH port number” are identical to the ones in
nc-rsync. In addition, one could set the “Sync periodicity”, which means rsync runs every X days, where X is the seleted sync periodicity.
Rsync will not run immediately, but it will scheduled to run at a specific time. Thus, when running rsync for the first time, make sure that your files are actually being transferred to the remote device by checking whether files are present on your device after 24 hours.
How to restore my data?
Hard drives will break sooner or later, and seemingly stable operating systems may suddenly refuse to boot after a power outage during a kernel upgrade. In such sitations, it would be desirable to reduce the downtime of your system to a minimum, and make the process of restoring backups as easy as possible. NCP offers several ways to restore snapshots, unfold backups and bring the Nextcloud-database to its most recent unbroken state. Which functionality suits you best, depends on your backup strategy. We will discuss the following restoration-methods:
- Restoring a snapshot
- Restoring a (full) backup
Restoring a snapshot
Note: make sure your (new) hard drive or SSD is formatted with BTRFS before proceeding.
Restoring a snapshot is particularly useful when your hard drive or solid state drive with
nc-datadir is broken. Snapshots maintain a 1-on-1 copy of your uploaded files at creation-time, and can be used for restoring your
nc-datadir. If you would like to restore your Nextcloud-configuration or database instead, please skip to the next section (restoring a backup).
The first step is: determine which snapshot to restore. In case you have used
nc-snapshot-auto, hourly, daily, weekly and monthly snapshots can be restored. Determine if you want to restore the most recent snapshot, a snapshot from several days ago, or maybe even a snapshot from a few months back.
It may be the case that your snapshots are stored on a remote disk (i.e. physically disconnected drive). In order to use them on your NCP-device, use btrfs-sync to copy them back to a drive connected to your NCP-instance. More on
btrfs-sync can be found in this article.
Once you have selected a snapshot to restore, and moved it to a BTRFS-drive that is physically connected to an NCP-instance, the prerequisites for restoring a snapshot are met. We can now use
nc-restore-snapshot to start the restoration process.
Note: it is recommended to read the shell script before performing this operation.
For restoring a snapshot via
nc-restore-snapshot, merely one argument is required: the location of the snapshot on the mounted drive. Hitting the apply button will create a snapshot of the current data in
nc-datadir (if applicable) and then overwrite
nc-datadir with the selected snapshot.
Restoring a (full) backup
As the title suggests, there are two options when it comes to restoring backups:
- Restoring Nextcloud configuration and database
- Restoring your files
These options are cumulative, meaning that it is possible to do either option (1), or both option (1) and (2). Option (1) is particularly interesting for people following a backup strategy with data-less backups. The second option could be relevant for people that used
nc-backup(-auto) with the option “Include Data”. For both of these options, we can use
One reason for you to restore a backup, could be that your NCP instance was broken. In that case, before making a restore-attempt, it is recommended to start with a clean installation of NextCloudPi. If you’ve manually edited config-files or performed database-manipulations, it could also be more time-efficient to start with a fresh NCP instance. Note: this doesn’t include wiping your data, but merely flashing your SD-card, eMMC-module or any other drive with a recent NCP-image, or creating a new Docker-container in case you use Docker.
Once Nextcloud is up and running (again), select which backup file you would like to restore. This could either be a data-less backup or a full backup. Then we can proceed to
nc-restore to restore the backup
Once again, it is recommended to read the shell script that is executed when pressing “Apply”.
A single argument is required to run this operation: a “Backup file”, located at one of your mounted disks. The script will automatically detect if this is a data-less backup (excluding files) or a full backup (including files). Data-less backups will only restore the Nextcloud configuration files and Nextcloud database.
nc-datadir remains untouched. Full backups will restore the Nextcloud configuration files, Nextcloud database and the entire data directory, thereby removing all of your current files in
nc-datadir. However, before they are removed, your files in
nc-datadir will be saved in a backup.
nc-export-ncp and nc-import-ncp
Once NextCloudPi (thus not Nextcloud) is configured to your liking, you could consider to use the
nc-export-ncp to export the current state of your NextCloudPi settings. After restoring a backup, use
nc-import-ncp to restore the NCP settings such as