Storage Backends and supported functions

Feature comparison

LXD supports using plain dirs, Btrfs, LVM, and ZFS for storage of images and containers.
Where possible, LXD tries to use the advanced features of each system to optimize operations.

Feature Directory Btrfs LVM ZFS
Optimized image storage no yes yes yes
Optimized container creation no yes yes yes
Optimized snapshot creation no yes yes yes
Optimized image transfer no yes no yes
Optimized container transfer no yes no yes
Copy on write no yes yes yes
Block based no no yes no
Instant cloning no yes yes yes
Nesting support yes yes no no
Restore from older snapshots (not latest) yes yes yes no
Storage quotas no yes no yes

Mixed storage

When switching storage backend after some containers or images already exist, LXD will create any new container
using the new backend and converting older images to the new backend as needed.

Non-optimized container transfer

When the filesystem on the source and target hosts differs or when there is no faster way,
rsync is used to transfer the container content across.

I/O limits

I/O limits in IOp/s or MB/s can be set on storage devices when attached to a container (see Containers).

Those are applied through the Linux blkio cgroup controller which makes it possible
to restrict I/O at the disk level (but nothing finer grained than that).

Because those apply to a whole physical disk rather than a partition or path, the following restrictions apply:

It's also worth noting that all I/O limits only apply to actual block device access,
so you will need to consider the filesystem's own overhead when setting limits.
This also means that access to cached data will not be affected by the limit.

Notes

Directory

Btrfs

LVM

ZFS

Also note that container copies use ZFS snapshots, so you also cannot restore a container to a snapshot taken before the last copy without having to also delete container copies.

Copying the wanted snapshot into a new container and then deleting the old container does however work, at the cost of losing any other snapshot the container may have had. - Note that LXD will assume it has full control over the zfs pool or dataset. It is recommended to not maintain any non-LXD owned filesystem entities in a LXD zfs pool or dataset since LXD might delete them. - I/O quotas (IOps/MBs) are unlikely to affect ZFS filesystems very much. That's because of ZFS being a port of a Solaris module (using SPL) and not a native Linux filesystem using the Linux VFS API which is where I/O limits are applied.

Growing a loop backed ZFS pool

LXD doesn't let you directly grow a loop backed ZFS pool, but you can do so with:

sudo truncate -s +5G /var/lib/lxd/zfs.img
sudo zpool set autoexpand=on lxd
sudo zpool online -e lxd /var/lib/lxd/zfs.img
sudo zpool set autoexpand=off lxd