Proxmox VE storage types: local, LVM-thin, ZFS, NFS and Ceph

In Proxmox VE a storage is a named pool where the system keeps VM disks, container volumes, ISO images, container templates, backups or snippets. All storage definitions are in one file, /etc/pve/storage.cfg. Because that file is inside /etc/pve, it is the same on every node of a cluster.

File level and block level

• File level storages (Directory, NFS, CIFS, CephFS, BTRFS) are normal file systems. They can hold any content type, including ISOs and backups. VM disks are stored as files, usually in raw or qcow2 format.
• Block level storages (LVM, LVM-thin, iSCSI, Ceph RBD, ZFS over iSCSI) hand out block devices for disks. They usually cannot hold ISOs or backup files.

Local ZFS (zfspool) is a special case: it stores VM disks as ZFS volumes (zvols) and container volumes as ZFS datasets.

The default storages

A fresh install creates local, a directory at /var/lib/vz for ISOs, templates and backups. Depending on the disk layout chosen in the installer, it also creates local-lvm (an LVM-thin pool on ext4/xfs installs) or local-zfs (on ZFS installs) for guest disks.

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

Each entry starts with <type>: <storage-id> followed by indented properties. The content property lists what the storage may hold: images (VM disks), rootdir (container volumes), iso, vztmpl (container templates), backup, snippets and import.

Comparing common types

• Directory (dir): simple, local, any content. Snapshots only with qcow2 files.
• LVM-thin (lvmthin): local block storage with thin provisioning, snapshots and fast linked clones. A good default for single nodes.
• LVM (lvm): thick block storage. On a shared SAN LUN (iSCSI, Fibre Channel) it can be used as shared storage in a cluster.
• ZFS (zfspool): local, with checksums, compression, snapshots and clones. Also enables storage replication between nodes.
• NFS and CIFS: shared file storage from a NAS. Easy way to give all nodes the same ISOs, backups or disks.
• Ceph RBD and CephFS: distributed, replicated storage, which Proxmox VE can also run on the cluster nodes themselves (hyper-converged). Best with at least three nodes and fast, dedicated networking.
• Proxmox Backup Server (pbs): a backup target, not a disk store.

Thin provisioning: the trap

Thin storage only uses space for blocks that guests actually write, so you can give guests more virtual disk space than you physically have. If the underlying storage then runs full, guests on it receive I/O errors, which can corrupt file systems. Avoid heavy over-provisioning and monitor free space, for example with pvesm status or lvs for LVM-thin.

Shared versus local

The shared flag tells Proxmox VE that all nodes see the same content (for example an NFS export). It does not make local storage shared. Shared storage lets live migration move only memory, and is a requirement for HA.

Adding storage with pvesm

You can add storages in the web UI under Datacenter → Storage or with pvesm:

pvesm scan nfs 192.0.2.50
pvesm add nfs nas-backup --server 192.0.2.50 --export /export/pve --content backup,iso
pvesm add dir bulk --path /mnt/bulk --content images,iso
pvesm status
pvesm list nas-backup --content backup

NFS storages are mounted automatically under /mnt/pve/<storage-id>. To restrict a storage to certain nodes, set --nodes pve1,pve2. pvesm remove <id> only removes the configuration and does not delete data, but pvesm free <volume> really destroys a volume.

Do not define two storages that point at the same place, and never use the same storage from two different clusters. Locking only works inside one cluster, so both can corrupt each other’s data.

Official documentation

• Proxmox VE Storage
• pvesm(1): Proxmox VE Storage Manager