OS-7915: bhyve should default to no zfs quota

Details

Issue Type:Bug
Priority:4 - Normal
Status:Open
Created at:2019-07-30T19:14:22.159Z
Updated at:2019-07-30T19:15:27.256Z

People

Created by:Former user
Reported by:Former user

Related Issues

Labels

bhyve

Description

To ensure that triton instance snapshot and similar do not allow cloud users to consume all space in a CN's zpool, we made it so that bhyve sets the zfs quota property on <pool>/<uuid> to the same value as reserved. The use of flexible_disk_size can be used to arbitrarily increase the amount of space reserved, thus increasing quota. This rigid alignment can be helpful to cloud operators that care about charging for all the space used and ensuring that storage is not oversubscribed.

Typical standalone SmartOS users and many Triton operators have no untrusted snapshot creators but rely on snapshots for various forms of data protection. We should shift our defaults to accommodate this more common use case.

SmartOS changes

The new default for bhyve instances will allow bhyve instances to create snapshots without quota restrictions.

This will be implemented with a new VM property quota_includes_snapshots, a boolean. If not specified, in a VM configuration (attr.quota-includes-snapshots), it is treated as true for backwards compatibility. If not specified in a vmadm or VM.js create payload, a value of false is implied.

The vmadm(1M) man page will be updated as follows.

       flexible_disk_size:

           This sets an upper bound for the amount of space that a bhyve
           instance may use for its disks and optionally snapshots of those
           disks. See quota_includes_snapshots. If this value is not set, it
           will not be possible to create snapshots of the instance.

           This value must be at least as large as the sum of all of the
           disk.*.size values.

           type: integer (number of MiB)
           vmtype: bhyve
           listable: yes
           create: yes
           update: yes (live update)

       quota_includes_snapshots:

           When this property is set to true, the ZFS quota property is set on
           the zone's top-level dataset.

           For bhyve VMs that have flexible_disk_size set, the value of the ZFS
           quota will be the sum of the the VM's quota property (ZFS refquota),
           the VM's flexible_disk_size, plus ZFS metadata overhead for each
           disk.  This implies that snapshots will only be possible if the
           amount of changed data since the last snapshot is less than the
           difference of flexible_disk_size and the sum of all disk.*.size.

           For bhyve VMs that do not have flexible_disk_size, the ZFS quota will
           be the sum of the the VM's quota property (ZFS refquota), plus the
           size of disks.*.size plus ZFS metadata overhead for each disk.  This
           leaves no space for snapshots, effectively disabling create-snapshot.

           When this property is set to false, a VM's that has frequent
           snapshots and a high rate of data change can consume an arbitrary
           amount of space.

           In a Triton environment that allows untrusted tenants to create
           snapshots, it is recommend that this property be set to true.
           If it is false, tenants that are allowed to create snapshots may
           consume all storage by alternately creating snapshots and writing
           large amounts of data to one or more disks.

           type: boolean
           vmtype: bhyve
           listable: yes
           create: yes
           update: yes (live update), but will fail if current usage is too high
           default: true, but if unspecified in a create payload false is used

VMAPI Changes

VMAPI will be updated to allow the operator to control whether snapshots space is constrained and whether snapshots of bhyve instances are allowed via new SAPI configuration options.

KeyTypeDescription
bhyve_quota_enforcedBooleanIs the ZFS quota set on a new instance's top-level dataset (default is false). It is recommended that this value is true if bhyve_snapshots_allowed is true. A setting of false allows the operator to take snapshots without being impacted by the ZFS quota. This setting can be overridden by vm.quota_includes_snapshots.
bhyve_snapshots_allowedBooleanAre new bhyve instances created in a way that allows snapshots (default is false)?

While the value of bhyve_quota_enforced may be changed from time to time by the operator, the setting that is in effect at the time of VM creation will be sticky with that VM via quota_includes_snapshots. Operators may adjust VMs with vmadm update <uuid> quota_includes_snapshots=<value>.

CN-Agent changes

During VM migration, if quota is set on an instance's top-level dataset, it must be relaxed so that an attempt to create a snapshot does not get ENOSPC or EDQUOT. During migration activation (e.g. sdc-migrate activate) the final vm-migrate snapshot is removed and quota is set if only if quota_includes_snapshots is true (explicitly or implicitly through its absence). quota is not set during other migration phases.

Comments