Issue Type: | Bug |
---|---|
Priority: | 4 - Normal |
Status: | Open |
Created at: | 2019-07-30T19:14:22.159Z |
Updated at: | 2019-07-30T19:15:27.256Z |
Created by: | Former user |
---|---|
Reported by: | Former user |
To ensure that triton instance snapshot
and similar do not allow cloud users to consume all space in a CN's zpool, we made it so that bhyve sets the zfs quota
property on <pool>/<uuid>
to the same value as reserved
. The use of flexible_disk_size
can be used to arbitrarily increase the amount of space reserved
, thus increasing quota
. This rigid alignment can be helpful to cloud operators that care about charging for all the space used and ensuring that storage is not oversubscribed.
Typical standalone SmartOS users and many Triton operators have no untrusted snapshot creators but rely on snapshots for various forms of data protection. We should shift our defaults to accommodate this more common use case.
The new default for bhyve instances will allow bhyve instances to create snapshots without quota
restrictions.
This will be implemented with a new VM property quota_includes_snapshots
, a boolean. If not specified, in a VM configuration (attr.quota-includes-snapshots
), it is treated as true
for backwards compatibility. If not specified in a vmadm
or VM.js
create payload, a value of false
is implied.
The vmadm(1M) man page will be updated as follows.
flexible_disk_size: This sets an upper bound for the amount of space that a bhyve instance may use for its disks and optionally snapshots of those disks. See quota_includes_snapshots. If this value is not set, it will not be possible to create snapshots of the instance. This value must be at least as large as the sum of all of the disk.*.size values. type: integer (number of MiB) vmtype: bhyve listable: yes create: yes update: yes (live update) quota_includes_snapshots: When this property is set to true, the ZFS quota property is set on the zone's top-level dataset. For bhyve VMs that have flexible_disk_size set, the value of the ZFS quota will be the sum of the the VM's quota property (ZFS refquota), the VM's flexible_disk_size, plus ZFS metadata overhead for each disk. This implies that snapshots will only be possible if the amount of changed data since the last snapshot is less than the difference of flexible_disk_size and the sum of all disk.*.size. For bhyve VMs that do not have flexible_disk_size, the ZFS quota will be the sum of the the VM's quota property (ZFS refquota), plus the size of disks.*.size plus ZFS metadata overhead for each disk. This leaves no space for snapshots, effectively disabling create-snapshot. When this property is set to false, a VM's that has frequent snapshots and a high rate of data change can consume an arbitrary amount of space. In a Triton environment that allows untrusted tenants to create snapshots, it is recommend that this property be set to true. If it is false, tenants that are allowed to create snapshots may consume all storage by alternately creating snapshots and writing large amounts of data to one or more disks. type: boolean vmtype: bhyve listable: yes create: yes update: yes (live update), but will fail if current usage is too high default: true, but if unspecified in a create payload false is used
VMAPI will be updated to allow the operator to control whether snapshots space is constrained and whether snapshots of bhyve instances are allowed via new SAPI configuration options.
Key | Type | Description |
---|---|---|
bhyve_quota_enforced | Boolean | Is the ZFS quota set on a new instance's top-level dataset (default is false). It is recommended that this value is true if bhyve_snapshots_allowed is true. A setting of false allows the operator to take snapshots without being impacted by the ZFS quota. This setting can be overridden by vm.quota_includes_snapshots . |
bhyve_snapshots_allowed | Boolean | Are new bhyve instances created in a way that allows snapshots (default is false)? |
While the value of bhyve_quota_enforced
may be changed from time to time by the operator, the setting that is in effect at the time of VM creation will be sticky with that VM via quota_includes_snapshots
. Operators may adjust VMs with vmadm update <uuid> quota_includes_snapshots=<value>
.
During VM migration, if quota
is set on an instance's top-level dataset, it must be relaxed so that an attempt to create a snapshot does not get ENOSPC
or EDQUOT
. During migration activation (e.g. sdc-migrate activate
) the final vm-migrate
snapshot is removed and quota
is set if only if quota_includes_snapshots
is true (explicitly or implicitly through its absence). quota
is not set during other migration phases.