OS-6832: reclaim memory when NOSLEEP bhyve allocation fails

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-06-25T15:09:44.188Z)

Fix Versions

2018-07-05 Villa (Release Date: 2018-07-05)

Related Links

Description

See OS-6826 for some background.

Comments

Comment by Mike Gerdts
Created at 2018-05-30T20:16:56.009Z
Updated at 2018-05-30T20:18:51.439Z
I have a test case that shows that what we have is not sufficient. This is done in a vmware instance that has 8 GiB RAM.
# zfs create zones/junk
# cp big.iso /zones/junk; cd /zones/junk; for i in {1..5}; do cp big.iso big.iso.$i; done (keep copying until vmstat shows that free becomes steady at some smallish number.)
# vmadm start $(vm test) (ram=2048 on this one). It succeeds.
# (with cwd not in zones/junk) vmadm create -f log.json This has ram=1024 and does not immediately succeed./zones/$uuid/logs/platform.log says Unable to allocate memory (1073741824), retrying in 1 secondn every second. vmstat 1 does not show free memory changing significantly.
# cd; zfs destroy zones/junk
# Observe that the vm creation now succeeded.

Since we don't set primarycache=metadata, the cached blocks from other VMs' zvols will be sufficient to fill the ARC.  That is, in the real world the synthetic "fill the arc with iso copies" part will not be needed to reproduce the inability to start a VM due to the ARC squatting on RAM.

Comment by Hans Rosenfeld
Created at 2018-06-25T10:00:35.936Z
Testing:

On a test machine with 128G of RAM, I created 24 VMs with 4GB each. With no VMs running I filled the ARC by copying a bunch of zvols into /dev/null, resulting in this memory use:
# echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    8356550             32642   25%
Boot pages                  70213               274    0%
ZFS File Data             9385524             36662   28%
Anon                       110474               431    0%
Exec and libs                3072                12    0%
Page cache                  21472                83    0%
Free (cachelist)            10117                39    0%
Free (freelist)          15560965             60785   46%

Total                    33518387            130931
Physical                 33518386            130931

Then I booted up all the VMs sequentially. After 14 of the 24 were booted, the system ran out of free memory:

Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    8545211             33379   25%
Boot pages                  70213               274    0%
ZFS File Data             9127806             35655   27%
VMM Memory               14739968             57578   44%
Anon                       142633               557    0%
Exec and libs                3083                12    0%
Page cache                  22488                87    0%
Free (cachelist)            11501                44    0%
Free (freelist)            855484              3341    3%

Total                    33518387            130931
Physical                 33518386            130931

The next VM took a bit longer to start as it was waiting for memory to be freed. Eventually the system could start 21 out of 24 VMs before it ran out of memory and the ARC couldn't be shrunk any further:

> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7087677             27686   21%
Boot pages                  70213               274    0%
ZFS File Data             1460043              5703    4%
VMM Memory               22116864             86394   66%
Anon                       114175               445    0%
Exec and libs                1107                 4    0%
Page cache                   7743                30    0%
Free (cachelist)            33537               131    0%
Free (freelist)           2627028             10261    8%

Total                    33518387            130931
Physical                 33518386            130931
> availrmem/X
availrmem:
availrmem:      12c3c3

Comment by Jira Bot
Created at 2018-06-25T15:03:27.132Z
illumos-joyent commit a449eb75c6b8137021ca42d49ee3753e36804dc7 (branch master, by Hans Rosenfeld)

OS-6832 reclaim memory when NOSLEEP bhyve allocation fails
Reviewed by: Mike Gerdts <mike.gerdts@joyent.com>
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Approved by: Jerry Jelinek <jerry.jelinek@joyent.com>