OS-5330

[root@headnode (coal) /zones/jj]# zoneadm -z ub64 mount
zone 'ub64': failed to determine the zone's default privilege set
zoneadm: zone 'ub64': call to zoneadmd failed

Former user commented on 2016-04-18T09:38:53.000-0400 (edited 2017-12-14T12:24:36.633-0500):

The first problem is that the SUNWdefault.xml file lists a non-existent brand. If we change that to joyent then the second problem (lx-specific) is that mounting "/usr/lib/fs/lofs/mount -o ro,nodevices /manifests/joyent /zones/ub15/lu/a/lib/svc/manifest" fails. This is due to the missing mountpoint within the Linux image. If we change the default brand to lx then we get an error with "mount of /dev failed: Invalid argument", but the mount does complete. However, zlogin into the zone will fail as a result of the missing dev (zlogin: failed to open /zones/ub15/lu/dev/pts/3: No such file or directory).

zoneadmd <- mount(/native/dev /dev lx_devfs)=-1

An additional bug is that unmounting the zone does not properly cleanup the dynamic vnic, so attempting to mount again, or to boot the zone, gives:

zone 'ub15': error creating VNIC eth0 (global NIC external)
zone 'ub15': msg: dladm: vnic creation over e1000g0 failed: MAC address is already in use
zone 'ub15': Failed cmd: dladm create-vnic -t -l e1000g0 -p  mtu=1500,zone=ub15 -m 02:08:20:e1:7b:15 tmp1053820
zoneadm: zone 'ub15': call to zoneadmd failed

Former user commented on 2016-09-08T16:10:38.000-0400:

The first problem is indeed caused by the default zone file
(SUNWdefault.xml) listing a nonexistent brand: liveimg. This can
be fixed by lofs mounting the Joyent zone file.

# mount -F lofs /etc/zones/Joyent.xml /etc/zones/SUNWdefault.xml

Retrying the mount at this point will complain about a missing
/manifests/joyent. This is because the joyent brand includes this
global mount:

<global_mount special="%P/manifests/joyent"
            directory="/lib/svc/manifest"
            opt="ro,nodevices" type="lofs" />

The %P is replaced with the value of _ZONEADM_ZPOOL (or empty
string if not set). I think this is supposed to be set by a
statechange script but we can also just set it ourselves before
running mount.

# export _ZONEADMD_ZPOOL=/zones

At this point I believe you should be able to mount a Joyent-brand
zone, but lx will continue to barf, this time about ccs. At this
point we can use lofs mount to have the default zone point to the lx
brand.

# mkdir -p /data/mock/etc/zones
# cp /etc/zones/Joyent.xml /data/mock/etc/zones/
# chmod u+w /data/mock/etc/zones/Joyent.xml
# vim /data/mock/etc/zones/Joyent.xml (change joyent to lx)
# umount /etc/zones/SUNWdefault.xml
# mount -F lofs /data/mock/etc/zones/Joyent.xml /etc/zones/SUNWdefault.xml

This will fix the ccs mount issue but now reports mount of /dev failed.

This is because lx devfs (and lx procfs) can only be mounted in an
lx-brand zone, but in this case we are mounting an lx-brand zone
inside a scratch zone, which is actually a native zone.

https://github.com/joyent/illumos-joyent/blob/master/usr/src/uts/common/brand/lx/devfs/lxd_vfsops.c#L391

To get around this create a mock lx-brand platform.xml and lofs mount
that.

# diff -u /usr/lib/brand/lx/platform.xml /data/mock/usr/lib/brand/lx/platform.xml
--- /usr/lib/brand/lx/platform.xml     	Wed Sep  7 03:35:05 2016
+++ /data/mock/usr/lib/brand/lx/platform.xml   	Thu Sep  8 19:43:36 2016
@@ -61,7 +61,7 @@
       	    directory="/native/etc/zones/%z.xml" opt="ro" type="lofs" />

       	<!-- Local filesystems to mount when booting the zone -->
-      	<mount special="/native/dev" directory="/dev" type="lx_devfs" />
+      	<!-- <mount special="%R/a/native/dev" directory="%R/a/dev" type="lx_devfs" /> -->
       	<mount special="proc" directory="/native/proc" type="proc" />
       	<mount special="swap" directory="/native/etc/svc/volatile"
       	    type="tmpfs" />
@@ -69,7 +69,7 @@
       	<mount special="objfs" directory="/system/object" type="objfs" />
       	<mount special="ctfs" directory="/system/contract" type="ctfs" />
       	<mount special="mnttab" directory="/etc/mnttab" type="mntfs" />
-      	<mount special="lxproc" directory="/proc" type="lx_proc" />
+      	<!-- <mount special="lxproc" directory="/proc" type="lx_proc" /> -->

       	<!-- Devices to create under /dev -->
       	<device match="arp" />

# mount -F lofs /data/mock/usr/lib/brand/lx/platform.xml /usr/lib/brand/lx/platform.xml

After this there should be no errors during mount (except maybe a
dladm error if a previous run managed to create a vnic, but you can
ignore that for now) but zlogin -S will fail.

# zlogin -S 710de071-78a4-c369-a8b6-b2414e17d7c6
[Connected to zone '710de071-78a4-c369-a8b6-b2414e17d7c6' pts/4]
zlogin: failed to open /zones/710de071-78a4-c369-a8b6-b2414e17d7c6/lu/dev/pts/4: No such file or directory

[Connection to zone '710de071-78a4-c369-a8b6-b2414e17d7c6' pts/4 closed]

The problem is that the scratch zone code needs to mount /dev
under /lu, not /lu/a (the actual zoneroot in a scratch zone)
but the code is hardcoded to check for the exact mount point /dev
and move it back to /lu, it doesn't notice lx's /native/dev.

https://github.com/joyent/illumos-joyent/blob/master/usr/src/cmd/zoneadmd/vplat.c#L1815-L1831

We can fix this by changing lx-brand's platform.xml to mount at
/dev/, not /native/dev.

# diff -u /usr/lib/brand/lx/platform.xml /data/mock/usr/lib/brand/lx/platform.xml
--- /usr/lib/brand/lx/platform.xml     	Wed Sep  7 03:35:05 2016
+++ /data/mock/usr/lib/brand/lx/platform.xml   	Thu Sep  8 19:52:58 2016
@@ -32,7 +32,7 @@

 <platform name="lx" allow-exclusive-ip="true">
       	<!-- Global filesystems to mount when booting the zone -->
-      	<global_mount special="/dev" directory="/native/dev" type="dev"
+      	<global_mount special="/dev" directory="/dev" type="dev"
       	    opt="attrdir=%R/dev" />

       	<!--
@@ -61,7 +61,7 @@
       	    directory="/native/etc/zones/%z.xml" opt="ro" type="lofs" />

       	<!-- Local filesystems to mount when booting the zone -->
-      	<mount special="/native/dev" directory="/dev" type="lx_devfs" />
+      	<!-- <mount special="%R/a/native/dev" directory="%R/a/dev" type="lx_devfs" /> -->
       	<mount special="proc" directory="/native/proc" type="proc" />
       	<mount special="swap" directory="/native/etc/svc/volatile"
       	    type="tmpfs" />
@@ -69,7 +69,7 @@
       	<mount special="objfs" directory="/system/object" type="objfs" />
       	<mount special="ctfs" directory="/system/contract" type="ctfs" />
       	<mount special="mnttab" directory="/etc/mnttab" type="mntfs" />
-      	<mount special="lxproc" directory="/proc" type="lx_proc" />
+      	<!-- <mount special="lxproc" directory="/proc" type="lx_proc" /> -->

       	<!-- Devices to create under /dev -->
       	<device match="arp" />

At this point mount will complain about /dev/fd, which can be
fixed with further platform munging, but this is good enough to
perform a zlogin -S.

# zlogin -S 710de071-78a4-c369-a8b6-b2414e17d7c6
[Connected to zone '710de071-78a4-c369-a8b6-b2414e17d7c6' pts/4]
@710de071-78a4-c369-a8b6-b2414e17d7c6:~$ ls -la /proc
total 16
drwxr-xr-x   2 root     root         117 Sep  8 20:01 .
drwxr-xr-x  16 root     root        1099 Sep  8 20:01 ..
@710de071-78a4-c369-a8b6-b2414e17d7c6:~$ ls -la /native/proc
total 142
dr-xr-xr-x  63 root     root       65216 Sep  8 20:09 .
drwxr-xr-x   5 root     root         298 Sep  8 20:01 ..
dr-x--x--x   5 root     root         896 Sep  8 20:01 18447
dr-x--x--x   5 root     root         896 Sep  8 20:09 18541
dr-x--x--x   5 root     root         896 Sep  8 20:09 18546
@710de071-78a4-c369-a8b6-b2414e17d7c6:~$ ls
a         etc       native    proc      tmp
bin       kernel    opt       sbin      usr
dev       lib       platform  system    var
@710de071-78a4-c369-a8b6-b2414e17d7c6:~$

However, this still isn't a fix as things aren't quite right. E.g.
/proc is actually mounted at /native/proc. I think the real solution
is to come up with something like a "scratch brand" that can be used
to mount any type of zone. This may be as simple as just creating a
new brand in /usr/lib/brand/scratch; or maybe some code changes
are required too. I'm not sure yet. I just wanted to give a status
update so my current knowledge isn't lost.

Former user commented on 2016-09-08T16:19:29.000-0400:

The dladm errors, both at mount and unmount, come from the statechange
script.

/usr/lib/brand/jcommon/statechange

This is invoked indirectly via the lx statechange script, e.g.
when the brand_poststatechg() function is invoked.

https://github.com/joyent/illumos-joyent/blob/master/usr/src/cmd/zoneadmd/zoneadmd.c#L564

In this call it will be invoked like so:

exec /usr/lib/brand/lx/poststate 710de071-78a4-c369-a8b6-b2414e17d7c6 /zones/710de071-78a4-c369-a8b6-b2414e17d7c6 2 0

Causing it to fall into this case in jcommon/statechange:

[[ "$subcommand" == "pre" && $cmd == 0 ]] && setup_fs
if [[ "$subcommand" == "post" && $cmd == 0 ]]; then
        [[ -n "$jst_showsnap" ]] && setup_snapshots
       setup_net
       setup_fw
fi

The cleanup path is similar:

if [[ "$subcommand" == "pre" && $cmd == 4 ]]; then
        [[ -n "$jst_snowsnap" ]] && cleanup_snapshots
        cleanup_net
fi

We'll either want to fix these or remove them altogether when
performing a scratch zone mount.

Former user commented on 2016-09-22T10:42:38.000-0400:

Former user commented on 2016-09-22T10:42:52.000-0400: