OS-7943: Want platform build test artifact

Details

Issue Type:Improvement
Priority:4 - Normal
Status:Resolved
Created at:2019-08-09T16:56:38.047Z
Updated at:2019-10-10T11:18:15.985Z

People

Created by:Former user
Reported by:Former user
Assigned to:Former user

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2019-10-10T11:18:15.969Z)

Fix Versions

2019-10-24 Double Dip (Release Date: 2019-10-24)

Description

Today the platform has a number of different test that are part of illumos that are not easy to run. Running them involves downloading the appropriate pkgsrc target, installing several packages, and then running the tests themselves.

To make this much simpler, it'd be great if the platform build had an additional output target that included the following:

As an example, we might say that we have a new tests-<buildstamp>.tgz artifact that includes all of the above and the contract is to extract it with -C / and then to run something like /opt/smartos-tests.

Comments

Comment by Former user
Created at 2019-09-20T11:02:08.874Z
Updated at 2019-09-20T11:02:18.717Z

Many of the tests we'd want to include in this artifact are listed in IPS package manifests in the illumos-joyent gate. A reasonable approach might be to first generate a smartos-style 'test.manifest' from these IPS manifests in order to deliver the required test artifact.

timf@smartos-build (master) pwd
/space/timf/test/illumos-joyent/usr/src/pkg/manifests
timf@smartos-build (master) ls  -1 *test*
system-bhyve-tests.mf
system-dtrace-tests.mf
system-file-system-zfs-tests.mf
system-io-tests.mf
system-test-cryptotest.mf
system-test-elftest.mf
system-test-libctest.mf
system-test-ostest.mf
system-test-smbclient.mf
system-test-testrunner.mf
system-test-utiltest.mf
system-test-zfstest.mf

We probably want to specify exactly which IPS manifests to parse. The list rm had in TOOLS-2306 seems like a good start.


Comment by Former user
Created at 2019-09-24T22:36:28.379Z

We have some tests that aren't yet packaged as IPS packages, so we likely need to add manifests for those as well as the scripting needed to configure the test machine.


Comment by Former user
Created at 2019-09-30T16:12:45.787Z

The approach we're suggesting here introduces a new wrapper script to the illumos-joyent gate, /opt/smartos-test/bin/smartos-test that has this usage message:

[root@kura ~]# /opt/smartos-test/bin/smartos-test -h
Usage: smartos-test [-h] [-c] [-e] [-r] <path to tests.tgz>

At least one of -c, -e, -r is required.

  -h       print usage
  -c       configure the system for testing
  -e       execute known tests
  -f       skip the check to ensure platform version == test version
  -r       roll back to the zones/opt@system-smartos-test snapshot
           before doing any system configuration or test execution

Running it with the -r -c -e arguments we'll do the following:

While TOOLS-2306 describes an approach of having one wrapper per test suite, we believe it would still be useful to have something in illumos-joyent that can take most of the pain out of setting up and installing a system to prepare for testing, rather than just something that works only for globe-theatre.

We specifically want scripting that enforces that the version of the platform we're running is identical to the version of the tests we propose to run.

The changes (in progress) add a 'tests-tar' Makefile target to smartos-live, which will be run by the ./tools/build_jenkins wrapper, and thus be built by the platform build automatically. Note that this is a separate build artifact, and doesn't change the contents of the existing platform-<buildstamp>.tgz


Comment by Former user
Created at 2019-10-02T10:29:49.302Z

test-out.txt shows the tests being installed and executed on our development machine.

We've also run these tests on our coal instance that was configured as a Manta . There, most of the tests pass, other than a few of the libc tests which occasionally time-out and are killed by the test harness. Bumping the timeout in /opt/libc-tests/runfiles/default.run from 60 to 180 worked around the problem.


Comment by Former user
Created at 2019-10-02T11:04:19.573Z

During the course of this work, we fixed several related problems that were either preventing tests from running via the /opt/test-runner/bin/run script, or where for some suites, it marked tests as "failing validation" - this was due to tests being included in the "default.run" file that the test runner consumes, but not being packaged in the IPS package manifest (which are used by the smartos-live build to determine the contents of the test archive)

We also noticed a few IPS manifests that had duplicate entries, which the smartos-live 'builder' tool would complain about.

Finally, due to gcc7 producing 64-bit objects by default on pkgsrc 2018Q4 x86_64 installs, a few tests needed "-m32" to produce object files of the correct bitness.


Comment by Jira Bot
Created at 2019-10-10T10:31:36.154Z

illumos-joyent commit 967c11b83ffd09b85f4de9e51b3d819de9846d78 (branch master, by Tim Foster)

OS-7943 Want platform build test artifact
Reviewed by: John Levon <john.levon@joyent.com>
Reviewed by: Ryan Zezeski <rpz@joyent.com>
Approved by: John Levon <john.levon@joyent.com>


Comment by Jira Bot
Created at 2019-10-10T10:32:19.520Z

smartos-live commit 7c7ba26873ee667b40b0d42f2a20e733ee747742 (branch master, by Tim Foster)

OS-7943 Want platform build test artifact
Reviewed by: John Levon <john.levon@joyent.com>
Reviewed by: Ryan Zezeski <rpz@joyent.com>
Approved by: John Levon <john.levon@joyent.com>