TRITON-884: remove cloud analytics (ca) from Triton

Details

Issue Type:Bug
Priority:4 - Normal
Status:Resolved
Created at:2018-10-16T23:59:58.197Z
Updated at:2019-04-08T19:59:37.266Z

People

Created by:Former user
Reported by:Former user
Assigned to:Former user

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2019-03-19T19:08:08.864Z)

Fix Versions

2019-03-28 Operation Righteous Cowboy Lightning (Release Date: 2019-03-28)

Related Issues

Description

This ticket will coordinate removing Cloud Analytics (ca) from the
Triton product.

More details and justification to follow in separate comments.

Comments

Comment by Jira Bot
Created at 2018-10-18T20:07:47.518Z

globe-theatre commit fa7acec2af29d9b0ffb21bcc7dfc0361716159da (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton

Early workaround cainstsvc being faily (TRITON-766). We remove
cainstsvc everywhere so it doesn't cause spurious healthcheck failures.


Comment by Former user
Created at 2018-10-18T22:42:44.783Z
Updated at 2018-10-22T23:52:47.001Z

justification tl;dr


Comment by Jira Bot
Created at 2018-10-19T12:45:35.982Z

globe-theatre commit e3bcb49d432e75c22f8e4586b181fb3c43fc5a82 (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton (correct path to sdc-oneachnode)


Comment by Former user
Created at 2018-10-22T16:41:12.316Z

Note: "test-system" is failing in nightly-1 and nightly-2 because cainstsvc is removed. E.g. from https://jenkins.joyent.us/view/nightly2/job/nightly2-020-test-system/702/console

ok 7 should be truthy
#  create instrumentation
not ok 8 InvalidArgumentError: value "fs.logical_ops" for field "module.stat" is invalid: not a valid module/stat pair
  ---
    operator: error
    expected: |-
...

Comment by Former user
Created at 2018-11-26T22:09:49.133Z

partial workaround

echo "TRITON-884 countermeasures: remove faily cainstsvc service"
echo "so 'sdcadm health' and 'sdc-healthcheck' will pass"
/opt/smartdc/bin/sdc-oneachnode -a \
    'svcadm disable cainstsvc; svccfg delete cainstsvc; rm -f /opt/smartdc/agents/smf/cainstsvc-default.xml'

This is the current workaround that the nightly setup is using to stop failures in health checks and things like `svcs -Zx`. Note that this might be a bit heavy-handed in that it'll remove 'cainstsvc' from CNs where it isn't yet falling over.


Comment by Former user
Created at 2019-02-15T02:31:45.603Z

debugging: how to setup cainstsvc on all mock CNs in a mockcloud zone

sdc-login -l mock

function mockcloud_add_cainstsvc {
    # Add a cainstsvc mock agent to each server in the this mockcloud zone.
    local image_uuid
    image_uuid=$(uuid)

    ls /data/mockcloud/servers/ | while read server; do
        cd /data/mockcloud/servers/$server/agents
        if [[ ! -d cainstsvc ]]; then
            mkdir cainstsvc
            echo $image_uuid > cainstsvc/image_uuid
            uuid > cainstsvc/instance_uuid
            echo '{"name":"cainstsvc","version":"0.0.3vrelease-20181122-20181121T194844Z-g92307e6"}' > cainstsvc/package.json
        fi
    done

    svcadm restart cn-agent
}

mockcloud_add_cainstsvc

Comment by Former user
Created at 2019-03-13T03:36:24.728Z

FWIW here is a test-cloudapi run on nightly-1 after installing a grr-TRITON-884 branch build of cloudapi: https://gist.github.com/trentm/c996b4b39d4d04f59148e6cb1fa3b1eb

...
# time=15962.682ms
#
# TESTS COMPLETE IN 6023 SECONDS, SUMMARY:
#
#   PASS: 606 / 683
#   FAIL: 77 / 683
#
# FAILED TESTS:
#
#   - test/machines.71.test.js
#   - test/machines.test.js
#

There are failures there, but then test-cloudapi is having failures against #master in nightly-1 these days, so it is a pain to compare.


Comment by Jira Bot
Created at 2019-03-15T18:14:10.957Z

node-sdc-clients commit 97ab67eaecfe9a66c7cda1e6c29b1276ed1d156b (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton
Reviewed by: Marsell Kukuljevic <marsell@joyent.com>
Reviewed by: Pedro Palazón Candel <pedro@joyent.com>
Approved by: Marsell Kukuljevic <marsell@joyent.com>
Approved by: Pedro Palazón Candel <pedro@joyent.com>


Comment by Former user
Created at 2019-03-15T18:15:31.976Z
Updated at 2019-03-15T18:16:49.237Z

and sdc-clients@13.0.0 published (and tagged).

I merged sdc-clients first to get this into the sdc-system-tests build for nightly testing. I expect this to fix the "test-system" nightly failure.


Comment by Jira Bot
Created at 2019-03-15T23:27:17.885Z

sdc-sdc commit deb1ab8c895cfdbee0f4f73d65bcfa727a7c836f (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton
Reviewed by: Pedro Palazón Candel <pedro@joyent.com>
Approved by: Pedro Palazón Candel <pedro@joyent.com>


Comment by Jira Bot
Created at 2019-03-15T23:28:47.262Z

node-smartdc commit babf54683caf98c4d3943dc056a54f88d99b65d6 (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton
Reviewed by: Marsell Kukuljevic <marsell@joyent.com>
Reviewed by: Pedro Palazón Candel <pedro@joyent.com>
Approved by: Marsell Kukuljevic <marsell@joyent.com>
Approved by: Pedro Palazón Candel <pedro@joyent.com>


Comment by Former user
Created at 2019-03-15T23:30:45.425Z

smartdc@9.0.0 published


Comment by Jira Bot
Created at 2019-03-15T23:32:41.646Z

sdc-cloudapi commit b23fbcb19b93215c05480b7fd0712020ee79753c (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton
Reviewed by: Marsell Kukuljevic <marsell@joyent.com>
Reviewed by: Pedro Palazón Candel <pedro@joyent.com>
Approved by: Marsell Kukuljevic <marsell@joyent.com>
Approved by: Pedro Palazón Candel <pedro@joyent.com>


Comment by Jira Bot
Created at 2019-03-15T23:33:05.404Z

node-triton commit da8d7b14b5b5a630ee996ea8398da2a4707292c2 (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton
Reviewed by: Marsell Kukuljevic <marsell@joyent.com>
Reviewed by: Pedro Palazón Candel <pedro@joyent.com>
Approved by: Marsell Kukuljevic <marsell@joyent.com>
Approved by: Pedro Palazón Candel <pedro@joyent.com>


Comment by Former user
Created at 2019-03-15T23:34:08.905Z

triton@7.0.1 published to npm


Comment by Jira Bot
Created at 2019-03-15T23:34:53.092Z

sdcadm commit 0f411eee9840cba3b7b406fcc954b867092834ec (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton
Reviewed by: Pedro Palazón Candel <pedro@joyent.com>
Approved by: Pedro Palazón Candel <pedro@joyent.com>


Comment by Jira Bot
Created at 2019-03-16T04:42:15.885Z

sdc-headnode commit ef1ab0c41f84c9cbdc0a3ff67f976a700152c701 (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton
Reviewed by: Josh Wilsdon <josh@wilsdon.ca>
Approved by: Josh Wilsdon <josh@wilsdon.ca>


Comment by Former user
Created at 2019-03-16T04:46:06.725Z

Ran on nightly-1 (which it still has a ca0 zone):

[root@headnode (nightly-1) ~]# sdcadm experimental remove-ca
sdcadm experimental: error (UnknownCommand): unknown command: "remove-ca"
[root@headnode (nightly-1) ~]# sdcadm self-update --latest
Using channel dev
Update to sdcadm 1.25.0 (master-20190315T233505Z-g0f411ee)
Download update from https://updates.joyent.com
Run sdcadm installer (log at /var/sdcadm/self-updates/20190316T044306Z/install.log)
Updated to sdcadm 1.25.0 (master-20190315T233505Z-g0f411ee, elapsed 72s)
[root@headnode (nightly-1) ~]# sdcadm experimental remove-ca
Gathering SAPI service data.
Gathering SAPI instance data.
Gathering server agent data.
Gathering VM instance data.

This will make the following changes:
    - Remove "ca" service: SAPI records, 1 vm instance
        - VM e97f7d18-1c9e-4ca5-82ba-4dbf2ee2c1cc (ca0)
    - Remove "cabase" service: nothing to do
    - Remove "cainstsvc" service: nothing to do

Would you like to continue? [y/N] y

remove SAPI instances         [===================================================>] 100%        1
remove SAPI services          [===================================================>] 100%        1
Completed successfully (elapsed 27s).
[root@headnode (nightly-1) ~]#
[root@headnode (nightly-1) ~]# sdcadm experimental remove-ca
Gathering SAPI service data.
Gathering SAPI instance data.
Gathering server agent data.
Gathering VM instance data.

Nothing to do.
[root@headnode (nightly-1) ~]#

Comment by Jira Bot
Created at 2019-03-16T05:38:53.071Z

node-sdc-clients commit ddd9bbcfd5081573a82d203e0e008bb970393207 (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton (missing CA ref in test setup)
Reviewed by: Marsell Kukuljevic <marsell@joyent.com>
Approved by: Marsell Kukuljevic <marsell@joyent.com>


Comment by Jira Bot
Created at 2019-03-18T16:31:54.479Z

sdc-cloud-analytics commit 57736ae61cc3ffbdff5ff2af507da9f3aac8f924 (branch master, by Tim Foster)

TRITON-884 remove cloud analytics (ca) from Triton
Reviewed by: Trent Mick <trentm@gmail.com>
Approved by: Trent Mick <trentm@gmail.com>


Comment by Jira Bot
Created at 2019-03-18T17:10:04.839Z

mountain-gorilla commit 6e513c06a7766942aa746c50dd9626ac733f8694 (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton
Reviewed by: Tim Foster <tim.foster@joyent.com>
Approved by: Tim Foster <tim.foster@joyent.com>


Comment by Jira Bot
Created at 2019-03-18T17:16:02.011Z

sdcadm commit a75f9f88b5553410ef48cfcad02ae764ff8c14a2 (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton (missing ca ref in docs)
Reviewed by: Pedro Palazón Candel <pedro@joyent.com>
Approved by: Pedro Palazón Candel <pedro@joyent.com>


Comment by Jira Bot
Created at 2019-03-18T17:21:50.319Z

rfd commit 7efac180b2844b9218cd7b2debfe4e87d8aaa084 (branch master, by Trent Mick)

RFD 12: Drop CA related cloudapi endpoints from needed node-triton work (per TRITON-884)


Comment by Jira Bot
Created at 2019-03-19T17:53:58.837Z

documentation commit d9442708e5050d1bd0a363142118bfc65d487d43 (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton (#1215)


Comment by Former user
Created at 2019-03-19T18:57:00.109Z

Notice to operators

The `sdcadm experimental remove-ca` command can be used to remove CA services and instances from a DC. It will remove the "ca" service and zone, and any remaining "cainstsvc" and "cabase" agents on all setup CNs. Note that running this command may require an update to relatively recent "cnapi" and "cn-agent" services. However, the 'sdcadm experimental remove-ca' command will warn gracefully and is re-runnable (idempotent).


Comment by Jira Bot
Created at 2019-04-08T19:59:37.266Z

globe-theatre commit 825a73dabd8ee4ffbcc9de7df50d4614bc6c0adf (branch master, by Trent Mick)

TRITON-884 remove cloud analytics (ca) from Triton (drop these countermeasures now that ca has been removed)