ZAPI-770: changefeed events should be published for every changes made to VM objects in moray

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2017-03-22T01:18:06.000Z)

Fix Versions

2017-03-30 MOROCCAN MINIMAL (Release Date: 2017-03-30)

Description

Currently, VMAPI publishes changefeed events at the HTTP API layer, in specific restify handlers. However, as is described by ZAPI-769, some restify handlers that actually make changes to VM objects in moray do not publish changefeed events.

The same applies to any program (such as with tools/fix-no-owner.js) that is part of VMAPI and that uses its moray client layer to make changes to VM objects.

We could instead publish changefeed events from the moray client layer (in the putVm function at lib/apis/moray.js). This way, any existing and future code that makes changes to VM objects in moray would publish a changefeed event.

Comments

Comment by Julien Gilli [X]
Created at 2017-03-13T23:10:13.000Z
CR at cr.joyent.us/#/c/1655.

Comment by Julien Gilli [X]
Created at 2017-03-15T23:06:33.000Z
ZAPI-770 requires changes to node programs located in the tools/ directory. Without being able to run style checks and a linter on these programs, it won't be possible to make sure these changes are correct.

Comment by Julien Gilli [X]
Created at 2017-03-21T21:56:08.000Z
The CR at cr.joyent.us/#/c/1655 was tested so far only by running the full integration tests suite that comes with VMAPI.

Comment by Julien Gilli [X]
Created at 2017-03-21T23:34:19.000Z
With the latest patchset in that CR (patchset #10), I also ran one manual test to make sure that triton-cns was working as expected. I created a VM with node-triton, then ran:

$ ./bin/triton instance tag set -w 52fbab5e triton.cns.services=service-name     
{
    "triton.cns.services": "service-name"
}
$ ./bin/triton instance get 52fbab5e                    
{
    "id": "52fbab5e-4822-cf71-8ee2-e1d9b8b52302",
    "name": "52fbab5e",
    "type": "smartmachine",
    "brand": "joyent",
    "state": "running",
    "image": "1f32508c-e6e9-11e6-bc05-8fea9e979940",
    "ips": [
        "10.88.88.186",
        "192.168.128.68"
    ],
    "memory": 128,
    "disk": 3072,
    "metadata": {
        "root_authorized_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgqsBhk3FNGwaMo0VmNbjKn69K7+Qmxi8xEYEq4XvEv8pgm9UHz5GHC0xK4Y66l5+llhazTBFzHnzMGs5JUhe+Hcf09oWm02bRWpAweuUsIp+5vVHAH50OqYCzVF5J5SEay2wkckU5zoG7f9iYdXq/QvDqb02lOx2MWVvERAzxDP+wB/iDDo3Fkpy7KAkWyLNjf9/6qiLGH57JOJ+Q4yPT28jNPN6wfFQ/T/S5a7bTy06vz2C4GfsoX0UjBl3/R0cValjm8m7WuilUfCNRk8blwwlItyucMKbyf0UrLKsDX+MjnPtUBNmURwlDbaaFNGeuXkBVtIzQmI1TP58xot8v JulienGilli@joyentmac2011.local\n"
    },
    "tags": {
        "triton.cns.services": "service-name"
    },
    "created": "2017-03-21T22:19:52.057Z",
    "updated": "2017-03-21T23:28:30.000Z",
    "networks": [
        "034cb4bb-e82a-4aed-befe-6a479ebcd216",
        "a92e60de-1272-4b3c-a4a8-8be0147dc0d6"
    ],
    "primaryIp": "10.88.88.186",
    "firewall_enabled": false,
    "compute_node": "564d937f-54f0-546f-5792-2911e88639db",
    "package": "sample-128M",
    "dns_names": [
        "52fbab5e-4822-cf71-8ee2-e1d9b8b52302.inst.1cf780e0-f301-4e78-90da-7c3c8b4912d8.coal.cns.joyent.us",
        "52fbab5e.inst.1cf780e0-f301-4e78-90da-7c3c8b4912d8.coal.cns.joyent.us",
        "service-name.svc.1cf780e0-f301-4e78-90da-7c3c8b4912d8.coal.cns.joyent.us"
    ]
}
$

The node-triton instance get output above shows that the dns_names property was updated correctly. Moreover, the cns-updater logs in the cns0 zone show:

[2017-03-21T23:28:31.943Z] DEBUG: cns/58986 on 476df4e6-673f-439e-94be-be2e75344e32: updating vm (stage=UpdateStream)
    info: {
      "vm": "52fbab5e-4822-cf71-8ee2-e1d9b8b52302",
      "why": [],
      "l_s": true,
      "l_i": true,
      "svcs": [
        {
          "name": "service-name",
          "ports": []
        }
      ],
      "c": {
        "coal.cns.joyent.us": 4,
        "88.88.10.in-addr.arpa": 1,
        "128.168.192.in-addr.arpa": 1
      },
      "o": "cf"
    }

which is consistent with the change that was made.

Comment by Julien Gilli [X]
Created at 2017-03-22T00:45:29.000Z
Finally, tested patchset #10 with cmon by creating the following VM after setting up CMON in my COAL:

$ ./bin/triton instance create --name test-changefeed-cmon base-64-lts sample-128M
Creating instance test-changefeed-cmon (529c33d5-5b81-e047-d970-8ce3c9aafa47, base-64-lts@16.4.1)
$ ./bin/triton instance get test-changefeed-cmon
{
    "id": "529c33d5-5b81-e047-d970-8ce3c9aafa47",
    "name": "test-changefeed-cmon",
    "type": "smartmachine",
    "brand": "joyent",
    "state": "running",
    "image": "1f32508c-e6e9-11e6-bc05-8fea9e979940",
    "ips": [
        "10.88.88.20",
        "192.168.128.70"
    ],
    "memory": 128,
    "disk": 3072,
    "metadata": {
        "root_authorized_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgqsBhk3FNGwaMo0VmNbjKn69K7+Qmxi8xEYEq4XvEv8pgm9UHz5GHC0xK4Y66l5+llhazTBFzHnzMGs5JUhe+Hcf09oWm02bRWpAweuUsIp+5vVHAH50OqYCzVF5J5SEay2wkckU5zoG7f9iYdXq/QvDqb02lOx2MWVvERAzxDP+wB/iDDo3Fkpy7KAkWyLNjf9/6qiLGH57JOJ+Q4yPT28jNPN6wfFQ/T/S5a7bTy06vz2C4GfsoX0UjBl3/R0cValjm8m7WuilUfCNRk8blwwlItyucMKbyf0UrLKsDX+MjnPtUBNmURwlDbaaFNGeuXkBVtIzQmI1TP58xot8v JulienGilli@joyentmac2011.local\n"
    },
    "tags": {},
    "created": "2017-03-22T00:39:25.708Z",
    "updated": "2017-03-22T00:40:49.000Z",
    "networks": [
        "034cb4bb-e82a-4aed-befe-6a479ebcd216",
        "a92e60de-1272-4b3c-a4a8-8be0147dc0d6"
    ],
    "primaryIp": "10.88.88.20",
    "firewall_enabled": false,
    "compute_node": "564d937f-54f0-546f-5792-2911e88639db",
    "package": "sample-128M",
    "dns_names": [
        "529c33d5-5b81-e047-d970-8ce3c9aafa47.inst.1cf780e0-f301-4e78-90da-7c3c8b4912d8.coal.cns.joyent.us",
        "test-changefeed-cmon.inst.1cf780e0-f301-4e78-90da-7c3c8b4912d8.coal.cns.joyent.us"
    ]
}
$
then running the following command in my COAL's cmon0 zone:
[root@211ed307-54f1-477a-92fb-545fdc581df4 (coal:cmon0) ~]# curl -s --insecure --cert-type pem --cert /var/tmp/cert.pem --key /var/tmp/key.pem https://cmon.coal.joyent.us:9163/v1/discover | json -Ha containers | json -a -c 'vm_alias==="test-changefeed-cmon"'
{
  "server_uuid": "564d937f-54f0-546f-5792-2911e88639db",
  "source": "Changefeed",
  "vm_alias": "test-changefeed-cmon",
  "vm_image_uuid": "1f32508c-e6e9-11e6-bc05-8fea9e979940",
  "vm_owner_uuid": "1cf780e0-f301-4e78-90da-7c3c8b4912d8",
  "vm_uuid": "529c33d5-5b81-e047-d970-8ce3c9aafa47",
  "cached_date": 1490143252203
}
[root@211ed307-54f1-477a-92fb-545fdc581df4 (coal:cmon0) ~]#

Comment by Bot Bot [X]
Created at 2017-03-22T01:16:20.000Z

sdc-vmapi commit 0ca1d9b (branch master, by Julien Gilli)

ZAPI-770 changefeed events should be published for every changes made to VM objects in moray
    Reviewed by: Richard Kiene <richard.kiene@joyent.com>
    Reviewed by: Josh Wilsdon <jwilsdon@joyent.com>
    Approved by: Richard Kiene <richard.kiene@joyent.com>


Comment by Julien Gilli [X]
Created at 2017-03-22T01:17:46.000Z
FWIW, I've also tested that deleting a VM updates the CNS records properly by deleting a VM that was registered in CNS and running cnsadm vm vm-uuid for that VM and verifying that CNS had unregistered it like following:

[root@476df4e6-673f-439e-94be-be2e75344e32 (coal:cns0) ~]# cnsadm vm 22fe40dc-7266-4108-96f1-b43cf84dd984
uuid:        22fe40dc-7266-4108-96f1-b43cf84dd984
last_visit:  17 secs ago
names:       []
[root@476df4e6-673f-439e-94be-be2e75344e32 (coal:cns0) ~]#