ZAPI-662: Update stream to support more efficient queries by Triton Name Service

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2015-11-28T23:52:05.000Z)

Fix Versions

2015-12-10 Dino (Release Date: 2015-10-12)

Comments

Comment by Richard Kiene
Created at 2015-09-14T21:25:05.000Z
Updated at 2015-11-09T19:26:55.000Z
TCNS needs updates for:

nics, tags, owner_uuid, server_uuid, destroyed, alias, state, internal_metadata, and customer_metadata

See RFD 0005 for implementation details.

Comment by Trent Mick
Created at 2015-10-08T17:35:35.000Z
https://github.com/joyent/node-sdc-changefeed/commit/69a390269efd6385d12b3bf682f4430c68acdadb was the commit for this ticket, BTW. (note to self while adding hooks)

Comment by Bot Bot [X]
Created at 2015-10-29T21:12:23.000Z

node-sdc-changefeed commit 9590e79 (branch master, by Richard Kiene)

ZAPI-662: Initial changefeed module implementation


Comment by Bot Bot [X]
Created at 2015-10-31T01:22:29.000Z

node-sdc-changefeed commit 533f399 (branch master, by Richard Kiene)

ZAPI-662: Update dependencies, naming changes, and prep for npm publish.


Comment by Bot Bot [X]
Created at 2015-11-02T19:36:43.000Z

node-sdc-changefeed commit 348d74f (branch master, by Richard Kiene)

ZAPI-662: Add documentation and detailed comments for public methods.


Comment by Bot Bot [X]
Created at 2015-11-04T04:24:58.000Z

node-sdc-changefeed commit 3e939d4 (branch master, by Richard Kiene)

ZAPI-662: Add changefeedsnoop


Comment by Bot Bot [X]
Created at 2015-11-04T04:24:59.000Z

node-sdc-changefeed commit ff16836 (branch master, by Richard Kiene)

ZAPI-662: Add bucket GC and configurable endpoint port.

    This commit also changes the publisher options slightly. Instead of
    individual moray properties, there is a moray object in the options
    object with moray specific properties. This should allow us to add
    other storage mechanisms in the same way, and keep things clean.


Comment by Bot Bot [X]
Created at 2015-11-04T04:29:14.000Z

node-sdc-changefeed commit f3a0848 (branch master, by Richard Kiene)

ZAPI-662: Update documentation for publisher options


Comment by Bot Bot [X]
Created at 2015-11-04T21:11:11.000Z

node-sdc-changefeed commit 6b7acc9 (branch master, by Richard Kiene)

ZAPI-662: Add missing main in package.json / fix npm install


Comment by Bot Bot [X]
Created at 2015-11-07T00:46:26.000Z

node-sdc-changefeed commit bfa969a (branch master, by Richard Kiene)

ZAPI-662: Add listener heartbeat to keep idle connections alive


Comment by Bot Bot [X]
Created at 2015-11-27T05:40:10.000Z

sdc-vmapi commit 58f775d (branch master, by Richard Kiene)

ZAPI-662: Add changefeed support
    ZAPI-684: Replace common.objectDiff with deep-diff
    ZAPI-687: Allow VM creation via PUT /vms


Comment by Richard Kiene
Created at 2015-11-27T05:45:40.000Z
https://github.com/joyent/sdc-vmapi/commit/58f775dba128d5b43052b21d5633c1ac69b3d691

Comment by Bot Bot [X]
Created at 2015-11-27T17:24:05.000Z

sdc-vmapi commit 8db33c7 (branch master, by Trent Mick)

ZAPI-662: Add changefeed support (fix sapi template syntax)


Comment by Bot Bot [X]
Created at 2015-11-27T19:38:28.000Z

sdc-vmapi commit 4461eab (branch master, by Josh Wilsdon)

ZAPI-662: Add changefeed support (needs more work)
    ZAPI-684: Replace common.objectDiff with deep-diff (needs more work)
    ZAPI-687: Allow VM creation via PUT /vms (needs more work)


Comment by Josh Wilsdon
Created at 2015-11-27T19:43:27.000Z
Updated at 2018-05-07T17:50:52.055Z
This commit needs some more work. We ran into two different problems this morning. First:

[2015-11-27T17:28:01.962Z]  INFO: vmapi/api/61777 on caf5f79a-3ec9-4a97-8356-51f91011f7a2:  (req_id=35df6280-952c-11e5-a547-7f79a1f124b9)
    evt: {
      "ph": "b",
      "name": "putvms",
      "req_seq": 1448645281955001
    }
Uncaught TypeError: Cannot read property '0' of undefined

FROM
_diff (/opt/smartdc/vmapi/lib/endpoints/vms.js:1069:41)
Function.deepDiff [as observableDiff] (/opt/smartdc/vmapi/node_modules/deep-diff/index.js:151:9)
_diffVms (/opt/smartdc/vmapi/lib/endpoints/vms.js:1068:26)
fn (/opt/smartdc/vmapi/node_modules/async/lib/async.js:573:34)
/opt/smartdc/vmapi/node_modules/async/lib/async.js:489:34
process._tickDomainCallback (node.js:459:13)
[ Nov 27 17:28:02 Stopping because all processes in service exited. ]
[ Nov 27 17:28:02 Executing stop method (:kill). ]
[ Nov 27 17:28:02 Restarting too quickly, changing state to maintenance. ]

which was caused because of this code:

        var oldVm = req.vms[uuid];
        async.waterfall([
            function _morayPut(cb2) {
                req.app.moray.putVm(uuid, vm, cb2);
            },
            function _diffVms(etag, cb2) {
                var diffs = [];
                deepDiff.observableDiff(oldVm, vm, function _diff(diff) {
                    diffs.push(diff.path[0]);
                });
                cb2(null, diffs);
            },

in lib/endpoints/vms.js putVms().

This can be triggered by stopping vm-agent, then running:

echo '{"image_uuid": "fd2cc906-8938-11e3-beab-4359c665ac99", "brand": "joyent-minimal", "alias": "breaking-vmapi"}' | vmadm create

to create a new VM and then starting vm-agent. This will crash vmapi with an error that looks like the above. The reason it's crashing is that oldVm here is `undefined` and when you do:

> var observableDiff = require('deep-diff').observableDiff;
undefined
> observableDiff(undefined, {a: 'a'}, function (d) { console.log('d: ' + JSON.stringify(d)); });
d: {"kind":"N","rhs":{"a":"a"}}
undefined
>

you indeed don't get a path member.

While testing this I also found another problem which is caused by a regular PUT /vms/:uuid for a new VM. If you run the same command above to create a VM while vm-agent is running, it crashes with:

Uncaught AssertionError: "undefined" == true

FROM
fail (assert.js:92:9)
Function.ok (assert.js:112:17)
Object.translateVm (/opt/smartdc/vmapi/lib/common/vm-common.js:53:12)
onGetVm (/opt/smartdc/vmapi/lib/endpoints/vms.js:1187:25)
EventEmitter.<anonymous> (/opt/smartdc/vmapi/lib/apis/moray.js:214:16)
EventEmitter.g (events.js:180:16)
EventEmitter.emit (events.js:92:17)
EventEmitter.<anonymous> (/opt/smartdc/vmapi/node_modules/moray/lib/objects.js:197:13)
EventEmitter.g (events.js:180:16)
EventEmitter.emit (events.js:92:17)
Client._handleMessage (/opt/smartdc/vmapi/node_modules/moray/node_modules/fast/lib/client.js:378:21)
MessageDecoder.onMessage (/opt/smartdc/vmapi/node_modules/moray/node_modules/fast/lib/client.js:446:14)
MessageDecoder.EventEmitter.emit (events.js:95:17)
MessageDecoder._write (/opt/smartdc/vmapi/node_modules/moray/node_modules/fast/lib/protocol/message_decoder.js:121:14)
doWrite (/opt/smartdc/vmapi/node_modules/moray/node_modules/fast/node_modules/readable-stream/lib/_stream_writable.js:263:12)
writeOrBuffer (/opt/smartdc/vmapi/node_modules/moray/node_modules/fast/node_modules/readable-stream/lib/_stream_writable.js:250:5)
MessageDecoder.Writable.write (/opt/smartdc/vmapi/node_modules/moray/node_modules/fast/node_modules/readable-stream/lib/_stream_writable.js:197:11)
write (_stream_readable.js:583:24)
flow (_stream_readable.js:592:7)
Socket.pipeOnReadable (_stream_readable.js:624:5)
Socket.EventEmitter.emit (events.js:92:17)
emitReadable_ (_stream_readable.js:408:10)
emitReadable (_stream_readable.js:404:5)
readableAddChunk (_stream_readable.js:165:9)
Socket.Readable.push (_stream_readable.js:127:10)
TCP.onread (net.js:528:21)
[ Nov 27 19:29:36 Stopping because all processes in service exited. ]
[ Nov 27 19:29:36 Executing stop method (:kill). ]
[ Nov 27 19:29:36 Restarting too quickly, changing state to maintenance. ]
[ Nov 27 19:41:30 Leaving maintenance because clear requested. ]
[ Nov 27 19:41:31 Enabled. ]
[ Nov 27 19:41:31 Executing start method ("/opt/smartdc/vmapi/build/node/bin/node --abort_on_uncaught_exception /opt/smartdc/vmapi/server.js &"). ]
[ Nov 27 19:41:31 Method "start" exited with status 0. ]



Comment by Bot Bot [X]
Created at 2015-11-28T08:39:05.000Z

sdc-vmapi commit 92e4682 (branch master, by Richard Kiene)

ZAPI-662: Add changefeed support
    ZAPI-684: Replace common.objectDiff with deep-diff
    ZAPI-687: Allow VM creation via PUT /vms


Comment by Richard Kiene
Created at 2015-11-28T23:52:05.000Z
@josh thanks for the stack trace and the repro steps, that made fixing the bug pretty straightforward. @trent.mick thanks for fixing the SAPI template. I've definitely learned a few areas I need to test from now on, and I've applied them to this latest update.

The main issue with the initial set of work was my assumption that VMAPI would always have a moray entry for a given PUT request. I've accounted for a VM not existing in VMAPI's moray bucket, tested a fresh deployment via `sdcadm update`, and tested that a fresh CoaL build works and passes all tests. In addition, I added tests which make PUT requests for VMs that do not exist yet.