ZAPI-690: PUT /vms[/:uuid] should check timestamp to prevent out-of-order replacement

Description

When we do a PUT on /vms or /vms?server_uuid=<uuid>, we are replacing the list of VMs with what's on the CN. We have two paths for this:

The problem here is that we may have multiple actions in-flight at a time and since each of these is replacing the VM's state with what is the current state, it's possible (especially with the workflow actions which involve multiple moving parts) that we could PUT an older state than what as already been PUT to VMAPI.

To resolve this, I'd like to make sure that any time we do a PUT we include a timestamp (since we're on the same CN these timestamps I believe can be assumed monotonic so long as they're generated at the time of gathering the data). This way, VMAPI can ignore PUT requests where the timestamp is older than the current object.

Thinking about this, the best thing to use may be the last_modified timestamp since this already is in the VM objects. However, there are a few problems with this:

As such what I'd like to do is add a new property to all VM objects which will be required for a PUT to succeed. This should:

I'm currently thinking to call this field `update_timestamp` but would be willing to consider other names.

The first step here will be to start including this field in all updates from vm-agent and from the objects returned by cn-agent tasks. (AGENT-954)

Comments

Comment by Josh Wilsdon
Created at 2015-11-18T06:09:34.000Z
Updated at 2018-05-07T17:50:46.568Z
Suggested name alternatives:

The latter is nice because it makes it even more clear but it is kinda long.

Important note here is that this data will serve 2 purposes:

1) it will allow us to prevent the out-of-order replacement
2) it will help with debugging to know when exactly the record that exists in VMAPI got there

Comment by Josh Wilsdon
Created at 2015-11-18T17:07:30.000Z
Updated at 2018-05-07T17:50:46.955Z
Another suggestion was: remote_timestamp.

Comment by Josh Wilsdon
Created at 2018-01-06T00:14:27.071Z
I'm not reasonably going to find time to do work on this anytime soon, so unassigning in case someone wants to pick this up.

Comment by Josh Wilsdon
Created at 2018-01-25T19:05:36.610Z
In discussion today with @trent.mick and @julien.gilli the prospect was raised that we could also consider using an ETag here. And storing that with the VM. That is another thing to investigate.