MANTA-3246: manta-init writes new key to local UFDS instead of UFDS primary


Under OPS-2822, we discovered that the first "manta-init" appears to write the newly-created SSH key into the local UFDS. That's wrong -- it needs to go to the UFDS primary so that it can be replicated to all other datacenters.


Comment by David Pacheco
Created at 2017-05-04T17:56:28.000Z
Updated at 2017-07-31T20:37:58.000Z
Symptoms: The way this manifests is that in a multi-DC deployment, if you ran the first "manta-init" in a DC other than the UFDS primary for that cloud, the key is only present in the DC that ran the first "manta-init". In the others, the key is not in UFDS. Only the mahis in that same first datacenter will end up containing the key. As a result, most of poseidon's requests to itself fail with a 403 ("Forbidden"). Among other issues, log uploading fails, and zones start running low of disk space.

Workaround: If you run into this, the procedure to repair the problem is to remove the key from the UFDS datacenter to which it was added erroneously and then add it by hand in the datacenter hosting the UFDS primary.

Suggested fix: The manta-deployment zone (the "manta" zone within the "sdc" application) likely needs new pieces of SAPI metadata for the remote UFDS (URL and credentials). (This may already be in the "sdc" application under ufds_remote_\{ip,ldap_root_pw\}.) This should make its way into the sdc-manta config file, and manta-init should use this when creating the key.

Also, as described in MANTA-2116, we could check the "sdc" application's SAPI metadata to confirm that we're talking to the correct UFDS.

Comment by Trent Mick
Created at 2017-05-04T18:11:51.000Z
FWIW, the following parts of manta-init will also write to the local UFDS. They should be updated to also use the UFDS master if there is one (it wouldn't have to be on this same ticket, tho):

        function getOrCreatePoseidon(cb) {
		getOrCreateUser(user, cb);

	function updatePoseidonEmail(user, cb) {
                updateEmail(user, ARGV.e, function (err)  {

	function addPoseidonToOperators(cb) {
                ufds.modify(operatorsdn, entry, function (err, res) {