Objects that are overwritten (MPUT an object at the same path as the first one) are not garbage collected. The first object should be garbage collected as it no longer exists, as it was overwritten by the second object put.
A bug report and a solution are in this PR
Carlos Neira commented on 2025-07-24T12:03:50.469-0400:
I tried the tests specified in the bug report with the following results
mput -i -f testfile1 ~~/stor/testfile0; mrm -i ~~/stor/testfile0;Then searching on the manta_fastdeletequeue table should give us a record that contains this filename (testfile0) as name in the _value column
moray=# select _key, jsonb_pretty(_value::jsonb) from manta_fastdelete_queue order by _mtime desc fetch first row only;
-[ RECORD 1 ]+-------------------------------------------------------------------
_key | 2c55f643-b302-436e-948f-20b57973bf50
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfile0",+
| "etag": "2c55f643-b302-436e-948f-20b57973bf50", +
| "name": "testfile0", +
| "type": "object", +
| "_etag": "92CA00DA", +
| "mtime": 1753360883895, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "3.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "2.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "2c55f643-b302-436e-948f-20b57973bf50", +
| "contentMD5": "w1oJ20z/DTShSkgUnZna0g==", +
| "contentType": "application/octet-stream", +
| "contentLength": 384 +
| }
This is expected as mantav2 does not have snaplinks, and every delete from an account that doesn’t have snaplinks activated should go into the fast queue (manta_fastdelete_queue). This should be the case for every Mantav2 account, according to this document mantav2-migration.md#step-3-snaplink-cleanup a migration from Manta V1 to V2 implies the removal of snaplinks.
Now let’s test the replace object path, which should remove the old object to replace it with the new one.
$ ~/ mput -i -f testfile1 ~~/stor/testfileXX; mput -i -f testfile2 ~~/stor/testfileXX
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
Now the same query should return a _value column with an object whose name is testfileXX
moray=# select _key, jsonb_pretty(_value::jsonb) from manta_fastdelete_queue order by _mtime desc fetch first row only;
-[ RECORD 1 ]+-------------------------------------------------------------------
_key | 2c55f643-b302-436e-948f-20b57973bf50
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfile0",+
| "etag": "2c55f643-b302-436e-948f-20b57973bf50", +
| "name": "testfile0", +
| "type": "object", +
| "_etag": "92CA00DA", +
| "mtime": 1753360883895, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "3.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "2.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "2c55f643-b302-436e-948f-20b57973bf50", +
| "contentMD5": "w1oJ20z/DTShSkgUnZna0g==", +
| "contentType": "application/octet-stream", +
| "contentLength": 384 +
| }However, we still see the previous object on the fast queue, which means we lost 384B of storage from testfile1 in the first copy to testfileXX.
Why is the space lost?. Because, as previously mentioned, MantaV2 does not have snaplink, so all object deletion should land on the fastqueue; if not, then the GC is not able to reclaim the space, hence we lose storage every time an object is overwritten using the same path of a previously existing object.
The bug report accurately describes the situation; for example, doing the mrm operation again results in a record added to the fastqueue as intended by MantaV2.
$ mput -i -f testfile1 ~~/stor/testfileYY; mrm -i ~~/stor/testfileYY;
/neirac/stor/testfileYY [=================================================================================================================================================================>] 100% 384B
moray=# select _key, jsonb_pretty(_value::jsonb) from manta_fastdelete_queue order by _mtime desc fetch first row only;
-[ RECORD 1 ]+--------------------------------------------------------------------
_key | ed95615f-a865-4fe4-9828-29a51eb9c5a5
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileYY",+
| "etag": "ed95615f-a865-4fe4-9828-29a51eb9c5a5", +
| "name": "testfileYY", +
| "type": "object", +
| "_etag": "CA44C4A1", +
| "mtime": 1753364297164, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "3.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "2.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "ed95615f-a865-4fe4-9828-29a51eb9c5a5", +
| "contentMD5": "w1oJ20z/DTShSkgUnZna0g==", +
| "contentType": "application/octet-stream", +
| "contentLength": 384 +
| }Let’s see how this reflects in the storage zones.
[root@headnode (coal) ~]# manta-adm show -a -o storage_id,datacenter,zonename storage
STORAGE ID DATACENTER ZONENAME
1.stor.coal.joyent.us coal b948d68a-c26c-4c5b-9c55-be01ab91d004
2.stor.coal.joyent.us coal d72c96b9-c770-407c-a8b2-f43f6d14db42
3.stor.coal.joyent.us coal ec1ea297-215b-4fc0-b89e-f019b34a8420
[root@headnode (coal) ~]# manta-login b948d68a-c26c-4c5b-9c55-be01ab91d004
(root@10.99.99.7) Password:
(root@10.99.99.7) Password:
[Connected to zone 'b948d68a-c26c-4c5b-9c55-be01ab91d004' pts/8]
Last login: Thu Jul 24 14:23:36 on pts/8
= J O Y E N T =
mantav2-storage (release-20250529-20250529T014855Z-g27a9a87)
https://github.com/tritondatacenter/manta-mako.git
triton-origin-x86_64-19.4.0@master-20200130T200825Z-gbb45b8d
[root@b948d68a (storage) ~]$ cd /manta/c116efce-086f-455e-9ae4-26d49551428d/
[root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ du -hs
76K .
[root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 60
-rw-r--r-- 1 nobody nobody 4790 Jul 18 19:20 7e5a660e-8e03-4b96-9c32-3a3a26aea1f5
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:12 4ffe36e1-87cc-4cbe-a8d4-c77b74068e65
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 14f1b546-cc66-463e-aee4-e47979005ddb
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 bc8c2198-05be-4125-8375-770ab2cce3a5
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 83a1933e-1174-4bd5-9dd5-1dd6e0bf8cdc
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 c56f7538-a076-4539-8d61-960109f31bb1
By looking at the timestamp, we can see that the 384 object is duplicated, GC should the objects on the fast queue table (GC is turned off at this point). Now turning on GC results in:
[root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 43
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 14f1b546-cc66-463e-aee4-e47979005ddb
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 bc8c2198-05be-4125-8375-770ab2cce3a5
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 83a1933e-1174-4bd5-9dd5-1dd6e0bf8cdc
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 c56f7538-a076-4539-8d61-960109f31bb1
All these are the leaked objects (384 bytes) that are wasting storage.
Let’s do a final test with GC now running to see the differences.
$ echo "hello this is a really simple test" > testfile2
$ mput -i -f testfile2 ~~/stor/testfileZZ
/neirac/stor/testfileZZ [=================================================================================================================================================================>] 100% 35B
Let’s check where the object landed
[root@headnode (coal) ~]# manta-login webapi
(root@10.99.99.7) Password:
[Connected to zone '4215a6eb-2603-4b17-904d-641c157432b4' pts/13]
= J O Y E N T =
mantav2-webapi (release-20250529-20250529T000353Z-gc9eec89)
https://github.com/tritondatacenter/manta-muskie.git
triton-origin-x86_64-21.4.0@master-20220322T012137Z-g9382491
[root@4215a6eb (webapi) ~]$ /opt/smartdc/muskie/bin/mlocate /neirac/stor/testfileZZ | json
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":39549,"component":"CueBallDNSResolver","domain":"nam
ice.coal.joyent.us","level":30,"removed":["10.99.99.11"],"msg":"removed 1 resolvers from bootstrap","time":"2025-07-24T15:14:3
Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":39549,"component":"MorayClient","domain":"electric-m
coal.joyent.us","local":"10.77.77.15:51411","remote":"10.77.77.10:2021","key":"uQDi5tBvD0N9/pY692gSGq0rmo0=.1","level":30,"msg
w connection","time":"2025-07-24T15:14:31.017Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":39549,"component":"MorayIndexClient","level":30,"att
:1,"msg":"libmanta.Moray.initAttempt: done","time":"2025-07-24T15:14:31.017Z","v":0}
{
"dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor",
"key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileZZ",
"headers": {},
"mtime": 1753369887817,
"name": "testfileZZ",
"creator": "c116efce-086f-455e-9ae4-26d49551428d",
"owner": "c116efce-086f-455e-9ae4-26d49551428d",
"roles": [],
"type": "object",
"contentLength": 35,
"contentMD5": "xsZNCjmZFNv1CVToYMhR/A==",
"contentType": "application/octet-stream",
"etag": "12d84d92-b25d-40d4-b7aa-df82679bdab7",
"objectId": "12d84d92-b25d-40d4-b7aa-df82679bdab7",
"sharks": [
{
"datacenter": "coal",
"manta_storage_id": "2.stor.coal.joyent.us"
},
{
"datacenter": "coal",
"manta_storage_id": "3.stor.coal.joyent.us"
}
],
"_node": {
"pnode": "tcp://1.moray.coal.joyent.us:2020",
"vnode": 62534,
"data": 1
}
}
[root@4215a6eb (webapi) ~]$
Then we use the manta_storage_id to zlogin into the correct zone that has the object data
[root@headnode (coal) ~]# manta-adm show -a -o storage_id,datacenter,zonename storage
STORAGE ID DATACENTER ZONENAME
1.stor.coal.joyent.us coal b948d68a-c26c-4c5b-9c55-be01ab91d004
2.stor.coal.joyent.us coal d72c96b9-c770-407c-a8b2-f43f6d14db42
3.stor.coal.joyent.us coal ec1ea297-215b-4fc0-b89e-f019b34a8420
[root@headnode (coal) ~]# manta-login d72c96b9-c770-407c-a8b2-f43f6d14db42
(root@10.99.99.7) Password:
[Connected to zone 'd72c96b9-c770-407c-a8b2-f43f6d14db42' pts/8]
= J O Y E N T =
mantav2-storage (release-20250529-20250529T014855Z-g27a9a87)
https://github.com/tritondatacenter/manta-mako.git
triton-origin-x86_64-19.4.0@master-20200130T200825Z-gbb45b8d
[root@d72c96b9 (storage) ~]$ cd /manta/c116efce-086f-455e-9ae4-26d49551428d/
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 26
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 5dbebe63-9fa3-4301-832b-866789933c73
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 2cb5e086-b626-46e6-bff1-fc9171503eb5
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:11 12d84d92-b25d-40d4-b7aa-df82679bdab7
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ cat 12d84d92-b25d-40d4-b7aa-df82679bdab7
hello this is a really simple test
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$
Now let’s remove the object GC should delete file 12d84d92-b25d-40d4-b7aa-df82679bdab7, which is the object we just pushed (notice the other objects with a size of 384 bytes, those are the leaked objects from previous tests, they won't be processed by GC, so we lost that storage).
mrm -i ~~/stor/testfileZZ
moray=# select _key, jsonb_pretty(_value::jsonb) from manta_fastdelete_queue order by _mtime desc fetch first row only;
-[ RECORD 1 ]+--------------------------------------------------------------------
_key | 12d84d92-b25d-40d4-b7aa-df82679bdab7
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileZZ",+
| "etag": "12d84d92-b25d-40d4-b7aa-df82679bdab7", +
| "name": "testfileZZ", +
| "type": "object", +
| "_etag": "EA03ABAD", +
| "mtime": 1753369887817, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "2.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "3.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "12d84d92-b25d-40d4-b7aa-df82679bdab7", +
| "contentMD5": "xsZNCjmZFNv1CVToYMhR/A==", +
| "contentType": "application/octet-stream", +
| "contentLength": 35 +
| }
moray=#
The delete operation is on the fast queue, so GC should pick this up; the object is gone now.
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 26
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 5dbebe63-9fa3-4301-832b-866789933c73
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 2cb5e086-b626-46e6-bff1-fc9171503eb5
And if we check the fastqueue, the record is also gone as expected.
moray=# select _key, jsonb_pretty(_value::jsonb) from manta_fastdelete_queue order by _mtime desc fetch first row only;
(0 rows)Carlos Neira commented on 2025-07-24T12:03:59.079-0400:
Leaking Storage
Now let’s perform an object replacement. This scenario is not triggering the addition of a record in the fastqueue hence GC has nothing to process.
mput -i -f testfile1 ~~/stor/testfileXX; mput -i -f testfile2 ~~/stor/testfileXX
moray=# select _key, jsonb_pretty(_value::jsonb) from manta_fastdelete_queue order by _mtime desc fetch first row only;
(0 rows)We have a new file in storage node 2
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 34
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 5dbebe63-9fa3-4301-832b-866789933c73
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 2cb5e086-b626-46e6-bff1-fc9171503eb5
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:39 37ab057a-b924-4f6b-baf7-9aa5b4b13d59 --> this one is new
Let’s see where the object landed
[root@4215a6eb (webapi) ~]$ /opt/smartdc/muskie/bin/mlocate /neirac/stor/testfileXX | json
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":48546,"component":"CueBallDNSResolver","domain":"nameservice.coal.joyent.us","level":30,"removed":["10.99.99.11"],"msg":"removed 1 resolvers from bootstrap","time":"2025-07-24T15:42:34.867Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":48546,"component":"MorayClient","domain":"electric-moray.coal.joyent.us","local":"10.77.77.15:34186","remote":"10.77.77.10:2021","key":"uQDi5tBvD0N9/pY692gSGq0rmo0=.1","level":30,"msg":"new connection","time":"2025-07-24T15:42:34.882Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":48546,"component":"MorayIndexClient","level":30,"attempt":1,"msg":"libmanta.Moray.initAttempt: done","time":"2025-07-24T15:42:34.883Z","v":0}
{
"dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor",
"key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileXX",
"headers": {},
"mtime": 1753371545672,
"name": "testfileXX",
"creator": "c116efce-086f-455e-9ae4-26d49551428d",
"owner": "c116efce-086f-455e-9ae4-26d49551428d",
"roles": [],
"type": "object",
"contentLength": 35,
"contentMD5": "xsZNCjmZFNv1CVToYMhR/A==",
"contentType": "application/octet-stream",
"etag": "96ca17fb-9697-4903-9d89-07589bd7f241",
"objectId": "96ca17fb-9697-4903-9d89-07589bd7f241",
"sharks": [
{
"datacenter": "coal",
"manta_storage_id": "3.stor.coal.joyent.us"
},
{
"datacenter": "coal",
"manta_storage_id": "1.stor.coal.joyent.us"
}
],
"_node": {
"pnode": "tcp://1.moray.coal.joyent.us:2020",
"vnode": 62534,
"data": 1
}
}
[root@4215a6eb (webapi) ~]$
Object should be in 3 and 1 storage zones.
[root@headnode (coal) ~]# manta-adm show -a -o storage_id,datacenter,zonename storage
STORAGE ID DATACENTER ZONENAME
1.stor.coal.joyent.us coal b948d68a-c26c-4c5b-9c55-be01ab91d004
2.stor.coal.joyent.us coal d72c96b9-c770-407c-a8b2-f43f6d14db42
3.stor.coal.joyent.us coal ec1ea297-215b-4fc0-b89e-f019b34a8420
[root@headnode (coal) ~]# zlogin b948d68a-c26c-4c5b-9c55-be01ab91d004
[Connected to zone 'b948d68a-c26c-4c5b-9c55-be01ab91d004' pts/6]
Last login: Thu Jul 24 14:29:29 on pts/8
= J O Y E N T =
mantav2-storage (release-20250529-20250529T014855Z-g27a9a87)
https://github.com/tritondatacenter/manta-mako.git
triton-origin-x86_64-19.4.0@master-20200130T200825Z-gbb45b8d
[root@b948d68a (storage) ~]$ cd /manta/c116efce-086f-455e-9ae4-26d49551428d/
[root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 43
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 14f1b546-cc66-463e-aee4-e47979005ddb
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 bc8c2198-05be-4125-8375-770ab2cce3a5
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 83a1933e-1174-4bd5-9dd5-1dd6e0bf8cdc
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 c56f7538-a076-4539-8d61-960109f31bb1
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:39 96ca17fb-9697-4903-9d89-07589bd7f241
[root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ cat 96ca17fb-9697-4903-9d89-07589bd7f241
hello this is a really simple test
[root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$
That’s the file from the last mput (testfile 2 has a string content) but in storage node 2 we still have the previous data from first mput (mput -i -f testfile1 ~~/stor/testfileXX;)
[root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 43
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 14f1b546-cc66-463e-aee4-e47979005ddb
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 bc8c2198-05be-4125-8375-770ab2cce3a5
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 83a1933e-1174-4bd5-9dd5-1dd6e0bf8cdc
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 c56f7538-a076-4539-8d61-960109f31bb1
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:39 96ca17fb-9697-4903-9d89-07589bd7f241
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 34
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 5dbebe63-9fa3-4301-832b-866789933c73
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 2cb5e086-b626-46e6-bff1-fc9171503eb5
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:39 37ab057a-b924-4f6b-baf7-9aa5b4b13d59
Notice that in storage zone 2 (d72c96b9) we have the same timestamp as in storage zone 3 (b948d68a), which confirms that we are leaking store
space.
Let’s do the same operation 10 times in a row; we should see new files being added instead of just one being overwritten.
~/ for x in {1..10}; do mput -i -f testfile1 ~~/stor/testfileXX; mput -i -f testfile2 ~~/stor/testfileXX; done ;
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 97
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 5dbebe63-9fa3-4301-832b-866789933c73
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 2cb5e086-b626-46e6-bff1-fc9171503eb5
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:39 37ab057a-b924-4f6b-baf7-9aa5b4b13d59
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 15c7bde6-b796-46d8-af28-c556bdc6a50f
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 6a619c4d-de28-492c-a854-dedf2cc66236
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 f1856f55-eeab-4afe-8e65-5b4822f3932d
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 e6204b51-1d74-4a2e-a7e1-6f23bc5a6ca9
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 ff70c630-8989-44be-96f1-8a305b148e16
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 e6a60518-9111-4e71-81ce-721706bc1d68
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 5f004fbd-dee0-4411-8144-1f624e7114d3
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 cc9b79f0-fd89-46fa-b9f0-8ebabce713ae
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 5eee3095-7a5f-43a8-874f-ef522ea6faf7
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 c8dfc3a5-b01f-48a0-a8a2-638d2a7ec283
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 7562bd0f-7fa3-4adf-86a8-9c054a49d8af
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 cb48b965-5c4e-4b21-831f-83669f3c4388
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 fe496245-8ecc-4b3a-a140-55b6778263b2
[root@ec1ea297 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 123
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 5dbebe63-9fa3-4301-832b-866789933c73
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 2cb5e086-b626-46e6-bff1-fc9171503eb5
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 14f1b546-cc66-463e-aee4-e47979005ddb
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 bc8c2198-05be-4125-8375-770ab2cce3a5
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 83a1933e-1174-4bd5-9dd5-1dd6e0bf8cdc
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 c56f7538-a076-4539-8d61-960109f31bb1
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:39 37ab057a-b924-4f6b-baf7-9aa5b4b13d59
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:39 96ca17fb-9697-4903-9d89-07589bd7f241
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 15c7bde6-b796-46d8-af28-c556bdc6a50f
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 f1856f55-eeab-4afe-8e65-5b4822f3932d
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 e6204b51-1d74-4a2e-a7e1-6f23bc5a6ca9
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 508f5bea-8112-4880-ba0b-b002e0cdb750
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 6182beaf-ab7d-427b-aefd-c057bb7dfd3e
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 3a44579a-bf28-440c-bd08-7765554df45f
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 e6a60518-9111-4e71-81ce-721706bc1d68
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 e9defe6e-a6d0-4ce6-8115-03c6da6068b7
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 ac833f1a-f005-44c5-8141-899ff516351f
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 5f004fbd-dee0-4411-8144-1f624e7114d3
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 3502324f-8ba5-448e-aa36-75b98dd4d9f8
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 fe496245-8ecc-4b3a-a140-55b6778263b2
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 4e34ef05-80d5-4711-b04c-9ed14157b175Given the current situation, a user could overwrite a sufficiently large object to deplete storage.
Carlos Neira commented on 2025-07-24T14:59:28.270-0400:
Now testing the change at MANTA-5480 and MANTA-5480
Let’s try to leak an object again
$ ~/ mput -i -f testfile1 ~~/stor/testfileXX; mput -i -f testfile2 ~~/stor/testfileXX
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
In the previous test, we had no records on the fastqueue, but after this change, we do.
moray=# select _key, jsonb_pretty(_value::jsonb) from manta_fastdelete_queue order by _mtime desc fetch first row only;
-[ RECORD 1 ]+--------------------------------------------------------------------
_key | 018a591c-33ae-49ed-bae6-cfbf75e9969f
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileXX",+
| "etag": "018a591c-33ae-49ed-bae6-cfbf75e9969f", +
| "name": "testfileXX", +
| "type": "object", +
| "_etag": "E5661622", +
| "mtime": 1753381281656, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "1.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "3.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "018a591c-33ae-49ed-bae6-cfbf75e9969f", +
| "contentMD5": "w1oJ20z/DTShSkgUnZna0g==", +
| "contentType": "application/octet-stream", +
| "contentLength": 384 +
| }
Again, let’s try a loop overwriting the same object 10 times, and check the storages nodes.
Before running this test storage nodes had the following contents (this test only has three storage nodes):
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
19
[root@ec1ea297 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
22
[root@ec1ea297 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
22 ~/ for x in {1..10}; do mput -i -f testfile1 ~~/stor/testfileXX; mput -i -f testfile2 ~~/stor/testfileXX; done ;
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
Let’s check where the last mput object landed
rac/stor/testfileXX | json /opt/smartdc/muskie/bin/mlocate /neir
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":48035,"component":"CueBallDNSResolver","domain":"nameservice.coal.joyent.us","level":30,"removed":["10.99.99.11"],"msg":"removed 1 resolvers from bootstrap","time":"2025-07-24T18:40:32.037Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":48035,"component":"MorayClient","domain":"electric-moray.coal.joyent.us","local":"10.77.77.15:49480","remote":"10.77.77.10:2021","key":"uQDi5tBvD0N9/pY692gSGq0rmo0=.1","level":30,"msg":"new connection","time":"2025-07-24T18:40:32.046Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":48035,"component":"MorayIndexClient","level":30,"attempt":1,"msg":"libmanta.Moray.initAttempt: done","time":"2025-07-24T18:40:32.047Z","v":0}
{
"dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor",
"key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileXX",
"headers": {},
"mtime": 1753382307047,
"name": "testfileXX",
"creator": "c116efce-086f-455e-9ae4-26d49551428d",
"owner": "c116efce-086f-455e-9ae4-26d49551428d",
"roles": [],
"type": "object",
"contentLength": 35,
"contentMD5": "xsZNCjmZFNv1CVToYMhR/A==",
"contentType": "application/octet-stream",
"etag": "a0bf3a1b-7b2b-4566-929d-b11ce776c4ed",
"objectId": "a0bf3a1b-7b2b-4566-929d-b11ce776c4ed",
"sharks": [
{
"datacenter": "coal",
"manta_storage_id": "1.stor.coal.joyent.us"
},
{
"datacenter": "coal",
"manta_storage_id": "3.stor.coal.joyent.us"
}
],
"_node": {
"pnode": "tcp://1.moray.coal.joyent.us:2020",
"vnode": 62534,
"data": 1
}
}Storage after test
[root@ec1ea297 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ cat a0bf3a1b-7b2b-4566-929d-b11ce776c4ed
hello this is a really simple test
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
18
[root@ec1ea297 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
22
[root@ec1ea297 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
22Let’s try with a different file
mput -i -f testfile1 ~~/stor/testfileWWW; mput -i -f testfile2 ~~/stor/testfileWWW
/neirac/stor/testfileWWW [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileWWW [=================================================================================================================================================================>] 100% 35B
moray=# select _key, jsonb_pretty(_value::jsonb) from manta_fastdelete_queue order by _mtime desc;
-[ RECORD 1 ]+---------------------------------------------------------------------
_key | 701f632c-79f1-467d-afe4-2172f32c02c0
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileWWW",+
| "etag": "701f632c-79f1-467d-afe4-2172f32c02c0", +
| "name": "testfileWWW", +
| "type": "object", +
| "_etag": "10D9E1BD", +
| "mtime": 1753382642612, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "2.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "3.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "701f632c-79f1-467d-afe4-2172f32c02c0", +
| "contentMD5": "w1oJ20z/DTShSkgUnZna0g==", +
| "contentType": "application/octet-stream", +
| "contentLength": 384 +
| }
moray=#
[root@4215a6eb (webapi) ~]$ /opt/smartdc/muskie/bin/mlocate /neirac/stor/testfileWWW | json
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":49499,"component":"CueBallDNSResolver","domain":"nameservice.coal.joyent.us","level":30,"removed":["10.99.99.11"],"msg":"removed 1 resolvers from bootstrap","time":"2025-07-24T18:45:14.696Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":49499,"component":"MorayClient","domain":"electric-moray.coal.joyent.us","local":"10.77.77.15:64176","remote":"10.77.77.10:2021","key":"uQDi5tBvD0N9/pY692gSGq0rmo0=.1","level":30,"msg":"new connection","time":"2025-07-24T18:45:14.711Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":49499,"component":"MorayIndexClient","level":30,"attempt":1,"msg":"libmanta.Moray.initAttempt: done","time":"2025-07-24T18:45:14.712Z","v":0}
{
"dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor",
"key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileWWW",
"headers": {},
"mtime": 1753382643395,
"name": "testfileWWW",
"creator": "c116efce-086f-455e-9ae4-26d49551428d",
"owner": "c116efce-086f-455e-9ae4-26d49551428d",
"roles": [],
"type": "object",
"contentLength": 35,
"contentMD5": "xsZNCjmZFNv1CVToYMhR/A==",
"contentType": "application/octet-stream",
"etag": "909013b2-cb38-4db1-8579-a00cda825135",
"objectId": "909013b2-cb38-4db1-8579-a00cda825135",
"sharks": [
{
"datacenter": "coal",
"manta_storage_id": "1.stor.coal.joyent.us"
},
{
"datacenter": "coal",
"manta_storage_id": "2.stor.coal.joyent.us"
}
],
"_node": {
"pnode": "tcp://1.moray.coal.joyent.us:2020",
"vnode": 62534,
"data": 1
}
}
[root@4215a6eb (webapi) ~]$
And the objects that currently exist on disk are the table.
root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 99
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 14f1b546-cc66-463e-aee4-e47979005ddb
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 bc8c2198-05be-4125-8375-770ab2cce3a5
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 83a1933e-1174-4bd5-9dd5-1dd6e0bf8cdc
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 c56f7538-a076-4539-8d61-960109f31bb1
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:39 96ca17fb-9697-4903-9d89-07589bd7f241
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 6a619c4d-de28-492c-a854-dedf2cc66236
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 508f5bea-8112-4880-ba0b-b002e0cdb750
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 6182beaf-ab7d-427b-aefd-c057bb7dfd3e
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 ff70c630-8989-44be-96f1-8a305b148e16
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 3a44579a-bf28-440c-bd08-7765554df45f
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 e9defe6e-a6d0-4ce6-8115-03c6da6068b7
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 ac833f1a-f005-44c5-8141-899ff516351f
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 cc9b79f0-fd89-46fa-b9f0-8ebabce713ae
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 5eee3095-7a5f-43a8-874f-ef522ea6faf7
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 c8dfc3a5-b01f-48a0-a8a2-638d2a7ec283
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 7562bd0f-7fa3-4adf-86a8-9c054a49d8af
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 3502324f-8ba5-448e-aa36-75b98dd4d9f8
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 cb48b965-5c4e-4b21-831f-83669f3c4388
-rw-r--r-- 1 nobody nobody 35 Jul 24 18:38 a0bf3a1b-7b2b-4566-929d-b11ce776c4ed
-rw-r--r-- 1 nobody nobody 35 Jul 24 18:44 909013b2-cb38-4db1-8579-a00cda825135
[root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ cat 909013b2-cb38-4db1-8579-a00cda825135
hello this is a really simple test
[root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 97
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 5dbebe63-9fa3-4301-832b-866789933c73
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 2cb5e086-b626-46e6-bff1-fc9171503eb5
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:39 37ab057a-b924-4f6b-baf7-9aa5b4b13d59
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 15c7bde6-b796-46d8-af28-c556bdc6a50f
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 6a619c4d-de28-492c-a854-dedf2cc66236
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 f1856f55-eeab-4afe-8e65-5b4822f3932d
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 e6204b51-1d74-4a2e-a7e1-6f23bc5a6ca9
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 ff70c630-8989-44be-96f1-8a305b148e16
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 e6a60518-9111-4e71-81ce-721706bc1d68
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 5f004fbd-dee0-4411-8144-1f624e7114d3
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 cc9b79f0-fd89-46fa-b9f0-8ebabce713ae
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 5eee3095-7a5f-43a8-874f-ef522ea6faf7
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 c8dfc3a5-b01f-48a0-a8a2-638d2a7ec283
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 7562bd0f-7fa3-4adf-86a8-9c054a49d8af
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 cb48b965-5c4e-4b21-831f-83669f3c4388
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 fe496245-8ecc-4b3a-a140-55b6778263b2
-rw-r--r-- 1 nobody nobody 35 Jul 24 18:44 909013b2-cb38-4db1-8579-a00cda825135
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ cat 909013b2-cb38-4db1-8579-a00cda825135
hello this is a really simple test
moray=# select _key, jsonb_pretty(_value::jsonb) from manta order by _mtime desc;
-[ RECORD 1 ]+-------------------------------------------------------------------------------------------------------------------------------------------
_key | /c116efce-086f-455e-9ae4-26d49551428d/stor/testfileWWW
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileWWW", +
| "etag": "909013b2-cb38-4db1-8579-a00cda825135", +
| "name": "testfileWWW", +
| "type": "object", +
| "mtime": 1753382643395, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
+ --More--
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "1.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "2.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "909013b2-cb38-4db1-8579-a00cda825135", +
| "contentMD5": "xsZNCjmZFNv1CVToYMhR/A==", +
| "contentType": "application/octet-stream", +
| "contentLength": 35 +
| }
-[ RECORD 2 ]+-------------------------------------------------------------------------------------------------------------------------------------------
_key | /c116efce-086f-455e-9ae4-26d49551428d/stor/testfileXX
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileXX", +
| "etag": "a0bf3a1b-7b2b-4566-929d-b11ce776c4ed", +
| "name": "testfileXX", +
| "type": "object", +
| "mtime": 1753382307047, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
| "vnode": 62534, +
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "1.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "3.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "a0bf3a1b-7b2b-4566-929d-b11ce776c4ed", +
| "contentMD5": "xsZNCjmZFNv1CVToYMhR/A==", +
| "contentType": "application/octet-stream", +
| "contentLength": 35 +
| }
-[ RECORD 3 ]+-------------------------------------------------------------------------------------------------------------------------------------------
Carlos Neira commented on 2025-07-24T15:00:10.421-0400:
Now testing the change at MANTA-5480 and MANTA-5480
Let’s try to leak an object again
$ ~/ mput -i -f testfile1 ~~/stor/testfileXX; mput -i -f testfile2 ~~/stor/testfileXX
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
In the previous test, we had no records on the fastqueue, but after this change, we do.
moray=# select _key, jsonb_pretty(_value::jsonb) from manta_fastdelete_queue order by _mtime desc fetch first row only;
-[ RECORD 1 ]+--------------------------------------------------------------------
_key | 018a591c-33ae-49ed-bae6-cfbf75e9969f
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileXX",+
| "etag": "018a591c-33ae-49ed-bae6-cfbf75e9969f", +
| "name": "testfileXX", +
| "type": "object", +
| "_etag": "E5661622", +
| "mtime": 1753381281656, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "1.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "3.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "018a591c-33ae-49ed-bae6-cfbf75e9969f", +
| "contentMD5": "w1oJ20z/DTShSkgUnZna0g==", +
| "contentType": "application/octet-stream", +
| "contentLength": 384 +
| }
Again, let’s try a loop overwriting the same object 10 times, and check the storages nodes.
Before running this test storage nodes had the following contents (this test only has three storage nodes):
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
19
[root@ec1ea297 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
22
[root@ec1ea297 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
22 ~/ for x in {1..10}; do mput -i -f testfile1 ~~/stor/testfileXX; mput -i -f testfile2 ~~/stor/testfileXX; done ;
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileXX [=================================================================================================================================================================>] 100% 35B
Let’s check where the last mput object landed
rac/stor/testfileXX | json /opt/smartdc/muskie/bin/mlocate /neir
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":48035,"component":"CueBallDNSResolver","domain":"nameservice.coal.joyent.us","level":30,"removed":["10.99.99.11"],"msg":"removed 1 resolvers from bootstrap","time":"2025-07-24T18:40:32.037Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":48035,"component":"MorayClient","domain":"electric-moray.coal.joyent.us","local":"10.77.77.15:49480","remote":"10.77.77.10:2021","key":"uQDi5tBvD0N9/pY692gSGq0rmo0=.1","level":30,"msg":"new connection","time":"2025-07-24T18:40:32.046Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":48035,"component":"MorayIndexClient","level":30,"attempt":1,"msg":"libmanta.Moray.initAttempt: done","time":"2025-07-24T18:40:32.047Z","v":0}
{
"dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor",
"key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileXX",
"headers": {},
"mtime": 1753382307047,
"name": "testfileXX",
"creator": "c116efce-086f-455e-9ae4-26d49551428d",
"owner": "c116efce-086f-455e-9ae4-26d49551428d",
"roles": [],
"type": "object",
"contentLength": 35,
"contentMD5": "xsZNCjmZFNv1CVToYMhR/A==",
"contentType": "application/octet-stream",
"etag": "a0bf3a1b-7b2b-4566-929d-b11ce776c4ed",
"objectId": "a0bf3a1b-7b2b-4566-929d-b11ce776c4ed",
"sharks": [
{
"datacenter": "coal",
"manta_storage_id": "1.stor.coal.joyent.us"
},
{
"datacenter": "coal",
"manta_storage_id": "3.stor.coal.joyent.us"
}
],
"_node": {
"pnode": "tcp://1.moray.coal.joyent.us:2020",
"vnode": 62534,
"data": 1
}
}Storage after test
[root@ec1ea297 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ cat a0bf3a1b-7b2b-4566-929d-b11ce776c4ed
hello this is a really simple test
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
18
[root@ec1ea297 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
22
[root@ec1ea297 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt | wc -l
22Let’s try with a different file
mput -i -f testfile1 ~~/stor/testfileWWW; mput -i -f testfile2 ~~/stor/testfileWWW
/neirac/stor/testfileWWW [=================================================================================================================================================================>] 100% 384B
/neirac/stor/testfileWWW [=================================================================================================================================================================>] 100% 35B
moray=# select _key, jsonb_pretty(_value::jsonb) from manta_fastdelete_queue order by _mtime desc;
-[ RECORD 1 ]+---------------------------------------------------------------------
_key | 701f632c-79f1-467d-afe4-2172f32c02c0
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileWWW",+
| "etag": "701f632c-79f1-467d-afe4-2172f32c02c0", +
| "name": "testfileWWW", +
| "type": "object", +
| "_etag": "10D9E1BD", +
| "mtime": 1753382642612, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "2.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "3.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "701f632c-79f1-467d-afe4-2172f32c02c0", +
| "contentMD5": "w1oJ20z/DTShSkgUnZna0g==", +
| "contentType": "application/octet-stream", +
| "contentLength": 384 +
| }
moray=#
[root@4215a6eb (webapi) ~]$ /opt/smartdc/muskie/bin/mlocate /neirac/stor/testfileWWW | json
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":49499,"component":"CueBallDNSResolver","domain":"nameservice.coal.joyent.us","level":30,"removed":["10.99.99.11"],"msg":"removed 1 resolvers from bootstrap","time":"2025-07-24T18:45:14.696Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":49499,"component":"MorayClient","domain":"electric-moray.coal.joyent.us","local":"10.77.77.15:64176","remote":"10.77.77.10:2021","key":"uQDi5tBvD0N9/pY692gSGq0rmo0=.1","level":30,"msg":"new connection","time":"2025-07-24T18:45:14.711Z","v":0}
{"name":"mlocate","hostname":"4215a6eb-2603-4b17-904d-641c157432b4","pid":49499,"component":"MorayIndexClient","level":30,"attempt":1,"msg":"libmanta.Moray.initAttempt: done","time":"2025-07-24T18:45:14.712Z","v":0}
{
"dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor",
"key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileWWW",
"headers": {},
"mtime": 1753382643395,
"name": "testfileWWW",
"creator": "c116efce-086f-455e-9ae4-26d49551428d",
"owner": "c116efce-086f-455e-9ae4-26d49551428d",
"roles": [],
"type": "object",
"contentLength": 35,
"contentMD5": "xsZNCjmZFNv1CVToYMhR/A==",
"contentType": "application/octet-stream",
"etag": "909013b2-cb38-4db1-8579-a00cda825135",
"objectId": "909013b2-cb38-4db1-8579-a00cda825135",
"sharks": [
{
"datacenter": "coal",
"manta_storage_id": "1.stor.coal.joyent.us"
},
{
"datacenter": "coal",
"manta_storage_id": "2.stor.coal.joyent.us"
}
],
"_node": {
"pnode": "tcp://1.moray.coal.joyent.us:2020",
"vnode": 62534,
"data": 1
}
}
[root@4215a6eb (webapi) ~]$
Carlos Neira commented on 2025-07-24T15:00:29.474-0400:
And the objects that currently exist on disk are the table.
root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 99
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 14f1b546-cc66-463e-aee4-e47979005ddb
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:54 bc8c2198-05be-4125-8375-770ab2cce3a5
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 83a1933e-1174-4bd5-9dd5-1dd6e0bf8cdc
-rw-r--r-- 1 nobody nobody 384 Jul 24 13:31 c56f7538-a076-4539-8d61-960109f31bb1
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:39 96ca17fb-9697-4903-9d89-07589bd7f241
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 6a619c4d-de28-492c-a854-dedf2cc66236
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 508f5bea-8112-4880-ba0b-b002e0cdb750
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 6182beaf-ab7d-427b-aefd-c057bb7dfd3e
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 ff70c630-8989-44be-96f1-8a305b148e16
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 3a44579a-bf28-440c-bd08-7765554df45f
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 e9defe6e-a6d0-4ce6-8115-03c6da6068b7
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 ac833f1a-f005-44c5-8141-899ff516351f
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 cc9b79f0-fd89-46fa-b9f0-8ebabce713ae
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 5eee3095-7a5f-43a8-874f-ef522ea6faf7
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 c8dfc3a5-b01f-48a0-a8a2-638d2a7ec283
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 7562bd0f-7fa3-4adf-86a8-9c054a49d8af
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 3502324f-8ba5-448e-aa36-75b98dd4d9f8
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 cb48b965-5c4e-4b21-831f-83669f3c4388
-rw-r--r-- 1 nobody nobody 35 Jul 24 18:38 a0bf3a1b-7b2b-4566-929d-b11ce776c4ed
-rw-r--r-- 1 nobody nobody 35 Jul 24 18:44 909013b2-cb38-4db1-8579-a00cda825135
[root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ cat 909013b2-cb38-4db1-8579-a00cda825135
hello this is a really simple test
[root@b948d68a (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ ls -lrt
total 97
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 5dbebe63-9fa3-4301-832b-866789933c73
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 4ca12887-2951-44ad-900a-80f8e08c2ef4
-rw-r--r-- 1 nobody nobody 384 Jul 23 23:53 2cb5e086-b626-46e6-bff1-fc9171503eb5
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:39 37ab057a-b924-4f6b-baf7-9aa5b4b13d59
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 15c7bde6-b796-46d8-af28-c556bdc6a50f
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 6a619c4d-de28-492c-a854-dedf2cc66236
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 f1856f55-eeab-4afe-8e65-5b4822f3932d
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:52 e6204b51-1d74-4a2e-a7e1-6f23bc5a6ca9
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 ff70c630-8989-44be-96f1-8a305b148e16
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:52 e6a60518-9111-4e71-81ce-721706bc1d68
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 5f004fbd-dee0-4411-8144-1f624e7114d3
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 cc9b79f0-fd89-46fa-b9f0-8ebabce713ae
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 5eee3095-7a5f-43a8-874f-ef522ea6faf7
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 c8dfc3a5-b01f-48a0-a8a2-638d2a7ec283
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 7562bd0f-7fa3-4adf-86a8-9c054a49d8af
-rw-r--r-- 1 nobody nobody 35 Jul 24 15:53 cb48b965-5c4e-4b21-831f-83669f3c4388
-rw-r--r-- 1 nobody nobody 384 Jul 24 15:53 fe496245-8ecc-4b3a-a140-55b6778263b2
-rw-r--r-- 1 nobody nobody 35 Jul 24 18:44 909013b2-cb38-4db1-8579-a00cda825135
[root@d72c96b9 (storage) /manta/c116efce-086f-455e-9ae4-26d49551428d]$ cat 909013b2-cb38-4db1-8579-a00cda825135
hello this is a really simple test
moray=# select _key, jsonb_pretty(_value::jsonb) from manta order by _mtime desc;
-[ RECORD 1 ]+-------------------------------------------------------------------------------------------------------------------------------------------
_key | /c116efce-086f-455e-9ae4-26d49551428d/stor/testfileWWW
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileWWW", +
| "etag": "909013b2-cb38-4db1-8579-a00cda825135", +
| "name": "testfileWWW", +
| "type": "object", +
| "mtime": 1753382643395, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
+ --More--
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "1.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "2.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "909013b2-cb38-4db1-8579-a00cda825135", +
| "contentMD5": "xsZNCjmZFNv1CVToYMhR/A==", +
| "contentType": "application/octet-stream", +
| "contentLength": 35 +
| }
-[ RECORD 2 ]+-------------------------------------------------------------------------------------------------------------------------------------------
_key | /c116efce-086f-455e-9ae4-26d49551428d/stor/testfileXX
jsonb_pretty | { +
| "key": "/c116efce-086f-455e-9ae4-26d49551428d/stor/testfileXX", +
| "etag": "a0bf3a1b-7b2b-4566-929d-b11ce776c4ed", +
| "name": "testfileXX", +
| "type": "object", +
| "mtime": 1753382307047, +
| "owner": "c116efce-086f-455e-9ae4-26d49551428d", +
| "roles": [ +
| ], +
| "vnode": 62534, +
| "sharks": [ +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "1.stor.coal.joyent.us" +
| }, +
| { +
| "datacenter": "coal", +
| "manta_storage_id": "3.stor.coal.joyent.us" +
| } +
| ], +
| "creator": "c116efce-086f-455e-9ae4-26d49551428d", +
| "dirname": "/c116efce-086f-455e-9ae4-26d49551428d/stor", +
| "headers": { +
| }, +
| "objectId": "a0bf3a1b-7b2b-4566-929d-b11ce776c4ed", +
| "contentMD5": "xsZNCjmZFNv1CVToYMhR/A==", +
| "contentType": "application/octet-stream", +
| "contentLength": 35 +
| }
-[ RECORD 3 ]+-------------------------------------------------------------------------------------------------------------------------------------------
Carlos Neira commented on 2025-07-24T16:41:04.911-0400:
I have talked with goekesmi and he agreed that I could push his changes, but I’ll put him as the author of this work.
Dan McDonald commented on 2025-07-24T17:14:00.632-0400:
Copying the analysis from the github issue filed by Jeff Goeke-Smith (goekesmi):
While working on my MantaV2, I noted an unexpected behavior in the garbage-collector. When replacing an object, no garbage was being collected. I have traced this to a set of functions that appear to have never been completed in this new context. Functional, but rough, patches have been created.
Test case:
dd if=/dev/urandom of=testfile1 bs=128 count=3;
mput -f testfile1 ~~/stor/testfile1; mrm ~~/stor/testfile1
If watching any part of the garbage-collection system (moray, postgres, the garbage-collection jobs) you should, and do see, an object get enqueued for deletion, and then deleted from the storage nodes.
Broken case:
dd if=/dev/urandom of=testfile1 bs=128 count=3;
dd if=/dev/urandom of=testfile2 bs=128 count=3;
mput -f testfile1 ~~/stor/testfile; mput -f testfile2 ~~/stor/testfile
If watching any part of the garbage-collection system (moray, postgres, the garbage-collection jobs) you should but do not see, an object get enqueued for deletion, and then deleted from the storage nodes. That object being the first time testfile was uploaded, now replaced by the second time testfile was uploaded.
Both of these paths are relying on the post function in the Manta's Moray, as documented at index.md#triggers . In my Manta's moray, I find:
[root@56abbdbb (moray) ~]$ getbucket manta
{
"name": "manta",
"index": {
"dirname": {
"type": "string"
},
"name": {
"type": "string"
},
"owner": {
"type": "string"
},
"objectId": {
"type": "string"
},
"type": {
"type": "string"
}
},
"pre": [],
"post": [
"function recordDeleteLog(req, cb) {\n var microtime = require('microtime');\n var crc = require('crc');\n\n var log = req.log;\n log.debug({\n id: req.id,\n bucket: req.bucket,\n key: req.key,\n value: req.value,\n headers: req.headers\n }, 'recordDeleteLog entered.');\n var prevmd = req.headers['x-muskie-prev-metadata'];\n if (!prevmd || !prevmd.objectId) {\n log.debug('not logging without previous metadata');\n cb();\n return;\n }\n var prevObjectId = prevmd.objectId;\n\n if (req.value && req.value.objectId &&\n prevObjectId === req.value.objectId) {\n log.debug('not logging since object === prev object');\n cb();\n return;\n }\n log.debug('object ' + prevObjectId + ' is candidate for deletion.');\n\n // now log to the manta_delete_log table or the manta_fastdelete_queue...\n var now = Math.round((microtime.now() / 1000));\n var _key = '/' + prevObjectId + '/' + now;\n var _value = JSON.stringify(prevmd);\n var _etag = crc.hex32(crc.crc32(_value));\n var _mtime = now;\n var sql = '';\n var values = [];\n\n // If snaplinks are disabled use the fastdelete_queue rather than delete_log\n if (req.headers['x-muskie-snaplinks-disabled']) {\n log.debug('object ' + prevObjectId + ' being added to fastdelete.');\n sql = 'INSERT INTO manta_fastdelete_queue (_key, _value, _etag, ' +\n '_mtime) VALUES ($1, $2, $3, $4)';\n values = [prevObjectId, _value, _etag, _mtime];\n } else {\n sql = 'INSERT INTO manta_delete_log (_key, _value, _etag, ' +\n '_mtime, objectId) VALUES ($1, $2, $3, $4, $5)';\n values = [_key, _value, _etag, _mtime, prevObjectId];\n }\n\n // execute\n var q = req.pg.query(sql, values);\n q.once('error', function (err) {\n log.debug(err, 'manta delete log insert: failed');\n cb(err);\n });\n q.once('end', function () {\n log.debug('manta delete log insert: done');\n cb();\n });\n}"
],
"options": {
"version": 2
},
"mtime": "2019-04-02T17:42:48.794Z"
}
which has a post function, which is much easier to read here:
Relevant here is the line
which looks for the header x-muskie-snaplinks-disabled which then chooses to use the manta_fastdelete_queue which is the expected way garbage collection works in MantaV2.
So, implicitly this is getting set during the mrm operation, but not the mput operation, why is that? What even sets that?
Again, node-libmanta.
but notably, only in the delMetadata function. delMetadata is called by
Line 1036 in c9eec89
req.moray.delMetadata(opts, function (err) {
muskie at deletePointer. This explains why mrm works.
So, why doesn't this work for the replacement operation?
Because
Line 640 in c9eec89
function saveMetadata(req, res, next) {
saveMetadata doesn't call delMetadata. saveMetadata calls moray.putMetadata
Line 656 in c9eec89
req.moray.putMetadata(opts, function (err2, obj, data) {
moray-libmanta's putMetadata has not handling for adding the header x-muskie-snaplinks-disabled at all.
So, in the case where an object is replaced with another object, the header never gets attached, the post code takes the other path, and the delete get written to the log, rather than the fastdelete table.
Add options on the calls on the replacement path similar if not identical to the delete path.
I have two branches on the two projects that have a rough patch, which copies in the logic for snaplink detection and relaying that information to moray. They are
I am running this on one of my webapi instances via in place editing, and it does not appear to be malfunctioning. I have not setup a more extensive test environment.