Manta issues status code 413 ("Request Entity Too Large") with an error called "MaxContentLengthExceededError" if you attempt a streaming upload with an insufficient max-content-length header. Here's a trivial example:
$ echo 1234567890 | mput -q -H 'max-content-length: 7' /dap/public/foo
mput: MaxContentLengthExceededError: request has exceeded 11 bytesHere's the Muskie log entry for a similar request:
[2019-04-10T23:12:58.060Z] INFO: muskie/HttpServer/587413 on 380920d9-ed44-4bcd-b61c-4b99f49c1329: handled: 413 (audit=true, _audit=true, operation=putpublicobject, billable_operation=PUT, bytesTransferred=11, logicalRemoteAddress=172.20.5.18, remoteAddress=127.0.0.1, remotePort=33151, reqHeaderLength=599, resHeaderLength=238, latency=156, objectId=930e230c-313a-6df6-a34e-c4226cedde25, route=putpublicobject, latencyToFirstByte=145, req.owner=bc8cd146-fecb-11e1-bd8a-bb6f54b49808)
PUT /dap/public/foo HTTP/1.1
max-content-length: 7
accept: application/json
content-type: application/octet-stream
expect: 100-continue
x-request-id: 7cc9b334-2730-4ff1-af39-56477f3a8739
x-durability-level: 2
transfer-encoding: chunked
date: Wed, 10 Apr 2019 23:12:57 GMT
authorization: Signature keyId="/dap/keys/c0:24:6d:54:c4:5e:72:15:b5:65:21:80:69:81:cb:14",algorithm="ecdsa-sha256",headers="date",signature="MEUCIQCXFchvlp7B0wsK7iBhO7XMYAx4c0L2p47qQiiXkoJPGQIgZcG+Bk5ogSAYbgFzaWfLwl/7K0E3/XxR3nEys4kYne4="
user-agent: restify/1.4.1 (x64-darwin; v8/3.14.5.9; OpenSSL/1.0.1t) node/0.10.45
accept-version: ~1.0
host: manta.staging.joyent.us
connection: keep-alive
--
HTTP/1.1 413 Request Entity Too Large
content-type: application/json
content-length: 77
content-md5: hLMqq7KoRHKipRAvj3s5Ew==
date: Wed, 10 Apr 2019 23:12:58 GMT
server: Manta
x-request-id: 7cc9b334-2730-4ff1-af39-56477f3a8739
x-response-time: 156
x-server-name: 380920d9-ed44-4bcd-b61c-4b99f49c1329
--
MaxSizeExceededError: request has exceeded 11 bytes
at CheckStream.<anonymous> (/opt/smartdc/muskie/lib/obj.js:588:18)
at CheckStream.g (events.js:180:16)
at CheckStream.emit (events.js:95:17)
at CheckStream._write (/opt/smartdc/muskie/lib/check_stream.js:125:14)
at doWrite (_stream_writable.js:226:10)
at writeOrBuffer (_stream_writable.js:216:5)
at CheckStream.Writable.write (_stream_writable.js:183:11)
at write (_stream_readable.js:602:24)
at Array.forEach (native)
at flow (_stream_readable.js:613:19)
at IncomingMessage.pipeOnReadable (_stream_readable.js:643:5)
at IncomingMessage.emit (events.js:92:17)
--
sharksContacted: [
{
"shark": "1.stor.staging.joyent.us",
"result": null,
"timeToFirstByte": 3,
"timeTotal": null,
"_startTime": 1554937978043
},
{
"shark": "2.stor.staging.joyent.us",
"result": null,
"timeToFirstByte": 3,
"timeTotal": null,
"_startTime": 1554937978044
}
]
--
req.caller: {
"login": "dap",
"uuid": "bc8cd146-fecb-11e1-bd8a-bb6f54b49808",
"groups": [],
"user": null
}
--
req.timers: {
"earlySetup": 75,
"parseDate": 18,
"parseQueryString": 30,
"handler-3": 163,
"checkIfPresigned": 8,
"enforceSSL": 6,
"ensureDependencies": 7,
"_authSetup": 9,
"preSignedUrl": 6,
"checkAuthzScheme": 13,
"parseAuthTokenHandler": 55,
"signatureHandler": 612,
"parseKeyId": 126,
"loadCaller": 294,
"verifySignature": 2666,
"parseHttpAuthToken": 16,
"loadOwner": 243,
"getActiveRoles": 47,
"gatherContext": 64,
"setup": 387,
"getMetadata": 11622,
"storageContext": 29,
"authorize": 248,
"checkIfMatch": 18,
"checkIfNoneMatch": 13,
"checkIfModified": 15,
"checkIfUnmodified": 7,
"ensureNotRoot": 3,
"parseArguments": 176,
"ensureNotDirectory": 8,
"ensureParent": 100,
"enforceDirectoryCount": 121301,
"findSharks": 562,
"startSharkStreams": 3854,
"sharkStreams": 11931
}There are several goofy things about how this works, ranging from confusing to downright buggy:
In Muskie, the error name is {{MaxContentLengthExceededError}} (and so that's what goes out to the client), but the error class is called MaxSizeExceededError. Not only is that confusing for people looking at the source to understand Muskie behavior, but also the class is what winds up in the log entry, so somebody looking in the logs for MaxContentLengthExceededError (because a user reported it) won't find it. This is confusing for operators trying to understand the system's behavior. This could also affect metrics created under MANTA-3858.
Notice the message says "request has exceeded 11 bytes". That's flat out wrong. The request body was 11 bytes, and the problem is that it exceeded the maximum of 7. This is likely to be quite confusing for end users that hit this problem. I looked briefly at this, and it looks to me like inside the check stream (where we detect this condition), we should be reporting this.maxBytes, but we're using this.bytes instead. Note that there are two different cases where we create this Error, which leads me to the next item.
There are two places in Muskie where we instantiate MaxSizeExceededError: one directly inside the check stream if we hit this condition when the caller hasn't called abandon(); and one in the event listener for the length_exceeded event, which is emitted immediately prior to that. This is pretty confusing and could use a comment – as best as I can tell, the only reason we don't invoke the callback twice every single time we hit this case is that the event listener itself abandons the check stream. It would be reasonable for a caller to think it could defer that a bit (e.g., for something else to happen), but it's not clear that would actually be safe.