|Issue Type:||New Feature|
|Priority:||4 - Normal|
|Created by:||Former user|
|Reported by:||Former user|
|Assigned to:||Ashwin Nair|
After doing a fair bit of experimentation, I found out that Manta doesn't send compressed data to the requester when the Accept-Encoding header is set to deflate or gzip. Nor does it allow for uploading (POST/PUT) using deflate or gzip.
This feature is very very desirable because it allows us to send data that is highly compressible much faster. In very preliminary benchmarks with Apache Drill and the Hadoop Filesystem driver, I was seeing a 3x speed improvement on data transfer when sending pre-gzipped files.
I believe that all we would need to do is to tweak our HAProxy settings to support inbound and outbound compression.
Turning on compressed downloads is trivial. I have a PR her: https://github.com/joyent/manta-muskie/pull/82
The caveat with that, is that getting compressed downloads to work with node-manta (i.e., restify-clients) is not trivial. I tested this change by hot patching us-west-a manta (though with slightly different syntax because west-a is manta v1). Compressed downloads work with curl and browsers, but node-manta will still be uncompressed. As far as the manta client is concerned, there's no change in behavior.
Compressed uploads, are a different matter. From my initial investigation, it seems like there would need to be non-trivial changes to restify and node-manta. Changes to restify are particularly problematic because we're already on a very old version. An update to support compressed uploads would most certainly not be back ported to the version we're using, so we would either need to go through the effort of upgrading the version we're using, or forking it.
manta-muskie commit c49136302a8079f48b9093e9ab30d7ec565fe700 (branch master, by Brian Bennett)
MANTA-2926 Support Accept-Encoding / Content-Encoding gzip, deflate (#82)
Reviewed by: Ashwin Nair
Leaving this unassigned and open for any potential upload and/or client work.
The change here caused problems for existing java-manta clients as described in MANTA-5433. I think we should back this change out until we're able to update this to not break existing clients.
manta-muskie commit 8fee49b164f7141a7a2c87e7a24c547c06398b0b (branch master, by Josh Wilsdon)
MANTA-2926 Support Accept-Encoding / Content-Encoding gzip, deflate (revert: needs more work) (#86)
This reverts MANTA-2926 / c491363 pending more work, since that change caused unanticipated issues for some users of the java-manta client.
FWIW. java-manta already supports compressed uploads since Apache HttpClient 4.1 supports content compression out of the box along with many other features that were previously considered out of scope. java-manta-3.4.3 uses Apache http client
Dependent java-manta work and issue documented in the above referenced link.
After [https://mnx.atlassian.net/browse/MANTA-5433|https://mnx.atlassian.net/browse/MANTA-5433|smart-link], I’ve taken another shot at this and reworked it so that compression can now be enabled behind a SAPI flag. In the new implementation, setting
metadata.WEBAPI_USE_COMPRESSION in for the webapi service in SAPI will enable compression.
This way, operators can take advantage of compression if: