MANTA-2926: Support Accept-Encoding / Content-Encoding gzip, deflate


Issue Type:New Feature
Priority:4 - Normal
Created at:2016-06-17T22:32:54.000Z
Updated at:2022-06-02T22:17:22.594Z


Created by:Former user
Reported by:Former user
Assigned to:Ashwin Nair

Related Links


After doing a fair bit of experimentation, I found out that Manta doesn't send compressed data to the requester when the Accept-Encoding header is set to deflate or gzip. Nor does it allow for uploading (POST/PUT) using deflate or gzip.

This feature is very very desirable because it allows us to send data that is highly compressible much faster. In very preliminary benchmarks with Apache Drill and the Hadoop Filesystem driver, I was seeing a 3x speed improvement on data transfer when sending pre-gzipped files.

I believe that all we would need to do is to tweak our HAProxy settings to support inbound and outbound compression.


Comment by Brian Bennett
Created at 2020-10-07T23:05:22.415Z

Turning on compressed downloads is trivial. I have a PR her:

The caveat with that, is that getting compressed downloads to work with node-manta (i.e., restify-clients) is not trivial. I tested this change by hot patching us-west-a manta (though with slightly different syntax because west-a is manta v1). Compressed downloads work with curl and browsers, but node-manta will still be uncompressed. As far as the manta client is concerned, there's no change in behavior.

Comment by Brian Bennett
Created at 2020-10-07T23:08:50.853Z

Compressed uploads, are a different matter. From my initial investigation, it seems like there would need to be non-trivial changes to restify and node-manta. Changes to restify are particularly problematic because we're already on a very old version. An update to support compressed uploads would most certainly not be back ported to the version we're using, so we would either need to go through the effort of upgrading the version we're using, or forking it.

Comment by Jira Bot
Created at 2020-10-08T00:01:02.766Z

manta-muskie commit c49136302a8079f48b9093e9ab30d7ec565fe700 (branch master, by Brian Bennett)

MANTA-2926 Support Accept-Encoding / Content-Encoding gzip, deflate (#82)

Reviewed by: Ashwin Nair

Comment by Brian Bennett
Created at 2020-10-08T00:12:29.411Z

Leaving this unassigned and open for any potential upload and/or client work.

Comment by Josh Wilsdon
Created at 2020-12-03T01:50:33.552Z

The change here caused problems for existing java-manta clients as described in MANTA-5433. I think we should back this change out until we're able to update this to not break existing clients.

Comment by Jira Bot
Created at 2020-12-03T06:00:58.832Z

manta-muskie commit 8fee49b164f7141a7a2c87e7a24c547c06398b0b (branch master, by Josh Wilsdon)

MANTA-2926 Support Accept-Encoding / Content-Encoding gzip, deflate (revert: needs more work) (#86)

This reverts MANTA-2926 / c491363 pending more work, since that change caused unanticipated issues for some users of the java-manta client.

Comment by Ashwin Nair
Created at 2020-12-03T23:19:09.179Z
Updated at 2020-12-03T23:19:32.588Z

FWIW. java-manta already supports compressed uploads since Apache HttpClient 4.1 supports content compression out of the box along with many other features that were previously considered out of scope. java-manta-3.4.3 uses Apache http client 4.5.7

Comment by Ashwin Nair
Created at 2020-12-09T18:15:10.165Z

Dependent java-manta work and issue documented in the above referenced link.

Comment by Brian Bennett
Created at 2022-06-02T21:54:43.911Z

After [||smart-link], I’ve taken another shot at this and reworked it so that compression can now be enabled behind a SAPI flag. In the new implementation, setting metadata.WEBAPI_USE_COMPRESSION in for the webapi service in SAPI will enable compression.

This way, operators can take advantage of compression if: