OS-6485: CTF conversion fails with large files


Issue Type:Bug
Priority:4 - Normal
Created at:2017-12-05T09:55:19.274Z
Updated at:2019-11-07T21:22:09.035Z


Created by:Jonathan Perkin
Reported by:Jonathan Perkin
Assigned to:Jonathan Perkin

Related Links


During the work to CTF convert pkgsrc, libicudata.so exposed memory scaling issues with ctfconvert. It is a 29MB shared library containing 3,449 DIEs.

The current conversion process allocates memory as follows:

With all these allocations, a 32-bit ctfconvert process runs out of available memory while processing libicudata.so.

While investigating solutions for this issue, we should bear in mind much larger objects than even libicudata.so such as libLLVM.so, which as of version 3.7 is 681MB and contains 43,149 DIEs. A fix should be able to handle files as large as that, even though we can't currently process it due to it being C++.


Comment by Jonathan Perkin
Created at 2017-12-05T10:25:33.430Z
Updated at 2017-12-05T10:30:17.570Z

With the proposed patch the conversion process is changed to be as follows:

The default batchsize of 256 is based on a few constraints:

Timing a patched ctfconvert on libicudata.so resulted in the following build times:

Time (seconds)4530231917161413Failed ENOMEM

Further analysis across pkgsrc may be helpful to determine whether we should consider changing the default.

Comment by Jonathan Perkin
Created at 2017-12-05T13:15:33.726Z

For the record, due to the fact both tickets change similar code and are somewhat interdependent, this ticket must be rebased and checked after OS-6428 has been pushed.  In particular, as this change pre-initialises cdp->cd_elf the new check in OS-6428's ctf_dwarf_free_dies becomes useless, and we might want a cleaner way to avoid trying to free dies that have already been freed.

Comment by Robert Mustacchi [X]
Created at 2017-12-05T22:00:59.230Z

Looking at this more, I think the way we're processing the symbol table is fine, but what we're doing with it isn't. We should probably try to convert it and ignore failures in those dies maybe.

Comment by Jonathan Perkin
Created at 2017-12-06T12:37:29.995Z

I guess you meant this comment for a different ticket?