maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity

Avoid parsing large YAML file. 

YAML files tend to be very large compared to binary formats because of ASCII
format inefficiency. And the YAML reader consumes an excessively large amount
of memory when parsing a large file. It's very slow too.

For example, I observed that 6MB executable became 120MB YAML file, and the
YAML reader consumed more than 1.5GB memory to load it. The YAML reader even
caused OOM error on 32 bit, causing the entire process to fail.

This patch sets the limit on the YAML file size the linker will try to load in
the RoundTripYAML test as a safeguard.

llvm-svn: 194666
This commit is contained in:
Rui Ueyama 2013-11-14 05:57:54 +00:00
parent ec6e392975
commit 404e97c3a2
1 changed files with 10 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
lld/lib/Passes/RoundTripYAMLPass.cpp
View File

@ -15,6 +15,11 @@
#include "llvm/Support/Path.h"
// Skip YAML files larger than this to avoid OOM error. The YAML reader consumes
// excessively large amount of memory when parsing a large file.
// TODO: Fix the YAML reader to reduce memory footprint.
static const size_t MAX_YAML_FILE_SIZE = 50 * 1024 * 1024;
using namespace lld;
/// Perform the actual pass
@ -34,10 +39,11 @@ void RoundTripYAMLPass::perform(std::unique_ptr<MutableFile> &mergedFile) {
if (MemoryBuffer::getFileOrSTDIN(tmpYAMLFile.str(), buff))
return;
std::unique_ptr<MemoryBuffer> mb(buff.take());
_context.getYAMLReader().parseFile(mb, _yamlFile);
mergedFile.reset(new FileToMutable(_context, *_yamlFile[0].get()));
if (buff->getBufferSize() < MAX_YAML_FILE_SIZE) {
std::unique_ptr<MemoryBuffer> mb(buff.take());
_context.getYAMLReader().parseFile(mb, _yamlFile);
mergedFile.reset(new FileToMutable(_context, *_yamlFile[0].get()));
}
llvm::sys::fs::remove(tmpYAMLFile.str());
}