llvm-project/lld/ELF/InputFiles.cpp

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

1720 lines
62 KiB
C++
Raw Normal View History

//===- InputFiles.cpp -----------------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
#include "InputFiles.h"
[ELF] Implement Dependent Libraries Feature This patch implements a limited form of autolinking primarily designed to allow either the --dependent-library compiler option, or "comment lib" pragmas ( https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) in C/C++ e.g. #pragma comment(lib, "foo"), to cause an ELF linker to automatically add the specified library to the link when processing the input file generated by the compiler. Currently this extension is unique to LLVM and LLD. However, care has been taken to design this feature so that it could be supported by other ELF linkers. The design goals were to provide: - A simple linking model for developers to reason about. - The ability to to override autolinking from the linker command line. - Source code compatibility, where possible, with "comment lib" pragmas in other environments (MSVC in particular). Dependent library support is implemented differently for ELF platforms than on the other platforms. Primarily this difference is that on ELF we pass the dependent library specifiers directly to the linker without manipulating them. This is in contrast to other platforms where they are mapped to a specific linker option by the compiler. This difference is a result of the greater variety of ELF linkers and the fact that ELF linkers tend to handle libraries in a more complicated fashion than on other platforms. This forces us to defer handling the specifiers to the linker. In order to achieve a level of source code compatibility with other platforms we have restricted this feature to work with libraries that meet the following "reasonable" requirements: 1. There are no competing defined symbols in a given set of libraries, or if they exist, the program owner doesn't care which is linked to their program. 2. There may be circular dependencies between libraries. The binary representation is a mergeable string section (SHF_MERGE, SHF_STRINGS), called .deplibs, with custom type SHT_LLVM_DEPENDENT_LIBRARIES (0x6fff4c04). The compiler forms this section by concatenating the arguments of the "comment lib" pragmas and --dependent-library options in the order they are encountered. Partial (-r, -Ur) links are handled by concatenating .deplibs sections with the normal mergeable string section rules. As an example, #pragma comment(lib, "foo") would result in: .section ".deplibs","MS",@llvm_dependent_libraries,1 .asciz "foo" For LTO, equivalent information to the contents of a the .deplibs section can be retrieved by the LLD for bitcode input files. LLD processes the dependent library specifiers in the following way: 1. Dependent libraries which are found from the specifiers in .deplibs sections of relocatable object files are added when the linker decides to include that file (which could itself be in a library) in the link. Dependent libraries behave as if they were appended to the command line after all other options. As a consequence the set of dependent libraries are searched last to resolve symbols. 2. It is an error if a file cannot be found for a given specifier. 3. Any command line options in effect at the end of the command line parsing apply to the dependent libraries, e.g. --whole-archive. 4. The linker tries to add a library or relocatable object file from each of the strings in a .deplibs section by; first, handling the string as if it was specified on the command line; second, by looking for the string in each of the library search paths in turn; third, by looking for a lib<string>.a or lib<string>.so (depending on the current mode of the linker) in each of the library search paths. 5. A new command line option --no-dependent-libraries tells LLD to ignore the dependent libraries. Rationale for the above points: 1. Adding the dependent libraries last makes the process simple to understand from a developers perspective. All linkers are able to implement this scheme. 2. Error-ing for libraries that are not found seems like better behavior than failing the link during symbol resolution. 3. It seems useful for the user to be able to apply command line options which will affect all of the dependent libraries. There is a potential problem of surprise for developers, who might not realize that these options would apply to these "invisible" input files; however, despite the potential for surprise, this is easy for developers to reason about and gives developers the control that they may require. 4. This algorithm takes into account all of the different ways that ELF linkers find input files. The different search methods are tried by the linker in most obvious to least obvious order. 5. I considered adding finer grained control over which dependent libraries were ignored (e.g. MSVC has /nodefaultlib:<library>); however, I concluded that this is not necessary: if finer control is required developers can fall back to using the command line directly. RFC thread: http://lists.llvm.org/pipermail/llvm-dev/2019-March/131004.html. Differential Revision: https://reviews.llvm.org/D60274 llvm-svn: 360984
2019-05-17 11:44:15 +08:00
#include "Driver.h"
2016-02-11 23:24:48 +08:00
#include "InputSection.h"
#include "LinkerScript.h"
ELF: New symbol table design. This patch implements a new design for the symbol table that stores SymbolBodies within a memory region of the Symbol object. Symbols are mutated by constructing SymbolBodies in place over existing SymbolBodies, rather than by mutating pointers. As mentioned in the initial proposal [1], this memory layout helps reduce the cache miss rate by improving memory locality. Performance numbers: old(s) new(s) Without debug info: chrome 7.178 6.432 (-11.5%) LLVMgold.so 0.505 0.502 (-0.5%) clang 0.954 0.827 (-15.4%) llvm-as 0.052 0.045 (-15.5%) With debug info: scylla 5.695 5.613 (-1.5%) clang 14.396 14.143 (-1.8%) Performance counter results show that the fewer required indirections is indeed the cause of the improved performance. For example, when linking chrome, stalled cycles decreases from 14,556,444,002 to 12,959,238,310, and instructions per cycle increases from 0.78 to 0.83. We are also executing many fewer instructions (15,516,401,933 down to 15,002,434,310), probably because we spend less time allocating SymbolBodies. The new mechanism by which symbols are added to the symbol table is by calling add* functions on the SymbolTable. In this patch, I handle local symbols by storing them inside "unparented" SymbolBodies. This is suboptimal, but if we do want to try to avoid allocating these SymbolBodies, we can probably do that separately. I also removed a few members from the SymbolBody class that were only being used to pass information from the input file to the symbol table. This patch implements the new design for the ELF linker only. I intend to prepare a similar patch for the COFF linker. [1] http://lists.llvm.org/pipermail/llvm-dev/2016-April/098832.html Differential Revision: http://reviews.llvm.org/D19752 llvm-svn: 268178
2016-05-01 12:55:03 +08:00
#include "SymbolTable.h"
#include "Symbols.h"
#include "SyntheticSections.h"
#include "lld/Common/DWARF.h"
#include "lld/Common/ErrorHandler.h"
#include "lld/Common/Memory.h"
#include "llvm/ADT/STLExtras.h"
#include "llvm/CodeGen/Analysis.h"
#include "llvm/IR/LLVMContext.h"
2016-03-02 23:43:50 +08:00
#include "llvm/IR/Module.h"
#include "llvm/LTO/LTO.h"
#include "llvm/MC/StringTableBuilder.h"
#include "llvm/Object/ELFObjectFile.h"
#include "llvm/Support/ARMAttributeParser.h"
#include "llvm/Support/ARMBuildAttributes.h"
#include "llvm/Support/Endian.h"
#include "llvm/Support/Path.h"
#include "llvm/Support/TarWriter.h"
#include "llvm/Support/raw_ostream.h"
using namespace llvm;
using namespace llvm::ELF;
using namespace llvm::object;
using namespace llvm::sys;
using namespace llvm::sys::fs;
using namespace llvm::support::endian;
using namespace lld;
using namespace lld::elf;
bool InputFile::isInGroup;
uint32_t InputFile::nextGroupId;
std::vector<ArchiveFile *> elf::archiveFiles;
std::vector<BinaryFile *> elf::binaryFiles;
std::vector<BitcodeFile *> elf::bitcodeFiles;
std::vector<LazyObjFile *> elf::lazyObjFiles;
std::vector<InputFile *> elf::objectFiles;
std::vector<SharedFile *> elf::sharedFiles;
std::unique_ptr<TarWriter> elf::tar;
// Returns "<internal>", "foo.a(bar.o)" or "baz.o".
std::string lld::toString(const InputFile *f) {
if (!f)
return "<internal>";
if (f->toStringCache.empty()) {
if (f->archiveName.empty())
f->toStringCache = std::string(f->getName());
else
f->toStringCache = (f->archiveName + "(" + f->getName() + ")").str();
}
return f->toStringCache;
}
static ELFKind getELFKind(MemoryBufferRef mb, StringRef archiveName) {
unsigned char size;
unsigned char endian;
std::tie(size, endian) = getElfArchType(mb.getBuffer());
[Coding style change] Rename variables so that they start with a lowercase letter This patch is mechanically generated by clang-llvm-rename tool that I wrote using Clang Refactoring Engine just for creating this patch. You can see the source code of the tool at https://reviews.llvm.org/D64123. There's no manual post-processing; you can generate the same patch by re-running the tool against lld's code base. Here is the main discussion thread to change the LLVM coding style: https://lists.llvm.org/pipermail/llvm-dev/2019-February/130083.html In the discussion thread, I proposed we use lld as a testbed for variable naming scheme change, and this patch does that. I chose to rename variables so that they are in camelCase, just because that is a minimal change to make variables to start with a lowercase letter. Note to downstream patch maintainers: if you are maintaining a downstream lld repo, just rebasing ahead of this commit would cause massive merge conflicts because this patch essentially changes every line in the lld subdirectory. But there's a remedy. clang-llvm-rename tool is a batch tool, so you can rename variables in your downstream repo with the tool. Given that, here is how to rebase your repo to a commit after the mass renaming: 1. rebase to the commit just before the mass variable renaming, 2. apply the tool to your downstream repo to mass-rename variables locally, and 3. rebase again to the head. Most changes made by the tool should be identical for a downstream repo and for the head, so at the step 3, almost all changes should be merged and disappear. I'd expect that there would be some lines that you need to merge by hand, but that shouldn't be too many. Differential Revision: https://reviews.llvm.org/D64121 llvm-svn: 365595
2019-07-10 13:00:37 +08:00
auto report = [&](StringRef msg) {
StringRef filename = mb.getBufferIdentifier();
if (archiveName.empty())
fatal(filename + ": " + msg);
else
fatal(archiveName + "(" + filename + "): " + msg);
};
if (!mb.getBuffer().startswith(ElfMagic))
report("not an ELF file");
if (endian != ELFDATA2LSB && endian != ELFDATA2MSB)
report("corrupted ELF file: invalid data encoding");
if (size != ELFCLASS32 && size != ELFCLASS64)
report("corrupted ELF file: invalid file class");
[Coding style change] Rename variables so that they start with a lowercase letter This patch is mechanically generated by clang-llvm-rename tool that I wrote using Clang Refactoring Engine just for creating this patch. You can see the source code of the tool at https://reviews.llvm.org/D64123. There's no manual post-processing; you can generate the same patch by re-running the tool against lld's code base. Here is the main discussion thread to change the LLVM coding style: https://lists.llvm.org/pipermail/llvm-dev/2019-February/130083.html In the discussion thread, I proposed we use lld as a testbed for variable naming scheme change, and this patch does that. I chose to rename variables so that they are in camelCase, just because that is a minimal change to make variables to start with a lowercase letter. Note to downstream patch maintainers: if you are maintaining a downstream lld repo, just rebasing ahead of this commit would cause massive merge conflicts because this patch essentially changes every line in the lld subdirectory. But there's a remedy. clang-llvm-rename tool is a batch tool, so you can rename variables in your downstream repo with the tool. Given that, here is how to rebase your repo to a commit after the mass renaming: 1. rebase to the commit just before the mass variable renaming, 2. apply the tool to your downstream repo to mass-rename variables locally, and 3. rebase again to the head. Most changes made by the tool should be identical for a downstream repo and for the head, so at the step 3, almost all changes should be merged and disappear. I'd expect that there would be some lines that you need to merge by hand, but that shouldn't be too many. Differential Revision: https://reviews.llvm.org/D64121 llvm-svn: 365595
2019-07-10 13:00:37 +08:00
size_t bufSize = mb.getBuffer().size();
if ((size == ELFCLASS32 && bufSize < sizeof(Elf32_Ehdr)) ||
(size == ELFCLASS64 && bufSize < sizeof(Elf64_Ehdr)))
report("corrupted ELF file: file is too short");
[Coding style change] Rename variables so that they start with a lowercase letter This patch is mechanically generated by clang-llvm-rename tool that I wrote using Clang Refactoring Engine just for creating this patch. You can see the source code of the tool at https://reviews.llvm.org/D64123. There's no manual post-processing; you can generate the same patch by re-running the tool against lld's code base. Here is the main discussion thread to change the LLVM coding style: https://lists.llvm.org/pipermail/llvm-dev/2019-February/130083.html In the discussion thread, I proposed we use lld as a testbed for variable naming scheme change, and this patch does that. I chose to rename variables so that they are in camelCase, just because that is a minimal change to make variables to start with a lowercase letter. Note to downstream patch maintainers: if you are maintaining a downstream lld repo, just rebasing ahead of this commit would cause massive merge conflicts because this patch essentially changes every line in the lld subdirectory. But there's a remedy. clang-llvm-rename tool is a batch tool, so you can rename variables in your downstream repo with the tool. Given that, here is how to rebase your repo to a commit after the mass renaming: 1. rebase to the commit just before the mass variable renaming, 2. apply the tool to your downstream repo to mass-rename variables locally, and 3. rebase again to the head. Most changes made by the tool should be identical for a downstream repo and for the head, so at the step 3, almost all changes should be merged and disappear. I'd expect that there would be some lines that you need to merge by hand, but that shouldn't be too many. Differential Revision: https://reviews.llvm.org/D64121 llvm-svn: 365595
2019-07-10 13:00:37 +08:00
if (size == ELFCLASS32)
return (endian == ELFDATA2LSB) ? ELF32LEKind : ELF32BEKind;
return (endian == ELFDATA2LSB) ? ELF64LEKind : ELF64BEKind;
}
Add --warn-backrefs to maintain compatibility with other linkers I'm proposing a new command line flag, --warn-backrefs in this patch. The flag and the feature proposed below don't exist in GNU linkers nor the current lld. --warn-backrefs is an option to detect reverse or cyclic dependencies between static archives, and it can be used to keep your program compatible with GNU linkers after you switch to lld. I'll explain the feature and why you may find it useful below. lld's symbol resolution semantics is more relaxed than traditional Unix linkers. Therefore, ld.lld foo.a bar.o succeeds even if bar.o contains an undefined symbol that have to be resolved by some object file in foo.a. Traditional Unix linkers don't allow this kind of backward reference, as they visit each file only once from left to right in the command line while resolving all undefined symbol at the moment of visiting. In the above case, since there's no undefined symbol when a linker visits foo.a, no files are pulled out from foo.a, and because the linker forgets about foo.a after visiting, it can't resolve undefined symbols that could have been resolved otherwise. That lld accepts more relaxed form means (besides it makes more sense) that you can accidentally write a command line or a build file that works only with lld, even if you have a plan to distribute it to wider users who may be using GNU linkers. With --check-library-dependency, you can detect a library order that doesn't work with other Unix linkers. The option is also useful to detect cyclic dependencies between static archives. Again, lld accepts ld.lld foo.a bar.a even if foo.a and bar.a depend on each other. With --warn-backrefs it is handled as an error. Here is how the option works. We assign a group ID to each file. A file with a smaller group ID can pull out object files from an archive file with an equal or greater group ID. Otherwise, it is a reverse dependency and an error. A file outside --{start,end}-group gets a fresh ID when instantiated. All files within the same --{start,end}-group get the same group ID. E.g. ld.lld A B --start-group C D --end-group E A and B form group 0, C, D and their member object files form group 1, and E forms group 2. I think that you can see how this group assignment rule simulates the traditional linker's semantics. Differential Revision: https://reviews.llvm.org/D45195 llvm-svn: 329636
2018-04-10 07:05:48 +08:00
InputFile::InputFile(Kind k, MemoryBufferRef m)
: mb(m), groupId(nextGroupId), fileKind(k) {
// All files within the same --{start,end}-group get the same group ID.
// Otherwise, a new file will get a new group ID.
if (!isInGroup)
++nextGroupId;
}
Optional<MemoryBufferRef> elf::readFile(StringRef path) {
// The --chroot option changes our virtual root directory.
// This is useful when you are dealing with files created by --reproduce.
if (!config->chroot.empty() && path.startswith("/"))
path = saver.save(config->chroot + path);
log(path);
auto mbOrErr = MemoryBuffer::getFile(path, -1, false);
if (auto ec = mbOrErr.getError()) {
error("cannot open " + path + ": " + ec.message());
return None;
}
std::unique_ptr<MemoryBuffer> &mb = *mbOrErr;
MemoryBufferRef mbref = mb->getMemBufferRef();
make<std::unique_ptr<MemoryBuffer>>(std::move(mb)); // take MB ownership
if (tar)
tar->append(relativeToRoot(path), mbref.getBuffer());
return mbref;
}
// All input object files must be for the same architecture
// (e.g. it does not make sense to link x86 object files with
// MIPS object files.) This function checks for that error.
static bool isCompatible(InputFile *file) {
if (!file->isElf() && !isa<BitcodeFile>(file))
return true;
if (file->ekind == config->ekind && file->emachine == config->emachine) {
if (config->emachine != EM_MIPS)
return true;
if (isMipsN32Abi(file) == config->mipsN32Abi)
return true;
}
StringRef target =
!config->bfdname.empty() ? config->bfdname : config->emulation;
if (!target.empty()) {
error(toString(file) + " is incompatible with " + target);
2019-07-29 13:24:51 +08:00
return false;
}
2019-07-29 13:24:51 +08:00
InputFile *existing;
if (!objectFiles.empty())
existing = objectFiles[0];
else if (!sharedFiles.empty())
existing = sharedFiles[0];
else if (!bitcodeFiles.empty())
2019-07-29 13:24:51 +08:00
existing = bitcodeFiles[0];
else
llvm_unreachable("Must have -m, OUTPUT_FORMAT or existing input file to "
"determine target emulation");
2019-07-29 13:24:51 +08:00
error(toString(file) + " is incompatible with " + toString(existing));
return false;
}
template <class ELFT> static void doParseFile(InputFile *file) {
if (!isCompatible(file))
return;
// Binary file
if (auto *f = dyn_cast<BinaryFile>(file)) {
binaryFiles.push_back(f);
f->parse();
return;
}
// .a file
if (auto *f = dyn_cast<ArchiveFile>(file)) {
archiveFiles.push_back(f);
f->parse();
return;
}
// Lazy object file
if (auto *f = dyn_cast<LazyObjFile>(file)) {
lazyObjFiles.push_back(f);
f->parse<ELFT>();
return;
}
if (config->trace)
message(toString(file));
// .so file
if (auto *f = dyn_cast<SharedFile>(file)) {
f->parse<ELFT>();
return;
}
// LLVM bitcode file
if (auto *f = dyn_cast<BitcodeFile>(file)) {
bitcodeFiles.push_back(f);
f->parse<ELFT>();
return;
}
// Regular object file
objectFiles.push_back(file);
cast<ObjFile<ELFT>>(file)->parse();
}
// Add symbols in File to the symbol table.
void elf::parseFile(InputFile *file) {
switch (config->ekind) {
case ELF32LEKind:
doParseFile<ELF32LE>(file);
return;
case ELF32BEKind:
doParseFile<ELF32BE>(file);
return;
case ELF64LEKind:
doParseFile<ELF64LE>(file);
return;
case ELF64BEKind:
doParseFile<ELF64BE>(file);
return;
default:
llvm_unreachable("unknown ELFT");
}
}
// Concatenates arguments to construct a string representing an error location.
static std::string createFileLineMsg(StringRef path, unsigned line) {
std::string filename = std::string(path::filename(path));
std::string lineno = ":" + std::to_string(line);
if (filename == path)
return filename + lineno;
return filename + lineno + " (" + path.str() + lineno + ")";
}
template <class ELFT>
static std::string getSrcMsgAux(ObjFile<ELFT> &file, const Symbol &sym,
InputSectionBase &sec, uint64_t offset) {
// In DWARF, functions and variables are stored to different places.
// First, lookup a function for a given offset.
if (Optional<DILineInfo> info = file.getDILineInfo(&sec, offset))
return createFileLineMsg(info->FileName, info->Line);
// If it failed, lookup again as a variable.
if (Optional<std::pair<std::string, unsigned>> fileLine =
file.getVariableLoc(sym.getName()))
return createFileLineMsg(fileLine->first, fileLine->second);
// File.sourceFile contains STT_FILE symbol, and that is a last resort.
return std::string(file.sourceFile);
}
std::string InputFile::getSrcMsg(const Symbol &sym, InputSectionBase &sec,
uint64_t offset) {
if (kind() != ObjKind)
return "";
switch (config->ekind) {
default:
llvm_unreachable("Invalid kind");
case ELF32LEKind:
return getSrcMsgAux(cast<ObjFile<ELF32LE>>(*this), sym, sec, offset);
case ELF32BEKind:
return getSrcMsgAux(cast<ObjFile<ELF32BE>>(*this), sym, sec, offset);
case ELF64LEKind:
return getSrcMsgAux(cast<ObjFile<ELF64LE>>(*this), sym, sec, offset);
case ELF64BEKind:
return getSrcMsgAux(cast<ObjFile<ELF64BE>>(*this), sym, sec, offset);
}
}
template <class ELFT> DWARFCache *ObjFile<ELFT>::getDwarf() {
llvm::call_once(initDwarf, [this]() {
dwarf = std::make_unique<DWARFCache>(std::make_unique<DWARFContext>(
std::make_unique<LLDDwarfObj<ELFT>>(this), "",
[&](Error err) { warn(getName() + ": " + toString(std::move(err))); },
[&](Error warning) {
warn(getName() + ": " + toString(std::move(warning)));
}));
});
return dwarf.get();
}
// Returns the pair of file name and line number describing location of data
// object (variable, array, etc) definition.
template <class ELFT>
Optional<std::pair<std::string, unsigned>>
ObjFile<ELFT>::getVariableLoc(StringRef name) {
return getDwarf()->getVariableLoc(name);
}
// Returns source line information for a given offset
// using DWARF debug info.
template <class ELFT>
Optional<DILineInfo> ObjFile<ELFT>::getDILineInfo(InputSectionBase *s,
uint64_t offset) {
// Detect SectionIndex for specified section.
uint64_t sectionIndex = object::SectionedAddress::UndefSection;
ArrayRef<InputSectionBase *> sections = s->file->getSections();
for (uint64_t curIndex = 0; curIndex < sections.size(); ++curIndex) {
if (s == sections[curIndex]) {
sectionIndex = curIndex;
break;
}
}
return getDwarf()->getDILineInfo(offset, sectionIndex);
}
ELFFileBase::ELFFileBase(Kind k, MemoryBufferRef mb) : InputFile(k, mb) {
ekind = getELFKind(mb, "");
switch (ekind) {
case ELF32LEKind:
init<ELF32LE>();
break;
case ELF32BEKind:
init<ELF32BE>();
break;
case ELF64LEKind:
init<ELF64LE>();
break;
case ELF64BEKind:
init<ELF64BE>();
break;
default:
llvm_unreachable("getELFKind");
}
}
template <typename Elf_Shdr>
static const Elf_Shdr *findSection(ArrayRef<Elf_Shdr> sections, uint32_t type) {
for (const Elf_Shdr &sec : sections)
if (sec.sh_type == type)
return &sec;
return nullptr;
}
template <class ELFT> void ELFFileBase::init() {
using Elf_Shdr = typename ELFT::Shdr;
using Elf_Sym = typename ELFT::Sym;
// Initialize trivial attributes.
const ELFFile<ELFT> &obj = getObj<ELFT>();
emachine = obj.getHeader()->e_machine;
osabi = obj.getHeader()->e_ident[llvm::ELF::EI_OSABI];
abiVersion = obj.getHeader()->e_ident[llvm::ELF::EI_ABIVERSION];
ArrayRef<Elf_Shdr> sections = CHECK(obj.sections(), this);
// Find a symbol table.
bool isDSO =
(identify_magic(mb.getBuffer()) == file_magic::elf_shared_object);
const Elf_Shdr *symtabSec =
findSection(sections, isDSO ? SHT_DYNSYM : SHT_SYMTAB);
if (!symtabSec)
return;
// Initialize members corresponding to a symbol table.
firstGlobal = symtabSec->sh_info;
ArrayRef<Elf_Sym> eSyms = CHECK(obj.symbols(symtabSec), this);
if (firstGlobal == 0 || firstGlobal > eSyms.size())
fatal(toString(this) + ": invalid sh_info in symbol table");
elfSyms = reinterpret_cast<const void *>(eSyms.data());
numELFSyms = eSyms.size();
stringTable = CHECK(obj.getStringTableForSymtab(*symtabSec, sections), this);
}
template <class ELFT>
uint32_t ObjFile<ELFT>::getSectionIndex(const Elf_Sym &sym) const {
return CHECK(
this->getObj().getSectionIndex(&sym, getELFSyms<ELFT>(), shndxTable),
this);
}
template <class ELFT> ArrayRef<Symbol *> ObjFile<ELFT>::getLocalSymbols() {
if (this->symbols.empty())
return {};
return makeArrayRef(this->symbols).slice(1, this->firstGlobal - 1);
}
template <class ELFT> ArrayRef<Symbol *> ObjFile<ELFT>::getGlobalSymbols() {
return makeArrayRef(this->symbols).slice(this->firstGlobal);
}
template <class ELFT> void ObjFile<ELFT>::parse(bool ignoreComdats) {
// Read a section table. justSymbols is usually false.
if (this->justSymbols)
initializeJustSymbols();
else
initializeSections(ignoreComdats);
// Read a symbol table.
initializeSymbols();
}
// Sections with SHT_GROUP and comdat bits define comdat section groups.
// They are identified and deduplicated by group name. This function
// returns a group name.
template <class ELFT>
StringRef ObjFile<ELFT>::getShtGroupSignature(ArrayRef<Elf_Shdr> sections,
const Elf_Shdr &sec) {
typename ELFT::SymRange symbols = this->getELFSyms<ELFT>();
if (sec.sh_info >= symbols.size())
fatal(toString(this) + ": invalid symbol index");
const typename ELFT::Sym &sym = symbols[sec.sh_info];
StringRef signature = CHECK(sym.getName(this->stringTable), this);
// As a special case, if a symbol is a section symbol and has no name,
// we use a section name as a signature.
//
// Such SHT_GROUP sections are invalid from the perspective of the ELF
// standard, but GNU gold 1.14 (the newest version as of July 2017) or
// older produce such sections as outputs for the -r option, so we need
// a bug-compatibility.
if (signature.empty() && sym.getType() == STT_SECTION)
return getSectionName(sec);
return signature;
}
template <class ELFT>
bool ObjFile<ELFT>::shouldMerge(const Elf_Shdr &sec, StringRef name) {
if (!(sec.sh_flags & SHF_MERGE))
return false;
// On a regular link we don't merge sections if -O0 (default is -O1). This
// sometimes makes the linker significantly faster, although the output will
// be bigger.
//
// Doing the same for -r would create a problem as it would combine sections
// with different sh_entsize. One option would be to just copy every SHF_MERGE
// section as is to the output. While this would produce a valid ELF file with
// usable SHF_MERGE sections, tools like (llvm-)?dwarfdump get confused when
// they see two .debug_str. We could have separate logic for combining
// SHF_MERGE sections based both on their name and sh_entsize, but that seems
// to be more trouble than it is worth. Instead, we just use the regular (-O1)
// logic for -r.
if (config->optimize == 0 && !config->relocatable)
return false;
// A mergeable section with size 0 is useless because they don't have
// any data to merge. A mergeable string section with size 0 can be
// argued as invalid because it doesn't end with a null character.
// We'll avoid a mess by handling them as if they were non-mergeable.
if (sec.sh_size == 0)
return false;
// Check for sh_entsize. The ELF spec is not clear about the zero
// sh_entsize. It says that "the member [sh_entsize] contains 0 if
// the section does not hold a table of fixed-size entries". We know
// that Rust 1.13 produces a string mergeable section with a zero
// sh_entsize. Here we just accept it rather than being picky about it.
uint64_t entSize = sec.sh_entsize;
if (entSize == 0)
return false;
if (sec.sh_size % entSize)
fatal(toString(this) + ":(" + name + "): SHF_MERGE section size (" +
Twine(sec.sh_size) + ") must be a multiple of sh_entsize (" +
Twine(entSize) + ")");
if (sec.sh_flags & SHF_WRITE)
fatal(toString(this) + ":(" + name +
"): writable SHF_MERGE section is not supported");
return true;
}
// This is for --just-symbols.
//
// --just-symbols is a very minor feature that allows you to link your
// output against other existing program, so that if you load both your
// program and the other program into memory, your output can refer the
// other program's symbols.
//
// When the option is given, we link "just symbols". The section table is
// initialized with null pointers.
template <class ELFT> void ObjFile<ELFT>::initializeJustSymbols() {
ArrayRef<Elf_Shdr> sections = CHECK(this->getObj().sections(), this);
this->sections.resize(sections.size());
}
[ELF] Implement Dependent Libraries Feature This patch implements a limited form of autolinking primarily designed to allow either the --dependent-library compiler option, or "comment lib" pragmas ( https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) in C/C++ e.g. #pragma comment(lib, "foo"), to cause an ELF linker to automatically add the specified library to the link when processing the input file generated by the compiler. Currently this extension is unique to LLVM and LLD. However, care has been taken to design this feature so that it could be supported by other ELF linkers. The design goals were to provide: - A simple linking model for developers to reason about. - The ability to to override autolinking from the linker command line. - Source code compatibility, where possible, with "comment lib" pragmas in other environments (MSVC in particular). Dependent library support is implemented differently for ELF platforms than on the other platforms. Primarily this difference is that on ELF we pass the dependent library specifiers directly to the linker without manipulating them. This is in contrast to other platforms where they are mapped to a specific linker option by the compiler. This difference is a result of the greater variety of ELF linkers and the fact that ELF linkers tend to handle libraries in a more complicated fashion than on other platforms. This forces us to defer handling the specifiers to the linker. In order to achieve a level of source code compatibility with other platforms we have restricted this feature to work with libraries that meet the following "reasonable" requirements: 1. There are no competing defined symbols in a given set of libraries, or if they exist, the program owner doesn't care which is linked to their program. 2. There may be circular dependencies between libraries. The binary representation is a mergeable string section (SHF_MERGE, SHF_STRINGS), called .deplibs, with custom type SHT_LLVM_DEPENDENT_LIBRARIES (0x6fff4c04). The compiler forms this section by concatenating the arguments of the "comment lib" pragmas and --dependent-library options in the order they are encountered. Partial (-r, -Ur) links are handled by concatenating .deplibs sections with the normal mergeable string section rules. As an example, #pragma comment(lib, "foo") would result in: .section ".deplibs","MS",@llvm_dependent_libraries,1 .asciz "foo" For LTO, equivalent information to the contents of a the .deplibs section can be retrieved by the LLD for bitcode input files. LLD processes the dependent library specifiers in the following way: 1. Dependent libraries which are found from the specifiers in .deplibs sections of relocatable object files are added when the linker decides to include that file (which could itself be in a library) in the link. Dependent libraries behave as if they were appended to the command line after all other options. As a consequence the set of dependent libraries are searched last to resolve symbols. 2. It is an error if a file cannot be found for a given specifier. 3. Any command line options in effect at the end of the command line parsing apply to the dependent libraries, e.g. --whole-archive. 4. The linker tries to add a library or relocatable object file from each of the strings in a .deplibs section by; first, handling the string as if it was specified on the command line; second, by looking for the string in each of the library search paths in turn; third, by looking for a lib<string>.a or lib<string>.so (depending on the current mode of the linker) in each of the library search paths. 5. A new command line option --no-dependent-libraries tells LLD to ignore the dependent libraries. Rationale for the above points: 1. Adding the dependent libraries last makes the process simple to understand from a developers perspective. All linkers are able to implement this scheme. 2. Error-ing for libraries that are not found seems like better behavior than failing the link during symbol resolution. 3. It seems useful for the user to be able to apply command line options which will affect all of the dependent libraries. There is a potential problem of surprise for developers, who might not realize that these options would apply to these "invisible" input files; however, despite the potential for surprise, this is easy for developers to reason about and gives developers the control that they may require. 4. This algorithm takes into account all of the different ways that ELF linkers find input files. The different search methods are tried by the linker in most obvious to least obvious order. 5. I considered adding finer grained control over which dependent libraries were ignored (e.g. MSVC has /nodefaultlib:<library>); however, I concluded that this is not necessary: if finer control is required developers can fall back to using the command line directly. RFC thread: http://lists.llvm.org/pipermail/llvm-dev/2019-March/131004.html. Differential Revision: https://reviews.llvm.org/D60274 llvm-svn: 360984
2019-05-17 11:44:15 +08:00
// An ELF object file may contain a `.deplibs` section. If it exists, the
// section contains a list of library specifiers such as `m` for libm. This
// function resolves a given name by finding the first matching library checking
// the various ways that a library can be specified to LLD. This ELF extension
// is a form of autolinking and is called `dependent libraries`. It is currently
// unique to LLVM and lld.
static void addDependentLibrary(StringRef specifier, const InputFile *f) {
if (!config->dependentLibraries)
return;
if (fs::exists(specifier))
driver->addFile(specifier, /*withLOption=*/false);
[ELF] Implement Dependent Libraries Feature This patch implements a limited form of autolinking primarily designed to allow either the --dependent-library compiler option, or "comment lib" pragmas ( https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) in C/C++ e.g. #pragma comment(lib, "foo"), to cause an ELF linker to automatically add the specified library to the link when processing the input file generated by the compiler. Currently this extension is unique to LLVM and LLD. However, care has been taken to design this feature so that it could be supported by other ELF linkers. The design goals were to provide: - A simple linking model for developers to reason about. - The ability to to override autolinking from the linker command line. - Source code compatibility, where possible, with "comment lib" pragmas in other environments (MSVC in particular). Dependent library support is implemented differently for ELF platforms than on the other platforms. Primarily this difference is that on ELF we pass the dependent library specifiers directly to the linker without manipulating them. This is in contrast to other platforms where they are mapped to a specific linker option by the compiler. This difference is a result of the greater variety of ELF linkers and the fact that ELF linkers tend to handle libraries in a more complicated fashion than on other platforms. This forces us to defer handling the specifiers to the linker. In order to achieve a level of source code compatibility with other platforms we have restricted this feature to work with libraries that meet the following "reasonable" requirements: 1. There are no competing defined symbols in a given set of libraries, or if they exist, the program owner doesn't care which is linked to their program. 2. There may be circular dependencies between libraries. The binary representation is a mergeable string section (SHF_MERGE, SHF_STRINGS), called .deplibs, with custom type SHT_LLVM_DEPENDENT_LIBRARIES (0x6fff4c04). The compiler forms this section by concatenating the arguments of the "comment lib" pragmas and --dependent-library options in the order they are encountered. Partial (-r, -Ur) links are handled by concatenating .deplibs sections with the normal mergeable string section rules. As an example, #pragma comment(lib, "foo") would result in: .section ".deplibs","MS",@llvm_dependent_libraries,1 .asciz "foo" For LTO, equivalent information to the contents of a the .deplibs section can be retrieved by the LLD for bitcode input files. LLD processes the dependent library specifiers in the following way: 1. Dependent libraries which are found from the specifiers in .deplibs sections of relocatable object files are added when the linker decides to include that file (which could itself be in a library) in the link. Dependent libraries behave as if they were appended to the command line after all other options. As a consequence the set of dependent libraries are searched last to resolve symbols. 2. It is an error if a file cannot be found for a given specifier. 3. Any command line options in effect at the end of the command line parsing apply to the dependent libraries, e.g. --whole-archive. 4. The linker tries to add a library or relocatable object file from each of the strings in a .deplibs section by; first, handling the string as if it was specified on the command line; second, by looking for the string in each of the library search paths in turn; third, by looking for a lib<string>.a or lib<string>.so (depending on the current mode of the linker) in each of the library search paths. 5. A new command line option --no-dependent-libraries tells LLD to ignore the dependent libraries. Rationale for the above points: 1. Adding the dependent libraries last makes the process simple to understand from a developers perspective. All linkers are able to implement this scheme. 2. Error-ing for libraries that are not found seems like better behavior than failing the link during symbol resolution. 3. It seems useful for the user to be able to apply command line options which will affect all of the dependent libraries. There is a potential problem of surprise for developers, who might not realize that these options would apply to these "invisible" input files; however, despite the potential for surprise, this is easy for developers to reason about and gives developers the control that they may require. 4. This algorithm takes into account all of the different ways that ELF linkers find input files. The different search methods are tried by the linker in most obvious to least obvious order. 5. I considered adding finer grained control over which dependent libraries were ignored (e.g. MSVC has /nodefaultlib:<library>); however, I concluded that this is not necessary: if finer control is required developers can fall back to using the command line directly. RFC thread: http://lists.llvm.org/pipermail/llvm-dev/2019-March/131004.html. Differential Revision: https://reviews.llvm.org/D60274 llvm-svn: 360984
2019-05-17 11:44:15 +08:00
else if (Optional<std::string> s = findFromSearchPaths(specifier))
driver->addFile(*s, /*withLOption=*/true);
[ELF] Implement Dependent Libraries Feature This patch implements a limited form of autolinking primarily designed to allow either the --dependent-library compiler option, or "comment lib" pragmas ( https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) in C/C++ e.g. #pragma comment(lib, "foo"), to cause an ELF linker to automatically add the specified library to the link when processing the input file generated by the compiler. Currently this extension is unique to LLVM and LLD. However, care has been taken to design this feature so that it could be supported by other ELF linkers. The design goals were to provide: - A simple linking model for developers to reason about. - The ability to to override autolinking from the linker command line. - Source code compatibility, where possible, with "comment lib" pragmas in other environments (MSVC in particular). Dependent library support is implemented differently for ELF platforms than on the other platforms. Primarily this difference is that on ELF we pass the dependent library specifiers directly to the linker without manipulating them. This is in contrast to other platforms where they are mapped to a specific linker option by the compiler. This difference is a result of the greater variety of ELF linkers and the fact that ELF linkers tend to handle libraries in a more complicated fashion than on other platforms. This forces us to defer handling the specifiers to the linker. In order to achieve a level of source code compatibility with other platforms we have restricted this feature to work with libraries that meet the following "reasonable" requirements: 1. There are no competing defined symbols in a given set of libraries, or if they exist, the program owner doesn't care which is linked to their program. 2. There may be circular dependencies between libraries. The binary representation is a mergeable string section (SHF_MERGE, SHF_STRINGS), called .deplibs, with custom type SHT_LLVM_DEPENDENT_LIBRARIES (0x6fff4c04). The compiler forms this section by concatenating the arguments of the "comment lib" pragmas and --dependent-library options in the order they are encountered. Partial (-r, -Ur) links are handled by concatenating .deplibs sections with the normal mergeable string section rules. As an example, #pragma comment(lib, "foo") would result in: .section ".deplibs","MS",@llvm_dependent_libraries,1 .asciz "foo" For LTO, equivalent information to the contents of a the .deplibs section can be retrieved by the LLD for bitcode input files. LLD processes the dependent library specifiers in the following way: 1. Dependent libraries which are found from the specifiers in .deplibs sections of relocatable object files are added when the linker decides to include that file (which could itself be in a library) in the link. Dependent libraries behave as if they were appended to the command line after all other options. As a consequence the set of dependent libraries are searched last to resolve symbols. 2. It is an error if a file cannot be found for a given specifier. 3. Any command line options in effect at the end of the command line parsing apply to the dependent libraries, e.g. --whole-archive. 4. The linker tries to add a library or relocatable object file from each of the strings in a .deplibs section by; first, handling the string as if it was specified on the command line; second, by looking for the string in each of the library search paths in turn; third, by looking for a lib<string>.a or lib<string>.so (depending on the current mode of the linker) in each of the library search paths. 5. A new command line option --no-dependent-libraries tells LLD to ignore the dependent libraries. Rationale for the above points: 1. Adding the dependent libraries last makes the process simple to understand from a developers perspective. All linkers are able to implement this scheme. 2. Error-ing for libraries that are not found seems like better behavior than failing the link during symbol resolution. 3. It seems useful for the user to be able to apply command line options which will affect all of the dependent libraries. There is a potential problem of surprise for developers, who might not realize that these options would apply to these "invisible" input files; however, despite the potential for surprise, this is easy for developers to reason about and gives developers the control that they may require. 4. This algorithm takes into account all of the different ways that ELF linkers find input files. The different search methods are tried by the linker in most obvious to least obvious order. 5. I considered adding finer grained control over which dependent libraries were ignored (e.g. MSVC has /nodefaultlib:<library>); however, I concluded that this is not necessary: if finer control is required developers can fall back to using the command line directly. RFC thread: http://lists.llvm.org/pipermail/llvm-dev/2019-March/131004.html. Differential Revision: https://reviews.llvm.org/D60274 llvm-svn: 360984
2019-05-17 11:44:15 +08:00
else if (Optional<std::string> s = searchLibraryBaseName(specifier))
driver->addFile(*s, /*withLOption=*/true);
[ELF] Implement Dependent Libraries Feature This patch implements a limited form of autolinking primarily designed to allow either the --dependent-library compiler option, or "comment lib" pragmas ( https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) in C/C++ e.g. #pragma comment(lib, "foo"), to cause an ELF linker to automatically add the specified library to the link when processing the input file generated by the compiler. Currently this extension is unique to LLVM and LLD. However, care has been taken to design this feature so that it could be supported by other ELF linkers. The design goals were to provide: - A simple linking model for developers to reason about. - The ability to to override autolinking from the linker command line. - Source code compatibility, where possible, with "comment lib" pragmas in other environments (MSVC in particular). Dependent library support is implemented differently for ELF platforms than on the other platforms. Primarily this difference is that on ELF we pass the dependent library specifiers directly to the linker without manipulating them. This is in contrast to other platforms where they are mapped to a specific linker option by the compiler. This difference is a result of the greater variety of ELF linkers and the fact that ELF linkers tend to handle libraries in a more complicated fashion than on other platforms. This forces us to defer handling the specifiers to the linker. In order to achieve a level of source code compatibility with other platforms we have restricted this feature to work with libraries that meet the following "reasonable" requirements: 1. There are no competing defined symbols in a given set of libraries, or if they exist, the program owner doesn't care which is linked to their program. 2. There may be circular dependencies between libraries. The binary representation is a mergeable string section (SHF_MERGE, SHF_STRINGS), called .deplibs, with custom type SHT_LLVM_DEPENDENT_LIBRARIES (0x6fff4c04). The compiler forms this section by concatenating the arguments of the "comment lib" pragmas and --dependent-library options in the order they are encountered. Partial (-r, -Ur) links are handled by concatenating .deplibs sections with the normal mergeable string section rules. As an example, #pragma comment(lib, "foo") would result in: .section ".deplibs","MS",@llvm_dependent_libraries,1 .asciz "foo" For LTO, equivalent information to the contents of a the .deplibs section can be retrieved by the LLD for bitcode input files. LLD processes the dependent library specifiers in the following way: 1. Dependent libraries which are found from the specifiers in .deplibs sections of relocatable object files are added when the linker decides to include that file (which could itself be in a library) in the link. Dependent libraries behave as if they were appended to the command line after all other options. As a consequence the set of dependent libraries are searched last to resolve symbols. 2. It is an error if a file cannot be found for a given specifier. 3. Any command line options in effect at the end of the command line parsing apply to the dependent libraries, e.g. --whole-archive. 4. The linker tries to add a library or relocatable object file from each of the strings in a .deplibs section by; first, handling the string as if it was specified on the command line; second, by looking for the string in each of the library search paths in turn; third, by looking for a lib<string>.a or lib<string>.so (depending on the current mode of the linker) in each of the library search paths. 5. A new command line option --no-dependent-libraries tells LLD to ignore the dependent libraries. Rationale for the above points: 1. Adding the dependent libraries last makes the process simple to understand from a developers perspective. All linkers are able to implement this scheme. 2. Error-ing for libraries that are not found seems like better behavior than failing the link during symbol resolution. 3. It seems useful for the user to be able to apply command line options which will affect all of the dependent libraries. There is a potential problem of surprise for developers, who might not realize that these options would apply to these "invisible" input files; however, despite the potential for surprise, this is easy for developers to reason about and gives developers the control that they may require. 4. This algorithm takes into account all of the different ways that ELF linkers find input files. The different search methods are tried by the linker in most obvious to least obvious order. 5. I considered adding finer grained control over which dependent libraries were ignored (e.g. MSVC has /nodefaultlib:<library>); however, I concluded that this is not necessary: if finer control is required developers can fall back to using the command line directly. RFC thread: http://lists.llvm.org/pipermail/llvm-dev/2019-March/131004.html. Differential Revision: https://reviews.llvm.org/D60274 llvm-svn: 360984
2019-05-17 11:44:15 +08:00
else
error(toString(f) +
": unable to find library from dependent library specifier: " +
specifier);
}
// Record the membership of a section group so that in the garbage collection
// pass, section group members are kept or discarded as a unit.
template <class ELFT>
static void handleSectionGroup(ArrayRef<InputSectionBase *> sections,
ArrayRef<typename ELFT::Word> entries) {
bool hasAlloc = false;
for (uint32_t index : entries.slice(1)) {
if (index >= sections.size())
return;
if (InputSectionBase *s = sections[index])
if (s != &InputSection::discarded && s->flags & SHF_ALLOC)
hasAlloc = true;
}
// If any member has the SHF_ALLOC flag, the whole group is subject to garbage
// collection. See the comment in markLive(). This rule retains .debug_types
// and .rela.debug_types.
if (!hasAlloc)
return;
// Connect the members in a circular doubly-linked list via
// nextInSectionGroup.
InputSectionBase *head;
InputSectionBase *prev = nullptr;
for (uint32_t index : entries.slice(1)) {
InputSectionBase *s = sections[index];
if (!s || s == &InputSection::discarded)
continue;
if (prev)
prev->nextInSectionGroup = s;
else
head = s;
prev = s;
}
if (prev)
prev->nextInSectionGroup = head;
}
template <class ELFT>
void ObjFile<ELFT>::initializeSections(bool ignoreComdats) {
2017-05-26 10:17:30 +08:00
const ELFFile<ELFT> &obj = this->getObj();
ArrayRef<Elf_Shdr> objSections = CHECK(obj.sections(), this);
uint64_t size = objSections.size();
this->sections.resize(size);
this->sectionStringTable =
CHECK(obj.getSectionStringTable(objSections), this);
2017-05-26 10:17:30 +08:00
[ELF] Improve --gc-sections compatibility with GNU ld regarding section groups Based on D70020 by serge-sans-paille. The ELF spec says: > Furthermore, there may be internal references among these sections that would not make sense if one of the sections were removed or replaced by a duplicate from another object. Therefore, such groups must be included or omitted from the linked object as a unit. A section cannot be a member of more than one group. GNU ld has 2 behaviors that we don't have: - Group members (nextInSectionGroup != nullptr) are subject to garbage collection. This includes non-SHF_ALLOC SHT_NOTE sections. In particular, discarding non-SHF_ALLOC SHT_NOTE sections is an expected behavior by the Annobin project. See https://developers.redhat.com/blog/2018/02/20/annobin-storing-information-binaries/ for more information. - Groups members are retained or discarded as a unit. Members may have internal references that are not expressed as SHF_LINK_ORDER, relocations, etc. It seems that we should be more conservative here: if a section is marked live, mark all the other member within the group. Both behaviors are reasonable. This patch implements them. A new field InputSectionBase::nextInSectionGroup tracks the next member within a group. on ELF64, this increases sizeof(InputSectionBase) froms 144 to 152. InputSectionBase::dependentSections tracks section dependencies, which is used by both --gc-sections and /DISCARD/. We can't overload it for the "next member" semantic, because we should allow /DISCARD/ to discard sections independent of --gc-sections (GNU ld behavior). This behavior may be reasonably used by `/DISCARD/ : { *(.ARM.exidx*) }` or `/DISCARD/ : { *(.note*) }` (new test `linkerscript/discard-group.s`). Reviewed By: ruiu Differential Revision: https://reviews.llvm.org/D70146
2019-11-19 13:56:58 +08:00
std::vector<ArrayRef<Elf_Word>> selectedGroups;
for (size_t i = 0, e = objSections.size(); i < e; ++i) {
if (this->sections[i] == &InputSection::discarded)
continue;
2017-05-26 10:17:30 +08:00
const Elf_Shdr &sec = objSections[i];
if (sec.sh_type == ELF::SHT_LLVM_CALL_GRAPH_PROFILE)
2019-01-26 05:25:25 +08:00
cgProfile =
check(obj.template getSectionContentsAsArray<Elf_CGProfile>(&sec));
2016-10-05 00:47:49 +08:00
// SHF_EXCLUDE'ed sections are discarded by the linker. However,
// if -r is given, we'll let the final link discard such sections.
// This is compatible with GNU.
if ((sec.sh_flags & SHF_EXCLUDE) && !config->relocatable) {
if (sec.sh_type == SHT_LLVM_ADDRSIG) {
// We ignore the address-significance table if we know that the object
// file was created by objcopy or ld -r. This is because these tools
// will reorder the symbols in the symbol table, invalidating the data
// in the address-significance table, which refers to symbols by index.
if (sec.sh_link != 0)
this->addrsigSec = &sec;
else if (config->icf == ICFLevel::Safe)
warn(toString(this) + ": --icf=safe is incompatible with object "
"files created using objcopy or ld -r");
}
this->sections[i] = &InputSection::discarded;
continue;
}
switch (sec.sh_type) {
case SHT_GROUP: {
2017-06-09 10:42:20 +08:00
// De-duplicate section groups by their signatures.
StringRef signature = getShtGroupSignature(objSections, sec);
this->sections[i] = &InputSection::discarded;
ArrayRef<Elf_Word> entries =
CHECK(obj.template getSectionContentsAsArray<Elf_Word>(&sec), this);
if (entries.empty())
fatal(toString(this) + ": empty SHT_GROUP");
// The first word of a SHT_GROUP section contains flags. Currently,
// the standard defines only "GRP_COMDAT" flag for the COMDAT group.
// An group with the empty flag doesn't define anything; such sections
// are just skipped.
if (entries[0] == 0)
continue;
if (entries[0] != GRP_COMDAT)
fatal(toString(this) + ": unsupported SHT_GROUP format");
bool isNew =
ignoreComdats ||
symtab->comdatGroups.try_emplace(CachedHashStringRef(signature), this)
.second;
2017-06-09 10:42:20 +08:00
if (isNew) {
if (config->relocatable)
this->sections[i] = createInputSection(sec);
[ELF] Improve --gc-sections compatibility with GNU ld regarding section groups Based on D70020 by serge-sans-paille. The ELF spec says: > Furthermore, there may be internal references among these sections that would not make sense if one of the sections were removed or replaced by a duplicate from another object. Therefore, such groups must be included or omitted from the linked object as a unit. A section cannot be a member of more than one group. GNU ld has 2 behaviors that we don't have: - Group members (nextInSectionGroup != nullptr) are subject to garbage collection. This includes non-SHF_ALLOC SHT_NOTE sections. In particular, discarding non-SHF_ALLOC SHT_NOTE sections is an expected behavior by the Annobin project. See https://developers.redhat.com/blog/2018/02/20/annobin-storing-information-binaries/ for more information. - Groups members are retained or discarded as a unit. Members may have internal references that are not expressed as SHF_LINK_ORDER, relocations, etc. It seems that we should be more conservative here: if a section is marked live, mark all the other member within the group. Both behaviors are reasonable. This patch implements them. A new field InputSectionBase::nextInSectionGroup tracks the next member within a group. on ELF64, this increases sizeof(InputSectionBase) froms 144 to 152. InputSectionBase::dependentSections tracks section dependencies, which is used by both --gc-sections and /DISCARD/. We can't overload it for the "next member" semantic, because we should allow /DISCARD/ to discard sections independent of --gc-sections (GNU ld behavior). This behavior may be reasonably used by `/DISCARD/ : { *(.ARM.exidx*) }` or `/DISCARD/ : { *(.note*) }` (new test `linkerscript/discard-group.s`). Reviewed By: ruiu Differential Revision: https://reviews.llvm.org/D70146
2019-11-19 13:56:58 +08:00
selectedGroups.push_back(entries);
2019-01-25 02:17:17 +08:00
continue;
2017-06-09 10:42:20 +08:00
}
2017-06-09 10:42:20 +08:00
// Otherwise, discard group members.
for (uint32_t secIndex : entries.slice(1)) {
if (secIndex >= size)
fatal(toString(this) +
": invalid section index in group: " + Twine(secIndex));
this->sections[secIndex] = &InputSection::discarded;
}
break;
}
case SHT_SYMTAB_SHNDX:
shndxTable = CHECK(obj.getSHNDXTable(sec, objSections), this);
break;
case SHT_SYMTAB:
case SHT_STRTAB:
case SHT_NULL:
break;
default:
this->sections[i] = createInputSection(sec);
}
}
[ELF] Improve --gc-sections compatibility with GNU ld regarding section groups Based on D70020 by serge-sans-paille. The ELF spec says: > Furthermore, there may be internal references among these sections that would not make sense if one of the sections were removed or replaced by a duplicate from another object. Therefore, such groups must be included or omitted from the linked object as a unit. A section cannot be a member of more than one group. GNU ld has 2 behaviors that we don't have: - Group members (nextInSectionGroup != nullptr) are subject to garbage collection. This includes non-SHF_ALLOC SHT_NOTE sections. In particular, discarding non-SHF_ALLOC SHT_NOTE sections is an expected behavior by the Annobin project. See https://developers.redhat.com/blog/2018/02/20/annobin-storing-information-binaries/ for more information. - Groups members are retained or discarded as a unit. Members may have internal references that are not expressed as SHF_LINK_ORDER, relocations, etc. It seems that we should be more conservative here: if a section is marked live, mark all the other member within the group. Both behaviors are reasonable. This patch implements them. A new field InputSectionBase::nextInSectionGroup tracks the next member within a group. on ELF64, this increases sizeof(InputSectionBase) froms 144 to 152. InputSectionBase::dependentSections tracks section dependencies, which is used by both --gc-sections and /DISCARD/. We can't overload it for the "next member" semantic, because we should allow /DISCARD/ to discard sections independent of --gc-sections (GNU ld behavior). This behavior may be reasonably used by `/DISCARD/ : { *(.ARM.exidx*) }` or `/DISCARD/ : { *(.note*) }` (new test `linkerscript/discard-group.s`). Reviewed By: ruiu Differential Revision: https://reviews.llvm.org/D70146
2019-11-19 13:56:58 +08:00
// This block handles SHF_LINK_ORDER.
for (size_t i = 0, e = objSections.size(); i < e; ++i) {
if (this->sections[i] == &InputSection::discarded)
continue;
const Elf_Shdr &sec = objSections[i];
if (!(sec.sh_flags & SHF_LINK_ORDER))
continue;
// .ARM.exidx sections have a reverse dependency on the InputSection they
// have a SHF_LINK_ORDER dependency, this is identified by the sh_link.
InputSectionBase *linkSec = nullptr;
if (sec.sh_link < this->sections.size())
linkSec = this->sections[sec.sh_link];
if (!linkSec)
fatal(toString(this) + ": invalid sh_link index: " + Twine(sec.sh_link));
InputSection *isec = cast<InputSection>(this->sections[i]);
linkSec->dependentSections.push_back(isec);
if (!isa<InputSection>(linkSec))
error("a section " + isec->name +
" with SHF_LINK_ORDER should not refer a non-regular section: " +
toString(linkSec));
}
[ELF] Improve --gc-sections compatibility with GNU ld regarding section groups Based on D70020 by serge-sans-paille. The ELF spec says: > Furthermore, there may be internal references among these sections that would not make sense if one of the sections were removed or replaced by a duplicate from another object. Therefore, such groups must be included or omitted from the linked object as a unit. A section cannot be a member of more than one group. GNU ld has 2 behaviors that we don't have: - Group members (nextInSectionGroup != nullptr) are subject to garbage collection. This includes non-SHF_ALLOC SHT_NOTE sections. In particular, discarding non-SHF_ALLOC SHT_NOTE sections is an expected behavior by the Annobin project. See https://developers.redhat.com/blog/2018/02/20/annobin-storing-information-binaries/ for more information. - Groups members are retained or discarded as a unit. Members may have internal references that are not expressed as SHF_LINK_ORDER, relocations, etc. It seems that we should be more conservative here: if a section is marked live, mark all the other member within the group. Both behaviors are reasonable. This patch implements them. A new field InputSectionBase::nextInSectionGroup tracks the next member within a group. on ELF64, this increases sizeof(InputSectionBase) froms 144 to 152. InputSectionBase::dependentSections tracks section dependencies, which is used by both --gc-sections and /DISCARD/. We can't overload it for the "next member" semantic, because we should allow /DISCARD/ to discard sections independent of --gc-sections (GNU ld behavior). This behavior may be reasonably used by `/DISCARD/ : { *(.ARM.exidx*) }` or `/DISCARD/ : { *(.note*) }` (new test `linkerscript/discard-group.s`). Reviewed By: ruiu Differential Revision: https://reviews.llvm.org/D70146
2019-11-19 13:56:58 +08:00
for (ArrayRef<Elf_Word> entries : selectedGroups)
handleSectionGroup<ELFT>(this->sections, entries);
}
// For ARM only, to set the EF_ARM_ABI_FLOAT_SOFT or EF_ARM_ABI_FLOAT_HARD
// flag in the ELF Header we need to look at Tag_ABI_VFP_args to find out how
// the input objects have been compiled.
static void updateARMVFPArgs(const ARMAttributeParser &attributes,
const InputFile *f) {
Optional<unsigned> attr =
attributes.getAttributeValue(ARMBuildAttrs::ABI_VFP_args);
if (!attr.hasValue())
// If an ABI tag isn't present then it is implicitly given the value of 0
// which maps to ARMBuildAttrs::BaseAAPCS. However many assembler files,
// including some in glibc that don't use FP args (and should have value 3)
// don't have the attribute so we do not consider an implicit value of 0
// as a clash.
return;
unsigned vfpArgs = attr.getValue();
ARMVFPArgKind arg;
switch (vfpArgs) {
case ARMBuildAttrs::BaseAAPCS:
arg = ARMVFPArgKind::Base;
break;
case ARMBuildAttrs::HardFPAAPCS:
arg = ARMVFPArgKind::VFP;
break;
case ARMBuildAttrs::ToolChainFPPCS:
// Tool chain specific convention that conforms to neither AAPCS variant.
arg = ARMVFPArgKind::ToolChain;
break;
case ARMBuildAttrs::CompatibleFPAAPCS:
// Object compatible with all conventions.
return;
default:
error(toString(f) + ": unknown Tag_ABI_VFP_args value: " + Twine(vfpArgs));
return;
}
// Follow ld.bfd and error if there is a mix of calling conventions.
if (config->armVFPArgs != arg && config->armVFPArgs != ARMVFPArgKind::Default)
error(toString(f) + ": incompatible Tag_ABI_VFP_args");
else
config->armVFPArgs = arg;
}
// The ARM support in lld makes some use of instructions that are not available
// on all ARM architectures. Namely:
// - Use of BLX instruction for interworking between ARM and Thumb state.
// - Use of the extended Thumb branch encoding in relocation.
// - Use of the MOVT/MOVW instructions in Thumb Thunks.
// The ARM Attributes section contains information about the architecture chosen
// at compile time. We follow the convention that if at least one input object
// is compiled with an architecture that supports these features then lld is
// permitted to use them.
static void updateSupportedARMFeatures(const ARMAttributeParser &attributes) {
Optional<unsigned> attr =
attributes.getAttributeValue(ARMBuildAttrs::CPU_arch);
if (!attr.hasValue())
return;
auto arch = attr.getValue();
switch (arch) {
case ARMBuildAttrs::Pre_v4:
case ARMBuildAttrs::v4:
case ARMBuildAttrs::v4T:
// Architectures prior to v5 do not support BLX instruction
break;
case ARMBuildAttrs::v5T:
case ARMBuildAttrs::v5TE:
case ARMBuildAttrs::v5TEJ:
case ARMBuildAttrs::v6:
case ARMBuildAttrs::v6KZ:
case ARMBuildAttrs::v6K:
config->armHasBlx = true;
// Architectures used in pre-Cortex processors do not support
// The J1 = 1 J2 = 1 Thumb branch range extension, with the exception
// of Architecture v6T2 (arm1156t2-s and arm1156t2f-s) that do.
break;
default:
// All other Architectures have BLX and extended branch encoding
config->armHasBlx = true;
config->armJ1J2BranchEncoding = true;
if (arch != ARMBuildAttrs::v6_M && arch != ARMBuildAttrs::v6S_M)
// All Architectures used in Cortex processors with the exception
// of v6-M and v6S-M have the MOVT and MOVW instructions.
config->armHasMovtMovw = true;
break;
}
}
// If a source file is compiled with x86 hardware-assisted call flow control
// enabled, the generated object file contains feature flags indicating that
// fact. This function reads the feature flags and returns it.
//
// Essentially we want to read a single 32-bit value in this function, but this
// function is rather complicated because the value is buried deep inside a
// .note.gnu.property section.
//
// The section consists of one or more NOTE records. Each NOTE record consists
// of zero or more type-length-value fields. We want to find a field of a
// certain type. It seems a bit too much to just store a 32-bit value, perhaps
// the ABI is unnecessarily complicated.
template <class ELFT>
static uint32_t readAndFeatures(ObjFile<ELFT> *obj, ArrayRef<uint8_t> data) {
using Elf_Nhdr = typename ELFT::Nhdr;
using Elf_Note = typename ELFT::Note;
uint32_t featuresSet = 0;
while (!data.empty()) {
// Read one NOTE record.
if (data.size() < sizeof(Elf_Nhdr))
fatal(toString(obj) + ": .note.gnu.property: section too short");
auto *nhdr = reinterpret_cast<const Elf_Nhdr *>(data.data());
if (data.size() < nhdr->getSize())
fatal(toString(obj) + ": .note.gnu.property: section too short");
Elf_Note note(*nhdr);
if (nhdr->n_type != NT_GNU_PROPERTY_TYPE_0 || note.getName() != "GNU") {
data = data.slice(nhdr->getSize());
continue;
}
[ELF][AArch64] Support for BTI and PAC Branch Target Identification (BTI) and Pointer Authentication (PAC) are architecture features introduced in v8.5a and 8.3a respectively. The new instructions have been added in the hint space so that binaries take advantage of support where it exists yet still run on older hardware. The impact of each feature is: BTI: For executable pages that have been guarded, all indirect branches must have a destination that is a BTI instruction of the appropriate type. For the static linker, this means that PLT entries must have a "BTI c" as the first instruction in the sequence. BTI is an all or nothing property for a link unit, any indirect branch not landing on a valid destination will cause a Branch Target Exception. PAC: The dynamic loader encodes with PACIA the address of the destination that the PLT entry will load from the .plt.got, placing the result in a subset of the top-bits that are not valid virtual addresses. The PLT entry may authenticate these top-bits using the AUTIA instruction before branching to the destination. Use of PAC in PLT sequences is a contract between the dynamic loader and the static linker, it is independent of whether the relocatable objects use PAC. BTI and PAC are independent features that can be combined. So we can have several combinations of PLT: - Standard with no BTI or PAC - BTI PLT with "BTI c" as first instruction. - PAC PLT with "AUTIA1716" before the indirect branch to X17. - BTIPAC PLT with "BTI c" as first instruction and "AUTIA1716" before the first indirect branch to X17. The use of BTI and PAC in relocatable object files are encoded by feature bits in the .note.gnu.property section in a similar way to Intel CET. There is one AArch64 specific program property GNU_PROPERTY_AARCH64_FEATURE_1_AND and two target feature bits defined: - GNU_PROPERTY_AARCH64_FEATURE_1_BTI -- All executable sections are compatible with BTI. - GNU_PROPERTY_AARCH64_FEATURE_1_PAC -- All executable sections have return address signing enabled. Due to the properties of FEATURE_1_AND the static linker can tell when all input relocatable objects have the BTI and PAC feature bits set. The static linker uses this to enable the appropriate PLT sequence. Neither -> standard PLT GNU_PROPERTY_AARCH64_FEATURE_1_BTI -> BTI PLT GNU_PROPERTY_AARCH64_FEATURE_1_PAC -> PAC PLT Both properties -> BTIPAC PLT In addition to the .note.gnu.properties there are two new command line options: --force-bti : Act as if all relocatable inputs had GNU_PROPERTY_AARCH64_FEATURE_1_BTI and warn for every relocatable object that does not. --pac-plt : Act as if all relocatable inputs had GNU_PROPERTY_AARCH64_FEATURE_1_PAC. As PAC is a contract between the loader and static linker no warning is given if it is not present in an input. Two processor specific dynamic tags are used to communicate that a non standard PLT sequence is being used. DTI_AARCH64_BTI_PLT and DTI_AARCH64_BTI_PAC. Differential Revision: https://reviews.llvm.org/D62609 llvm-svn: 362793
2019-06-07 21:00:17 +08:00
uint32_t featureAndType = config->emachine == EM_AARCH64
? GNU_PROPERTY_AARCH64_FEATURE_1_AND
: GNU_PROPERTY_X86_FEATURE_1_AND;
// Read a body of a NOTE record, which consists of type-length-value fields.
ArrayRef<uint8_t> desc = note.getDesc();
while (!desc.empty()) {
if (desc.size() < 8)
fatal(toString(obj) + ": .note.gnu.property: section too short");
uint32_t type = read32le(desc.data());
uint32_t size = read32le(desc.data() + 4);
[ELF][AArch64] Support for BTI and PAC Branch Target Identification (BTI) and Pointer Authentication (PAC) are architecture features introduced in v8.5a and 8.3a respectively. The new instructions have been added in the hint space so that binaries take advantage of support where it exists yet still run on older hardware. The impact of each feature is: BTI: For executable pages that have been guarded, all indirect branches must have a destination that is a BTI instruction of the appropriate type. For the static linker, this means that PLT entries must have a "BTI c" as the first instruction in the sequence. BTI is an all or nothing property for a link unit, any indirect branch not landing on a valid destination will cause a Branch Target Exception. PAC: The dynamic loader encodes with PACIA the address of the destination that the PLT entry will load from the .plt.got, placing the result in a subset of the top-bits that are not valid virtual addresses. The PLT entry may authenticate these top-bits using the AUTIA instruction before branching to the destination. Use of PAC in PLT sequences is a contract between the dynamic loader and the static linker, it is independent of whether the relocatable objects use PAC. BTI and PAC are independent features that can be combined. So we can have several combinations of PLT: - Standard with no BTI or PAC - BTI PLT with "BTI c" as first instruction. - PAC PLT with "AUTIA1716" before the indirect branch to X17. - BTIPAC PLT with "BTI c" as first instruction and "AUTIA1716" before the first indirect branch to X17. The use of BTI and PAC in relocatable object files are encoded by feature bits in the .note.gnu.property section in a similar way to Intel CET. There is one AArch64 specific program property GNU_PROPERTY_AARCH64_FEATURE_1_AND and two target feature bits defined: - GNU_PROPERTY_AARCH64_FEATURE_1_BTI -- All executable sections are compatible with BTI. - GNU_PROPERTY_AARCH64_FEATURE_1_PAC -- All executable sections have return address signing enabled. Due to the properties of FEATURE_1_AND the static linker can tell when all input relocatable objects have the BTI and PAC feature bits set. The static linker uses this to enable the appropriate PLT sequence. Neither -> standard PLT GNU_PROPERTY_AARCH64_FEATURE_1_BTI -> BTI PLT GNU_PROPERTY_AARCH64_FEATURE_1_PAC -> PAC PLT Both properties -> BTIPAC PLT In addition to the .note.gnu.properties there are two new command line options: --force-bti : Act as if all relocatable inputs had GNU_PROPERTY_AARCH64_FEATURE_1_BTI and warn for every relocatable object that does not. --pac-plt : Act as if all relocatable inputs had GNU_PROPERTY_AARCH64_FEATURE_1_PAC. As PAC is a contract between the loader and static linker no warning is given if it is not present in an input. Two processor specific dynamic tags are used to communicate that a non standard PLT sequence is being used. DTI_AARCH64_BTI_PLT and DTI_AARCH64_BTI_PAC. Differential Revision: https://reviews.llvm.org/D62609 llvm-svn: 362793
2019-06-07 21:00:17 +08:00
if (type == featureAndType) {
// We found a FEATURE_1_AND field. There may be more than one of these
// in a .note.gnu.property section, for a relocatable object we
// accumulate the bits set.
featuresSet |= read32le(desc.data() + 8);
}
// On 64-bit, a payload may be followed by a 4-byte padding to make its
// size a multiple of 8.
if (ELFT::Is64Bits)
size = alignTo(size, 8);
desc = desc.slice(size + 8); // +8 for Type and Size
}
// Go to next NOTE record to look for more FEATURE_1_AND descriptions.
data = data.slice(nhdr->getSize());
}
return featuresSet;
}
template <class ELFT>
InputSectionBase *ObjFile<ELFT>::getRelocTarget(const Elf_Shdr &sec) {
uint32_t idx = sec.sh_info;
if (idx >= this->sections.size())
fatal(toString(this) + ": invalid relocated section index: " + Twine(idx));
InputSectionBase *target = this->sections[idx];
// Strictly speaking, a relocation section must be included in the
// group of the section it relocates. However, LLVM 3.3 and earlier
// would fail to do so, so we gracefully handle that case.
if (target == &InputSection::discarded)
return nullptr;
if (!target)
fatal(toString(this) + ": unsupported relocation reference");
return target;
}
// Create a regular InputSection class that has the same contents
// as a given section.
static InputSection *toRegularSection(MergeInputSection *sec) {
return make<InputSection>(sec->file, sec->flags, sec->type, sec->alignment,
sec->data(), sec->name);
}
template <class ELFT>
InputSectionBase *ObjFile<ELFT>::createInputSection(const Elf_Shdr &sec) {
StringRef name = getSectionName(sec);
switch (sec.sh_type) {
case SHT_ARM_ATTRIBUTES: {
if (config->emachine != EM_ARM)
break;
ARMAttributeParser attributes;
ArrayRef<uint8_t> contents = check(this->getObj().getSectionContents(&sec));
if (Error e = attributes.parse(contents, config->ekind == ELF32LEKind
? support::little
: support::big)) {
auto *isec = make<InputSection>(*this, sec, name);
warn(toString(isec) + ": " + llvm::toString(std::move(e)));
break;
}
updateSupportedARMFeatures(attributes);
updateARMVFPArgs(attributes, this);
// FIXME: Retain the first attribute section we see. The eglibc ARM
// dynamic loaders require the presence of an attribute section for dlopen
// to work. In a full implementation we would merge all attribute sections.
if (in.armAttributes == nullptr) {
in.armAttributes = make<InputSection>(*this, sec, name);
return in.armAttributes;
}
return &InputSection::discarded;
}
[ELF] Implement Dependent Libraries Feature This patch implements a limited form of autolinking primarily designed to allow either the --dependent-library compiler option, or "comment lib" pragmas ( https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) in C/C++ e.g. #pragma comment(lib, "foo"), to cause an ELF linker to automatically add the specified library to the link when processing the input file generated by the compiler. Currently this extension is unique to LLVM and LLD. However, care has been taken to design this feature so that it could be supported by other ELF linkers. The design goals were to provide: - A simple linking model for developers to reason about. - The ability to to override autolinking from the linker command line. - Source code compatibility, where possible, with "comment lib" pragmas in other environments (MSVC in particular). Dependent library support is implemented differently for ELF platforms than on the other platforms. Primarily this difference is that on ELF we pass the dependent library specifiers directly to the linker without manipulating them. This is in contrast to other platforms where they are mapped to a specific linker option by the compiler. This difference is a result of the greater variety of ELF linkers and the fact that ELF linkers tend to handle libraries in a more complicated fashion than on other platforms. This forces us to defer handling the specifiers to the linker. In order to achieve a level of source code compatibility with other platforms we have restricted this feature to work with libraries that meet the following "reasonable" requirements: 1. There are no competing defined symbols in a given set of libraries, or if they exist, the program owner doesn't care which is linked to their program. 2. There may be circular dependencies between libraries. The binary representation is a mergeable string section (SHF_MERGE, SHF_STRINGS), called .deplibs, with custom type SHT_LLVM_DEPENDENT_LIBRARIES (0x6fff4c04). The compiler forms this section by concatenating the arguments of the "comment lib" pragmas and --dependent-library options in the order they are encountered. Partial (-r, -Ur) links are handled by concatenating .deplibs sections with the normal mergeable string section rules. As an example, #pragma comment(lib, "foo") would result in: .section ".deplibs","MS",@llvm_dependent_libraries,1 .asciz "foo" For LTO, equivalent information to the contents of a the .deplibs section can be retrieved by the LLD for bitcode input files. LLD processes the dependent library specifiers in the following way: 1. Dependent libraries which are found from the specifiers in .deplibs sections of relocatable object files are added when the linker decides to include that file (which could itself be in a library) in the link. Dependent libraries behave as if they were appended to the command line after all other options. As a consequence the set of dependent libraries are searched last to resolve symbols. 2. It is an error if a file cannot be found for a given specifier. 3. Any command line options in effect at the end of the command line parsing apply to the dependent libraries, e.g. --whole-archive. 4. The linker tries to add a library or relocatable object file from each of the strings in a .deplibs section by; first, handling the string as if it was specified on the command line; second, by looking for the string in each of the library search paths in turn; third, by looking for a lib<string>.a or lib<string>.so (depending on the current mode of the linker) in each of the library search paths. 5. A new command line option --no-dependent-libraries tells LLD to ignore the dependent libraries. Rationale for the above points: 1. Adding the dependent libraries last makes the process simple to understand from a developers perspective. All linkers are able to implement this scheme. 2. Error-ing for libraries that are not found seems like better behavior than failing the link during symbol resolution. 3. It seems useful for the user to be able to apply command line options which will affect all of the dependent libraries. There is a potential problem of surprise for developers, who might not realize that these options would apply to these "invisible" input files; however, despite the potential for surprise, this is easy for developers to reason about and gives developers the control that they may require. 4. This algorithm takes into account all of the different ways that ELF linkers find input files. The different search methods are tried by the linker in most obvious to least obvious order. 5. I considered adding finer grained control over which dependent libraries were ignored (e.g. MSVC has /nodefaultlib:<library>); however, I concluded that this is not necessary: if finer control is required developers can fall back to using the command line directly. RFC thread: http://lists.llvm.org/pipermail/llvm-dev/2019-March/131004.html. Differential Revision: https://reviews.llvm.org/D60274 llvm-svn: 360984
2019-05-17 11:44:15 +08:00
case SHT_LLVM_DEPENDENT_LIBRARIES: {
if (config->relocatable)
break;
ArrayRef<char> data =
CHECK(this->getObj().template getSectionContentsAsArray<char>(&sec), this);
if (!data.empty() && data.back() != '\0') {
error(toString(this) +
": corrupted dependent libraries section (unterminated string): " +
name);
return &InputSection::discarded;
}
for (const char *d = data.begin(), *e = data.end(); d < e;) {
StringRef s(d);
addDependentLibrary(s, this);
d += s.size() + 1;
}
return &InputSection::discarded;
}
case SHT_RELA:
case SHT_REL: {
2018-03-27 10:52:58 +08:00
// Find a relocation target section and associate this section with that.
// Target may have been discarded if it is in a different section group
// and the group is discarded, even though it's a violation of the
// spec. We handle that situation gracefully by discarding dangling
// relocation sections.
InputSectionBase *target = getRelocTarget(sec);
if (!target)
return nullptr;
// ELF spec allows mergeable sections with relocations, but they are
// rare, and it is in practice hard to merge such sections by contents,
// because applying relocations at end of linking changes section
// contents. So, we simply handle such sections as non-mergeable ones.
// Degrading like this is acceptable because section merging is optional.
if (auto *ms = dyn_cast<MergeInputSection>(target)) {
target = toRegularSection(ms);
this->sections[sec.sh_info] = target;
}
// This section contains relocation information.
// If -r is given, we do not interpret or apply relocation
// but just copy relocation sections to output.
if (config->relocatable) {
InputSection *relocSec = make<InputSection>(*this, sec, name);
// We want to add a dependency to target, similar like we do for
// -emit-relocs below. This is useful for the case when linker script
// contains the "/DISCARD/". It is perhaps uncommon to use a script with
// -r, but we faced it in the Linux kernel and have to handle such case
// and not to crash.
target->dependentSections.push_back(relocSec);
return relocSec;
}
if (target->firstRelocation)
fatal(toString(this) +
": multiple relocation sections to one section are not supported");
if (sec.sh_type == SHT_RELA) {
ArrayRef<Elf_Rela> rels = CHECK(getObj().relas(&sec), this);
target->firstRelocation = rels.begin();
target->numRelocations = rels.size();
target->areRelocsRela = true;
} else {
ArrayRef<Elf_Rel> rels = CHECK(getObj().rels(&sec), this);
target->firstRelocation = rels.begin();
target->numRelocations = rels.size();
target->areRelocsRela = false;
}
assert(isUInt<31>(target->numRelocations));
// Relocation sections processed by the linker are usually removed
// from the output, so returning `nullptr` for the normal case.
// However, if -emit-relocs is given, we need to leave them in the output.
// (Some post link analysis tools need this information.)
if (config->emitRelocs) {
InputSection *relocSec = make<InputSection>(*this, sec, name);
// We will not emit relocation section if target was discarded.
target->dependentSections.push_back(relocSec);
return relocSec;
}
return nullptr;
}
}
// The GNU linker uses .note.GNU-stack section as a marker indicating
// that the code in the object file does not expect that the stack is
// executable (in terms of NX bit). If all input files have the marker,
// the GNU linker adds a PT_GNU_STACK segment to tells the loader to
// make the stack non-executable. Most object files have this section as
// of 2017.
//
// But making the stack non-executable is a norm today for security
// reasons. Failure to do so may result in a serious security issue.
// Therefore, we make LLD always add PT_GNU_STACK unless it is
// explicitly told to do otherwise (by -z execstack). Because the stack
// executable-ness is controlled solely by command line options,
// .note.GNU-stack sections are simply ignored.
if (name == ".note.GNU-stack")
return &InputSection::discarded;
[ELF][AArch64] Support for BTI and PAC Branch Target Identification (BTI) and Pointer Authentication (PAC) are architecture features introduced in v8.5a and 8.3a respectively. The new instructions have been added in the hint space so that binaries take advantage of support where it exists yet still run on older hardware. The impact of each feature is: BTI: For executable pages that have been guarded, all indirect branches must have a destination that is a BTI instruction of the appropriate type. For the static linker, this means that PLT entries must have a "BTI c" as the first instruction in the sequence. BTI is an all or nothing property for a link unit, any indirect branch not landing on a valid destination will cause a Branch Target Exception. PAC: The dynamic loader encodes with PACIA the address of the destination that the PLT entry will load from the .plt.got, placing the result in a subset of the top-bits that are not valid virtual addresses. The PLT entry may authenticate these top-bits using the AUTIA instruction before branching to the destination. Use of PAC in PLT sequences is a contract between the dynamic loader and the static linker, it is independent of whether the relocatable objects use PAC. BTI and PAC are independent features that can be combined. So we can have several combinations of PLT: - Standard with no BTI or PAC - BTI PLT with "BTI c" as first instruction. - PAC PLT with "AUTIA1716" before the indirect branch to X17. - BTIPAC PLT with "BTI c" as first instruction and "AUTIA1716" before the first indirect branch to X17. The use of BTI and PAC in relocatable object files are encoded by feature bits in the .note.gnu.property section in a similar way to Intel CET. There is one AArch64 specific program property GNU_PROPERTY_AARCH64_FEATURE_1_AND and two target feature bits defined: - GNU_PROPERTY_AARCH64_FEATURE_1_BTI -- All executable sections are compatible with BTI. - GNU_PROPERTY_AARCH64_FEATURE_1_PAC -- All executable sections have return address signing enabled. Due to the properties of FEATURE_1_AND the static linker can tell when all input relocatable objects have the BTI and PAC feature bits set. The static linker uses this to enable the appropriate PLT sequence. Neither -> standard PLT GNU_PROPERTY_AARCH64_FEATURE_1_BTI -> BTI PLT GNU_PROPERTY_AARCH64_FEATURE_1_PAC -> PAC PLT Both properties -> BTIPAC PLT In addition to the .note.gnu.properties there are two new command line options: --force-bti : Act as if all relocatable inputs had GNU_PROPERTY_AARCH64_FEATURE_1_BTI and warn for every relocatable object that does not. --pac-plt : Act as if all relocatable inputs had GNU_PROPERTY_AARCH64_FEATURE_1_PAC. As PAC is a contract between the loader and static linker no warning is given if it is not present in an input. Two processor specific dynamic tags are used to communicate that a non standard PLT sequence is being used. DTI_AARCH64_BTI_PLT and DTI_AARCH64_BTI_PAC. Differential Revision: https://reviews.llvm.org/D62609 llvm-svn: 362793
2019-06-07 21:00:17 +08:00
// Object files that use processor features such as Intel Control-Flow
// Enforcement (CET) or AArch64 Branch Target Identification BTI, use a
// .note.gnu.property section containing a bitfield of feature bits like the
// GNU_PROPERTY_X86_FEATURE_1_IBT flag. Read a bitmap containing the flag.
//
// Since we merge bitmaps from multiple object files to create a new
// .note.gnu.property containing a single AND'ed bitmap, we discard an input
// file's .note.gnu.property section.
if (name == ".note.gnu.property") {
ArrayRef<uint8_t> contents = check(this->getObj().getSectionContents(&sec));
this->andFeatures = readAndFeatures(this, contents);
return &InputSection::discarded;
}
// Split stacks is a feature to support a discontiguous stack,
// commonly used in the programming language Go. For the details,
// see https://gcc.gnu.org/wiki/SplitStacks. An object file compiled
// for split stack will include a .note.GNU-split-stack section.
if (name == ".note.GNU-split-stack") {
if (config->relocatable) {
error("cannot mix split-stack and non-split-stack in a relocatable link");
return &InputSection::discarded;
}
this->splitStack = true;
return &InputSection::discarded;
}
// An object file cmpiled for split stack, but where some of the
// functions were compiled with the no_split_stack_attribute will
// include a .note.GNU-no-split-stack section.
if (name == ".note.GNU-no-split-stack") {
this->someNoSplitStack = true;
return &InputSection::discarded;
}
// The linkonce feature is a sort of proto-comdat. Some glibc i386 object
// files contain definitions of symbol "__x86.get_pc_thunk.bx" in linkonce
// sections. Drop those sections to avoid duplicate symbol errors.
// FIXME: This is glibc PR20543, we should remove this hack once that has been
// fixed for a while.
if (name == ".gnu.linkonce.t.__x86.get_pc_thunk.bx" ||
name == ".gnu.linkonce.t.__i686.get_pc_thunk.bx")
return &InputSection::discarded;
// If we are creating a new .build-id section, strip existing .build-id
// sections so that the output won't have more than one .build-id.
// This is not usually a problem because input object files normally don't
// have .build-id sections, but you can create such files by
// "ld.{bfd,gold,lld} -r --build-id", and we want to guard against it.
if (name == ".note.gnu.build-id" && config->buildId != BuildIdKind::None)
return &InputSection::discarded;
2016-07-15 12:57:44 +08:00
// The linker merges EH (exception handling) frames and creates a
// .eh_frame_hdr section for runtime. So we handle them with a special
// class. For relocatable outputs, they are just passed through.
if (name == ".eh_frame" && !config->relocatable)
return make<EhInputSection>(*this, sec, name);
2016-07-15 12:57:44 +08:00
if (shouldMerge(sec, name))
return make<MergeInputSection>(*this, sec, name);
return make<InputSection>(*this, sec, name);
}
template <class ELFT>
StringRef ObjFile<ELFT>::getSectionName(const Elf_Shdr &sec) {
return CHECK(getObj().getSectionName(&sec, sectionStringTable), this);
}
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
// Initialize this->Symbols. this->Symbols is a parallel array as
// its corresponding ELF symbol table.
template <class ELFT> void ObjFile<ELFT>::initializeSymbols() {
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
ArrayRef<Elf_Sym> eSyms = this->getELFSyms<ELFT>();
this->symbols.resize(eSyms.size());
// Our symbol table may have already been partially initialized
// because of LazyObjFile.
for (size_t i = 0, end = eSyms.size(); i != end; ++i)
if (!this->symbols[i] && eSyms[i].getBinding() != STB_LOCAL)
this->symbols[i] =
symtab->insert(CHECK(eSyms[i].getName(this->stringTable), this));
// Fill this->Symbols. A symbol is either local or global.
for (size_t i = 0, end = eSyms.size(); i != end; ++i) {
const Elf_Sym &eSym = eSyms[i];
// Read symbol attributes.
uint32_t secIdx = getSectionIndex(eSym);
if (secIdx >= this->sections.size())
fatal(toString(this) + ": invalid section index: " + Twine(secIdx));
[Coding style change] Rename variables so that they start with a lowercase letter This patch is mechanically generated by clang-llvm-rename tool that I wrote using Clang Refactoring Engine just for creating this patch. You can see the source code of the tool at https://reviews.llvm.org/D64123. There's no manual post-processing; you can generate the same patch by re-running the tool against lld's code base. Here is the main discussion thread to change the LLVM coding style: https://lists.llvm.org/pipermail/llvm-dev/2019-February/130083.html In the discussion thread, I proposed we use lld as a testbed for variable naming scheme change, and this patch does that. I chose to rename variables so that they are in camelCase, just because that is a minimal change to make variables to start with a lowercase letter. Note to downstream patch maintainers: if you are maintaining a downstream lld repo, just rebasing ahead of this commit would cause massive merge conflicts because this patch essentially changes every line in the lld subdirectory. But there's a remedy. clang-llvm-rename tool is a batch tool, so you can rename variables in your downstream repo with the tool. Given that, here is how to rebase your repo to a commit after the mass renaming: 1. rebase to the commit just before the mass variable renaming, 2. apply the tool to your downstream repo to mass-rename variables locally, and 3. rebase again to the head. Most changes made by the tool should be identical for a downstream repo and for the head, so at the step 3, almost all changes should be merged and disappear. I'd expect that there would be some lines that you need to merge by hand, but that shouldn't be too many. Differential Revision: https://reviews.llvm.org/D64121 llvm-svn: 365595
2019-07-10 13:00:37 +08:00
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
InputSectionBase *sec = this->sections[secIdx];
uint8_t binding = eSym.getBinding();
uint8_t stOther = eSym.st_other;
uint8_t type = eSym.getType();
uint64_t value = eSym.st_value;
uint64_t size = eSym.st_size;
StringRefZ name = this->stringTable.data() + eSym.st_name;
// Handle local symbols. Local symbols are not added to the symbol
// table because they are not visible from other object files. We
// allocate symbol instances and add their pointers to Symbols.
if (binding == STB_LOCAL) {
if (eSym.getType() == STT_FILE)
sourceFile = CHECK(eSym.getName(this->stringTable), this);
if (this->stringTable.size() <= eSym.st_name)
fatal(toString(this) + ": invalid symbol name offset");
if (eSym.st_shndx == SHN_UNDEF)
this->symbols[i] = make<Undefined>(this, name, binding, stOther, type);
Reland D61583 [ELF] Error on relocations to STT_SECTION symbols if the sections were discarded This restores r361830 "[ELF] Error on relocations to STT_SECTION symbols if the sections were discarded" and dependent commits (r362218, r362497) which were reverted by r364321, with a fix of a --gdb-index issue. .rela.debug_ranges contains relocations of range list entries: // start address of a range list entry // old: 0; after r361830: 0 00000000000033a0 R_X86_64_64 .text._ZN2v88internal7Isolate7factoryEv + 0 // end address of a range list entry // old: 0xe; after r361830: 0 00000000000033a8 R_X86_64_64 .text._ZN2v88internal7Isolate7factoryEv + e If both start and end addresses of a range list entry resolve to 0, DWARFDebugRangeList::isEndOfListEntry() will return true, then the .debug_range decoding loop will terminate prematurely: while (true) { decode StartAddress decode EndAddress if (Entry.isEndOfListEntry()) // prematurely break; Entries.push_back(Entry); } In lld/ELF/SyntheticSections.cpp, readAddressAreas() will read incomplete address ranges and the resulting .gdb_index will be incomplete. For files that gdb hasn't loaded their debug info, gdb uses .gdb_index to map addresses to CUs. The absent entries make gdb fail to symbolize some addresses. To address this issue, we simply allow relocations to undefined symbols in DWARF.cpp:findAux() and let RelocationResolver resolve them. This patch should fix: [1] http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20190603/659848.html [2] https://bugs.chromium.org/p/chromium/issues/detail?id=978067 llvm-svn: 364391
2019-06-26 16:09:08 +08:00
else if (sec == &InputSection::discarded)
this->symbols[i] = make<Undefined>(this, name, binding, stOther, type,
/*DiscardedSecIdx=*/secIdx);
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
else
this->symbols[i] =
make<Defined>(this, name, binding, stOther, type, value, size, sec);
continue;
}
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
// Handle global undefined symbols.
if (eSym.st_shndx == SHN_UNDEF) {
this->symbols[i]->resolve(Undefined{this, name, binding, stOther, type});
[ELF] Make binding (weak or non-weak) logic consistent for Undefined and SharedSymbol This is a case missed by D64136. If %t1.o has a weak reference on foo, and %t2.so has a non-weak reference on foo: ``` 0. ld.lld %t1.o %t2.so # ok; STB_WEAK; accepted since D64136 1. ld.lld %t2.so %t1.o # undefined symbol: foo; STB_GLOBAL 2. gold %t1.o %t2.so # ok; STB_WEAK 3. gold %t2.so %t1.o # undefined reference to 'foo'; STB_GLOBAL 4. ld.bfd %t1.o %t2.so # undefined reference to `foo'; STB_WEAK 5. ld.bfd %t2.so %t1.o # undefined reference to `foo'; STB_WEAK ``` It can be argued that in both cases, the binding of the undefined foo should be set to STB_WEAK, because the binding should not be affected by referenced from shared objects. --allow-shlib-undefined doesn't suppress errors (3,4,5), but -shared or --noinhibit-exec allows ld.bfd/gold to produce a binary: ``` 3. gold -shared %t2.so %t1.o # ok; STB_GLOBAL 4. ld.bfd -shared %t2.so %t1.o # ok; STB_WEAK 5. ld.bfd -shared %t1.o %t1.o # ok; STB_WEAK ``` If %t2.so has DT_NEEDED entries, ld.bfd will load them (lld/gold don't have the behavior). If one of the DSO defines foo and it is in the link-time search path (e.g. DT_NEEDED entry is an absolute path, via -rpath=, via -rpath-link=, etc), `ld.bfd %t1.o %t2.so` and `ld.bfd %t1.o %t2.so` will not error. In this patch, we make Undefined and SharedSymbol share the same binding computing logic. Case 1 will be allowed: ``` 0. ld.lld %t1.o %t2.so # ok; STB_WEAK; accepted since D64136 1. ld.lld %t2.so %t1.o # ok; STB_WEAK; changed by this patch ``` In the future, we can explore the option that turns both (0,1) into errors if --no-allow-shlib-undefined (default when linking an executable) is in action. Reviewed By: ruiu Differential Revision: https://reviews.llvm.org/D65584 llvm-svn: 368038
2019-08-06 22:03:45 +08:00
this->symbols[i]->referenced = true;
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
continue;
}
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
// Handle global common symbols.
if (eSym.st_shndx == SHN_COMMON) {
if (value == 0 || value >= UINT32_MAX)
fatal(toString(this) + ": common symbol '" + StringRef(name.data) +
"' has invalid alignment: " + Twine(value));
this->symbols[i]->resolve(
CommonSymbol{this, name, binding, stOther, type, value, size});
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
continue;
}
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
// If a defined symbol is in a discarded section, handle it as if it
// were an undefined symbol. Such symbol doesn't comply with the
// standard, but in practice, a .eh_frame often directly refer
// COMDAT member sections, and if a comdat group is discarded, some
// defined symbol in a .eh_frame becomes dangling symbols.
if (sec == &InputSection::discarded) {
this->symbols[i]->resolve(
Undefined{this, name, binding, stOther, type, secIdx});
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
continue;
}
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
// Handle global defined symbols.
if (binding == STB_GLOBAL || binding == STB_WEAK ||
binding == STB_GNU_UNIQUE) {
this->symbols[i]->resolve(
Defined{this, name, binding, stOther, type, value, size, sec});
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
continue;
}
fatal(toString(this) + ": unexpected binding: " + Twine((int)binding));
}
}
ArchiveFile::ArchiveFile(std::unique_ptr<Archive> &&file)
: InputFile(ArchiveKind, file->getMemoryBufferRef()),
file(std::move(file)) {}
void ArchiveFile::parse() {
for (const Archive::Symbol &sym : file->symbols())
symtab->addSymbol(LazyArchive{*this, sym});
}
// Returns a buffer pointing to a member file containing a given symbol.
void ArchiveFile::fetch(const Archive::Symbol &sym) {
Archive::Child c =
CHECK(sym.getMember(), toString(this) +
": could not get the member for symbol " +
toELFString(sym));
if (!seen.insert(c.getChildOffset()).second)
return;
MemoryBufferRef mb =
CHECK(c.getMemoryBufferRef(),
toString(this) +
": could not get the buffer for the member defining symbol " +
toELFString(sym));
if (tar && c.getParent()->isThin())
tar->append(relativeToRoot(CHECK(c.getFullName(), this)), mb.getBuffer());
InputFile *file = createObjectFile(mb, getName(), c.getChildOffset());
Add --warn-backrefs to maintain compatibility with other linkers I'm proposing a new command line flag, --warn-backrefs in this patch. The flag and the feature proposed below don't exist in GNU linkers nor the current lld. --warn-backrefs is an option to detect reverse or cyclic dependencies between static archives, and it can be used to keep your program compatible with GNU linkers after you switch to lld. I'll explain the feature and why you may find it useful below. lld's symbol resolution semantics is more relaxed than traditional Unix linkers. Therefore, ld.lld foo.a bar.o succeeds even if bar.o contains an undefined symbol that have to be resolved by some object file in foo.a. Traditional Unix linkers don't allow this kind of backward reference, as they visit each file only once from left to right in the command line while resolving all undefined symbol at the moment of visiting. In the above case, since there's no undefined symbol when a linker visits foo.a, no files are pulled out from foo.a, and because the linker forgets about foo.a after visiting, it can't resolve undefined symbols that could have been resolved otherwise. That lld accepts more relaxed form means (besides it makes more sense) that you can accidentally write a command line or a build file that works only with lld, even if you have a plan to distribute it to wider users who may be using GNU linkers. With --check-library-dependency, you can detect a library order that doesn't work with other Unix linkers. The option is also useful to detect cyclic dependencies between static archives. Again, lld accepts ld.lld foo.a bar.a even if foo.a and bar.a depend on each other. With --warn-backrefs it is handled as an error. Here is how the option works. We assign a group ID to each file. A file with a smaller group ID can pull out object files from an archive file with an equal or greater group ID. Otherwise, it is a reverse dependency and an error. A file outside --{start,end}-group gets a fresh ID when instantiated. All files within the same --{start,end}-group get the same group ID. E.g. ld.lld A B --start-group C D --end-group E A and B form group 0, C, D and their member object files form group 1, and E forms group 2. I think that you can see how this group assignment rule simulates the traditional linker's semantics. Differential Revision: https://reviews.llvm.org/D45195 llvm-svn: 329636
2018-04-10 07:05:48 +08:00
file->groupId = groupId;
parseFile(file);
}
size_t ArchiveFile::getMemberCount() const {
size_t count = 0;
Error err = Error::success();
for (const Archive::Child &c : file->children(err)) {
(void)c;
++count;
}
// This function is used by --print-archive-stats=, where an error does not
// really matter.
consumeError(std::move(err));
return count;
}
unsigned SharedFile::vernauxNum;
// Parse the version definitions in the object file if present, and return a
// vector whose nth element contains a pointer to the Elf_Verdef for version
// identifier n. Version identifiers that are not definitions map to nullptr.
template <typename ELFT>
static std::vector<const void *> parseVerdefs(const uint8_t *base,
const typename ELFT::Shdr *sec) {
if (!sec)
return {};
// We cannot determine the largest verdef identifier without inspecting
// every Elf_Verdef, but both bfd and gold assign verdef identifiers
// sequentially starting from 1, so we predict that the largest identifier
// will be verdefCount.
unsigned verdefCount = sec->sh_info;
std::vector<const void *> verdefs(verdefCount + 1);
// Build the Verdefs array by following the chain of Elf_Verdef objects
// from the start of the .gnu.version_d section.
const uint8_t *verdef = base + sec->sh_offset;
for (unsigned i = 0; i != verdefCount; ++i) {
auto *curVerdef = reinterpret_cast<const typename ELFT::Verdef *>(verdef);
verdef += curVerdef->vd_next;
unsigned verdefIndex = curVerdef->vd_ndx;
verdefs.resize(verdefIndex + 1);
verdefs[verdefIndex] = curVerdef;
}
return verdefs;
}
// Parse SHT_GNU_verneed to properly set the name of a versioned undefined
// symbol. We detect fatal issues which would cause vulnerabilities, but do not
// implement sophisticated error checking like in llvm-readobj because the value
// of such diagnostics is low.
template <typename ELFT>
std::vector<uint32_t> SharedFile::parseVerneed(const ELFFile<ELFT> &obj,
const typename ELFT::Shdr *sec) {
if (!sec)
return {};
std::vector<uint32_t> verneeds;
ArrayRef<uint8_t> data = CHECK(obj.getSectionContents(sec), this);
const uint8_t *verneedBuf = data.begin();
for (unsigned i = 0; i != sec->sh_info; ++i) {
if (verneedBuf + sizeof(typename ELFT::Verneed) > data.end() ||
uintptr_t(verneedBuf) % sizeof(uint32_t) != 0)
fatal(toString(this) + " has an invalid Verneed");
auto *vn = reinterpret_cast<const typename ELFT::Verneed *>(verneedBuf);
const uint8_t *vernauxBuf = verneedBuf + vn->vn_aux;
for (unsigned j = 0; j != vn->vn_cnt; ++j) {
if (vernauxBuf + sizeof(typename ELFT::Vernaux) > data.end() ||
uintptr_t(vernauxBuf) % sizeof(uint32_t) != 0)
fatal(toString(this) + " has an invalid Vernaux");
auto *aux = reinterpret_cast<const typename ELFT::Vernaux *>(vernauxBuf);
if (aux->vna_name >= this->stringTable.size())
fatal(toString(this) + " has a Vernaux with an invalid vna_name");
uint16_t version = aux->vna_other & VERSYM_VERSION;
if (version >= verneeds.size())
verneeds.resize(version + 1);
verneeds[version] = aux->vna_name;
vernauxBuf += aux->vna_next;
}
verneedBuf += vn->vn_next;
}
return verneeds;
}
// We do not usually care about alignments of data in shared object
// files because the loader takes care of it. However, if we promote a
// DSO symbol to point to .bss due to copy relocation, we need to keep
// the original alignment requirements. We infer it in this function.
template <typename ELFT>
static uint64_t getAlignment(ArrayRef<typename ELFT::Shdr> sections,
const typename ELFT::Sym &sym) {
uint64_t ret = UINT64_MAX;
if (sym.st_value)
ret = 1ULL << countTrailingZeros((uint64_t)sym.st_value);
if (0 < sym.st_shndx && sym.st_shndx < sections.size())
ret = std::min<uint64_t>(ret, sections[sym.st_shndx].sh_addralign);
return (ret > UINT32_MAX) ? 0 : ret;
}
// Fully parse the shared object file.
//
// This function parses symbol versions. If a DSO has version information,
// the file has a ".gnu.version_d" section which contains symbol version
// definitions. Each symbol is associated to one version through a table in
// ".gnu.version" section. That table is a parallel array for the symbol
// table, and each table entry contains an index in ".gnu.version_d".
//
// The special index 0 is reserved for VERF_NDX_LOCAL and 1 is for
// VER_NDX_GLOBAL. There's no table entry for these special versions in
// ".gnu.version_d".
//
// The file format for symbol versioning is perhaps a bit more complicated
// than necessary, but you can easily understand the code if you wrap your
// head around the data structure described above.
template <class ELFT> void SharedFile::parse() {
using Elf_Dyn = typename ELFT::Dyn;
using Elf_Shdr = typename ELFT::Shdr;
using Elf_Sym = typename ELFT::Sym;
using Elf_Verdef = typename ELFT::Verdef;
using Elf_Versym = typename ELFT::Versym;
ArrayRef<Elf_Dyn> dynamicTags;
const ELFFile<ELFT> obj = this->getObj<ELFT>();
ArrayRef<Elf_Shdr> sections = CHECK(obj.sections(), this);
const Elf_Shdr *versymSec = nullptr;
const Elf_Shdr *verdefSec = nullptr;
const Elf_Shdr *verneedSec = nullptr;
// Search for .dynsym, .dynamic, .symtab, .gnu.version and .gnu.version_d.
for (const Elf_Shdr &sec : sections) {
switch (sec.sh_type) {
default:
continue;
case SHT_DYNAMIC:
dynamicTags =
CHECK(obj.template getSectionContentsAsArray<Elf_Dyn>(&sec), this);
break;
case SHT_GNU_versym:
versymSec = &sec;
break;
case SHT_GNU_verdef:
verdefSec = &sec;
break;
case SHT_GNU_verneed:
verneedSec = &sec;
break;
}
}
if (versymSec && numELFSyms == 0) {
error("SHT_GNU_versym should be associated with symbol table");
return;
}
// Search for a DT_SONAME tag to initialize this->soName.
for (const Elf_Dyn &dyn : dynamicTags) {
if (dyn.d_tag == DT_NEEDED) {
uint64_t val = dyn.getVal();
if (val >= this->stringTable.size())
fatal(toString(this) + ": invalid DT_NEEDED entry");
dtNeeded.push_back(this->stringTable.data() + val);
} else if (dyn.d_tag == DT_SONAME) {
uint64_t val = dyn.getVal();
2015-10-12 23:49:02 +08:00
if (val >= this->stringTable.size())
fatal(toString(this) + ": invalid DT_SONAME entry");
soName = this->stringTable.data() + val;
}
}
// DSOs are uniquified not by filename but by soname.
DenseMap<StringRef, SharedFile *>::iterator it;
bool wasInserted;
std::tie(it, wasInserted) = symtab->soNames.try_emplace(soName, this);
// If a DSO appears more than once on the command line with and without
// --as-needed, --no-as-needed takes precedence over --as-needed because a
// user can add an extra DSO with --no-as-needed to force it to be added to
// the dependency list.
it->second->isNeeded |= isNeeded;
if (!wasInserted)
return;
sharedFiles.push_back(this);
verdefs = parseVerdefs<ELFT>(obj.base(), verdefSec);
std::vector<uint32_t> verneeds = parseVerneed<ELFT>(obj, verneedSec);
// Parse ".gnu.version" section which is a parallel array for the symbol
// table. If a given file doesn't have a ".gnu.version" section, we use
// VER_NDX_GLOBAL.
size_t size = numELFSyms - firstGlobal;
std::vector<uint16_t> versyms(size, VER_NDX_GLOBAL);
if (versymSec) {
ArrayRef<Elf_Versym> versym =
CHECK(obj.template getSectionContentsAsArray<Elf_Versym>(versymSec),
this)
.slice(firstGlobal);
for (size_t i = 0; i < size; ++i)
versyms[i] = versym[i].vs_index;
}
// System libraries can have a lot of symbols with versions. Using a
// fixed buffer for computing the versions name (foo@ver) can save a
// lot of allocations.
SmallString<0> versionedNameBuffer;
// Add symbols to the symbol table.
ArrayRef<Elf_Sym> syms = this->getGlobalELFSyms<ELFT>();
for (size_t i = 0; i < syms.size(); ++i) {
const Elf_Sym &sym = syms[i];
// ELF spec requires that all local symbols precede weak or global
// symbols in each symbol table, and the index of first non-local symbol
// is stored to sh_info. If a local symbol appears after some non-local
// symbol, that's a violation of the spec.
StringRef name = CHECK(sym.getName(this->stringTable), this);
if (sym.getBinding() == STB_LOCAL) {
2017-12-15 08:07:15 +08:00
warn("found local symbol '" + name +
"' in global part of symbol table in file " + toString(this));
continue;
}
uint16_t idx = versyms[i] & ~VERSYM_HIDDEN;
if (sym.isUndefined()) {
// For unversioned undefined symbols, VER_NDX_GLOBAL makes more sense but
// as of binutils 2.34, GNU ld produces VER_NDX_LOCAL.
if (idx != VER_NDX_LOCAL && idx != VER_NDX_GLOBAL) {
if (idx >= verneeds.size()) {
error("corrupt input file: version need index " + Twine(idx) +
" for symbol " + name + " is out of bounds\n>>> defined in " +
toString(this));
continue;
}
StringRef verName = this->stringTable.data() + verneeds[idx];
versionedNameBuffer.clear();
name =
saver.save((name + "@" + verName).toStringRef(versionedNameBuffer));
}
Symbol *s = symtab->addSymbol(
Undefined{this, name, sym.getBinding(), sym.st_other, sym.getType()});
s->exportDynamic = true;
continue;
}
// MIPS BFD linker puts _gp_disp symbol into DSO files and incorrectly
// assigns VER_NDX_LOCAL to this section global symbol. Here is a
// workaround for this bug.
if (config->emachine == EM_MIPS && idx == VER_NDX_LOCAL &&
name == "_gp_disp")
continue;
uint32_t alignment = getAlignment<ELFT>(sections, sym);
if (!(versyms[i] & VERSYM_HIDDEN)) {
symtab->addSymbol(SharedSymbol{*this, name, sym.getBinding(),
sym.st_other, sym.getType(), sym.st_value,
sym.st_size, alignment, idx});
}
// Also add the symbol with the versioned name to handle undefined symbols
// with explicit versions.
if (idx == VER_NDX_GLOBAL)
continue;
if (idx >= verdefs.size() || idx == VER_NDX_LOCAL) {
error("corrupt input file: version definition index " + Twine(idx) +
" for symbol " + name + " is out of bounds\n>>> defined in " +
toString(this));
continue;
}
StringRef verName =
this->stringTable.data() +
reinterpret_cast<const Elf_Verdef *>(verdefs[idx])->getAux()->vda_name;
versionedNameBuffer.clear();
name = (name + "@" + verName).toStringRef(versionedNameBuffer);
symtab->addSymbol(SharedSymbol{*this, saver.save(name), sym.getBinding(),
sym.st_other, sym.getType(), sym.st_value,
sym.st_size, alignment, idx});
}
}
static ELFKind getBitcodeELFKind(const Triple &t) {
if (t.isLittleEndian())
return t.isArch64Bit() ? ELF64LEKind : ELF32LEKind;
return t.isArch64Bit() ? ELF64BEKind : ELF32BEKind;
}
static uint8_t getBitcodeMachineKind(StringRef path, const Triple &t) {
switch (t.getArch()) {
2016-07-07 10:46:30 +08:00
case Triple::aarch64:
return EM_AARCH64;
case Triple::amdgcn:
case Triple::r600:
return EM_AMDGPU;
2016-07-07 10:46:30 +08:00
case Triple::arm:
case Triple::thumb:
2016-07-07 10:46:30 +08:00
return EM_ARM;
case Triple::avr:
return EM_AVR;
2016-07-07 10:46:30 +08:00
case Triple::mips:
case Triple::mipsel:
case Triple::mips64:
case Triple::mips64el:
return EM_MIPS;
case Triple::msp430:
return EM_MSP430;
2016-07-07 10:46:30 +08:00
case Triple::ppc:
return EM_PPC;
case Triple::ppc64:
case Triple::ppc64le:
2016-07-07 10:46:30 +08:00
return EM_PPC64;
case Triple::riscv32:
case Triple::riscv64:
return EM_RISCV;
2016-07-07 10:46:30 +08:00
case Triple::x86:
return t.isOSIAMCU() ? EM_IAMCU : EM_386;
2016-07-07 10:46:30 +08:00
case Triple::x86_64:
return EM_X86_64;
default:
error(path + ": could not infer e_machine from bitcode target triple " +
t.str());
return EM_NONE;
}
}
BitcodeFile::BitcodeFile(MemoryBufferRef mb, StringRef archiveName,
uint64_t offsetInArchive)
: InputFile(BitcodeKind, mb) {
this->archiveName = std::string(archiveName);
std::string path = mb.getBufferIdentifier().str();
if (config->thinLTOIndexOnly)
path = replaceThinLTOSuffix(mb.getBufferIdentifier());
// ThinLTO assumes that all MemoryBufferRefs given to it have a unique
// name. If two archives define two members with the same name, this
// causes a collision which result in only one of the objects being taken
// into consideration at LTO time (which very likely causes undefined
// symbols later in the link stage). So we append file offset to make
// filename unique.
StringRef name =
archiveName.empty()
? saver.save(path)
: saver.save(archiveName + "(" + path::filename(path) + " at " +
utostr(offsetInArchive) + ")");
MemoryBufferRef mbref(mb.getBuffer(), name);
obj = CHECK(lto::InputFile::create(mbref), this);
Triple t(obj->getTargetTriple());
ekind = getBitcodeELFKind(t);
emachine = getBitcodeMachineKind(mb.getBufferIdentifier(), t);
}
static uint8_t mapVisibility(GlobalValue::VisibilityTypes gvVisibility) {
switch (gvVisibility) {
case GlobalValue::DefaultVisibility:
return STV_DEFAULT;
case GlobalValue::HiddenVisibility:
return STV_HIDDEN;
case GlobalValue::ProtectedVisibility:
return STV_PROTECTED;
}
llvm_unreachable("unknown visibility");
}
ELF: New symbol table design. This patch implements a new design for the symbol table that stores SymbolBodies within a memory region of the Symbol object. Symbols are mutated by constructing SymbolBodies in place over existing SymbolBodies, rather than by mutating pointers. As mentioned in the initial proposal [1], this memory layout helps reduce the cache miss rate by improving memory locality. Performance numbers: old(s) new(s) Without debug info: chrome 7.178 6.432 (-11.5%) LLVMgold.so 0.505 0.502 (-0.5%) clang 0.954 0.827 (-15.4%) llvm-as 0.052 0.045 (-15.5%) With debug info: scylla 5.695 5.613 (-1.5%) clang 14.396 14.143 (-1.8%) Performance counter results show that the fewer required indirections is indeed the cause of the improved performance. For example, when linking chrome, stalled cycles decreases from 14,556,444,002 to 12,959,238,310, and instructions per cycle increases from 0.78 to 0.83. We are also executing many fewer instructions (15,516,401,933 down to 15,002,434,310), probably because we spend less time allocating SymbolBodies. The new mechanism by which symbols are added to the symbol table is by calling add* functions on the SymbolTable. In this patch, I handle local symbols by storing them inside "unparented" SymbolBodies. This is suboptimal, but if we do want to try to avoid allocating these SymbolBodies, we can probably do that separately. I also removed a few members from the SymbolBody class that were only being used to pass information from the input file to the symbol table. This patch implements the new design for the ELF linker only. I intend to prepare a similar patch for the COFF linker. [1] http://lists.llvm.org/pipermail/llvm-dev/2016-April/098832.html Differential Revision: http://reviews.llvm.org/D19752 llvm-svn: 268178
2016-05-01 12:55:03 +08:00
template <class ELFT>
static Symbol *createBitcodeSymbol(const std::vector<bool> &keptComdats,
const lto::InputFile::Symbol &objSym,
BitcodeFile &f) {
StringRef name = saver.save(objSym.getName());
uint8_t binding = objSym.isWeak() ? STB_WEAK : STB_GLOBAL;
uint8_t type = objSym.isTLS() ? STT_TLS : STT_NOTYPE;
uint8_t visibility = mapVisibility(objSym.getVisibility());
bool canOmitFromDynSym = objSym.canBeOmittedFromSymbolTable();
[Coding style change] Rename variables so that they start with a lowercase letter This patch is mechanically generated by clang-llvm-rename tool that I wrote using Clang Refactoring Engine just for creating this patch. You can see the source code of the tool at https://reviews.llvm.org/D64123. There's no manual post-processing; you can generate the same patch by re-running the tool against lld's code base. Here is the main discussion thread to change the LLVM coding style: https://lists.llvm.org/pipermail/llvm-dev/2019-February/130083.html In the discussion thread, I proposed we use lld as a testbed for variable naming scheme change, and this patch does that. I chose to rename variables so that they are in camelCase, just because that is a minimal change to make variables to start with a lowercase letter. Note to downstream patch maintainers: if you are maintaining a downstream lld repo, just rebasing ahead of this commit would cause massive merge conflicts because this patch essentially changes every line in the lld subdirectory. But there's a remedy. clang-llvm-rename tool is a batch tool, so you can rename variables in your downstream repo with the tool. Given that, here is how to rebase your repo to a commit after the mass renaming: 1. rebase to the commit just before the mass variable renaming, 2. apply the tool to your downstream repo to mass-rename variables locally, and 3. rebase again to the head. Most changes made by the tool should be identical for a downstream repo and for the head, so at the step 3, almost all changes should be merged and disappear. I'd expect that there would be some lines that you need to merge by hand, but that shouldn't be too many. Differential Revision: https://reviews.llvm.org/D64121 llvm-svn: 365595
2019-07-10 13:00:37 +08:00
int c = objSym.getComdatIndex();
if (objSym.isUndefined() || (c != -1 && !keptComdats[c])) {
Undefined newSym(&f, name, binding, visibility, type);
if (canOmitFromDynSym)
newSym.exportDynamic = false;
Symbol *ret = symtab->addSymbol(newSym);
ret->referenced = true;
return ret;
}
if (objSym.isCommon())
return symtab->addSymbol(
CommonSymbol{&f, name, binding, visibility, STT_OBJECT,
objSym.getCommonAlignment(), objSym.getCommonSize()});
Defined newSym(&f, name, binding, visibility, type, 0, 0, nullptr);
if (canOmitFromDynSym)
newSym.exportDynamic = false;
return symtab->addSymbol(newSym);
}
template <class ELFT> void BitcodeFile::parse() {
std::vector<bool> keptComdats;
for (StringRef s : obj->getComdatTable())
keptComdats.push_back(
symtab->comdatGroups.try_emplace(CachedHashStringRef(s), this).second);
for (const lto::InputFile::Symbol &objSym : obj->symbols())
symbols.push_back(createBitcodeSymbol<ELFT>(keptComdats, objSym, *this));
[ELF] Implement Dependent Libraries Feature This patch implements a limited form of autolinking primarily designed to allow either the --dependent-library compiler option, or "comment lib" pragmas ( https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) in C/C++ e.g. #pragma comment(lib, "foo"), to cause an ELF linker to automatically add the specified library to the link when processing the input file generated by the compiler. Currently this extension is unique to LLVM and LLD. However, care has been taken to design this feature so that it could be supported by other ELF linkers. The design goals were to provide: - A simple linking model for developers to reason about. - The ability to to override autolinking from the linker command line. - Source code compatibility, where possible, with "comment lib" pragmas in other environments (MSVC in particular). Dependent library support is implemented differently for ELF platforms than on the other platforms. Primarily this difference is that on ELF we pass the dependent library specifiers directly to the linker without manipulating them. This is in contrast to other platforms where they are mapped to a specific linker option by the compiler. This difference is a result of the greater variety of ELF linkers and the fact that ELF linkers tend to handle libraries in a more complicated fashion than on other platforms. This forces us to defer handling the specifiers to the linker. In order to achieve a level of source code compatibility with other platforms we have restricted this feature to work with libraries that meet the following "reasonable" requirements: 1. There are no competing defined symbols in a given set of libraries, or if they exist, the program owner doesn't care which is linked to their program. 2. There may be circular dependencies between libraries. The binary representation is a mergeable string section (SHF_MERGE, SHF_STRINGS), called .deplibs, with custom type SHT_LLVM_DEPENDENT_LIBRARIES (0x6fff4c04). The compiler forms this section by concatenating the arguments of the "comment lib" pragmas and --dependent-library options in the order they are encountered. Partial (-r, -Ur) links are handled by concatenating .deplibs sections with the normal mergeable string section rules. As an example, #pragma comment(lib, "foo") would result in: .section ".deplibs","MS",@llvm_dependent_libraries,1 .asciz "foo" For LTO, equivalent information to the contents of a the .deplibs section can be retrieved by the LLD for bitcode input files. LLD processes the dependent library specifiers in the following way: 1. Dependent libraries which are found from the specifiers in .deplibs sections of relocatable object files are added when the linker decides to include that file (which could itself be in a library) in the link. Dependent libraries behave as if they were appended to the command line after all other options. As a consequence the set of dependent libraries are searched last to resolve symbols. 2. It is an error if a file cannot be found for a given specifier. 3. Any command line options in effect at the end of the command line parsing apply to the dependent libraries, e.g. --whole-archive. 4. The linker tries to add a library or relocatable object file from each of the strings in a .deplibs section by; first, handling the string as if it was specified on the command line; second, by looking for the string in each of the library search paths in turn; third, by looking for a lib<string>.a or lib<string>.so (depending on the current mode of the linker) in each of the library search paths. 5. A new command line option --no-dependent-libraries tells LLD to ignore the dependent libraries. Rationale for the above points: 1. Adding the dependent libraries last makes the process simple to understand from a developers perspective. All linkers are able to implement this scheme. 2. Error-ing for libraries that are not found seems like better behavior than failing the link during symbol resolution. 3. It seems useful for the user to be able to apply command line options which will affect all of the dependent libraries. There is a potential problem of surprise for developers, who might not realize that these options would apply to these "invisible" input files; however, despite the potential for surprise, this is easy for developers to reason about and gives developers the control that they may require. 4. This algorithm takes into account all of the different ways that ELF linkers find input files. The different search methods are tried by the linker in most obvious to least obvious order. 5. I considered adding finer grained control over which dependent libraries were ignored (e.g. MSVC has /nodefaultlib:<library>); however, I concluded that this is not necessary: if finer control is required developers can fall back to using the command line directly. RFC thread: http://lists.llvm.org/pipermail/llvm-dev/2019-March/131004.html. Differential Revision: https://reviews.llvm.org/D60274 llvm-svn: 360984
2019-05-17 11:44:15 +08:00
for (auto l : obj->getDependentLibraries())
addDependentLibrary(l, this);
}
void BinaryFile::parse() {
ArrayRef<uint8_t> data = arrayRefFromStringRef(mb.getBuffer());
auto *section = make<InputSection>(this, SHF_ALLOC | SHF_WRITE, SHT_PROGBITS,
8, data, ".data");
sections.push_back(section);
// For each input file foo that is embedded to a result as a binary
// blob, we define _binary_foo_{start,end,size} symbols, so that
// user programs can access blobs by name. Non-alphanumeric
// characters in a filename are replaced with underscore.
std::string s = "_binary_" + mb.getBufferIdentifier().str();
for (size_t i = 0; i < s.size(); ++i)
if (!isAlnum(s[i]))
s[i] = '_';
[Coding style change] Rename variables so that they start with a lowercase letter This patch is mechanically generated by clang-llvm-rename tool that I wrote using Clang Refactoring Engine just for creating this patch. You can see the source code of the tool at https://reviews.llvm.org/D64123. There's no manual post-processing; you can generate the same patch by re-running the tool against lld's code base. Here is the main discussion thread to change the LLVM coding style: https://lists.llvm.org/pipermail/llvm-dev/2019-February/130083.html In the discussion thread, I proposed we use lld as a testbed for variable naming scheme change, and this patch does that. I chose to rename variables so that they are in camelCase, just because that is a minimal change to make variables to start with a lowercase letter. Note to downstream patch maintainers: if you are maintaining a downstream lld repo, just rebasing ahead of this commit would cause massive merge conflicts because this patch essentially changes every line in the lld subdirectory. But there's a remedy. clang-llvm-rename tool is a batch tool, so you can rename variables in your downstream repo with the tool. Given that, here is how to rebase your repo to a commit after the mass renaming: 1. rebase to the commit just before the mass variable renaming, 2. apply the tool to your downstream repo to mass-rename variables locally, and 3. rebase again to the head. Most changes made by the tool should be identical for a downstream repo and for the head, so at the step 3, almost all changes should be merged and disappear. I'd expect that there would be some lines that you need to merge by hand, but that shouldn't be too many. Differential Revision: https://reviews.llvm.org/D64121 llvm-svn: 365595
2019-07-10 13:00:37 +08:00
symtab->addSymbol(Defined{nullptr, saver.save(s + "_start"), STB_GLOBAL,
STV_DEFAULT, STT_OBJECT, 0, 0, section});
symtab->addSymbol(Defined{nullptr, saver.save(s + "_end"), STB_GLOBAL,
STV_DEFAULT, STT_OBJECT, data.size(), 0, section});
symtab->addSymbol(Defined{nullptr, saver.save(s + "_size"), STB_GLOBAL,
STV_DEFAULT, STT_OBJECT, data.size(), 0, nullptr});
}
InputFile *elf::createObjectFile(MemoryBufferRef mb, StringRef archiveName,
uint64_t offsetInArchive) {
if (isBitcode(mb))
return make<BitcodeFile>(mb, archiveName, offsetInArchive);
switch (getELFKind(mb, archiveName)) {
case ELF32LEKind:
return make<ObjFile<ELF32LE>>(mb, archiveName);
case ELF32BEKind:
return make<ObjFile<ELF32BE>>(mb, archiveName);
case ELF64LEKind:
return make<ObjFile<ELF64LE>>(mb, archiveName);
case ELF64BEKind:
return make<ObjFile<ELF64BE>>(mb, archiveName);
default:
llvm_unreachable("getELFKind");
}
}
void LazyObjFile::fetch() {
if (mb.getBuffer().empty())
return;
Add --warn-backrefs to maintain compatibility with other linkers I'm proposing a new command line flag, --warn-backrefs in this patch. The flag and the feature proposed below don't exist in GNU linkers nor the current lld. --warn-backrefs is an option to detect reverse or cyclic dependencies between static archives, and it can be used to keep your program compatible with GNU linkers after you switch to lld. I'll explain the feature and why you may find it useful below. lld's symbol resolution semantics is more relaxed than traditional Unix linkers. Therefore, ld.lld foo.a bar.o succeeds even if bar.o contains an undefined symbol that have to be resolved by some object file in foo.a. Traditional Unix linkers don't allow this kind of backward reference, as they visit each file only once from left to right in the command line while resolving all undefined symbol at the moment of visiting. In the above case, since there's no undefined symbol when a linker visits foo.a, no files are pulled out from foo.a, and because the linker forgets about foo.a after visiting, it can't resolve undefined symbols that could have been resolved otherwise. That lld accepts more relaxed form means (besides it makes more sense) that you can accidentally write a command line or a build file that works only with lld, even if you have a plan to distribute it to wider users who may be using GNU linkers. With --check-library-dependency, you can detect a library order that doesn't work with other Unix linkers. The option is also useful to detect cyclic dependencies between static archives. Again, lld accepts ld.lld foo.a bar.a even if foo.a and bar.a depend on each other. With --warn-backrefs it is handled as an error. Here is how the option works. We assign a group ID to each file. A file with a smaller group ID can pull out object files from an archive file with an equal or greater group ID. Otherwise, it is a reverse dependency and an error. A file outside --{start,end}-group gets a fresh ID when instantiated. All files within the same --{start,end}-group get the same group ID. E.g. ld.lld A B --start-group C D --end-group E A and B form group 0, C, D and their member object files form group 1, and E forms group 2. I think that you can see how this group assignment rule simulates the traditional linker's semantics. Differential Revision: https://reviews.llvm.org/D45195 llvm-svn: 329636
2018-04-10 07:05:48 +08:00
InputFile *file = createObjectFile(mb, archiveName, offsetInArchive);
Add --warn-backrefs to maintain compatibility with other linkers I'm proposing a new command line flag, --warn-backrefs in this patch. The flag and the feature proposed below don't exist in GNU linkers nor the current lld. --warn-backrefs is an option to detect reverse or cyclic dependencies between static archives, and it can be used to keep your program compatible with GNU linkers after you switch to lld. I'll explain the feature and why you may find it useful below. lld's symbol resolution semantics is more relaxed than traditional Unix linkers. Therefore, ld.lld foo.a bar.o succeeds even if bar.o contains an undefined symbol that have to be resolved by some object file in foo.a. Traditional Unix linkers don't allow this kind of backward reference, as they visit each file only once from left to right in the command line while resolving all undefined symbol at the moment of visiting. In the above case, since there's no undefined symbol when a linker visits foo.a, no files are pulled out from foo.a, and because the linker forgets about foo.a after visiting, it can't resolve undefined symbols that could have been resolved otherwise. That lld accepts more relaxed form means (besides it makes more sense) that you can accidentally write a command line or a build file that works only with lld, even if you have a plan to distribute it to wider users who may be using GNU linkers. With --check-library-dependency, you can detect a library order that doesn't work with other Unix linkers. The option is also useful to detect cyclic dependencies between static archives. Again, lld accepts ld.lld foo.a bar.a even if foo.a and bar.a depend on each other. With --warn-backrefs it is handled as an error. Here is how the option works. We assign a group ID to each file. A file with a smaller group ID can pull out object files from an archive file with an equal or greater group ID. Otherwise, it is a reverse dependency and an error. A file outside --{start,end}-group gets a fresh ID when instantiated. All files within the same --{start,end}-group get the same group ID. E.g. ld.lld A B --start-group C D --end-group E A and B form group 0, C, D and their member object files form group 1, and E forms group 2. I think that you can see how this group assignment rule simulates the traditional linker's semantics. Differential Revision: https://reviews.llvm.org/D45195 llvm-svn: 329636
2018-04-10 07:05:48 +08:00
file->groupId = groupId;
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
mb = {};
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
// Copy symbol vector so that the new InputFile doesn't have to
// insert the same defined symbols to the symbol table again.
file->symbols = std::move(symbols);
parseFile(file);
}
template <class ELFT> void LazyObjFile::parse() {
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
using Elf_Sym = typename ELFT::Sym;
// A lazy object file wraps either a bitcode file or an ELF file.
if (isBitcode(this->mb)) {
std::unique_ptr<lto::InputFile> obj =
CHECK(lto::InputFile::create(this->mb), this);
for (const lto::InputFile::Symbol &sym : obj->symbols()) {
if (sym.isUndefined())
continue;
symtab->addSymbol(LazyObject{*this, saver.save(sym.getName())});
}
return;
}
if (getELFKind(this->mb, archiveName) != config->ekind) {
error("incompatible file: " + this->mb.getBufferIdentifier());
return;
}
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
// Find a symbol table.
ELFFile<ELFT> obj = check(ELFFile<ELFT>::create(mb.getBuffer()));
ArrayRef<typename ELFT::Shdr> sections = CHECK(obj.sections(), this);
for (const typename ELFT::Shdr &sec : sections) {
if (sec.sh_type != SHT_SYMTAB)
continue;
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
// A symbol table is found.
ArrayRef<Elf_Sym> eSyms = CHECK(obj.symbols(&sec), this);
uint32_t firstGlobal = sec.sh_info;
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
StringRef strtab = CHECK(obj.getStringTableForSymtab(sec, sections), this);
this->symbols.resize(eSyms.size());
// Get existing symbols or insert placeholder symbols.
for (size_t i = firstGlobal, end = eSyms.size(); i != end; ++i)
if (eSyms[i].st_shndx != SHN_UNDEF)
this->symbols[i] = symtab->insert(CHECK(eSyms[i].getName(strtab), this));
// Replace existing symbols with LazyObject symbols.
//
// resolve() may trigger this->fetch() if an existing symbol is an
// undefined symbol. If that happens, this LazyObjFile has served
// its purpose, and we can exit from the loop early.
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
for (Symbol *sym : this->symbols) {
if (!sym)
continue;
sym->resolve(LazyObject{*this, sym->getName()});
// MemoryBuffer is emptied if this file is instantiated as ObjFile.
if (mb.getBuffer().empty())
Speed up --start-lib and --end-lib. --{start,end}-lib give files grouped by the options the archive file semantics. That is, each object file between them acts as if it were in an archive file whose sole member is the file. Therefore, files between --{start,end}-lib are linked to the final output only if they are needed to resolve some undefined symbols. Previously, the feature was implemented this way: 1. We read a symbol table and insert defined symbols to the symbol table as lazy symbols. 2. If an undefind symbol is resolved to a lazy symbol, that lazy symbol instantiate ObjFile class for that symbol, which re-insert all defined symbols to the symbol table. So, if an ObjFile is instantiated, defined symbols are inserted to the symbol table twice. Since inserting long symbol names is not cheap, there's a room to optimize here. This patch optimzies it. Now, LazyObjFile remembers symbol handles and passed them over to a new ObjFile instance, so that the ObjFile doesn't insert the same strings. Here is a quick benchmark to link clang. "Original" is the original lld with unmodified command line options. For "Case 1" and "Case 2", I extracted all files from archive files and replace .a's in a command line with .o's wrapped with --{start,end}-lib. I used the original lld for Case 1" and use this patch for Case 2. Original: 5.892 Case 1: 6.001 (+1.8%) Case 2: 5.701 (-3.2%) So, interestingly, --{start,end}-lib are now faster than the regular linking scheme with archive files. That's perhaps not too surprising, though, because for regular archive files, we look up the symbol table with the same string twice. Differential Revision: https://reviews.llvm.org/D62188 llvm-svn: 361473
2019-05-23 17:53:30 +08:00
return;
}
return;
}
}
std::string elf::replaceThinLTOSuffix(StringRef path) {
StringRef suffix = config->thinLTOObjectSuffixReplace.first;
StringRef repl = config->thinLTOObjectSuffixReplace.second;
if (path.consume_back(suffix))
return (path + repl).str();
return std::string(path);
}
template void BitcodeFile::parse<ELF32LE>();
template void BitcodeFile::parse<ELF32BE>();
template void BitcodeFile::parse<ELF64LE>();
template void BitcodeFile::parse<ELF64BE>();
ELF: New symbol table design. This patch implements a new design for the symbol table that stores SymbolBodies within a memory region of the Symbol object. Symbols are mutated by constructing SymbolBodies in place over existing SymbolBodies, rather than by mutating pointers. As mentioned in the initial proposal [1], this memory layout helps reduce the cache miss rate by improving memory locality. Performance numbers: old(s) new(s) Without debug info: chrome 7.178 6.432 (-11.5%) LLVMgold.so 0.505 0.502 (-0.5%) clang 0.954 0.827 (-15.4%) llvm-as 0.052 0.045 (-15.5%) With debug info: scylla 5.695 5.613 (-1.5%) clang 14.396 14.143 (-1.8%) Performance counter results show that the fewer required indirections is indeed the cause of the improved performance. For example, when linking chrome, stalled cycles decreases from 14,556,444,002 to 12,959,238,310, and instructions per cycle increases from 0.78 to 0.83. We are also executing many fewer instructions (15,516,401,933 down to 15,002,434,310), probably because we spend less time allocating SymbolBodies. The new mechanism by which symbols are added to the symbol table is by calling add* functions on the SymbolTable. In this patch, I handle local symbols by storing them inside "unparented" SymbolBodies. This is suboptimal, but if we do want to try to avoid allocating these SymbolBodies, we can probably do that separately. I also removed a few members from the SymbolBody class that were only being used to pass information from the input file to the symbol table. This patch implements the new design for the ELF linker only. I intend to prepare a similar patch for the COFF linker. [1] http://lists.llvm.org/pipermail/llvm-dev/2016-April/098832.html Differential Revision: http://reviews.llvm.org/D19752 llvm-svn: 268178
2016-05-01 12:55:03 +08:00
template void LazyObjFile::parse<ELF32LE>();
template void LazyObjFile::parse<ELF32BE>();
template void LazyObjFile::parse<ELF64LE>();
template void LazyObjFile::parse<ELF64BE>();
ELF: New symbol table design. This patch implements a new design for the symbol table that stores SymbolBodies within a memory region of the Symbol object. Symbols are mutated by constructing SymbolBodies in place over existing SymbolBodies, rather than by mutating pointers. As mentioned in the initial proposal [1], this memory layout helps reduce the cache miss rate by improving memory locality. Performance numbers: old(s) new(s) Without debug info: chrome 7.178 6.432 (-11.5%) LLVMgold.so 0.505 0.502 (-0.5%) clang 0.954 0.827 (-15.4%) llvm-as 0.052 0.045 (-15.5%) With debug info: scylla 5.695 5.613 (-1.5%) clang 14.396 14.143 (-1.8%) Performance counter results show that the fewer required indirections is indeed the cause of the improved performance. For example, when linking chrome, stalled cycles decreases from 14,556,444,002 to 12,959,238,310, and instructions per cycle increases from 0.78 to 0.83. We are also executing many fewer instructions (15,516,401,933 down to 15,002,434,310), probably because we spend less time allocating SymbolBodies. The new mechanism by which symbols are added to the symbol table is by calling add* functions on the SymbolTable. In this patch, I handle local symbols by storing them inside "unparented" SymbolBodies. This is suboptimal, but if we do want to try to avoid allocating these SymbolBodies, we can probably do that separately. I also removed a few members from the SymbolBody class that were only being used to pass information from the input file to the symbol table. This patch implements the new design for the ELF linker only. I intend to prepare a similar patch for the COFF linker. [1] http://lists.llvm.org/pipermail/llvm-dev/2016-April/098832.html Differential Revision: http://reviews.llvm.org/D19752 llvm-svn: 268178
2016-05-01 12:55:03 +08:00
template class elf::ObjFile<ELF32LE>;
template class elf::ObjFile<ELF32BE>;
template class elf::ObjFile<ELF64LE>;
template class elf::ObjFile<ELF64BE>;
template void SharedFile::parse<ELF32LE>();
template void SharedFile::parse<ELF32BE>();
template void SharedFile::parse<ELF64LE>();
template void SharedFile::parse<ELF64BE>();