Commit Graph

259 Commits

Author SHA1 Message Date
Zachary Turner c762666e87 Resubmit [pdb] Change /DEBUG:GHASH to emit 8 byte hashes."
This fixes the remaining failing tests, so resubmitting with no
functional change.

llvm-svn: 332676
2018-05-17 22:55:15 +00:00
Zachary Turner 1de9fce151 Revert "[pdb] Change /DEBUG:GHASH to emit 8 byte hashes."
A few tests haven't been properly updated, so reverting while
I have time to investigate proper fixes.

llvm-svn: 332672
2018-05-17 21:49:25 +00:00
Zachary Turner 3c4c8a0937 [pdb] Change /DEBUG:GHASH to emit 8 byte hashes.
Previously we emitted 20-byte SHA1 hashes.  This is overkill
for identifying debug info records, and has the negative side
effect of making object files bigger and links slower.  By
using only the last 8 bytes of a SHA1, we get smaller object
files and ~10% faster links.

This modifies the format of the .debug$H section by adding a new
value for the hash algorithm field, so that the linker will still
work when its object files have an old format.

Differential Revision: https://reviews.llvm.org/D46855

llvm-svn: 332669
2018-05-17 21:22:48 +00:00
Reid Kleckner f40f85868e [codeview] Include record prefix in global type hashing
The prefix includes type kind, which is important to preserve. Two
different type leafs can easily have the same interior record contents
as another type.

We ran into this issue in PR37492 where a bitfield type record collided
with a const modifier record. Their contents were bitwise identical, but
their kinds were different.

llvm-svn: 332664
2018-05-17 20:47:22 +00:00
Brock Wyma 94ece8fbc9 [CodeView] Initial support for emitting S_THUNK32 symbols for compiler...
When emitting CodeView debug information, compiler-generated thunk routines
should be emitted using S_THUNK32 symbols instead of S_GPROC32_ID symbols so
Visual Studio can properly step into the user code.  This initial support only
handles standard thunk ordinals.

Differential Revision: https://reviews.llvm.org/D43838

llvm-svn: 330132
2018-04-16 16:53:57 +00:00
Mandeep Singh Grang 0f035ebed2 [DebugInfo] Change std::sort to llvm::sort in response to r327219
r327219 added wrappers to std::sort which randomly shuffle the container before
sorting.  This will help in uncovering non-determinism caused due to undefined
sorting order of objects having the same key.

To make use of that infrastructure we need to invoke llvm::sort instead of
std::sort.

Note: This patch is one of a series of patches to replace *all* std::sort to
llvm::sort.  Refer the comments section in D44363 for a list of all the
required patches.

llvm-svn: 330061
2018-04-13 19:50:51 +00:00
Alexandre Ganea 08df84e4f0 [DebugInfo][COFF] Fix reading variable-length encoded records
While reading Codeview records which contain variable-length encoded integers,
such as LF_BCLASS, LF_ENUMERATE, LF_MEMBER, LF_VBCLASS or LF_IVBCLASS,
the record's size would be improperly calculated in cases where the value was
indeed of a variable length (>= LF_NUMERIC). This caused a bad alignement on
the next record, which would/might crash later on.

Differential Revision: https://reviews.llvm.org/D45104

llvm-svn: 329659
2018-04-10 01:58:45 +00:00
Alexandre Ganea 3241cec577 Fix line endings (CR/LF -> LF) introduced by rL329613
reviewer: zturner
llvm-svn: 329646
2018-04-10 00:09:15 +00:00
Alexandre Ganea d9e96741c4 [Debuginfo][COFF] Minimal serialization support for precompiled types records
This change adds support for the LF_PRECOMP and LF_ENDPRECOMP records required
to read/write Microsoft precompiled types .objs.
See https://en.wikipedia.org/wiki/Precompiled_header#Microsoft_Visual_C_and_C++

This also adds handling for the .debug$P section, which is actually a .debug$T
section in disguise, found only in precompiled .objs.

Differential Revision: https://reviews.llvm.org/D45283

llvm-svn: 329613
2018-04-09 20:17:56 +00:00
Mandeep Singh Grang fe1d28e83d [DebugInfo] Change std::sort to llvm::sort in response to r327219
Summary:
r327219 added wrappers to std::sort which randomly shuffle the container before sorting.
This will help in uncovering non-determinism caused due to undefined sorting
order of objects having the same key.

To make use of that infrastructure we need to invoke llvm::sort instead of std::sort.

Note: This patch is one of a series of patches to replace *all* std::sort to llvm::sort.
Refer the comments section in D44363 for a list of all the required patches.

Reviewers: echristo, zturner, samsonov

Reviewed By: echristo

Subscribers: JDevlieghere, llvm-commits

Differential Revision: https://reviews.llvm.org/D45134

llvm-svn: 328935
2018-04-01 16:18:49 +00:00
Zachary Turner a6fb536e5b [PDB] Make our PDBs look more like MS PDBs.
When investigating bugs in PDB generation, the first step is
often to do the same link with link.exe and then compare PDBs.

But comparing PDBs is hard because two completely different byte
sequences can both be correct, so it hampers the investigation when
you also have to spend time figuring out not just which bytes are
different, but also if the difference is meaningful.

This patch fixes a couple of cases related to string table emission,
hash table emission, and the order in which we emit strings that
makes more of our bytes the same as the bytes generated by MS PDBs.

Differential Revision: https://reviews.llvm.org/D44810

llvm-svn: 328348
2018-03-23 18:43:39 +00:00
Zachary Turner 71d36ad9f9 [Codeview/PDB] Rename some methods for clarity.
NFC, this just renames some methods to better express what they
do, and also adds a few helper methods to add some symmetry to the
API in a few places (for example there was a getStringFromId but not
a getIdFromString method in the string table).

llvm-svn: 328221
2018-03-22 17:37:28 +00:00
Zachary Turner fced530650 Revert "Resubmit "Support embedding natvis files in PDBs.""
This is still failing on a different bot this time due to some
issue related to hashing absolute paths.  Reverting until I can
figure it out.

llvm-svn: 328014
2018-03-20 18:37:03 +00:00
Zachary Turner 132d7a134f Resubmit "Support embedding natvis files in PDBs."
The issue causing this to fail in certain configurations
should be fixed.

It was due to the fact that DIA apparently expects there to be
a null string at ID 1 in the string table.  I'm not sure why this
is important but it seems to make a difference, so set it.

llvm-svn: 328002
2018-03-20 17:06:39 +00:00
Zachary Turner a21558897b Revert "Support embedding natvis files in PDBs."
This is causing a test failure on a certain bot, so I'm removing
this temporarily until we can figure out the source of the error.

llvm-svn: 327903
2018-03-19 20:41:59 +00:00
Zachary Turner de53aaf132 Support embedding natvis files in PDBs.
Natvis is a debug language supported by Visual Studio for
specifying custom visualizers.  The /NATVIS option is an
undocumented link.exe flag which will take a .natvis file
and "inject" it into the PDB.  This way, you can ship the
debug visualizers for a program along with the PDB, which
is very useful for postmortem debugging.

This is implemented by adding a new "named stream" to the
PDB with a special name of /src/files/<natvis file name>
and simply copying the contents of the xml into this file.

Additionally, we need to emit a single stream named
/src/headerblock which contains a hash table of embedded
files to records describing them.

This patch adds this functionality, including the /NATVIS
option to lld-link.

Differential Revision: https://reviews.llvm.org/D44328

llvm-svn: 327895
2018-03-19 19:53:51 +00:00
Reid Kleckner 3acdc67734 [CodeView] Lower __restrict and other pointer qualifiers correctly
Qualifiers on a pointer or reference type may apply to either the
pointee or the pointer itself. Consider 'const char *' and 'char *
const'. In the first example, the pointee data may not be modified
without casts, and in the second example, the pointer may not be updated
to point to new data.

In the general case, qualifiers are applied to types with LF_MODIFIER
records, which support the usual const and volatile qualifiers as well
as the __unaligned extension qualifier.

However, LF_POINTER records, which are used for pointers, references,
and member pointers, have flags for qualifiers applying to the
*pointer*. In fact, this is the only way to represent the restrict
qualifier, which can only apply to pointers, and cannot qualify regular
data types.

This patch causes LLVM to correctly fold 'const' and 'volatile' pointer
qualifiers into the pointer record, as well as adding support for
'__restrict' qualifiers in the same place.

Based on a patch from Aaron Smith

Differential Revision: https://reviews.llvm.org/D43060

llvm-svn: 326260
2018-02-27 22:08:15 +00:00
Reid Kleckner 22d838cd31 [codeview] Remove unused variable
llvm-svn: 326253
2018-02-27 21:46:40 +00:00
Zachary Turner 07d803777c [CodeView] Micro-optimizations to speed up type merging.
Based on a profile, a couple of hot spots were identified in the
main type merging loop.  The code was simplified, a few loops
were re-arranged, and some outlined functions were inlined.  This
speeds up type merging by a decent amount, shaving around 3-4 seconds
off of a 40 second link in my test case.

Differential Revision: https://reviews.llvm.org/D42559

llvm-svn: 323790
2018-01-30 17:12:04 +00:00
Zachary Turner 1bc2ce6b9b Speed up iteration of CodeView record streams.
There's some abstraction overhead in the underlying
mechanisms that were being used, and it was leading to an
abundance of small but not-free copies being made.  This
showed up on a profile.  Eliminating this and going back to
a low-level byte-based implementation speeds up lld with
/DEBUG between 10 and 15%.

Differential Revision: https://reviews.llvm.org/D42148

llvm-svn: 322871
2018-01-18 18:35:01 +00:00
Zachary Turner 0d07a8e948 [COFF] Teach LLD to use the COFF .debug$H section.
This adds the /DEBUG:GHASH option to LLD which will look for
the existence of .debug$H sections in linker inputs and use them
to accelerate type merging.  The clang-cl side has already been
added, so this completes the work necessary to begin experimenting
with this feature.

Differential Revision: https://reviews.llvm.org/D40980

llvm-svn: 320719
2017-12-14 18:07:04 +00:00
Zachary Turner 048f8f99bf [CodeView] Teach clang to emit the .debug$H COFF section.
Currently this is an LLVM extension to the COFF spec which is
experimental and intended to speed up linking.  For now it is
behind a hidden cl::opt flag, but in the future we can move it
to a "real" cc1 flag and have the driver pass it through whenever
it is appropriate.

The patch to actually make use of this section in lld will come
in a followup.

Differential Revision: https://reviews.llvm.org/D40917

llvm-svn: 320649
2017-12-13 22:33:58 +00:00
Michael Zolotukhin 0c169bf7f7 Remove redundant includes from lib/DebugInfo.
llvm-svn: 320620
2017-12-13 21:30:49 +00:00
Zachary Turner ecd2684ed7 [DebugInfo] Fix register variables not showing up in pdb.
Previously, when linking against libcmt from the MSVC runtime,
lld-link /verbose would show "Ignoring unknown symbol record
with kind 0x1006".  It turns out this was because
TypeIndexDiscovery did not handle S_REGISTER records, so these
records were not getting properly remapped.

Patch by: Alexnadre Ganea
Differential Revision: https://reviews.llvm.org/D40919

llvm-svn: 320108
2017-12-07 22:51:16 +00:00
Zachary Turner 376d437776 Teach llvm-pdbutil to dump types from object files.
llvm-svn: 319859
2017-12-05 23:58:18 +00:00
Zachary Turner 023f88ef42 Fix -Wmissing-braces error.
llvm-svn: 319855
2017-12-05 23:19:33 +00:00
Zachary Turner 87b78e9d1e [CodeView] Add support for content hashing CodeView type records.
Currently nothing uses this, but this at least gets the core
algorithm in, and adds some test to demonstrate correctness.

Differential Revision: https://reviews.llvm.org/D40736

llvm-svn: 319854
2017-12-05 23:08:58 +00:00
Zachary Turner f0e4c6a819 Simplify the DenseSet used for hashing CodeView records.
This was storing the hash alongside the key so that the hash
doesn't need to be re-computed every time, but in doing so it
was allocating a structure to keep the key size small in the
DenseMap.  This is a noble goal, but it also leads to a pointer
indirection on every probe, and this cost of this pointer
indirection ends up being higher than the cost of having a
slightly larger entry in the hash table.  Removing this not only
simplifies the code, but yields a small but noticeable
performance improvement in the type merging algorithm.

llvm-svn: 319493
2017-11-30 23:00:30 +00:00
Zachary Turner ca6dbf1440 Split TypeTableBuilder into two classes.
llvm-svn: 319456
2017-11-30 18:39:50 +00:00
Zachary Turner 52d036e693 [CodeView] Factor some code out of TypeTableBuilder.
This class had some code that would automatically remap type
indices before hashing and serializing.  The only caller of
this method was the TypeStreamMerger anyway, and the method
doesn't make general sense, and prevents making certain future
improvements to the class.  So, factoring this up one level
into the TypeStreamMerger where it belongs.

llvm-svn: 319377
2017-11-29 22:41:56 +00:00
Zachary Turner 3e3936da93 Make TypeTableBuilder inherit from TypeCollection.
A couple of places in LLD were passing references to
TypeTableCollections around, which makes it hard to change the
implementation at runtime.  However, these cases only needed to
iterate over the types in the collection, and TypeCollection
already provides a handy abstract interface for this purpose.

By implementing this interface, we can get rid of the need to
pass TypeTableBuilder references around, which should allow us
to swap the implementation at runtime in subsequent patches.

llvm-svn: 319345
2017-11-29 19:35:21 +00:00
Zachary Turner 4c1fa68590 Fix a warning.
llvm-svn: 319263
2017-11-29 00:13:44 +00:00
Zachary Turner 29b081dcd1 [NFC] Minor cleanups in CodeView TypeTableBuilder.
llvm-svn: 319260
2017-11-28 23:57:13 +00:00
Rafael Espindola bba7f862d8 Fix non assert build warnings.
llvm-svn: 319200
2017-11-28 18:50:08 +00:00
Zachary Turner 6900de1dfb [CodeView] Refactor / Rewrite TypeSerializer and TypeTableBuilder.
The motivation behind this patch is that future directions require us to
be able to compute the hash value of records independently of actually
using them for de-duplication.

The current structure of TypeSerializer / TypeTableBuilder being a
single entry point that takes an unserialized type record, and then
hashes and de-duplicates it is not flexible enough to allow this.

At the same time, the existing TypeSerializer is already extremely
complex for this very reason -- it tries to be too many things. In
addition to serializing, hashing, and de-duplicating, ti also supports
splitting up field list records and adding continuations. All of this
functionality crammed into this one class makes it very complicated to
work with and hard to maintain.

To solve all of these problems, I've re-written everything from scratch
and split the functionality into separate pieces that can easily be
reused. The end result is that one class TypeSerializer is turned into 3
new classes SimpleTypeSerializer, ContinuationRecordBuilder, and
TypeTableBuilder, each of which in isolation is simple and
straightforward.

A quick summary of these new classes and their responsibilities are:

- SimpleTypeSerializer : Turns a non-FieldList leaf type into a series of
  bytes. Does not do any hashing. Every time you call it, it will
  re-serialize and return bytes again. The same instance can be re-used
  over and over to avoid re-allocations, and in exchange for this
  optimization the bytes returned by the serializer only live until the
  caller attempts to serialize a new record.

- ContinuationRecordBuilder : Turns a FieldList-like record into a series
  of fragments. Does not do any hashing. Like SimpleTypeSerializer,
  returns references to privately owned bytes, so the storage is
  invalidated as soon as the caller tries to re-use the instance. Works
  equally well for LF_FIELDLIST as it does for LF_METHODLIST, solving a
  long-standing theoretical limitation of the previous implementation.

- TypeTableBuilder : Accepts sequences of bytes that the user has already
  serialized, and inserts them by de-duplicating with a hash table. For
  the sake of convenience and efficiency, this class internally stores a
  SimpleTypeSerializer so that it can accept unserialized records. The
  same is not true of ContinuationRecordBuilder. The user is required to
  create their own instance of ContinuationRecordBuilder.

Differential Revision: https://reviews.llvm.org/D40518

llvm-svn: 319198
2017-11-28 18:33:17 +00:00
Reid Kleckner 8aa32ffbad [codeview] Fix handling of S_HEAPALLOCSITE
The type index is from the TPI stream, not the IPI stream. Fix the
dumper, fix type index discovery, and add a test in LLD.

Also improve the log message we emit when we fail to rewrite type
indices in LLD. That's how I found this bug.

llvm-svn: 316461
2017-10-24 17:02:40 +00:00
Reid Kleckner 0e88118dd7 [codeview] Add support for inlinee lists
This adds type index discovery and dumper support for symbol record kind
0x1168, which is a list of inlined function ids. This symbol kind is
undocumented, but S_INLINEES is consistent with the existing
nomenclature.

Fixes PR34222

llvm-svn: 316398
2017-10-23 23:43:40 +00:00
Reid Kleckner ecddee27a8 [codeview] Recognize two records with no type index fields
Thunk records do not have types and frame cookies do not have types.

These were found while linking libconcrt.lib from MSVC.

llvm-svn: 316385
2017-10-23 22:44:24 +00:00
Hans Wennborg 660531085a CodeView: Provide a .def file with the register ids
The list of register ids was previously written out in a couple of dirrent
places. This puts it in a .def file and also adds a few more registers (e.g.
the x87 regs) which should lead to more readable dumps, but I didn't include
the whole list since that seems unnecessary.

X86_MC::initLLVMToSEHAndCVRegMapping is pretty ugly, but at least it's not
relying on magic constants anymore. The TODO of using tablegen still stands.

Differential revision: https://reviews.llvm.org/D38480

llvm-svn: 314821
2017-10-03 18:27:22 +00:00
Hans Wennborg ea89ff7c25 CodeView symbol dumper: use symbolic names for registers
https://reviews.llvm.org/D38469

llvm-svn: 314690
2017-10-02 17:44:47 +00:00
Zachary Turner abb17cc084 [llvm-pdbutil] Support dumping CodeView from object files.
We have llvm-readobj for dumping CodeView from object files, and
llvm-pdbutil has always been more focused on PDB.  However,
llvm-pdbutil has a lot of useful options for summarizing debug
information in aggregate and presenting high level statistical
views.  Furthermore, it's arguably better as a testing tool since
we don't have to write tests to conform to a state-machine like
structure where you match multiple lines in succession, each
depending on a previous match.  llvm-pdbutil dumps much more
concisely, so it's possible to use single-line matches in many
cases where as with readobj tests you have to use multi-line
matches with an implicit state machine.

Because of this, I'm adding object file support to llvm-pdbutil.
In fact, this mirrors the cvdump tool from Microsoft, which also
supports both object files and pdb files.  In the future we could
perhaps rename this tool llvm-cvutil.

In the meantime, this allows us to deep dive into object files
the same way we already can with PDB files.

llvm-svn: 312358
2017-09-01 20:06:56 +00:00
Zachary Turner 5641c07d6b [PDB] Serialize records into a stack-allocated buffer.
We were using a std::vector<> and resizing to MaxRecordLength,
which is ~64KB.  We would then do this repeatedly often many
times in a tight loop, which was causing measurable performance
impact when linking PDBs.

Patch by Alex Telishev
Differential Revision: https://reviews.llvm.org/D36940

llvm-svn: 311375
2017-08-21 20:17:19 +00:00
Zachary Turner 197bba0028 Remove unused variable.
llvm-svn: 311119
2017-08-17 20:18:36 +00:00
Zachary Turner 96bcd6a37a [llvm-pdbutil] Fix some dumping issues.
When dumping, we were treating the S_INLINESITESYM as referring
to a type record, when it actually refers to an id record.  We
had this correct in TypeIndexDiscovery, so our merging algorithm
should be fine, but we had it wrong in the dumper, which means it
would appear to work most of the time, unless the index was out
of bounds in the type stream, when it would fail.  Fixed this, and
audited a few other cases to make them match the behavior in
TypeIndexDiscovery.

Also, I've now observed a new symbol record with kind 0x1168 which
I have no clue what it is, so to avoid crashing we have to just
print "Unknown Symbol Kind".

llvm-svn: 311117
2017-08-17 20:04:51 +00:00
Zachary Turner ee9906d884 [LLD/PDB] Write actual records to the globals stream.
Previously we were writing an empty globals stream.  Windows
tools interpret this as "private symbols are not present in
this PDB", even when they are, so we need to fix this.  Regardless,
without it we don't have information about global variables, so
we need to fix it anyway.  This patch does that.

With this patch, the "lm" command in WinDbg correctly reports
that we have private symbols available, but the "dv" command
still refuses to display local variables.

Differential Revision: https://reviews.llvm.org/D36535

llvm-svn: 310743
2017-08-11 19:00:03 +00:00
Zachary Turner 59e3ae827d [PDB] Fix linking of function symbols and local variables.
The compiler outputs PROC32_ID symbols into the object files
for functions, and these symbols have an embedded type index
which, when copied to the PDB, refer to the IPI stream.  However,
the symbols themselves are also converted into regular symbols
(e.g. S_GPROC32_ID -> S_GPROC32), and type indices in the regular
symbol records refer to the TPI stream.  So this patch applies
two fixes to function records.
  1. It converts ID symbols to the proper non-ID record type.
  2. After remapping the type index from the object file's index
     space to the PDB file/IPI stream's index space, it then
     remaps that index to the TPI stream's index space by.

Besides functions, during the remapping process we were also
discarding symbol record types which we did not recognize.
In particular, we were discarding S_BPREL32 records, which is
what MSVC uses to describe local variables on the stack.  So
this patch fixes that as well by copying them to the PDB.

Differential Revision: https://reviews.llvm.org/D36426

llvm-svn: 310394
2017-08-08 18:34:44 +00:00
Reid Kleckner 14d90fd05c [PDB] Improve GSI hash table dumping for publics and globals
The PDB "symbol stream" actually contains symbol records for the publics
and the globals stream. The globals and publics streams are essentially
hash tables that point into a single stream of records. In order to
match cvdump's behavior, we need to only dump symbol records referenced
from the hash table. This patch implements that, and then implements
global stream dumping, since it's just a subset of public stream
dumping.

Now we shouldn't see S_PROCREF or S_GDATA32 records when dumping
publics, and instead we should see those record in the globals stream.

llvm-svn: 309066
2017-07-26 00:40:36 +00:00
Reid Kleckner 898ddf61c0 [codeview] Emit 'D' as the cv source language for D code
This matches DMD:
522263965c/src/ddmd/backend/cv8.c (L199)

Fixes PR33899.

llvm-svn: 308890
2017-07-24 16:16:42 +00:00
Reid Kleckner 67653ee086 [codeview] Fix YAML for LF_TYPESERVER2 by hoisting PDB_UniqueId
Summary:
We were treating the GUIDs in TypeServer2Record as strings, and the
non-ASCII bytes in the GUID would not round-trip through YAML.

We already had the PDB_UniqueId type portably represent a Windows GUID,
but we need to hoist that up to the DebugInfo/CodeView library so that
we can use it in the TypeServer2Record as well as in PDB parsing code.

Reviewers: inglorion, amccarth

Subscribers: llvm-commits, hiraditya

Differential Revision: https://reviews.llvm.org/D35495

llvm-svn: 308234
2017-07-17 23:59:44 +00:00
Reid Kleckner 3167480ca6 [codeview] Don't use the type visitor to merge types
Summary:
This didn't do much to speed things up, but it implements a FIXME, and I
think it's a nice simplification. We don't need the record kind switch.
We're doing that ourselves.

Reviewers: ruiu, inglorion

Subscribers: llvm-commits, hiraditya

Differential Revision: https://reviews.llvm.org/D35496

llvm-svn: 308213
2017-07-17 20:31:38 +00:00