Commit Graph

43 Commits

Author SHA1 Message Date
Puyan Lotfi c38e42527b [clang][IFS][test] Temporary work around for in-process cc1 ASAN issues.
When using in-process cc1, the Clang Interface Stubs pipeline setup
exposes an ASAN bug. I am still investigating this issue but want to
green the bots for now. I don't think this is a huge issue since the
Clang Interface Stubs Driver setup code is the only code path that sets
up such a pipeline (ie N cc1's for N c files followed by another N cc1's
for to generate stub files for the same N c files).

This issue is being discussed in https://reviews.llvm.org/D69825.
If a resolution is not found soon, a bugzilla filling will be in order.
2020-01-21 22:56:20 -05:00
Fangrui Song 80146fc13a [test] clang/test/InterfaceStubs/externstatic.c requires x86-registered-target 2020-01-18 09:54:35 -08:00
Fangrui Song d0038012fd [InterfaceStubs][test] Add -triple to clang/test/InterfaceStubs/externstatic.c to make it robust
llvm-nm on Linux prints 0 line while llvm-nm on macOS prints 1 line.
2020-01-17 10:32:20 -08:00
Fangrui Song a9f0025acd Reland "[llvm-nm] Don't report "no symbols" error for files that contain symbols" 2020-01-17 10:08:42 -08:00
Fangrui Song 932b5d6fca [test] Fix tests after D52810 2020-01-17 10:02:56 -08:00
Nico Weber 84baf123a5 try to fix InterfaceStubs/lambda.cpp on Windows after bd8c8827d9 2020-01-13 21:22:46 -05:00
Puyan Lotfi a506f7f910 [clang][IFS][test] Fixing mangled name of a test for Darwin.
Darwin adds an extra '_' before every C/global function mangled name and
because of this, this test was breaking on Darwin.

This is a fix for commit: https://reviews.llvm.org/D71301
2020-01-13 17:01:48 -05:00
Puyan Lotfi bd8c8827d9 [clang][IFS] Prevent Clang-IFS from Leaking symbols from inside a block.
Built libdispatch with clang interface stubs. Ran into some block
related issues. Basically VarDecl symbols can leak out because I wasn't
checking the case where a VarDecl is contained inside a BlockDecl
(versus a method or function).

This patch checks that a VarDecl is not a child decl of a BlockDecl.
This patch also does something very similar for c++ lambdas as well.

Differential Revision: https://reviews.llvm.org/D71301
2020-01-13 16:04:27 -05:00
Puyan Lotfi d694594d76 [clang][IFS] Allow 2 output files when using -o and -c with clang IFS stubs.
This patch allows for -o to be used with -c when compiling with clang
interface stubs enabled. This is because the second file will be an
intermediate ifs stubs file that is the text stub analog of the .o file.
Both get produces in this case, so two files.

Why are we doing this? Because we want to support the case where
interface stubs are used bu first invoking clang like so:

clang -c <other flags> -emit-interface-stubs foo.c -o foo.o
...
clang -emit-interface-stubs <.o files> -o libfoo.so

This should generate N .ifs files, and one .ifso file. Prior to this
patch, using -o with the -c invocation was not possible. Currently the
clang driver supports generating a a.out/.so file at the same time as a
merged ifs file / ifso file, but this is done by checking that the final
job is the IfsMerge job. When -c is used, the final job is a Compile job
so what this patch does is check to figure out of the job type is
TY_IFS_CPP.

Differential Revision: https://reviews.llvm.org/D70763
2019-12-09 14:47:17 -05:00
Puyan Lotfi 4ef9110b8d [clang][IFS] Ignoring -Xlinker/-Xclang arguments in InterfaceStubs pass for now.
Many of the inputs to clang driver toolchain tools can be things other
than files such as -Xclang and -Xlinker arguments. For now we don't take
such tool pass-through type arguments (although having an -Xifs would be nice to
have to replace things like -emit-merged-ifs). So because for the moment
we are not doing any sort of argument pass-through, I am going to have
InterfaceStubs Merger ignore any non-file type input arguments.

Differential Revision: https://reviews.llvm.org/D70838
2019-12-04 20:03:57 -05:00
Puyan Lotfi 07e445103e [clang][IFS] Adding support for new clang interface stubs decl types.
NamespaceAliasDecl UnresolvedUsingTypenameDecl CXXDeductionGuideDecl
ConstructorUsingShadowDecl

Differential Revision: https://reviews.llvm.org/D70728
2019-12-04 19:10:47 -05:00
Puyan Lotfi 73429126c9 [clang][IFS] Driver Pipeline: generate stubs after standard pipeline (3)
Third Landing Attempt (dropping any linker invocation from clang driver):

Up until now, clang interface stubs has replaced the standard
PP -> C -> BE -> ASM -> LNK pipeline. With this change, it will happen in
conjunction with it. So what when you build your code you will get an
a.out or lib.so as well as an interface stub file.

Example:

clang -shared -o libfoo.so -emit-interface-stubs ...

will generate both a libfoo.so and a libfoo.ifso. The .so file will
contain the code from the standard compilation pipeline and the .ifso
file will contain the ELF stub library.

Note: For driver-test.c I've added -S in order to prevent any bot failures on
bots that don't have the proper linker for their native triple. You could always
specify a triple like x86_64-unknown-linux-gnu and on bots like x86_64-scei-ps4
the clang driver would invoke regular ld instead of getting the error
'Executable "orbis-ld" doesn't exist!' but on bots like ppc64be and s390x you'd
get an error "/usr/bin/ld: unrecognised emulation mode: elf_x86_64"

Differential Revision: https://reviews.llvm.org/D70274
2019-11-20 16:22:50 -05:00
Puyan Lotfi 9059854355 Revert "[clang][IFS] Driver Pipeline: generate stubs after standard pipeline (2)"
This reverts commit ea8e028223.
2019-11-19 21:59:50 -05:00
Puyan Lotfi 85a1662035 Revert "[clang][IFS] Fixing failing bots that do not have PPC target or "orbis-ld""
This reverts commit 377d70cdea.
2019-11-19 21:59:39 -05:00
Puyan Lotfi 0ce89e2a0d Revert "[clang][IFS] Attempting to fix missing 'orbis-ld' on scei-ps4-ubuntu bot."
This reverts commit 29fd1e1f4a.
2019-11-19 21:59:31 -05:00
Puyan Lotfi 6c6d34883a Revert "[clang][IFS] Fixing unsupported emulation mode on clang-ppc64be-linux bot."
This reverts commit 1b387484b9.
2019-11-19 21:59:22 -05:00
Puyan Lotfi 27b229dc17 Revert "[clang][IFS][test] Removing driver-test.c. Test is still too brittle."
This reverts commit f37356d6f6.
2019-11-19 21:59:10 -05:00
Puyan Lotfi f37356d6f6 [clang][IFS][test] Removing driver-test.c. Test is still too brittle.
Removing this test because if I add a triple then there are link falures
on targets like ppc and s390x. If I don't add a triple then on PS4
targets the clang driver tries to invoke orbis-ld which ends up being
not found.
2019-11-19 21:42:17 -05:00
Puyan Lotfi 1b387484b9 [clang][IFS] Fixing unsupported emulation mode on clang-ppc64be-linux bot.
I am in another pickle here where if I specify a triple, I get the wrong elf
target arch on the PPC bot (error from the PPC elf Linker). To avoid this I am
going to turn this test off on the PPC bots for now.
2019-11-19 19:27:46 -05:00
Puyan Lotfi 29fd1e1f4a [clang][IFS] Attempting to fix missing 'orbis-ld' on scei-ps4-ubuntu bot.
I want this test to run end to end, but I am still having trouble with
missing linkers on the scei-ps4 bot. Will remove this test if it
continues to be a source of brittle failures. Sorry for the noise.
2019-11-19 18:34:49 -05:00
Puyan Lotfi 377d70cdea [clang][IFS] Fixing failing bots that do not have PPC target or "orbis-ld" 2019-11-19 18:12:07 -05:00
Puyan Lotfi ea8e028223 [clang][IFS] Driver Pipeline: generate stubs after standard pipeline (2)
Second Landing Attempt:

Up until now, clang interface stubs has replaced the standard
PP -> C -> BE -> ASM -> LNK pipeline. With this change, it will happen in
conjunction with it. So what when you build your code you will get an
a.out or lib.so as well as an interface stub file.

Example:

clang -shared -o libfoo.so -emit-interface-stubs ...

will generate both a libfoo.so and a libfoo.ifso. The .so file will
contain the code from the standard compilation pipeline and the .ifso
file will contain the ELF stub library.

Differential Revision: https://reviews.llvm.org/D70274
2019-11-19 17:47:38 -05:00
Puyan Lotfi d044dcc5e4 Revert "[clang][IFS] Driver pipeline: generate interface stubs after standard pipeline."
This reverts commit 58ea00b51f.

Test for .o + .ifs sidecar files is brittle and failing on bots.
Reverting to unblock.
2019-11-19 02:08:22 -05:00
Puyan Lotfi 58ea00b51f [clang][IFS] Driver pipeline: generate interface stubs after standard pipeline.
Up until now, clang interface stubs has replaced the standard
PP -> C -> BE -> ASM -> LNK pipeline. With this change, it will happen in
conjunction with it. So what when you build your code you will get an
a.out or lib.so as well as an interface stub file.

Example:

clang -shared -o libfoo.so -emit-interface-stubs ...

will generate both a libfoo.so and a libfoo.ifso. The .so file will
contain the code from the standard compilation pipeline and the .ifso
file will contain the ELF stub library.

Differential Revision: https://reviews.llvm.org/D70274
2019-11-19 01:18:02 -05:00
Puyan Lotfi 4873319f97 [clang][IFS][test] Reverting driver-test.c to what it was prior to e782192d5e
My mistake. Changes I had in this test were for code changes that are
not landed yet. I am reverting driver-test.c back to what it was
originally.
2019-11-14 03:39:16 -05:00
Puyan Lotfi c72aef999c [clang][IFS][test] Removing -shared, fixing triple: driver-test.c/windows.cpp.
Removing -shared as it is not used on a lot of targets in order to green failing
bots with this change. Also, tiding up the windows.cpp test as the
triple compile out can look slightly different that what you specified
on a windows bot.
2019-11-14 02:43:40 -05:00
Puyan Lotfi e782192d5e [clang][IFS][test][NFC] Tightening up clang-ifs tests to use -cc1 more often.
Unless the test is explicitly testing a driver feature if clang
interface stubs I have changed the tests to use %clang_cc1. This should
make some changes I plan to make to the driver job pipeline cause fewer
test changes and breakages.
2019-11-14 02:25:22 -05:00
Puyan Lotfi 1257146eb4 [clang][IFS][test] Fixing buildbot test fails for clang-ifs.
Checking for the exact triple fails on many bots. Leaving the triple
check blank.
2019-11-08 21:06:23 -05:00
Puyan Lotfi 79e345fbcc [clang][IFS] Adds support for more decl types in clang interface stubs.
Adding support for processing the following Decls: NonTypeTemplateParmDecl,
CXXConversionDecl, UnresolvedUsingValueDecl, UsingDecl, UsingShadowDecl,
TypeAliasTemplateDecl, TypeAliasDecl, VarTemplateDecl,
VarTemplateSpecializationDecl, UsingDirectiveDecl, TemplateTemplateParmDecl,
ClassTemplatePartialSpecializationDecl, IndirectFieldDecl.

Also, this allows for processing NamedDecls that don't have an identifier and
skips over VarDecls that are dependent on template types.

Differential Revision: https://reviews.llvm.org/D69995
2019-11-08 20:27:25 -05:00
Puyan Lotfi 4e4b4f4099 [clang][IFS][test] Fixing lit test breakages on macOS due to r374798
Adding the quotes breaks tests because on Darwin the name mangling is prefixed
with an underscore.

llvm-svn: 374805
2019-10-14 18:57:29 +00:00
Puyan Lotfi 76f9869bf2 [clang][IFS] Escape mangled names so MS ABI doesn't break YAML parsing.
Microsoft's ABI mangles names differently than Itanium and this breaks the LLVM
yaml parser unless the name is escaped in quotes. Quotes are being added to the
mangled names of the IFS file generation so that llvm-ifs doesn't break when
Windows triples are passed to the driver.

Differential Revision: https://reviews.llvm.org/D68915

llvm-svn: 374798
2019-10-14 18:03:03 +00:00
Puyan Lotfi c0abc2e7f2 [clang][IFS] Updating tests to pass on -fvisibility=hidden builds (NFCi).
Special thanks to JamesNagurne who got to the bottom of this; landing this on
his behalf.

Differential Revision: https://reviews.llvm.org/D68897

llvm-svn: 374632
2019-10-12 02:46:57 +00:00
Puyan Lotfi e3388c42f3 [clang][IFS] Fixing assert in clang interface stubs for enums, records, typedefs
The clang IFS ASTConsumer was asserting on enums, records (struct definitions in
C), and typedefs. All it needs to do is skip them because the stub just needs to
expose global object instances and functions.

Differential Revision: https://reviews.llvm.org/D68859

llvm-svn: 374573
2019-10-11 17:24:11 +00:00
Puyan Lotfi c382d03ca8 [clang][ifs] Clang Interface Stubs ToolChain plumbing.
Second Landing Attempt:

This patch enables end to end support for generating ELF interface stubs
directly from clang. Now the following:

clang -emit-interface-stubs -o libfoo.so a.cpp b.cpp c.cpp

will product an ELF binary with visible symbols populated. Visibility attributes
and -fvisibility can be used to control what gets populated.

* Adding ToolChain support for clang Driver IFS Merge Phase
* Implementing a default InterfaceStubs Merge clang Tool, used by ToolChain
* Adds support for the clang Driver to involve llvm-ifs on ifs files.
* Adds -emit-merged-ifs flag, to tell llvm-ifs to emit a merged ifs text file
  instead of the final object format (normally ELF)


Differential Revision: https://reviews.llvm.org/D63978

llvm-svn: 374061
2019-10-08 15:23:14 +00:00
Nico Weber 6713f8235b Revert 373538 and follow-ups 373549 and 373552.
They break tests on (at least) macOS.

llvm-svn: 373556
2019-10-03 02:38:43 +00:00
Puyan Lotfi 30c8df02ba Fixing broken builds due to r373538, issues with filepath and hexagon toolchain.
It appears there are some issues with the hexagon toolchain, and also the file
path for the library file. If this doesn't fix the remaining breakages I will
attempt a revert.

llvm-svn: 373552
2019-10-03 01:19:51 +00:00
Puyan Lotfi 8581f860c7 Fixing broken builds due to r373538 due to test that should have been deleted.
test/InterfaceStubs/object.cpp should have been deleted.

llvm-svn: 373549
2019-10-03 00:41:13 +00:00
Puyan Lotfi 406de17b9b [clang][ifs] Clang Interface Stubs ToolChain plumbing.
This patch enables end to end support for generating ELF interface stubs
directly from clang. Now the following:

clang -emit-interface-stubs -o libfoo.so a.cpp b.cpp c.cpp

will product an ELF binary with visible symbols populated. Visibility attributes
and -fvisibility can be used to control what gets populated.

* Adding ToolChain support for clang Driver IFS Merge Phase
* Implementing a default InterfaceStubs Merge clang Tool, used by ToolChain
* Adds support for the clang Driver to involve llvm-ifs on ifs files.
* Adds -emit-merged-ifs flag, to tell llvm-ifs to emit a merged ifs text file
  instead of the final object format (normally ELF)


Differential Revision: https://reviews.llvm.org/D63978

llvm-svn: 373538
2019-10-02 22:50:07 +00:00
Puyan Lotfi 926f4f76c3 [clang][ifs] Dropping older experimental interface stub formats.
I've been working on a new tool, llvm-ifs, for merging interface stub files
generated by clang and I've iterated on my derivative format of TBE to a newer
format. llvm-ifs will only support the new format, so I am going to drop the
older experimental interface stubs formats in this commit to make things
simpler.

Differential Revision: https://reviews.llvm.org/D66573

llvm-svn: 369719
2019-08-22 23:44:34 +00:00
Puyan Lotfi 678e19d80c [NFC] Fix for InterfaceStubs tests (adding REQUIRES: x86-registered-target).
clang-hexagon-elf bot was failing with:

'No available targets are compatible with triple "x86_64-unknown-linux-gnu"'

Adding a "// REQUIRES: x86-registered-target" to these tests to quiet the bot.

llvm-svn: 363963
2019-06-20 18:28:21 +00:00
Puyan Lotfi 68f29dac4b [clang-ifs] Clang Interface Stubs, first version (second landing attempt).
This change reverts r363649; effectively re-landing r363626. At this point
clang::Index::CodegenNameGeneratorImpl has been refactored into
clang::AST::ASTNameGenerator. This makes it so that the previous circular link
dependency no longer exists, fixing the previous share lib
(-DBUILD_SHARED_LIBS=ON) build issue which was the reason for r363649.

Clang interface stubs (previously referred to as clang-ifsos) is a new frontend
action in clang that allows the generation of stub files that contain mangled
name info that can be used to produce a stub library. These stub libraries can
be useful for breaking up build dependencies and controlling access to a
library's internal symbols. Generation of these stubs can be invoked by:

clang -fvisibility=<visibility> -emit-interface-stubs \
                                -interface-stub-version=<interface format>

Notice that -fvisibility (along with use of visibility attributes) can be used
to control what symbols get generated. Currently the interface format is
experimental but there are a wide range of possibilities here.

Currently clang-ifs produces .ifs files that can be thought of as analogous to
object (.o) files, but just for the mangled symbol info. In a subsequent patch
I intend to add support for merging the .ifs files into one .ifs/.ifso file
that can be the input to something like llvm-elfabi to produce something like a
.so file or .dll (but without any of the code, just symbols).

Differential Revision: https://reviews.llvm.org/D60974

llvm-svn: 363948
2019-06-20 16:59:48 +00:00
Fangrui Song 2d94dd812f Revert D60974 "[clang-ifs] Clang Interface Stubs, first version."
This reverts commit rC363626.

clangIndex depends on clangFrontend. r363626 adds a dependency from
clangFrontend to clangIndex, which creates a circular dependency.

This is disallowed by -DBUILD_SHARED_LIBS=on builds:

    CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
      "clangFrontend" of type SHARED_LIBRARY
        depends on "clangIndex" (weak)
      "clangIndex" of type SHARED_LIBRARY
        depends on "clangFrontend" (weak)
    At least one of these targets is not a STATIC_LIBRARY.  Cyclic dependencies are allowed only among static libraries.

Note, the dependency on clangIndex cannot be removed because
libclangFrontend.so is linked with -Wl,-z,defs: a shared object must
have its full direct dependencies specified on the linker command line.

In -DBUILD_SHARED_LIBS=off builds, this appears to work when linking
`bin/clang-9`. However, it can cause trouble to downstream clang library
users. The llvm build system links libraries this way:

    clang main_program_object_file ... lib/libclangIndex.a ...  lib/libclangFrontend.a -o exe

libclangIndex.a etc are not wrapped in --start-group.

If the downstream application depends on libclangFrontend.a but not any
other clang libraries that depend on libclangIndex.a, this can cause undefined
reference errors when the linker is ld.bfd or gold.

The proper fix is to not include clangIndex files in clangFrontend.

llvm-svn: 363649
2019-06-18 05:52:39 +00:00
Puyan Lotfi 8df7f1a218 [clang-ifs] Clang Interface Stubs, first version.
Clang interface stubs (previously referred to as clang-ifsos) is a new frontend
action in clang that allows the generation of stub files that contain mangled
name info that can be used to produce a stub library. These stub libraries can
be useful for breaking up build dependencies and controlling access to a
library's internal symbols. Generation of these stubs can be invoked by:

clang -fvisibility=<visibility> -emit-interface-stubs \
                                -interface-stub-version=<interface format>

Notice that -fvisibility (along with use of visibility attributes) can be used
to control what symbols get generated. Currently the interface format is
experimental but there are a wide range of possibilities here.

Differential Revision: https://reviews.llvm.org/D60974

llvm-svn: 363626
2019-06-17 22:46:54 +00:00