The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.
== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.
By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.
This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.
== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std:🧵:hardware_concurrency() -- which can only return processors from the current "processor group".
== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).
When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.
When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.
Differential Revision: https://reviews.llvm.org/D71775
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
This reverts commit 3f76260dc0.
Breaks at least these tests on Windows:
Clang :: Driver/clang-offload-bundler.c
Clang :: Driver/clang-offload-wrapper.c
Use clang_target_link_libraries() in order to support linking against
libclang-cpp instead of static libraries.
Differential Revision: https://reviews.llvm.org/D68448
llvm-svn: 373786
"Bad block found.\n" -> "bad block found"
The lower cased form with no full stop or newline is more common in LLVM
tools.
Reviewed By: juliehockett
Differential Revision: https://reviews.llvm.org/D66783
llvm-svn: 370155
PR43039 reports hitting the assert on a very large file, so bumping this
to allow for larger files.
Differential Revision: https://reviews.llvm.org/D66681
llvm-svn: 369811
Summary:
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.
Patch produced by
for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done
[This is analogous to LLVM r331272 and CFE r331834]
Subscribers: srhines, nemanjai, javed.absar, kbarton, MaskRay, jkorous, arphaman, jfb, kadircet, jsji, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66578
llvm-svn: 369643
An implicit cast of std::string to llvm::SmallString<> was breaking GCC 5.4.0 builder.
A pair using llvm::SmallString<> now uses std::string.
Differential Revision: https://reviews.llvm.org/D66378
llvm-svn: 369182
The new design includes a header (contains the project name), a main section, and a footer.
The main section is divided into three subsections. Left, middle, right. The left section contains the general index, the middle contains the info's data, and the right contains the index for the info's content.
The CSS has been updated.
A flag --project-name is added.
The Attributes attribute of the TagNode struct is now a vector of pairs because these attributes should be rendered in the insertion order.
The functions (cpp and js) that converts an Index tree structure into HTML were slightly modified; the first ul tag created is now a ol tag. The inner lists are still ul.
Differential Revision: https://reviews.llvm.org/D66353
llvm-svn: 369139
When a Record is declared in the global namespace, clang-doc serializes
it as a child of the global namespace, so the global namespace is now
one if its parent namespaces. This namespace was not being included in
the list of namespaces of the Info causing paths to be incorrect and the
index rendered incorrectly.
Affected tests have been fixed.
Differential revision: https://reviews.llvm.org/D66298
llvm-svn: 369123
clang-doc now serializes the inherited attributes and methods, not only the name of the base class.
All inherited are tracked, if B:A and C:B, info of A is included in C.
This data is stored in attribute Bases in a RecordInfo.
Previously tracked inheritance data, stored in Parents and VParents, hasn't been removed to reduce review load.
Differential revision: https://reviews.llvm.org/D66238
llvm-svn: 369075
The value, if any, of --source-root flag was not being used.
This has been fixed and the logic was moved to the ClangDocContext
contructor.
Differential revision: https://reviews.llvm.org/D66268
llvm-svn: 369065
Bitcode writer was not emitting the corresponding record for the Access attribute of a FunctionInfo. This has been added.
AS_none was being used as the default value for any AcesssSpecifier attribute
(in FunctionInfo and MemberTypeInfo), this has been changed to AS_public
because this is the enum value that evaluates to 0.
The bitcode writer doesn't write values that are 0 so if an attribute
was set to AS_public, this value is not written and after reading the
bitcode it would have the default value which is AS_none. This is why
the default value is now AS_public.
Differential Revision: https://reviews.llvm.org/D66151
llvm-svn: 369063
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.
Differential revision: https://reviews.llvm.org/D66259
llvm-svn: 368944
Path is now stored in the references to the child while serializing,
then this path is used to generate the relative path in the HTML
generator.
Now some references have paths and some don't so in the reducing phase,
references are now properly merged checking for empty attributes.
Tests added for HTML and YAML generators, merging and serializing.
computeRelativePath function had a bug when the filepath is part of the
given directory; it returned a path that starts with a separator. This
has been fixed.
Differential Revision: https://reviews.llvm.org/D65987
llvm-svn: 368602
clang-doc now generates a file that contains only an index to all the
infos that can be used as the landing page for the generated website.
Differential Revision: https://reviews.llvm.org/D65918
llvm-svn: 368484
Two command line options have been added to clang-doc.
--repository=<string> - URL of repository that hosts code; used for links to definition locations.
--source-root=<string> - Directory where processed files are stored. Links to definition locations will only be generated if the file is in this dir.
If the file is in the source-root and a repository options is passed;
a link to the source code will be rendered by the HTML generator.
Differential Revision: https://reviews.llvm.org/D65483
llvm-svn: 368460
Idx in ClangDocContext instance was being modified by multiple threads
causing a seg fault.
A mutex is added to avoid this.
Differential Revision: https://reviews.llvm.org/D65915
llvm-svn: 368313
This new index contains links to the main section of infos: Namespaces, Records, Functions, Enums, Members.
Also to each child function or enum.
Index is currently rendered on top of the info content, this will be fixed later with CSS.
Depends on D65690.
Differential Revision: https://reviews.llvm.org/D65030
llvm-svn: 368209
Reduce phase has been parallelized and a execution time was reduced by
60% with this.
The reading of bitcode (bitcode -> Info) was moved to this segment of
code parallelized so it now happens just before reducing.
Differential Revision: https://reviews.llvm.org/D65628
llvm-svn: 368206
An index structure is created while generating the output file for each
info. This structure is parsed to JSON and written to a file in the
output directory. The html for the index is not rendered by clang-doc. A
Javascript file is included in the output directory, this will the JSON
file and insert HTML elements into the file.
Differential Revision: https://reviews.llvm.org/D65690
llvm-svn: 368070
Before making a link to a reference it is required to check that the
reference has a path (eg. primitives won't have paths).
This was done by checking if the path was empty; that worked because
when generating paths the outdirectory was included, so if the path was
assigned it had that outdirectory at least.
The path generation was changed, it's now only the composite of the
namespaces without the outdirectory. So if the info is in the global
namespace the path would be empty and the old check wouldn't work as expected.
A new attribute has been added to the Reference struct that indicates if
the info's parent is the global namespace.
Paths generation now fails if the path is empty and if the info
is not in the global namespace.
Differential Revision: https://reviews.llvm.org/D64958
llvm-svn: 367958
The tool used to stop execution if there was an error in the mapping
phase. It will now show the error but continue with the files that were
mapped correctly if the flag is true.
Differential revision: https://reviews.llvm.org/D65627
llvm-svn: 367729
Tests on Windows were failing due to path separator differences.
Links in HTML should use posix-style paths.
Differential Revision: https://reviews.llvm.org/D65419
llvm-svn: 367264
CSS files included in HTML should have a path in posix style, it should
not be different for Windows.
Differential Revision: https://reviews.llvm.org/D65309
llvm-svn: 367137
An option has been added to clang-doc to provide a list of css stylesheets that the user wants to use for the generated html docs.
Depends on D64539.
Differential Revision: https://reviews.llvm.org/D64938
llvm-svn: 367072
The children of a TagNode are rendered in the same line as the parent only if they are all TextNodes.
When children are not inline; two text nodes that are adjacent won't have a new line between them, each tag node is rendered in its own line.
Differential Revision: https://reviews.llvm.org/D65005
llvm-svn: 367050
Replace &, <, >, ", and ' with their corresponding html entities in text rendered
by HTML generator.
Differential Revision: https://reviews.llvm.org/D65107
llvm-svn: 367045
<a> tags are added for the parents and members of records and return type and
params of functions. The link redirects to the reference's info file.
The directory path where each info file will be saved is now generated in the
serialization phase and stored as an attribute in each Info.
Bitcode writer and reader were modified to handle the new attributes.
Committed on behalf of Diego Astiazarán (diegoaat97@gmail.com).
Differential Revision: https://reviews.llvm.org/D63663
llvm-svn: 365937
Without the fix gcc 7.4.0 complains with
/data/repo/master/clang-tools-extra/clang-doc/HTMLGenerator.cpp: In member function 'llvm::SmallString<16> clang::doc::{anonymous}::HTMLTag::ToString() const':
/data/repo/master/clang-tools-extra/clang-doc/HTMLGenerator.cpp:165:1: error: control reaches end of non-void function [-Werror=return-type]
}
^
/data/repo/master/clang-tools-extra/clang-doc/HTMLGenerator.cpp: In member function 'bool clang::doc::{anonymous}::HTMLTag::HasInlineChildren() const':
/data/repo/master/clang-tools-extra/clang-doc/HTMLGenerator.cpp:142:1: error: control reaches end of non-void function [-Werror=return-type]
}
^
/data/repo/master/clang-tools-extra/clang-doc/HTMLGenerator.cpp: In member function 'bool clang::doc::{anonymous}::HTMLTag::IsSelfClosing() const':
/data/repo/master/clang-tools-extra/clang-doc/HTMLGenerator.cpp:126:1: error: control reaches end of non-void function [-Werror=return-type]
}
^
cc1plus: all warnings being treated as errors
llvm-svn: 365736
Implements an HTML generator.
Nodes are used to represent each part of the HTML file. There are TagNodes that
represent every HTML tag (p, h1, div, ...) and they have children nodes, which
can be TagNodes or TextNodes (these nodes only have text).
Proper indentation is rendered within the files generated by tool.
No styling (CSS) is included.
Committed on behalf of Diego Astiazarán (diegoaat97@gmail.com)
Differential Revision: https://reviews.llvm.org/D63857
llvm-svn: 365687
Alexandre Ganea