maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/clang/lib/CodeGen/CGLoopInfo.cpp

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

842 lines
31 KiB
C++
Raw Normal View History

This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
//===---- CGLoopInfo.cpp - LLVM CodeGen for loop metadata -*- C++ -*-------===//
//
Update the file headers across all of the LLVM projects in the monorepo to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
2019-01-19 16:50:56 +08:00
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
//
//===----------------------------------------------------------------------===//
#include "CGLoopInfo.h"
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
#include "clang/AST/ASTContext.h"
Add assume_safety option for pragma loop vectorize and interleave. Specifying #pragma clang loop vectorize(assume_safety) on a loop adds the mem.parallel_loop_access metadata to each load/store operation in the loop. This metadata tells loop access analysis (LAA) to skip memory dependency checking. llvm-svn: 239572
2015-06-12 07:23:17 +08:00
#include "clang/AST/Attr.h"
Sink more Attr.h inline methods, NFC This has very little impact on build time, but is a mechanical pre-req to removing the OpenMPClause.h include, which matters. Most of these pretty print methods require Expr to be complete.
2020-03-12 11:22:14 +08:00
#include "clang/AST/Expr.h"
[Clang] Add llvm.loop.unroll.disable to loops with -fno-unroll-loops. Currently Clang does not respect -fno-unroll-loops during LTO. During D76916 it was suggested to respect -fno-unroll-loops on a TU basis. This patch uses the existing llvm.loop.unroll.disable metadata to disable loop unrolling explicitly for each loop in the TU if unrolling is disabled. This should ensure that loops from TUs compiled with -fno-unroll-loops are skipped by the unroller during LTO. This also means that if a loop from a TU with -fno-unroll-loops gets inlined into a TU without this option, the loop won't be unrolled. Due to the fact that some transforms might drop loop metadata, there potentially are cases in which we still unroll loops from TUs with -fno-unroll-loops. I think we should fix those issues rather than introducing a function attribute to disable loop unrolling during LTO. Improving the metadata handling will benefit other use cases, like various loop pragmas, too. And it is an improvement to clang completely ignoring -fno-unroll-loops during LTO. If that direction looks good, we can use a similar approach to also respect -fno-vectorize during LTO, at least for LoopVectorize. In the future, this might also allow us to remove the UnrollLoops option LLVM's PassManagerBuilder. Reviewers: Meinersbur, hfinkel, dexonsmith, tejohnson Reviewed By: Meinersbur, tejohnson Differential Revision: https://reviews.llvm.org/D77058
2020-04-07 20:43:48 +08:00
#include "clang/Basic/CodeGenOptions.h"
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
#include "llvm/IR/BasicBlock.h"
[TI removal] Test predicate rather than casting to detect a terminator and use the range based successor API. llvm-svn: 344730
2018-10-18 16:16:20 +08:00
#include "llvm/IR/CFG.h"
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
#include "llvm/IR/Constants.h"
#include "llvm/IR/InstrTypes.h"
#include "llvm/IR/Instructions.h"
#include "llvm/IR/Metadata.h"
Eliminate unnecessary namespace to prevent conflicts. llvm-svn: 239365
2015-06-09 07:27:35 +08:00
using namespace clang::CodeGen;
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
using namespace llvm;
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
MDNode *
LoopInfo::createLoopPropertiesMetadata(ArrayRef<Metadata *> LoopProperties) {
LLVMContext &Ctx = Header->getContext();
SmallVector<Metadata *, 4> NewLoopProperties;
Avoid unnecessary uses of `MDNode::getTemporary`, NFC This is a long-delayed follow-up to 5e5b85098dbeaea2cfa5d01695b5d2982634d7dd. `TempMDNode` includes a bunch of machinery for RAUW, and should only be used when necessary. RAUW wasn't being used in any of these cases... it was just a placeholder for a self-reference. Where the real node was using `MDNode::getDistinct`, just replace the temporary argument with `nullptr`. Where the real node was using `MDNode::get`, the `replaceOperandWith` call was "promoting" the node to a distinct one implicitly due to self-reference detection in `MDNode::handleChangedOperand`. The `TempMDNode` was serving a purpose by delaying uniquing, but it's way simpler to just call `MDNode::getDistinct` in the first place. Note that using a self-reference at all in these places is a hold-over from before `distinct` metadata existed. It was an old trick to create distinct nodes. It would be intrusive to change, including bitcode upgrades, etc., and it's harmless so I'm not sure there's much value in removing it from existing schemas. After this commit it still has a tiny memory cost (in the extra metadata operand) but no more overhead in construction. Differential Revision: https://reviews.llvm.org/D90079
2020-10-24 05:55:41 +08:00
NewLoopProperties.push_back(nullptr);
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
MDNode *LoopID = MDNode::getDistinct(Ctx, NewLoopProperties);
LoopID->replaceOperandWith(0, LoopID);
return LoopID;
}
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
MDNode *LoopInfo::createPipeliningMetadata(const LoopAttributes &Attrs,
ArrayRef<Metadata *> LoopProperties,
bool &HasUserTransforms) {
LLVMContext &Ctx = Header->getContext();
Optional<bool> Enabled;
if (Attrs.PipelineDisabled)
Enabled = false;
else if (Attrs.PipelineInitiationInterval != 0)
Enabled = true;
if (Enabled != true) {
SmallVector<Metadata *, 4> NewLoopProperties;
if (Enabled == false) {
NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
NewLoopProperties.push_back(
MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.pipeline.disable"),
ConstantAsMetadata::get(ConstantInt::get(
llvm::Type::getInt1Ty(Ctx), 1))}));
LoopProperties = NewLoopProperties;
}
return createLoopPropertiesMetadata(LoopProperties);
}
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
IR: Update clang for Metadata/Value split in r223802 Match LLVM API changes from r223802. llvm-svn: 223803
2014-12-10 02:39:32 +08:00
SmallVector<Metadata *, 4> Args;
Avoid unnecessary uses of `MDNode::getTemporary`, NFC This is a long-delayed follow-up to 5e5b85098dbeaea2cfa5d01695b5d2982634d7dd. `TempMDNode` includes a bunch of machinery for RAUW, and should only be used when necessary. RAUW wasn't being used in any of these cases... it was just a placeholder for a self-reference. Where the real node was using `MDNode::getDistinct`, just replace the temporary argument with `nullptr`. Where the real node was using `MDNode::get`, the `replaceOperandWith` call was "promoting" the node to a distinct one implicitly due to self-reference detection in `MDNode::handleChangedOperand`. The `TempMDNode` was serving a purpose by delaying uniquing, but it's way simpler to just call `MDNode::getDistinct` in the first place. Note that using a self-reference at all in these places is a hold-over from before `distinct` metadata existed. It was an old trick to create distinct nodes. It would be intrusive to change, including bitcode upgrades, etc., and it's harmless so I'm not sure there's much value in removing it from existing schemas. After this commit it still has a tiny memory cost (in the extra metadata operand) but no more overhead in construction. Differential Revision: https://reviews.llvm.org/D90079
2020-10-24 05:55:41 +08:00
Args.push_back(nullptr);
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
Args.append(LoopProperties.begin(), LoopProperties.end());
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
if (Attrs.PipelineInitiationInterval > 0) {
Metadata *Vals[] = {
MDString::get(Ctx, "llvm.loop.pipeline.initiationinterval"),
ConstantAsMetadata::get(ConstantInt::get(
llvm::Type::getInt32Ty(Ctx), Attrs.PipelineInitiationInterval))};
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
Args.push_back(MDNode::get(Ctx, Vals));
}
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
// No follow-up: This is the last transformation.
MDNode *LoopID = MDNode::getDistinct(Ctx, Args);
LoopID->replaceOperandWith(0, LoopID);
HasUserTransforms = true;
return LoopID;
}
MDNode *
LoopInfo::createPartialUnrollMetadata(const LoopAttributes &Attrs,
ArrayRef<Metadata *> LoopProperties,
bool &HasUserTransforms) {
LLVMContext &Ctx = Header->getContext();
Optional<bool> Enabled;
if (Attrs.UnrollEnable == LoopAttributes::Disable)
Enabled = false;
else if (Attrs.UnrollEnable == LoopAttributes::Full)
Enabled = None;
else if (Attrs.UnrollEnable != LoopAttributes::Unspecified ||
Attrs.UnrollCount != 0)
Enabled = true;
if (Enabled != true) {
// createFullUnrollMetadata will already have added llvm.loop.unroll.disable
// if unrolling is disabled.
return createPipeliningMetadata(Attrs, LoopProperties, HasUserTransforms);
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
}
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
SmallVector<Metadata *, 4> FollowupLoopProperties;
// Apply all loop properties to the unrolled loop.
FollowupLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
// Don't unroll an already unrolled loop.
FollowupLoopProperties.push_back(
MDNode::get(Ctx, MDString::get(Ctx, "llvm.loop.unroll.disable")));
bool FollowupHasTransforms = false;
MDNode *Followup = createPipeliningMetadata(Attrs, FollowupLoopProperties,
FollowupHasTransforms);
SmallVector<Metadata *, 4> Args;
Avoid unnecessary uses of `MDNode::getTemporary`, NFC This is a long-delayed follow-up to 5e5b85098dbeaea2cfa5d01695b5d2982634d7dd. `TempMDNode` includes a bunch of machinery for RAUW, and should only be used when necessary. RAUW wasn't being used in any of these cases... it was just a placeholder for a self-reference. Where the real node was using `MDNode::getDistinct`, just replace the temporary argument with `nullptr`. Where the real node was using `MDNode::get`, the `replaceOperandWith` call was "promoting" the node to a distinct one implicitly due to self-reference detection in `MDNode::handleChangedOperand`. The `TempMDNode` was serving a purpose by delaying uniquing, but it's way simpler to just call `MDNode::getDistinct` in the first place. Note that using a self-reference at all in these places is a hold-over from before `distinct` metadata existed. It was an old trick to create distinct nodes. It would be intrusive to change, including bitcode upgrades, etc., and it's harmless so I'm not sure there's much value in removing it from existing schemas. After this commit it still has a tiny memory cost (in the extra metadata operand) but no more overhead in construction. Differential Revision: https://reviews.llvm.org/D90079
2020-10-24 05:55:41 +08:00
Args.push_back(nullptr);
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
Args.append(LoopProperties.begin(), LoopProperties.end());
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
// Setting unroll.count
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
if (Attrs.UnrollCount > 0) {
Metadata *Vals[] = {MDString::get(Ctx, "llvm.loop.unroll.count"),
ConstantAsMetadata::get(ConstantInt::get(
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
llvm::Type::getInt32Ty(Ctx), Attrs.UnrollCount))};
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
Args.push_back(MDNode::get(Ctx, Vals));
}
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
// Setting unroll.full or unroll.disable
if (Attrs.UnrollEnable == LoopAttributes::Enable) {
Metadata *Vals[] = {MDString::get(Ctx, "llvm.loop.unroll.enable")};
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
Args.push_back(MDNode::get(Ctx, Vals));
}
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
if (FollowupHasTransforms)
Args.push_back(MDNode::get(
Ctx, {MDString::get(Ctx, "llvm.loop.unroll.followup_all"), Followup}));
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
MDNode *LoopID = MDNode::getDistinct(Ctx, Args);
LoopID->replaceOperandWith(0, LoopID);
HasUserTransforms = true;
return LoopID;
}
MDNode *
LoopInfo::createUnrollAndJamMetadata(const LoopAttributes &Attrs,
ArrayRef<Metadata *> LoopProperties,
bool &HasUserTransforms) {
LLVMContext &Ctx = Header->getContext();
Optional<bool> Enabled;
if (Attrs.UnrollAndJamEnable == LoopAttributes::Disable)
Enabled = false;
else if (Attrs.UnrollAndJamEnable == LoopAttributes::Enable ||
Attrs.UnrollAndJamCount != 0)
Enabled = true;
if (Enabled != true) {
SmallVector<Metadata *, 4> NewLoopProperties;
if (Enabled == false) {
NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
NewLoopProperties.push_back(MDNode::get(
Ctx, MDString::get(Ctx, "llvm.loop.unroll_and_jam.disable")));
LoopProperties = NewLoopProperties;
}
return createPartialUnrollMetadata(Attrs, LoopProperties,
HasUserTransforms);
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
}
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
SmallVector<Metadata *, 4> FollowupLoopProperties;
FollowupLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
FollowupLoopProperties.push_back(
MDNode::get(Ctx, MDString::get(Ctx, "llvm.loop.unroll_and_jam.disable")));
bool FollowupHasTransforms = false;
MDNode *Followup = createPartialUnrollMetadata(Attrs, FollowupLoopProperties,
FollowupHasTransforms);
SmallVector<Metadata *, 4> Args;
Avoid unnecessary uses of `MDNode::getTemporary`, NFC This is a long-delayed follow-up to 5e5b85098dbeaea2cfa5d01695b5d2982634d7dd. `TempMDNode` includes a bunch of machinery for RAUW, and should only be used when necessary. RAUW wasn't being used in any of these cases... it was just a placeholder for a self-reference. Where the real node was using `MDNode::getDistinct`, just replace the temporary argument with `nullptr`. Where the real node was using `MDNode::get`, the `replaceOperandWith` call was "promoting" the node to a distinct one implicitly due to self-reference detection in `MDNode::handleChangedOperand`. The `TempMDNode` was serving a purpose by delaying uniquing, but it's way simpler to just call `MDNode::getDistinct` in the first place. Note that using a self-reference at all in these places is a hold-over from before `distinct` metadata existed. It was an old trick to create distinct nodes. It would be intrusive to change, including bitcode upgrades, etc., and it's harmless so I'm not sure there's much value in removing it from existing schemas. After this commit it still has a tiny memory cost (in the extra metadata operand) but no more overhead in construction. Differential Revision: https://reviews.llvm.org/D90079
2020-10-24 05:55:41 +08:00
Args.push_back(nullptr);
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
Args.append(LoopProperties.begin(), LoopProperties.end());
// Setting unroll_and_jam.count
if (Attrs.UnrollAndJamCount > 0) {
Metadata *Vals[] = {
MDString::get(Ctx, "llvm.loop.unroll_and_jam.count"),
ConstantAsMetadata::get(ConstantInt::get(llvm::Type::getInt32Ty(Ctx),
Attrs.UnrollAndJamCount))};
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
Args.push_back(MDNode::get(Ctx, Vals));
}
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
if (Attrs.UnrollAndJamEnable == LoopAttributes::Enable) {
Metadata *Vals[] = {MDString::get(Ctx, "llvm.loop.unroll_and_jam.enable")};
Add loop pragma for Loop Distribution Summary: This is similar to other loop pragmas like 'vectorize'. Currently it only has state values: distribute(enable) and distribute(disable). When one of these is specified the corresponding loop metadata is generated: !{!"llvm.loop.distribute.enable", i1 true/false} As a result, loop distribution will be attempted on the loop even if Loop Distribution in not enabled globally. Analogously, with 'disable' distribution can be turned off for an individual loop even when the pass is otherwise enabled. There are some slight differences compared to the existing loop pragmas. 1. There is no 'assume_safety' variant which makes its handling slightly different from 'vectorize'/'interleave'. 2. Unlike the existing loop pragmas, it does not have a corresponding numeric pragma like 'vectorize' -> 'vectorize_width'. So for the consistency checks in CheckForIncompatibleAttributes we don't need to check it against other pragmas. We just need to check for duplicates of the same pragma. Reviewers: rsmith, dexonsmith, aaron.ballman Subscribers: bob.wilson, cfe-commits, hfinkel Differential Revision: http://reviews.llvm.org/D19403 llvm-svn: 272656
2016-06-14 20:04:26 +08:00
Args.push_back(MDNode::get(Ctx, Vals));
}
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
if (FollowupHasTransforms)
[CodeGen] Generate llvm.loop.parallel_accesses instead of llvm.mem.parallel_loop_access metadata. Instead of generating llvm.mem.parallel_loop_access metadata, generate llvm.access.group on instructions and llvm.loop.parallel_accesses on loops. There is one access group per generated loop. This is clang part of D52116/r349725. Differential Revision: https://reviews.llvm.org/D52117 llvm-svn: 349823
2018-12-21 05:24:54 +08:00
Args.push_back(MDNode::get(
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
Ctx, {MDString::get(Ctx, "llvm.loop.unroll_and_jam.followup_outer"),
Followup}));
if (UnrollAndJamInnerFollowup)
Args.push_back(MDNode::get(
Ctx, {MDString::get(Ctx, "llvm.loop.unroll_and_jam.followup_inner"),
UnrollAndJamInnerFollowup}));
MDNode *LoopID = MDNode::getDistinct(Ctx, Args);
LoopID->replaceOperandWith(0, LoopID);
HasUserTransforms = true;
return LoopID;
}
MDNode *
LoopInfo::createLoopVectorizeMetadata(const LoopAttributes &Attrs,
ArrayRef<Metadata *> LoopProperties,
bool &HasUserTransforms) {
LLVMContext &Ctx = Header->getContext();
Optional<bool> Enabled;
if (Attrs.VectorizeEnable == LoopAttributes::Disable)
Enabled = false;
else if (Attrs.VectorizeEnable != LoopAttributes::Unspecified ||
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
Attrs.VectorizePredicateEnable != LoopAttributes::Unspecified ||
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
Attrs.InterleaveCount != 0 || Attrs.VectorizeWidth != 0 ||
Attrs.VectorizeScalable != LoopAttributes::Unspecified)
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
Enabled = true;
if (Enabled != true) {
SmallVector<Metadata *, 4> NewLoopProperties;
if (Enabled == false) {
NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
NewLoopProperties.push_back(
MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
ConstantAsMetadata::get(ConstantInt::get(
llvm::Type::getInt1Ty(Ctx), 0))}));
LoopProperties = NewLoopProperties;
}
return createUnrollAndJamMetadata(Attrs, LoopProperties, HasUserTransforms);
}
// Apply all loop properties to the vectorized loop.
SmallVector<Metadata *, 4> FollowupLoopProperties;
FollowupLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
// Don't vectorize an already vectorized loop.
FollowupLoopProperties.push_back(
MDNode::get(Ctx, MDString::get(Ctx, "llvm.loop.isvectorized")));
bool FollowupHasTransforms = false;
MDNode *Followup = createUnrollAndJamMetadata(Attrs, FollowupLoopProperties,
FollowupHasTransforms);
SmallVector<Metadata *, 4> Args;
Avoid unnecessary uses of `MDNode::getTemporary`, NFC This is a long-delayed follow-up to 5e5b85098dbeaea2cfa5d01695b5d2982634d7dd. `TempMDNode` includes a bunch of machinery for RAUW, and should only be used when necessary. RAUW wasn't being used in any of these cases... it was just a placeholder for a self-reference. Where the real node was using `MDNode::getDistinct`, just replace the temporary argument with `nullptr`. Where the real node was using `MDNode::get`, the `replaceOperandWith` call was "promoting" the node to a distinct one implicitly due to self-reference detection in `MDNode::handleChangedOperand`. The `TempMDNode` was serving a purpose by delaying uniquing, but it's way simpler to just call `MDNode::getDistinct` in the first place. Note that using a self-reference at all in these places is a hold-over from before `distinct` metadata existed. It was an old trick to create distinct nodes. It would be intrusive to change, including bitcode upgrades, etc., and it's harmless so I'm not sure there's much value in removing it from existing schemas. After this commit it still has a tiny memory cost (in the extra metadata operand) but no more overhead in construction. Differential Revision: https://reviews.llvm.org/D90079
2020-10-24 05:55:41 +08:00
Args.push_back(nullptr);
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
Args.append(LoopProperties.begin(), LoopProperties.end());
[Clang] Ensure vector predication loop metadata is always emitted when pragma is specified. This patch ensures that vector predication and vectorization width pragmas work together correctly/as expected. Specifically, this patch fixes the issue that when vectorization_width > 1, the vector predication behaviour (this would matter if it has NOT been disabled explicitly by a pragma) was getting ignored, which was incorrect. The fix here removes the dependence of vector predication on the vectorization width. The loop metadata corresponding to clang loop pragma vectorize_predicate is always emitted, if the pragma is specified, even if vectorization is disabled by vectorize_width(1) or vectorize(disable) since the option is also used for interleaving by the LoopVectorize pass. Reviewed By: dmgreen, Meinersbur Differential Revision: https://reviews.llvm.org/D94779
2021-02-14 05:57:21 +08:00
// Setting vectorize.predicate when it has been specified and vectorization
// has not been disabled.
[Clang] Pragma vectorize_predicate implies vectorize New pragma "vectorize_predicate(enable)" now implies "vectorize(enable)", and it is ignored when vectorization is disabled with e.g. "vectorize(disable) vectorize_predicate(enable)". Differential Revision: https://reviews.llvm.org/D65776 llvm-svn: 368970
2019-08-15 14:24:40 +08:00
bool IsVectorPredicateEnabled = false;
[Clang] Ensure vector predication loop metadata is always emitted when pragma is specified. This patch ensures that vector predication and vectorization width pragmas work together correctly/as expected. Specifically, this patch fixes the issue that when vectorization_width > 1, the vector predication behaviour (this would matter if it has NOT been disabled explicitly by a pragma) was getting ignored, which was incorrect. The fix here removes the dependence of vector predication on the vectorization width. The loop metadata corresponding to clang loop pragma vectorize_predicate is always emitted, if the pragma is specified, even if vectorization is disabled by vectorize_width(1) or vectorize(disable) since the option is also used for interleaving by the LoopVectorize pass. Reviewed By: dmgreen, Meinersbur Differential Revision: https://reviews.llvm.org/D94779
2021-02-14 05:57:21 +08:00
if (Attrs.VectorizePredicateEnable != LoopAttributes::Unspecified) {
[Clang] Pragma vectorize_predicate implies vectorize New pragma "vectorize_predicate(enable)" now implies "vectorize(enable)", and it is ignored when vectorization is disabled with e.g. "vectorize(disable) vectorize_predicate(enable)". Differential Revision: https://reviews.llvm.org/D65776 llvm-svn: 368970
2019-08-15 14:24:40 +08:00
IsVectorPredicateEnabled =
(Attrs.VectorizePredicateEnable == LoopAttributes::Enable);
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
Metadata *Vals[] = {
MDString::get(Ctx, "llvm.loop.vectorize.predicate.enable"),
[Clang] Pragma vectorize_predicate implies vectorize New pragma "vectorize_predicate(enable)" now implies "vectorize(enable)", and it is ignored when vectorization is disabled with e.g. "vectorize(disable) vectorize_predicate(enable)". Differential Revision: https://reviews.llvm.org/D65776 llvm-svn: 368970
2019-08-15 14:24:40 +08:00
ConstantAsMetadata::get(ConstantInt::get(llvm::Type::getInt1Ty(Ctx),
IsVectorPredicateEnabled))};
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
Args.push_back(MDNode::get(Ctx, Vals));
}
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
// Setting vectorize.width
if (Attrs.VectorizeWidth > 0) {
Metadata *Vals[] = {
MDString::get(Ctx, "llvm.loop.vectorize.width"),
ConstantAsMetadata::get(ConstantInt::get(llvm::Type::getInt32Ty(Ctx),
Attrs.VectorizeWidth))};
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
Args.push_back(MDNode::get(Ctx, Vals));
}
if (Attrs.VectorizeScalable != LoopAttributes::Unspecified) {
bool IsScalable = Attrs.VectorizeScalable == LoopAttributes::Enable;
Metadata *Vals[] = {
MDString::get(Ctx, "llvm.loop.vectorize.scalable.enable"),
ConstantAsMetadata::get(
ConstantInt::get(llvm::Type::getInt1Ty(Ctx), IsScalable))};
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
Args.push_back(MDNode::get(Ctx, Vals));
[CodeGen] Generate llvm.loop.parallel_accesses instead of llvm.mem.parallel_loop_access metadata. Instead of generating llvm.mem.parallel_loop_access metadata, generate llvm.access.group on instructions and llvm.loop.parallel_accesses on loops. There is one access group per generated loop. This is clang part of D52116/r349725. Differential Revision: https://reviews.llvm.org/D52117 llvm-svn: 349823
2018-12-21 05:24:54 +08:00
}
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
// Setting interleave.count
if (Attrs.InterleaveCount > 0) {
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
Metadata *Vals[] = {
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
MDString::get(Ctx, "llvm.loop.interleave.count"),
ConstantAsMetadata::get(ConstantInt::get(llvm::Type::getInt32Ty(Ctx),
Attrs.InterleaveCount))};
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
Args.push_back(MDNode::get(Ctx, Vals));
}
[Clang] Pragma vectorize_width() implies vectorize(enable) Let's try this again; this has been reverted/recommited a few times. Last time this got reverted because for this loop: void a() { #pragma clang loop vectorize(disable) for (;;) ; } vectorisation was incorrectly enabled and the vectorize.enable metadata was set due to a logic error. But with this fixed, we now imply vectorisation when: 1) vectorisation is enabled, which means: VectorizeWidth > 1, 2) and don't want to add it when it is disabled or enabled, otherwise we would be incorrectly setting it or duplicating the metadata, respectively. This should fix PR27643. Differential Revision: https://reviews.llvm.org/D69628
2019-12-11 18:37:40 +08:00
// vectorize.enable is set if:
// 1) loop hint vectorize.enable is set, or
// 2) it is implied when vectorize.predicate is set, or
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
// 3) it is implied when vectorize.width is set to a value > 1
// 4) it is implied when vectorize.scalable.enable is true
// 5) it is implied when vectorize.width is unset (0) and the user
// explicitly requested fixed-width vectorization, i.e.
// vectorize.scalable.enable is false.
[Clang] Pragma vectorize_predicate implies vectorize New pragma "vectorize_predicate(enable)" now implies "vectorize(enable)", and it is ignored when vectorization is disabled with e.g. "vectorize(disable) vectorize_predicate(enable)". Differential Revision: https://reviews.llvm.org/D65776 llvm-svn: 368970
2019-08-15 14:24:40 +08:00
if (Attrs.VectorizeEnable != LoopAttributes::Unspecified ||
[Clang] Ensure vector predication loop metadata is always emitted when pragma is specified. This patch ensures that vector predication and vectorization width pragmas work together correctly/as expected. Specifically, this patch fixes the issue that when vectorization_width > 1, the vector predication behaviour (this would matter if it has NOT been disabled explicitly by a pragma) was getting ignored, which was incorrect. The fix here removes the dependence of vector predication on the vectorization width. The loop metadata corresponding to clang loop pragma vectorize_predicate is always emitted, if the pragma is specified, even if vectorization is disabled by vectorize_width(1) or vectorize(disable) since the option is also used for interleaving by the LoopVectorize pass. Reviewed By: dmgreen, Meinersbur Differential Revision: https://reviews.llvm.org/D94779
2021-02-14 05:57:21 +08:00
(IsVectorPredicateEnabled && Attrs.VectorizeWidth != 1) ||
Attrs.VectorizeWidth > 1 ||
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
Attrs.VectorizeScalable == LoopAttributes::Enable ||
(Attrs.VectorizeScalable == LoopAttributes::Disable &&
Attrs.VectorizeWidth != 1)) {
[Clang] Pragma vectorize_width() implies vectorize(enable) Let's try this again; this has been reverted/recommited a few times. Last time this got reverted because for this loop: void a() { #pragma clang loop vectorize(disable) for (;;) ; } vectorisation was incorrectly enabled and the vectorize.enable metadata was set due to a logic error. But with this fixed, we now imply vectorisation when: 1) vectorisation is enabled, which means: VectorizeWidth > 1, 2) and don't want to add it when it is disabled or enabled, otherwise we would be incorrectly setting it or duplicating the metadata, respectively. This should fix PR27643. Differential Revision: https://reviews.llvm.org/D69628
2019-12-11 18:37:40 +08:00
bool AttrVal = Attrs.VectorizeEnable != LoopAttributes::Disable;
Args.push_back(
MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
ConstantAsMetadata::get(ConstantInt::get(
llvm::Type::getInt1Ty(Ctx), AttrVal))}));
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
}
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
if (FollowupHasTransforms)
Args.push_back(MDNode::get(
Ctx,
{MDString::get(Ctx, "llvm.loop.vectorize.followup_all"), Followup}));
Avoid unnecessary uses of `MDNode::getTemporary`, NFC This is a long-delayed follow-up to 5e5b85098dbeaea2cfa5d01695b5d2982634d7dd. `TempMDNode` includes a bunch of machinery for RAUW, and should only be used when necessary. RAUW wasn't being used in any of these cases... it was just a placeholder for a self-reference. Where the real node was using `MDNode::getDistinct`, just replace the temporary argument with `nullptr`. Where the real node was using `MDNode::get`, the `replaceOperandWith` call was "promoting" the node to a distinct one implicitly due to self-reference detection in `MDNode::handleChangedOperand`. The `TempMDNode` was serving a purpose by delaying uniquing, but it's way simpler to just call `MDNode::getDistinct` in the first place. Note that using a self-reference at all in these places is a hold-over from before `distinct` metadata existed. It was an old trick to create distinct nodes. It would be intrusive to change, including bitcode upgrades, etc., and it's harmless so I'm not sure there's much value in removing it from existing schemas. After this commit it still has a tiny memory cost (in the extra metadata operand) but no more overhead in construction. Differential Revision: https://reviews.llvm.org/D90079
2020-10-24 05:55:41 +08:00
MDNode *LoopID = MDNode::getDistinct(Ctx, Args);
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
LoopID->replaceOperandWith(0, LoopID);
HasUserTransforms = true;
return LoopID;
}
MDNode *
LoopInfo::createLoopDistributeMetadata(const LoopAttributes &Attrs,
ArrayRef<Metadata *> LoopProperties,
bool &HasUserTransforms) {
LLVMContext &Ctx = Header->getContext();
Optional<bool> Enabled;
if (Attrs.DistributeEnable == LoopAttributes::Disable)
Enabled = false;
if (Attrs.DistributeEnable == LoopAttributes::Enable)
Enabled = true;
if (Enabled != true) {
SmallVector<Metadata *, 4> NewLoopProperties;
if (Enabled == false) {
NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
NewLoopProperties.push_back(
MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.distribute.enable"),
ConstantAsMetadata::get(ConstantInt::get(
llvm::Type::getInt1Ty(Ctx), 0))}));
LoopProperties = NewLoopProperties;
}
return createLoopVectorizeMetadata(Attrs, LoopProperties,
HasUserTransforms);
}
bool FollowupHasTransforms = false;
MDNode *Followup =
createLoopVectorizeMetadata(Attrs, LoopProperties, FollowupHasTransforms);
SmallVector<Metadata *, 4> Args;
Avoid unnecessary uses of `MDNode::getTemporary`, NFC This is a long-delayed follow-up to 5e5b85098dbeaea2cfa5d01695b5d2982634d7dd. `TempMDNode` includes a bunch of machinery for RAUW, and should only be used when necessary. RAUW wasn't being used in any of these cases... it was just a placeholder for a self-reference. Where the real node was using `MDNode::getDistinct`, just replace the temporary argument with `nullptr`. Where the real node was using `MDNode::get`, the `replaceOperandWith` call was "promoting" the node to a distinct one implicitly due to self-reference detection in `MDNode::handleChangedOperand`. The `TempMDNode` was serving a purpose by delaying uniquing, but it's way simpler to just call `MDNode::getDistinct` in the first place. Note that using a self-reference at all in these places is a hold-over from before `distinct` metadata existed. It was an old trick to create distinct nodes. It would be intrusive to change, including bitcode upgrades, etc., and it's harmless so I'm not sure there's much value in removing it from existing schemas. After this commit it still has a tiny memory cost (in the extra metadata operand) but no more overhead in construction. Differential Revision: https://reviews.llvm.org/D90079
2020-10-24 05:55:41 +08:00
Args.push_back(nullptr);
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
Args.append(LoopProperties.begin(), LoopProperties.end());
Metadata *Vals[] = {MDString::get(Ctx, "llvm.loop.distribute.enable"),
ConstantAsMetadata::get(ConstantInt::get(
llvm::Type::getInt1Ty(Ctx),
(Attrs.DistributeEnable == LoopAttributes::Enable)))};
Args.push_back(MDNode::get(Ctx, Vals));
if (FollowupHasTransforms)
Args.push_back(MDNode::get(
Ctx,
{MDString::get(Ctx, "llvm.loop.distribute.followup_all"), Followup}));
Avoid unnecessary uses of `MDNode::getTemporary`, NFC This is a long-delayed follow-up to 5e5b85098dbeaea2cfa5d01695b5d2982634d7dd. `TempMDNode` includes a bunch of machinery for RAUW, and should only be used when necessary. RAUW wasn't being used in any of these cases... it was just a placeholder for a self-reference. Where the real node was using `MDNode::getDistinct`, just replace the temporary argument with `nullptr`. Where the real node was using `MDNode::get`, the `replaceOperandWith` call was "promoting" the node to a distinct one implicitly due to self-reference detection in `MDNode::handleChangedOperand`. The `TempMDNode` was serving a purpose by delaying uniquing, but it's way simpler to just call `MDNode::getDistinct` in the first place. Note that using a self-reference at all in these places is a hold-over from before `distinct` metadata existed. It was an old trick to create distinct nodes. It would be intrusive to change, including bitcode upgrades, etc., and it's harmless so I'm not sure there's much value in removing it from existing schemas. After this commit it still has a tiny memory cost (in the extra metadata operand) but no more overhead in construction. Differential Revision: https://reviews.llvm.org/D90079
2020-10-24 05:55:41 +08:00
MDNode *LoopID = MDNode::getDistinct(Ctx, Args);
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
LoopID->replaceOperandWith(0, LoopID);
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
HasUserTransforms = true;
return LoopID;
}
MDNode *LoopInfo::createFullUnrollMetadata(const LoopAttributes &Attrs,
ArrayRef<Metadata *> LoopProperties,
bool &HasUserTransforms) {
LLVMContext &Ctx = Header->getContext();
Optional<bool> Enabled;
if (Attrs.UnrollEnable == LoopAttributes::Disable)
Enabled = false;
else if (Attrs.UnrollEnable == LoopAttributes::Full)
Enabled = true;
if (Enabled != true) {
SmallVector<Metadata *, 4> NewLoopProperties;
if (Enabled == false) {
NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
NewLoopProperties.push_back(
MDNode::get(Ctx, MDString::get(Ctx, "llvm.loop.unroll.disable")));
LoopProperties = NewLoopProperties;
}
return createLoopDistributeMetadata(Attrs, LoopProperties,
HasUserTransforms);
}
SmallVector<Metadata *, 4> Args;
Avoid unnecessary uses of `MDNode::getTemporary`, NFC This is a long-delayed follow-up to 5e5b85098dbeaea2cfa5d01695b5d2982634d7dd. `TempMDNode` includes a bunch of machinery for RAUW, and should only be used when necessary. RAUW wasn't being used in any of these cases... it was just a placeholder for a self-reference. Where the real node was using `MDNode::getDistinct`, just replace the temporary argument with `nullptr`. Where the real node was using `MDNode::get`, the `replaceOperandWith` call was "promoting" the node to a distinct one implicitly due to self-reference detection in `MDNode::handleChangedOperand`. The `TempMDNode` was serving a purpose by delaying uniquing, but it's way simpler to just call `MDNode::getDistinct` in the first place. Note that using a self-reference at all in these places is a hold-over from before `distinct` metadata existed. It was an old trick to create distinct nodes. It would be intrusive to change, including bitcode upgrades, etc., and it's harmless so I'm not sure there's much value in removing it from existing schemas. After this commit it still has a tiny memory cost (in the extra metadata operand) but no more overhead in construction. Differential Revision: https://reviews.llvm.org/D90079
2020-10-24 05:55:41 +08:00
Args.push_back(nullptr);
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
Args.append(LoopProperties.begin(), LoopProperties.end());
Args.push_back(MDNode::get(Ctx, MDString::get(Ctx, "llvm.loop.unroll.full")));
// No follow-up: there is no loop after full unrolling.
// TODO: Warn if there are transformations after full unrolling.
MDNode *LoopID = MDNode::getDistinct(Ctx, Args);
LoopID->replaceOperandWith(0, LoopID);
HasUserTransforms = true;
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
return LoopID;
}
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
MDNode *LoopInfo::createMetadata(
const LoopAttributes &Attrs,
llvm::ArrayRef<llvm::Metadata *> AdditionalLoopProperties,
bool &HasUserTransforms) {
SmallVector<Metadata *, 3> LoopProperties;
// If we have a valid start debug location for the loop, add it.
if (StartLoc) {
LoopProperties.push_back(StartLoc.getAsMDNode());
// If we also have a valid end debug location for the loop, add it.
if (EndLoc)
LoopProperties.push_back(EndLoc.getAsMDNode());
}
[clang] Add mustprogress and llvm.loop.mustprogress attribute deduction Since C++11, the C++ standard has a forward progress guarantee [intro.progress], so all such functions must have the `mustprogress` requirement. In addition, from C11 and onwards, loops without a non-zero constant conditional or no conditional are also required to make progress (C11 6.8.5p6). This patch implements these attribute deductions so they can be used by the optimization passes. Differential Revision: https://reviews.llvm.org/D86841
2020-11-03 05:03:21 +08:00
LLVMContext &Ctx = Header->getContext();
if (Attrs.MustProgress)
LoopProperties.push_back(
MDNode::get(Ctx, MDString::get(Ctx, "llvm.loop.mustprogress")));
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
assert(!!AccGroup == Attrs.IsParallel &&
"There must be an access group iff the loop is parallel");
if (Attrs.IsParallel) {
LoopProperties.push_back(MDNode::get(
Ctx, {MDString::get(Ctx, "llvm.loop.parallel_accesses"), AccGroup}));
}
LoopProperties.insert(LoopProperties.end(), AdditionalLoopProperties.begin(),
AdditionalLoopProperties.end());
return createFullUnrollMetadata(Attrs, LoopProperties, HasUserTransforms);
}
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
LoopAttributes::LoopAttributes(bool IsParallel)
Make the variable names match the name of the metadata they control. Rename Vectorizer to Vectorize and VectorizeUnroll to InterleaveCount. llvm-svn: 242241
2015-07-15 07:03:09 +08:00
: IsParallel(IsParallel), VectorizeEnable(LoopAttributes::Unspecified),
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
UnrollEnable(LoopAttributes::Unspecified),
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
UnrollAndJamEnable(LoopAttributes::Unspecified),
VectorizePredicateEnable(LoopAttributes::Unspecified), VectorizeWidth(0),
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
VectorizeScalable(LoopAttributes::Unspecified), InterleaveCount(0),
UnrollCount(0), UnrollAndJamCount(0),
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
DistributeEnable(LoopAttributes::Unspecified), PipelineDisabled(false),
[clang] Add mustprogress and llvm.loop.mustprogress attribute deduction Since C++11, the C++ standard has a forward progress guarantee [intro.progress], so all such functions must have the `mustprogress` requirement. In addition, from C11 and onwards, loops without a non-zero constant conditional or no conditional are also required to make progress (C11 6.8.5p6). This patch implements these attribute deductions so they can be used by the optimization passes. Differential Revision: https://reviews.llvm.org/D86841
2020-11-03 05:03:21 +08:00
PipelineInitiationInterval(0), MustProgress(false) {}
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
void LoopAttributes::clear() {
IsParallel = false;
Make the variable names match the name of the metadata they control. Rename Vectorizer to Vectorize and VectorizeUnroll to InterleaveCount. llvm-svn: 242241
2015-07-15 07:03:09 +08:00
VectorizeWidth = 0;
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
VectorizeScalable = LoopAttributes::Unspecified;
Make the variable names match the name of the metadata they control. Rename Vectorizer to Vectorize and VectorizeUnroll to InterleaveCount. llvm-svn: 242241
2015-07-15 07:03:09 +08:00
InterleaveCount = 0;
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
UnrollCount = 0;
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
UnrollAndJamCount = 0;
Make the variable names match the name of the metadata they control. Rename Vectorizer to Vectorize and VectorizeUnroll to InterleaveCount. llvm-svn: 242241
2015-07-15 07:03:09 +08:00
VectorizeEnable = LoopAttributes::Unspecified;
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
UnrollEnable = LoopAttributes::Unspecified;
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
UnrollAndJamEnable = LoopAttributes::Unspecified;
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
VectorizePredicateEnable = LoopAttributes::Unspecified;
[Pragma] Clear loop distribution attribute between loops llvm-svn: 279608
2016-08-24 12:31:56 +08:00
DistributeEnable = LoopAttributes::Unspecified;
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
PipelineDisabled = false;
PipelineInitiationInterval = 0;
[clang] Add mustprogress and llvm.loop.mustprogress attribute deduction Since C++11, the C++ standard has a forward progress guarantee [intro.progress], so all such functions must have the `mustprogress` requirement. In addition, from C11 and onwards, loops without a non-zero constant conditional or no conditional are also required to make progress (C11 6.8.5p6). This patch implements these attribute deductions so they can be used by the optimization passes. Differential Revision: https://reviews.llvm.org/D86841
2020-11-03 05:03:21 +08:00
MustProgress = false;
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
}
Add a loop's debug location to its llvm.loop metadata Getting accurate locations for loops is important, because those locations are used by the frontend to generate optimization remarks. Currently, optimization remarks for loops often appear on the wrong line, often the first line of the loop body instead of the loop itself. This is confusing because that line might itself be another loop, or might be somewhere else completely if the body was an inlined function call. This happens because of the way we find the loop's starting location. First, we look for a preheader, and if we find one, and its terminator has a debug location, then we use that. Otherwise, we look for a location on an instruction in the loop header. The fallback heuristic is not bad, but will almost always find the beginning of the body, and not the loop statement itself. The preheader location search often fails because there's often not a preheader, and even when there is a preheader, depending on how it was formed, it sometimes carries the location of some preceeding code. I don't see any good theoretical way to fix this problem. On the other hand, this seems like a straightforward solution: Put the debug location in the loop's llvm.loop metadata. When emitting debug information, this commit causes us to add the debug location as an operand to each loop's llvm.loop metadata. Thus, we now generate this metadata for all loops (not just loops with optimization hints) when we're otherwise generating debug information. The remark test case changes depend on the companion LLVM commit r270771. llvm-svn: 270772
2016-05-26 05:53:24 +08:00
LoopInfo::LoopInfo(BasicBlock *Header, const LoopAttributes &Attrs,
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
const llvm::DebugLoc &StartLoc, const llvm::DebugLoc &EndLoc,
LoopInfo *Parent)
: Header(Header), Attrs(Attrs), StartLoc(StartLoc), EndLoc(EndLoc),
Parent(Parent) {
if (Attrs.IsParallel) {
// Create an access group for this loop.
LLVMContext &Ctx = Header->getContext();
AccGroup = MDNode::getDistinct(Ctx, {});
}
if (!Attrs.IsParallel && Attrs.VectorizeWidth == 0 &&
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
Attrs.VectorizeScalable == LoopAttributes::Unspecified &&
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
Attrs.InterleaveCount == 0 && Attrs.UnrollCount == 0 &&
Attrs.UnrollAndJamCount == 0 && !Attrs.PipelineDisabled &&
Attrs.PipelineInitiationInterval == 0 &&
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
Attrs.VectorizePredicateEnable == LoopAttributes::Unspecified &&
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
Attrs.VectorizeEnable == LoopAttributes::Unspecified &&
Attrs.UnrollEnable == LoopAttributes::Unspecified &&
Attrs.UnrollAndJamEnable == LoopAttributes::Unspecified &&
Attrs.DistributeEnable == LoopAttributes::Unspecified && !StartLoc &&
[clang] Add mustprogress and llvm.loop.mustprogress attribute deduction Since C++11, the C++ standard has a forward progress guarantee [intro.progress], so all such functions must have the `mustprogress` requirement. In addition, from C11 and onwards, loops without a non-zero constant conditional or no conditional are also required to make progress (C11 6.8.5p6). This patch implements these attribute deductions so they can be used by the optimization passes. Differential Revision: https://reviews.llvm.org/D86841
2020-11-03 05:03:21 +08:00
!EndLoc && !Attrs.MustProgress)
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
return;
TempLoopID = MDNode::getTemporary(Header->getContext(), None);
}
void LoopInfo::finish() {
// We did not annotate the loop body instructions because there are no
// attributes for this loop.
if (!TempLoopID)
return;
MDNode *LoopID;
LoopAttributes CurLoopAttr = Attrs;
LLVMContext &Ctx = Header->getContext();
if (Parent && (Parent->Attrs.UnrollAndJamEnable ||
Parent->Attrs.UnrollAndJamCount != 0)) {
// Parent unroll-and-jams this loop.
// Split the transformations in those that happens before the unroll-and-jam
// and those after.
LoopAttributes BeforeJam, AfterJam;
BeforeJam.IsParallel = AfterJam.IsParallel = Attrs.IsParallel;
BeforeJam.VectorizeWidth = Attrs.VectorizeWidth;
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
BeforeJam.VectorizeScalable = Attrs.VectorizeScalable;
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
BeforeJam.InterleaveCount = Attrs.InterleaveCount;
BeforeJam.VectorizeEnable = Attrs.VectorizeEnable;
BeforeJam.DistributeEnable = Attrs.DistributeEnable;
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
BeforeJam.VectorizePredicateEnable = Attrs.VectorizePredicateEnable;
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
switch (Attrs.UnrollEnable) {
case LoopAttributes::Unspecified:
case LoopAttributes::Disable:
BeforeJam.UnrollEnable = Attrs.UnrollEnable;
AfterJam.UnrollEnable = Attrs.UnrollEnable;
break;
case LoopAttributes::Full:
BeforeJam.UnrollEnable = LoopAttributes::Full;
break;
case LoopAttributes::Enable:
AfterJam.UnrollEnable = LoopAttributes::Enable;
break;
}
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
AfterJam.VectorizePredicateEnable = Attrs.VectorizePredicateEnable;
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
AfterJam.UnrollCount = Attrs.UnrollCount;
AfterJam.PipelineDisabled = Attrs.PipelineDisabled;
AfterJam.PipelineInitiationInterval = Attrs.PipelineInitiationInterval;
// If this loop is subject of an unroll-and-jam by the parent loop, and has
// an unroll-and-jam annotation itself, we have to decide whether to first
// apply the parent's unroll-and-jam or this loop's unroll-and-jam. The
// UnrollAndJam pass processes loops from inner to outer, so we apply the
// inner first.
BeforeJam.UnrollAndJamCount = Attrs.UnrollAndJamCount;
BeforeJam.UnrollAndJamEnable = Attrs.UnrollAndJamEnable;
// Set the inner followup metadata to process by the outer loop. Only
// consider the first inner loop.
if (!Parent->UnrollAndJamInnerFollowup) {
// Splitting the attributes into a BeforeJam and an AfterJam part will
// stop 'llvm.loop.isvectorized' (generated by vectorization in BeforeJam)
// to be forwarded to the AfterJam part. We detect the situation here and
// add it manually.
SmallVector<Metadata *, 1> BeforeLoopProperties;
if (BeforeJam.VectorizeEnable != LoopAttributes::Unspecified ||
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
BeforeJam.VectorizePredicateEnable != LoopAttributes::Unspecified ||
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
BeforeJam.InterleaveCount != 0 || BeforeJam.VectorizeWidth != 0 ||
BeforeJam.VectorizeScalable == LoopAttributes::Enable)
[CodeGen] Generate follow-up metadata for loops with more than one transformation. Before this patch, CGLoop would dump all transformations for a loop into a single LoopID without encoding any order in which to apply them. rL348944 added the possibility to encode a transformation order using followup-attributes. When a loop has more than one transformation, use the follow-up attribute define the order in which they are applied. The emitted order is the defacto order as defined by the current LLVM pass pipeline, which is: LoopFullUnrollPass LoopDistributePass LoopVectorizePass LoopUnrollAndJamPass LoopUnrollPass MachinePipeliner This patch should therefore not change the assembly output, assuming that all explicit transformations can be applied, and no implicit transformations in-between. In the former case, WarnMissedTransformationsPass should emit a warning (except for MachinePipeliner which is not implemented yet). The latter could be avoided by adding 'llvm.loop.disable_nonforced' attributes. Because LoopUnrollAndJamPass processes a loop nest, generation of the MDNode is delayed to after the inner loop metadata have been processed. A temporary LoopID is therefore used to annotate instructions and RAUW'ed by the actual LoopID later. Differential Revision: https://reviews.llvm.org/D57978 llvm-svn: 357415
2019-04-02 01:47:41 +08:00
BeforeLoopProperties.push_back(
MDNode::get(Ctx, MDString::get(Ctx, "llvm.loop.isvectorized")));
bool InnerFollowupHasTransform = false;
MDNode *InnerFollowup = createMetadata(AfterJam, BeforeLoopProperties,
InnerFollowupHasTransform);
if (InnerFollowupHasTransform)
Parent->UnrollAndJamInnerFollowup = InnerFollowup;
}
CurLoopAttr = BeforeJam;
}
bool HasUserTransforms = false;
LoopID = createMetadata(CurLoopAttr, {}, HasUserTransforms);
TempLoopID->replaceAllUsesWith(LoopID);
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
}
[CodeGen] Pass objects that are expensive to copy by const ref. No functionality change. Found by clang-tidy's performance-unnecessary-value-param. llvm-svn: 287894
2016-11-25 00:01:20 +08:00
void LoopInfoStack::push(BasicBlock *Header, const llvm::DebugLoc &StartLoc,
const llvm::DebugLoc &EndLoc) {
Don't keep stale pointers to LoopInfos. CGLoopInfo was keeping pointers to parent loop LoopInfos, but when the loop info vector grew, it reallocated the storage and invalidated all of the parent pointers, causing use-after-free. Manage the lifetimes of the LoopInfos separately so that the pointers aren't stale. Patch by Bevin Hansson. llvm-svn: 369259
2019-08-19 21:37:41 +08:00
Active.emplace_back(
new LoopInfo(Header, StagedAttrs, StartLoc, EndLoc,
Active.empty() ? nullptr : Active.back().get()));
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
// Clear the attributes so nested loops do not inherit them.
StagedAttrs.clear();
}
void LoopInfoStack::push(BasicBlock *Header, clang::ASTContext &Ctx,
[Clang] Add llvm.loop.unroll.disable to loops with -fno-unroll-loops. Currently Clang does not respect -fno-unroll-loops during LTO. During D76916 it was suggested to respect -fno-unroll-loops on a TU basis. This patch uses the existing llvm.loop.unroll.disable metadata to disable loop unrolling explicitly for each loop in the TU if unrolling is disabled. This should ensure that loops from TUs compiled with -fno-unroll-loops are skipped by the unroller during LTO. This also means that if a loop from a TU with -fno-unroll-loops gets inlined into a TU without this option, the loop won't be unrolled. Due to the fact that some transforms might drop loop metadata, there potentially are cases in which we still unroll loops from TUs with -fno-unroll-loops. I think we should fix those issues rather than introducing a function attribute to disable loop unrolling during LTO. Improving the metadata handling will benefit other use cases, like various loop pragmas, too. And it is an improvement to clang completely ignoring -fno-unroll-loops during LTO. If that direction looks good, we can use a similar approach to also respect -fno-vectorize during LTO, at least for LoopVectorize. In the future, this might also allow us to remove the UnrollLoops option LLVM's PassManagerBuilder. Reviewers: Meinersbur, hfinkel, dexonsmith, tejohnson Reviewed By: Meinersbur, tejohnson Differential Revision: https://reviews.llvm.org/D77058
2020-04-07 20:43:48 +08:00
const clang::CodeGenOptions &CGOpts,
Add a loop's debug location to its llvm.loop metadata Getting accurate locations for loops is important, because those locations are used by the frontend to generate optimization remarks. Currently, optimization remarks for loops often appear on the wrong line, often the first line of the loop body instead of the loop itself. This is confusing because that line might itself be another loop, or might be somewhere else completely if the body was an inlined function call. This happens because of the way we find the loop's starting location. First, we look for a preheader, and if we find one, and its terminator has a debug location, then we use that. Otherwise, we look for a location on an instruction in the loop header. The fallback heuristic is not bad, but will almost always find the beginning of the body, and not the loop statement itself. The preheader location search often fails because there's often not a preheader, and even when there is a preheader, depending on how it was formed, it sometimes carries the location of some preceeding code. I don't see any good theoretical way to fix this problem. On the other hand, this seems like a straightforward solution: Put the debug location in the loop's llvm.loop metadata. When emitting debug information, this commit causes us to add the debug location as an operand to each loop's llvm.loop metadata. Thus, we now generate this metadata for all loops (not just loops with optimization hints) when we're otherwise generating debug information. The remark test case changes depend on the companion LLVM commit r270771. llvm-svn: 270772
2016-05-26 05:53:24 +08:00
ArrayRef<const clang::Attr *> Attrs,
[CodeGen] Pass objects that are expensive to copy by const ref. No functionality change. Found by clang-tidy's performance-unnecessary-value-param. llvm-svn: 287894
2016-11-25 00:01:20 +08:00
const llvm::DebugLoc &StartLoc,
[clang] Add mustprogress and llvm.loop.mustprogress attribute deduction Since C++11, the C++ standard has a forward progress guarantee [intro.progress], so all such functions must have the `mustprogress` requirement. In addition, from C11 and onwards, loops without a non-zero constant conditional or no conditional are also required to make progress (C11 6.8.5p6). This patch implements these attribute deductions so they can be used by the optimization passes. Differential Revision: https://reviews.llvm.org/D86841
2020-11-03 05:03:21 +08:00
const llvm::DebugLoc &EndLoc, bool MustProgress) {
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
// Identify loop hint attributes from Attrs.
Add assume_safety option for pragma loop vectorize and interleave. Specifying #pragma clang loop vectorize(assume_safety) on a loop adds the mem.parallel_loop_access metadata to each load/store operation in the loop. This metadata tells loop access analysis (LAA) to skip memory dependency checking. llvm-svn: 239572
2015-06-12 07:23:17 +08:00
for (const auto *Attr : Attrs) {
const LoopHintAttr *LH = dyn_cast<LoopHintAttr>(Attr);
[OpenCL] Generate metadata for opencl_unroll_hint attribute Add support for opencl_unroll_hint attribute from OpenCL v2.0 s6.11.5. Reusing most of metadata generation from CGLoopInfo helper class. The code is based on Khronos OpenCL compiler: https://github.com/KhronosGroup/SPIR/tree/spirv-1.0 Patch by Liu Yaxun (Sam)! Differential Revision: http://reviews.llvm.org/D16686 llvm-svn: 261350
2016-02-20 02:30:11 +08:00
const OpenCLUnrollHintAttr *OpenCLHint =
dyn_cast<OpenCLUnrollHintAttr>(Attr);
Add assume_safety option for pragma loop vectorize and interleave. Specifying #pragma clang loop vectorize(assume_safety) on a loop adds the mem.parallel_loop_access metadata to each load/store operation in the loop. This metadata tells loop access analysis (LAA) to skip memory dependency checking. llvm-svn: 239572
2015-06-12 07:23:17 +08:00
// Skip non loop hint attributes
[OpenCL] Generate metadata for opencl_unroll_hint attribute Add support for opencl_unroll_hint attribute from OpenCL v2.0 s6.11.5. Reusing most of metadata generation from CGLoopInfo helper class. The code is based on Khronos OpenCL compiler: https://github.com/KhronosGroup/SPIR/tree/spirv-1.0 Patch by Liu Yaxun (Sam)! Differential Revision: http://reviews.llvm.org/D16686 llvm-svn: 261350
2016-02-20 02:30:11 +08:00
if (!LH && !OpenCLHint) {
Add assume_safety option for pragma loop vectorize and interleave. Specifying #pragma clang loop vectorize(assume_safety) on a loop adds the mem.parallel_loop_access metadata to each load/store operation in the loop. This metadata tells loop access analysis (LAA) to skip memory dependency checking. llvm-svn: 239572
2015-06-12 07:23:17 +08:00
continue;
[OpenCL] Generate metadata for opencl_unroll_hint attribute Add support for opencl_unroll_hint attribute from OpenCL v2.0 s6.11.5. Reusing most of metadata generation from CGLoopInfo helper class. The code is based on Khronos OpenCL compiler: https://github.com/KhronosGroup/SPIR/tree/spirv-1.0 Patch by Liu Yaxun (Sam)! Differential Revision: http://reviews.llvm.org/D16686 llvm-svn: 261350
2016-02-20 02:30:11 +08:00
}
Add assume_safety option for pragma loop vectorize and interleave. Specifying #pragma clang loop vectorize(assume_safety) on a loop adds the mem.parallel_loop_access metadata to each load/store operation in the loop. This metadata tells loop access analysis (LAA) to skip memory dependency checking. llvm-svn: 239572
2015-06-12 07:23:17 +08:00
[OpenCL] Generate metadata for opencl_unroll_hint attribute Add support for opencl_unroll_hint attribute from OpenCL v2.0 s6.11.5. Reusing most of metadata generation from CGLoopInfo helper class. The code is based on Khronos OpenCL compiler: https://github.com/KhronosGroup/SPIR/tree/spirv-1.0 Patch by Liu Yaxun (Sam)! Differential Revision: http://reviews.llvm.org/D16686 llvm-svn: 261350
2016-02-20 02:30:11 +08:00
LoopHintAttr::OptionType Option = LoopHintAttr::Unroll;
LoopHintAttr::LoopHintState State = LoopHintAttr::Disable;
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
unsigned ValueInt = 1;
[OpenCL] Generate metadata for opencl_unroll_hint attribute Add support for opencl_unroll_hint attribute from OpenCL v2.0 s6.11.5. Reusing most of metadata generation from CGLoopInfo helper class. The code is based on Khronos OpenCL compiler: https://github.com/KhronosGroup/SPIR/tree/spirv-1.0 Patch by Liu Yaxun (Sam)! Differential Revision: http://reviews.llvm.org/D16686 llvm-svn: 261350
2016-02-20 02:30:11 +08:00
// Translate opencl_unroll_hint attribute argument to
// equivalent LoopHintAttr enums.
Remove trailing space sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h} llvm-svn: 338291
2018-07-31 03:24:48 +08:00
// OpenCL v2.0 s6.11.5:
[OpenCL] Generate 'unroll.enable' metadata for __attribute__((opencl_unroll_hint)) Summary: [OpenCL] Generate 'unroll.enable' metadata for __attribute__((opencl_unroll_hint)) For both !{!"llvm.loop.unroll.enable"} and !{!"llvm.loop.unroll.full"} the unroller will try to fully unroll a loop unless the trip count is not known at compile time. In that case for '.full' metadata no unrolling will be processed, while for '.enable' the loop will be partially unrolled with a heuristically chosen unroll factor. See: docs/LanguageExtensions.rst From https://www.khronos.org/registry/OpenCL/sdk/2.0/docs/man/xhtml/attributes-loopUnroll.html __attribute__((opencl_unroll_hint)) for (int i=0; i<2; i++) { ... } In the example above, the compiler will determine how much to unroll the loop. Before the patch for __attribute__((opencl_unroll_hint)) was generated metadata !{!"llvm.loop.unroll.full"}, which limits ability of loop unroller to decide, how much to unroll the loop. Reviewers: Anastasia, yaxunl Reviewed By: Anastasia Subscribers: zzheng, dmgreen, jdoerfert, cfe-commits, asavonic, AlexeySotkin Tags: #clang Differential Revision: https://reviews.llvm.org/D59493 llvm-svn: 356571
2019-03-21 00:43:07 +08:00
// 0 - enable unroll (no argument).
[OpenCL] Generate metadata for opencl_unroll_hint attribute Add support for opencl_unroll_hint attribute from OpenCL v2.0 s6.11.5. Reusing most of metadata generation from CGLoopInfo helper class. The code is based on Khronos OpenCL compiler: https://github.com/KhronosGroup/SPIR/tree/spirv-1.0 Patch by Liu Yaxun (Sam)! Differential Revision: http://reviews.llvm.org/D16686 llvm-svn: 261350
2016-02-20 02:30:11 +08:00
// 1 - disable unroll.
// other positive integer n - unroll by n.
if (OpenCLHint) {
ValueInt = OpenCLHint->getUnrollHint();
if (ValueInt == 0) {
[OpenCL] Generate 'unroll.enable' metadata for __attribute__((opencl_unroll_hint)) Summary: [OpenCL] Generate 'unroll.enable' metadata for __attribute__((opencl_unroll_hint)) For both !{!"llvm.loop.unroll.enable"} and !{!"llvm.loop.unroll.full"} the unroller will try to fully unroll a loop unless the trip count is not known at compile time. In that case for '.full' metadata no unrolling will be processed, while for '.enable' the loop will be partially unrolled with a heuristically chosen unroll factor. See: docs/LanguageExtensions.rst From https://www.khronos.org/registry/OpenCL/sdk/2.0/docs/man/xhtml/attributes-loopUnroll.html __attribute__((opencl_unroll_hint)) for (int i=0; i<2; i++) { ... } In the example above, the compiler will determine how much to unroll the loop. Before the patch for __attribute__((opencl_unroll_hint)) was generated metadata !{!"llvm.loop.unroll.full"}, which limits ability of loop unroller to decide, how much to unroll the loop. Reviewers: Anastasia, yaxunl Reviewed By: Anastasia Subscribers: zzheng, dmgreen, jdoerfert, cfe-commits, asavonic, AlexeySotkin Tags: #clang Differential Revision: https://reviews.llvm.org/D59493 llvm-svn: 356571
2019-03-21 00:43:07 +08:00
State = LoopHintAttr::Enable;
[OpenCL] Generate metadata for opencl_unroll_hint attribute Add support for opencl_unroll_hint attribute from OpenCL v2.0 s6.11.5. Reusing most of metadata generation from CGLoopInfo helper class. The code is based on Khronos OpenCL compiler: https://github.com/KhronosGroup/SPIR/tree/spirv-1.0 Patch by Liu Yaxun (Sam)! Differential Revision: http://reviews.llvm.org/D16686 llvm-svn: 261350
2016-02-20 02:30:11 +08:00
} else if (ValueInt != 1) {
Option = LoopHintAttr::UnrollCount;
State = LoopHintAttr::Numeric;
}
} else if (LH) {
auto *ValueExpr = LH->getValue();
if (ValueExpr) {
llvm::APSInt ValueAPS = ValueExpr->EvaluateKnownConstInt(Ctx);
ValueInt = ValueAPS.getSExtValue();
}
Option = LH->getOption();
State = LH->getState();
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
}
switch (State) {
case LoopHintAttr::Disable:
switch (Option) {
case LoopHintAttr::Vectorize:
// Disable vectorization by specifying a width of 1.
setVectorizeWidth(1);
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
setVectorizeScalable(LoopAttributes::Unspecified);
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
break;
case LoopHintAttr::Interleave:
// Disable interleaving by speciyfing a count of 1.
setInterleaveCount(1);
break;
case LoopHintAttr::Unroll:
Add new llvm.loop.unroll.enable metadata for use with "#pragma unroll". This change adds the new unroll metadata "llvm.loop.unroll.enable" which directs the optimizer to unroll a loop fully if the trip count is known at compile time, and unroll partially if the trip count is not known at compile time. This differs from "llvm.loop.unroll.full" which explicitly does not unroll a loop if the trip count is not known at compile time With this change "#pragma unroll" generates "llvm.loop.unroll.enable" rather than "llvm.loop.unroll.full" metadata. This changes the semantics of "#pragma unroll" slightly to mean "unroll aggressively (fully or partially)" rather than "unroll fully or not at all". The motivating example for this change was some internal code with a loop marked with "#pragma unroll" which only sometimes had a compile-time trip count depending on template magic. When the trip count was a compile-time constant, everything works as expected and the loop is fully unrolled. However, when the trip count was not a compile-time constant the "#pragma unroll" explicitly disabled unrolling of the loop(!). Removing "#pragma unroll" caused the loop to be unrolled partially which was desirable from a performance perspective. llvm-svn: 244467
2015-08-11 01:29:39 +08:00
setUnrollState(LoopAttributes::Disable);
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
break;
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
case LoopHintAttr::UnrollAndJam:
setUnrollAndJamState(LoopAttributes::Disable);
break;
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
case LoopHintAttr::VectorizePredicate:
setVectorizePredicateState(LoopAttributes::Disable);
break;
Add loop pragma for Loop Distribution Summary: This is similar to other loop pragmas like 'vectorize'. Currently it only has state values: distribute(enable) and distribute(disable). When one of these is specified the corresponding loop metadata is generated: !{!"llvm.loop.distribute.enable", i1 true/false} As a result, loop distribution will be attempted on the loop even if Loop Distribution in not enabled globally. Analogously, with 'disable' distribution can be turned off for an individual loop even when the pass is otherwise enabled. There are some slight differences compared to the existing loop pragmas. 1. There is no 'assume_safety' variant which makes its handling slightly different from 'vectorize'/'interleave'. 2. Unlike the existing loop pragmas, it does not have a corresponding numeric pragma like 'vectorize' -> 'vectorize_width'. So for the consistency checks in CheckForIncompatibleAttributes we don't need to check it against other pragmas. We just need to check for duplicates of the same pragma. Reviewers: rsmith, dexonsmith, aaron.ballman Subscribers: bob.wilson, cfe-commits, hfinkel Differential Revision: http://reviews.llvm.org/D19403 llvm-svn: 272656
2016-06-14 20:04:26 +08:00
case LoopHintAttr::Distribute:
setDistributeState(false);
break;
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
case LoopHintAttr::PipelineDisabled:
setPipelineDisabled(true);
break;
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
case LoopHintAttr::UnrollCount:
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
case LoopHintAttr::UnrollAndJamCount:
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
case LoopHintAttr::VectorizeWidth:
case LoopHintAttr::InterleaveCount:
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
case LoopHintAttr::PipelineInitiationInterval:
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
llvm_unreachable("Options cannot be disabled.");
break;
}
break;
case LoopHintAttr::Enable:
switch (Option) {
case LoopHintAttr::Vectorize:
case LoopHintAttr::Interleave:
setVectorizeEnable(true);
break;
case LoopHintAttr::Unroll:
Add new llvm.loop.unroll.enable metadata for use with "#pragma unroll". This change adds the new unroll metadata "llvm.loop.unroll.enable" which directs the optimizer to unroll a loop fully if the trip count is known at compile time, and unroll partially if the trip count is not known at compile time. This differs from "llvm.loop.unroll.full" which explicitly does not unroll a loop if the trip count is not known at compile time With this change "#pragma unroll" generates "llvm.loop.unroll.enable" rather than "llvm.loop.unroll.full" metadata. This changes the semantics of "#pragma unroll" slightly to mean "unroll aggressively (fully or partially)" rather than "unroll fully or not at all". The motivating example for this change was some internal code with a loop marked with "#pragma unroll" which only sometimes had a compile-time trip count depending on template magic. When the trip count was a compile-time constant, everything works as expected and the loop is fully unrolled. However, when the trip count was not a compile-time constant the "#pragma unroll" explicitly disabled unrolling of the loop(!). Removing "#pragma unroll" caused the loop to be unrolled partially which was desirable from a performance perspective. llvm-svn: 244467
2015-08-11 01:29:39 +08:00
setUnrollState(LoopAttributes::Enable);
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
break;
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
case LoopHintAttr::UnrollAndJam:
setUnrollAndJamState(LoopAttributes::Enable);
break;
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
case LoopHintAttr::VectorizePredicate:
setVectorizePredicateState(LoopAttributes::Enable);
break;
Add loop pragma for Loop Distribution Summary: This is similar to other loop pragmas like 'vectorize'. Currently it only has state values: distribute(enable) and distribute(disable). When one of these is specified the corresponding loop metadata is generated: !{!"llvm.loop.distribute.enable", i1 true/false} As a result, loop distribution will be attempted on the loop even if Loop Distribution in not enabled globally. Analogously, with 'disable' distribution can be turned off for an individual loop even when the pass is otherwise enabled. There are some slight differences compared to the existing loop pragmas. 1. There is no 'assume_safety' variant which makes its handling slightly different from 'vectorize'/'interleave'. 2. Unlike the existing loop pragmas, it does not have a corresponding numeric pragma like 'vectorize' -> 'vectorize_width'. So for the consistency checks in CheckForIncompatibleAttributes we don't need to check it against other pragmas. We just need to check for duplicates of the same pragma. Reviewers: rsmith, dexonsmith, aaron.ballman Subscribers: bob.wilson, cfe-commits, hfinkel Differential Revision: http://reviews.llvm.org/D19403 llvm-svn: 272656
2016-06-14 20:04:26 +08:00
case LoopHintAttr::Distribute:
setDistributeState(true);
break;
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
case LoopHintAttr::UnrollCount:
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
case LoopHintAttr::UnrollAndJamCount:
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
case LoopHintAttr::VectorizeWidth:
case LoopHintAttr::InterleaveCount:
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
case LoopHintAttr::PipelineDisabled:
case LoopHintAttr::PipelineInitiationInterval:
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
llvm_unreachable("Options cannot enabled.");
break;
}
break;
case LoopHintAttr::AssumeSafety:
switch (Option) {
case LoopHintAttr::Vectorize:
case LoopHintAttr::Interleave:
Add assume_safety option for pragma loop vectorize and interleave. Specifying #pragma clang loop vectorize(assume_safety) on a loop adds the mem.parallel_loop_access metadata to each load/store operation in the loop. This metadata tells loop access analysis (LAA) to skip memory dependency checking. llvm-svn: 239572
2015-06-12 07:23:17 +08:00
// Apply "llvm.mem.parallel_loop_access" metadata to load/stores.
setParallel(true);
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
setVectorizeEnable(true);
break;
case LoopHintAttr::Unroll:
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
case LoopHintAttr::UnrollAndJam:
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
case LoopHintAttr::VectorizePredicate:
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
case LoopHintAttr::UnrollCount:
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
case LoopHintAttr::UnrollAndJamCount:
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
case LoopHintAttr::VectorizeWidth:
case LoopHintAttr::InterleaveCount:
Add loop pragma for Loop Distribution Summary: This is similar to other loop pragmas like 'vectorize'. Currently it only has state values: distribute(enable) and distribute(disable). When one of these is specified the corresponding loop metadata is generated: !{!"llvm.loop.distribute.enable", i1 true/false} As a result, loop distribution will be attempted on the loop even if Loop Distribution in not enabled globally. Analogously, with 'disable' distribution can be turned off for an individual loop even when the pass is otherwise enabled. There are some slight differences compared to the existing loop pragmas. 1. There is no 'assume_safety' variant which makes its handling slightly different from 'vectorize'/'interleave'. 2. Unlike the existing loop pragmas, it does not have a corresponding numeric pragma like 'vectorize' -> 'vectorize_width'. So for the consistency checks in CheckForIncompatibleAttributes we don't need to check it against other pragmas. We just need to check for duplicates of the same pragma. Reviewers: rsmith, dexonsmith, aaron.ballman Subscribers: bob.wilson, cfe-commits, hfinkel Differential Revision: http://reviews.llvm.org/D19403 llvm-svn: 272656
2016-06-14 20:04:26 +08:00
case LoopHintAttr::Distribute:
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
case LoopHintAttr::PipelineDisabled:
case LoopHintAttr::PipelineInitiationInterval:
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
llvm_unreachable("Options cannot be used to assume mem safety.");
break;
Add assume_safety option for pragma loop vectorize and interleave. Specifying #pragma clang loop vectorize(assume_safety) on a loop adds the mem.parallel_loop_access metadata to each load/store operation in the loop. This metadata tells loop access analysis (LAA) to skip memory dependency checking. llvm-svn: 239572
2015-06-12 07:23:17 +08:00
}
break;
Add new llvm.loop.unroll.enable metadata for use with "#pragma unroll". This change adds the new unroll metadata "llvm.loop.unroll.enable" which directs the optimizer to unroll a loop fully if the trip count is known at compile time, and unroll partially if the trip count is not known at compile time. This differs from "llvm.loop.unroll.full" which explicitly does not unroll a loop if the trip count is not known at compile time With this change "#pragma unroll" generates "llvm.loop.unroll.enable" rather than "llvm.loop.unroll.full" metadata. This changes the semantics of "#pragma unroll" slightly to mean "unroll aggressively (fully or partially)" rather than "unroll fully or not at all". The motivating example for this change was some internal code with a loop marked with "#pragma unroll" which only sometimes had a compile-time trip count depending on template magic. When the trip count was a compile-time constant, everything works as expected and the loop is fully unrolled. However, when the trip count was not a compile-time constant the "#pragma unroll" explicitly disabled unrolling of the loop(!). Removing "#pragma unroll" caused the loop to be unrolled partially which was desirable from a performance perspective. llvm-svn: 244467
2015-08-11 01:29:39 +08:00
case LoopHintAttr::Full:
switch (Option) {
case LoopHintAttr::Unroll:
setUnrollState(LoopAttributes::Full);
break;
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
case LoopHintAttr::UnrollAndJam:
setUnrollAndJamState(LoopAttributes::Full);
break;
Add new llvm.loop.unroll.enable metadata for use with "#pragma unroll". This change adds the new unroll metadata "llvm.loop.unroll.enable" which directs the optimizer to unroll a loop fully if the trip count is known at compile time, and unroll partially if the trip count is not known at compile time. This differs from "llvm.loop.unroll.full" which explicitly does not unroll a loop if the trip count is not known at compile time With this change "#pragma unroll" generates "llvm.loop.unroll.enable" rather than "llvm.loop.unroll.full" metadata. This changes the semantics of "#pragma unroll" slightly to mean "unroll aggressively (fully or partially)" rather than "unroll fully or not at all". The motivating example for this change was some internal code with a loop marked with "#pragma unroll" which only sometimes had a compile-time trip count depending on template magic. When the trip count was a compile-time constant, everything works as expected and the loop is fully unrolled. However, when the trip count was not a compile-time constant the "#pragma unroll" explicitly disabled unrolling of the loop(!). Removing "#pragma unroll" caused the loop to be unrolled partially which was desirable from a performance perspective. llvm-svn: 244467
2015-08-11 01:29:39 +08:00
case LoopHintAttr::Vectorize:
case LoopHintAttr::Interleave:
case LoopHintAttr::UnrollCount:
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
case LoopHintAttr::UnrollAndJamCount:
Add new llvm.loop.unroll.enable metadata for use with "#pragma unroll". This change adds the new unroll metadata "llvm.loop.unroll.enable" which directs the optimizer to unroll a loop fully if the trip count is known at compile time, and unroll partially if the trip count is not known at compile time. This differs from "llvm.loop.unroll.full" which explicitly does not unroll a loop if the trip count is not known at compile time With this change "#pragma unroll" generates "llvm.loop.unroll.enable" rather than "llvm.loop.unroll.full" metadata. This changes the semantics of "#pragma unroll" slightly to mean "unroll aggressively (fully or partially)" rather than "unroll fully or not at all". The motivating example for this change was some internal code with a loop marked with "#pragma unroll" which only sometimes had a compile-time trip count depending on template magic. When the trip count was a compile-time constant, everything works as expected and the loop is fully unrolled. However, when the trip count was not a compile-time constant the "#pragma unroll" explicitly disabled unrolling of the loop(!). Removing "#pragma unroll" caused the loop to be unrolled partially which was desirable from a performance perspective. llvm-svn: 244467
2015-08-11 01:29:39 +08:00
case LoopHintAttr::VectorizeWidth:
case LoopHintAttr::InterleaveCount:
Add loop pragma for Loop Distribution Summary: This is similar to other loop pragmas like 'vectorize'. Currently it only has state values: distribute(enable) and distribute(disable). When one of these is specified the corresponding loop metadata is generated: !{!"llvm.loop.distribute.enable", i1 true/false} As a result, loop distribution will be attempted on the loop even if Loop Distribution in not enabled globally. Analogously, with 'disable' distribution can be turned off for an individual loop even when the pass is otherwise enabled. There are some slight differences compared to the existing loop pragmas. 1. There is no 'assume_safety' variant which makes its handling slightly different from 'vectorize'/'interleave'. 2. Unlike the existing loop pragmas, it does not have a corresponding numeric pragma like 'vectorize' -> 'vectorize_width'. So for the consistency checks in CheckForIncompatibleAttributes we don't need to check it against other pragmas. We just need to check for duplicates of the same pragma. Reviewers: rsmith, dexonsmith, aaron.ballman Subscribers: bob.wilson, cfe-commits, hfinkel Differential Revision: http://reviews.llvm.org/D19403 llvm-svn: 272656
2016-06-14 20:04:26 +08:00
case LoopHintAttr::Distribute:
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
case LoopHintAttr::PipelineDisabled:
case LoopHintAttr::PipelineInitiationInterval:
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
case LoopHintAttr::VectorizePredicate:
Add new llvm.loop.unroll.enable metadata for use with "#pragma unroll". This change adds the new unroll metadata "llvm.loop.unroll.enable" which directs the optimizer to unroll a loop fully if the trip count is known at compile time, and unroll partially if the trip count is not known at compile time. This differs from "llvm.loop.unroll.full" which explicitly does not unroll a loop if the trip count is not known at compile time With this change "#pragma unroll" generates "llvm.loop.unroll.enable" rather than "llvm.loop.unroll.full" metadata. This changes the semantics of "#pragma unroll" slightly to mean "unroll aggressively (fully or partially)" rather than "unroll fully or not at all". The motivating example for this change was some internal code with a loop marked with "#pragma unroll" which only sometimes had a compile-time trip count depending on template magic. When the trip count was a compile-time constant, everything works as expected and the loop is fully unrolled. However, when the trip count was not a compile-time constant the "#pragma unroll" explicitly disabled unrolling of the loop(!). Removing "#pragma unroll" caused the loop to be unrolled partially which was desirable from a performance perspective. llvm-svn: 244467
2015-08-11 01:29:39 +08:00
llvm_unreachable("Options cannot be used with 'full' hint.");
break;
}
break;
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
case LoopHintAttr::FixedWidth:
case LoopHintAttr::ScalableWidth:
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
switch (Option) {
case LoopHintAttr::VectorizeWidth:
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
setVectorizeScalable(State == LoopHintAttr::ScalableWidth
? LoopAttributes::Enable
: LoopAttributes::Disable);
if (LH->getValue())
setVectorizeWidth(ValueInt);
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
break;
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
default:
llvm_unreachable("Options cannot be used with 'scalable' hint.");
break;
}
break;
case LoopHintAttr::Numeric:
switch (Option) {
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
case LoopHintAttr::InterleaveCount:
setInterleaveCount(ValueInt);
break;
case LoopHintAttr::UnrollCount:
setUnrollCount(ValueInt);
break;
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
case LoopHintAttr::UnrollAndJamCount:
setUnrollAndJamCount(ValueInt);
break;
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
case LoopHintAttr::PipelineInitiationInterval:
setPipelineInitiationInterval(ValueInt);
break;
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
case LoopHintAttr::Unroll:
[UnrollAndJam] Add unroll_and_jam pragma handling This adds support for the unroll_and_jam pragma, to go with the recently added unroll and jam pass. The name of the pragma is the same as is used in the Intel compiler, and most of the code works the same as for unroll. #pragma clang loop unroll_and_jam has been separated into a different patch. This part adds #pragma unroll_and_jam with an optional count, and #pragma no_unroll_and_jam to disable the transform. Differential Revision: https://reviews.llvm.org/D47267 llvm-svn: 338566
2018-08-01 22:36:12 +08:00
case LoopHintAttr::UnrollAndJam:
[Clang] New loop pragma vectorize_predicate This adds a new vectorize predication loop hint: #pragma clang loop vectorize_predicate(enable) that can be used to indicate to the vectoriser that all (load/store) instructions should be predicated (masked). This allows, for example, folding of the remainder loop into the main loop. This patch will be followed up with D64916 and D65197. The former is a refactoring in the loopvectorizer and the groundwork to make tail loop folding a more general concept, and in the latter the actual tail loop folding transformation will be implemented. Differential Revision: https://reviews.llvm.org/D64744 llvm-svn: 366989
2019-07-25 15:33:13 +08:00
case LoopHintAttr::VectorizePredicate:
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
case LoopHintAttr::Vectorize:
[SVE] Add support to vectorize_width loop pragma for scalable vectors This patch adds support for two new variants of the vectorize_width pragma: 1. vectorize_width(X[, fixed|scalable]) where an optional second parameter is passed to the vectorize_width pragma, which indicates if the user wishes to use fixed width or scalable vectorization. For example the user can now write something like: #pragma clang loop vectorize_width(4, fixed) or #pragma clang loop vectorize_width(4, scalable) In the absence of a second parameter it is assumed the user wants fixed width vectorization, in order to maintain compatibility with existing code. 2. vectorize_width(fixed|scalable) where the width is left unspecified, but the user hints what type of vectorization they prefer, either fixed width or scalable. I have implemented this by making use of the LLVM loop hint attribute: llvm.loop.vectorize.scalable.enable Tests were added to clang/test/CodeGenCXX/pragma-loop.cpp for both the 'fixed' and 'scalable' optional parameter. See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html Differential Revision: https://reviews.llvm.org/D89031
2020-10-02 14:46:42 +08:00
case LoopHintAttr::VectorizeWidth:
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
case LoopHintAttr::Interleave:
Add loop pragma for Loop Distribution Summary: This is similar to other loop pragmas like 'vectorize'. Currently it only has state values: distribute(enable) and distribute(disable). When one of these is specified the corresponding loop metadata is generated: !{!"llvm.loop.distribute.enable", i1 true/false} As a result, loop distribution will be attempted on the loop even if Loop Distribution in not enabled globally. Analogously, with 'disable' distribution can be turned off for an individual loop even when the pass is otherwise enabled. There are some slight differences compared to the existing loop pragmas. 1. There is no 'assume_safety' variant which makes its handling slightly different from 'vectorize'/'interleave'. 2. Unlike the existing loop pragmas, it does not have a corresponding numeric pragma like 'vectorize' -> 'vectorize_width'. So for the consistency checks in CheckForIncompatibleAttributes we don't need to check it against other pragmas. We just need to check for duplicates of the same pragma. Reviewers: rsmith, dexonsmith, aaron.ballman Subscribers: bob.wilson, cfe-commits, hfinkel Differential Revision: http://reviews.llvm.org/D19403 llvm-svn: 272656
2016-06-14 20:04:26 +08:00
case LoopHintAttr::Distribute:
Add two new pragmas for controlling software pipelining optimizations. This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval. Patch by Alexey Lapshin. llvm-svn: 350414
2019-01-05 01:20:00 +08:00
case LoopHintAttr::PipelineDisabled:
Add new llvm.loop.unroll.enable metadata for use with "#pragma unroll". This change adds the new unroll metadata "llvm.loop.unroll.enable" which directs the optimizer to unroll a loop fully if the trip count is known at compile time, and unroll partially if the trip count is not known at compile time. This differs from "llvm.loop.unroll.full" which explicitly does not unroll a loop if the trip count is not known at compile time With this change "#pragma unroll" generates "llvm.loop.unroll.enable" rather than "llvm.loop.unroll.full" metadata. This changes the semantics of "#pragma unroll" slightly to mean "unroll aggressively (fully or partially)" rather than "unroll fully or not at all". The motivating example for this change was some internal code with a loop marked with "#pragma unroll" which only sometimes had a compile-time trip count depending on template magic. When the trip count was a compile-time constant, everything works as expected and the loop is fully unrolled. However, when the trip count was not a compile-time constant the "#pragma unroll" explicitly disabled unrolling of the loop(!). Removing "#pragma unroll" caused the loop to be unrolled partially which was desirable from a performance perspective. llvm-svn: 244467
2015-08-11 01:29:39 +08:00
llvm_unreachable("Options cannot be assigned a value.");
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
break;
}
Add assume_safety option for pragma loop vectorize and interleave. Specifying #pragma clang loop vectorize(assume_safety) on a loop adds the mem.parallel_loop_access metadata to each load/store operation in the loop. This metadata tells loop access analysis (LAA) to skip memory dependency checking. llvm-svn: 239572
2015-06-12 07:23:17 +08:00
break;
}
}
[clang] Add mustprogress and llvm.loop.mustprogress attribute deduction Since C++11, the C++ standard has a forward progress guarantee [intro.progress], so all such functions must have the `mustprogress` requirement. In addition, from C11 and onwards, loops without a non-zero constant conditional or no conditional are also required to make progress (C11 6.8.5p6). This patch implements these attribute deductions so they can be used by the optimization passes. Differential Revision: https://reviews.llvm.org/D86841
2020-11-03 05:03:21 +08:00
setMustProgress(MustProgress);
[Clang] Add llvm.loop.unroll.disable to loops with -fno-unroll-loops. Currently Clang does not respect -fno-unroll-loops during LTO. During D76916 it was suggested to respect -fno-unroll-loops on a TU basis. This patch uses the existing llvm.loop.unroll.disable metadata to disable loop unrolling explicitly for each loop in the TU if unrolling is disabled. This should ensure that loops from TUs compiled with -fno-unroll-loops are skipped by the unroller during LTO. This also means that if a loop from a TU with -fno-unroll-loops gets inlined into a TU without this option, the loop won't be unrolled. Due to the fact that some transforms might drop loop metadata, there potentially are cases in which we still unroll loops from TUs with -fno-unroll-loops. I think we should fix those issues rather than introducing a function attribute to disable loop unrolling during LTO. Improving the metadata handling will benefit other use cases, like various loop pragmas, too. And it is an improvement to clang completely ignoring -fno-unroll-loops during LTO. If that direction looks good, we can use a similar approach to also respect -fno-vectorize during LTO, at least for LoopVectorize. In the future, this might also allow us to remove the UnrollLoops option LLVM's PassManagerBuilder. Reviewers: Meinersbur, hfinkel, dexonsmith, tejohnson Reviewed By: Meinersbur, tejohnson Differential Revision: https://reviews.llvm.org/D77058
2020-04-07 20:43:48 +08:00
if (CGOpts.OptimizationLevel > 0)
// Disable unrolling for the loop, if unrolling is disabled (via
// -fno-unroll-loops) and no pragmas override the decision.
if (!CGOpts.UnrollLoops &&
(StagedAttrs.UnrollEnable == LoopAttributes::Unspecified &&
StagedAttrs.UnrollCount == 0))
setUnrollState(LoopAttributes::Disable);
Use CGLoopInfo to emit metadata for loop hint pragmas. When ‘#pragma clang loop vectorize(assume_safety)’ was specified on a loop other loop hints were lost. The problem is that CGLoopInfo attaches metadata differently than EmitCondBrHints in CGStmt. For do-loops CGLoopInfo attaches metadata to the br in the body block and for while and for loops, the inc block. EmitCondBrHints on the other hand always attaches data to the br in the cond block. When specifying assume_safety CGLoopInfo emits an empty llvm.loop metadata shadowing the metadata in the cond block. Loop transformations like rotate and unswitch would then eliminate the cond block and its non-empty metadata. This patch unifies both approaches for adding metadata and modifies the existing safety tests to include non-assume_safety loop hints. llvm-svn: 243315
2015-07-28 04:10:20 +08:00
/// Stage the attributes.
Add the loop end location to the loop metadata. This additional information can be used to improve the locations when generating remarks for loops. Depends on the companion LLVM change r286227. Patch by Florian Hahn. Differential Revision: https://reviews.llvm.org/D25764 llvm-svn: 286456
2016-11-10 22:44:30 +08:00
push(Header, StartLoc, EndLoc);
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
}
void LoopInfoStack::pop() {
assert(!Active.empty() && "No active loops to pop");
Don't keep stale pointers to LoopInfos. CGLoopInfo was keeping pointers to parent loop LoopInfos, but when the loop info vector grew, it reallocated the storage and invalidated all of the parent pointers, causing use-after-free. Manage the lifetimes of the LoopInfos separately so that the pointers aren't stale. Patch by Bevin Hansson. llvm-svn: 369259
2019-08-19 21:37:41 +08:00
Active.back()->finish();
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
Active.pop_back();
}
void LoopInfoStack::InsertHelper(Instruction *I) const {
[CodeGen] Generate llvm.loop.parallel_accesses instead of llvm.mem.parallel_loop_access metadata. Instead of generating llvm.mem.parallel_loop_access metadata, generate llvm.access.group on instructions and llvm.loop.parallel_accesses on loops. There is one access group per generated loop. This is clang part of D52116/r349725. Differential Revision: https://reviews.llvm.org/D52117 llvm-svn: 349823
2018-12-21 05:24:54 +08:00
if (I->mayReadOrWriteMemory()) {
SmallVector<Metadata *, 4> AccessGroups;
Don't keep stale pointers to LoopInfos. CGLoopInfo was keeping pointers to parent loop LoopInfos, but when the loop info vector grew, it reallocated the storage and invalidated all of the parent pointers, causing use-after-free. Manage the lifetimes of the LoopInfos separately so that the pointers aren't stale. Patch by Bevin Hansson. llvm-svn: 369259
2019-08-19 21:37:41 +08:00
for (const auto &AL : Active) {
[CodeGen] Generate llvm.loop.parallel_accesses instead of llvm.mem.parallel_loop_access metadata. Instead of generating llvm.mem.parallel_loop_access metadata, generate llvm.access.group on instructions and llvm.loop.parallel_accesses on loops. There is one access group per generated loop. This is clang part of D52116/r349725. Differential Revision: https://reviews.llvm.org/D52117 llvm-svn: 349823
2018-12-21 05:24:54 +08:00
// Here we assume that every loop that has an access group is parallel.
Don't keep stale pointers to LoopInfos. CGLoopInfo was keeping pointers to parent loop LoopInfos, but when the loop info vector grew, it reallocated the storage and invalidated all of the parent pointers, causing use-after-free. Manage the lifetimes of the LoopInfos separately so that the pointers aren't stale. Patch by Bevin Hansson. llvm-svn: 369259
2019-08-19 21:37:41 +08:00
if (MDNode *Group = AL->getAccessGroup())
[CodeGen] Generate llvm.loop.parallel_accesses instead of llvm.mem.parallel_loop_access metadata. Instead of generating llvm.mem.parallel_loop_access metadata, generate llvm.access.group on instructions and llvm.loop.parallel_accesses on loops. There is one access group per generated loop. This is clang part of D52116/r349725. Differential Revision: https://reviews.llvm.org/D52117 llvm-svn: 349823
2018-12-21 05:24:54 +08:00
AccessGroups.push_back(Group);
}
MDNode *UnionMD = nullptr;
if (AccessGroups.size() == 1)
UnionMD = cast<MDNode>(AccessGroups[0]);
else if (AccessGroups.size() >= 2)
UnionMD = MDNode::get(I->getContext(), AccessGroups);
I->setMetadata("llvm.access.group", UnionMD);
}
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
if (!hasInfo())
return;
const LoopInfo &L = getInfo();
if (!L.getLoopID())
return;
[TI removal] Test predicate rather than casting to detect a terminator and use the range based successor API. llvm-svn: 344730
2018-10-18 16:16:20 +08:00
if (I->isTerminator()) {
for (BasicBlock *Succ : successors(I))
if (Succ == L.getHeader()) {
I->setMetadata(llvm::LLVMContext::MD_loop, L.getLoopID());
This patch adds a helper class (CGLoopInfo) for marking memory instructions with llvm.mem.parallel_loop_access metadata. It also adds a simple initial version of codegen for pragma omp simd (it will change in the future to support all the clauses). Differential revision: http://reviews.llvm.org/D3644 llvm-svn: 209411
2014-05-22 16:54:05 +08:00
break;
}
return;
}
}