maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/lib/IR/AutoUpgrade.cpp

2333 lines
103 KiB
C++
Raw Normal View History

This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
//===-- AutoUpgrade.cpp - Implement auto-upgrade helper functions ---------===//
//
// The LLVM Compiler Infrastructure
//
Remove attribution from file headers, per discussion on llvmdev. llvm-svn: 45418
2007-12-30 04:36:04 +08:00
// This file is distributed under the University of Illinois Open Source
// License. See LICENSE.TXT for details.
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
//
//===----------------------------------------------------------------------===//
//
[X86, AVX] replace vinsertf128 intrinsics with generic shuffles We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is the sibling patch for the Clang half of this change: http://reviews.llvm.org/D8088 Differential Revision: http://reviews.llvm.org/D8086 llvm-svn: 231794
2015-03-11 00:08:36 +08:00
// This file implements the auto-upgrade helper functions.
// This is where deprecated IR intrinsics and other IR features are updated to
// current specifications.
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
//
//===----------------------------------------------------------------------===//
[Layering] Move AutoUpgrade.h into the IR library where its implementation already lives. llvm-svn: 202961
2014-03-05 18:34:14 +08:00
#include "llvm/IR/AutoUpgrade.h"
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 09:00:32 +08:00
#include "llvm/ADT/StringSwitch.h"
[Modules] Move CFG.h to the IR library as it defines graph traits over IR types. llvm-svn: 202827
2014-03-04 19:45:46 +08:00
#include "llvm/IR/CFG.h"
[Modules] Move CallSite into the IR library where it belogs. It is abstracting between a CallInst and an InvokeInst, both of which are IR concepts. llvm-svn: 202816
2014-03-04 19:01:28 +08:00
#include "llvm/IR/CallSite.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 19:36:10 +08:00
#include "llvm/IR/Constants.h"
[cleanup] Re-sort all the #include lines in LLVM using utils/sort_includes.py. I clearly haven't done this in a while, so more changed than usual. This even uncovered a missing include from the InstrProf library that I've added. No functionality changed here, just mechanical cleanup of the include order. llvm-svn: 225974
2015-01-14 19:23:27 +08:00
#include "llvm/IR/DIBuilder.h"
[Layering] Move DebugInfo.h into the IR library where its implementation already lives. llvm-svn: 203046
2014-03-06 08:46:21 +08:00
#include "llvm/IR/DebugInfo.h"
Report a warning when dropping outdated debug info metadata. Use DiagnosticInfo to emit the warning. llvm-svn: 199346
2014-01-16 09:51:12 +08:00
#include "llvm/IR/DiagnosticInfo.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 19:36:10 +08:00
#include "llvm/IR/Function.h"
#include "llvm/IR/IRBuilder.h"
#include "llvm/IR/Instruction.h"
#include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/LLVMContext.h"
#include "llvm/IR/Module.h"
assert(0) -> LLVM_UNREACHABLE. Make llvm_unreachable take an optional string, thus moving the cerr<< out of line. LLVM_UNREACHABLE is now a simple wrapper that makes the message go away for NDEBUG builds. llvm-svn: 75379
2009-07-12 04:10:48 +08:00
#include "llvm/Support/ErrorHandling.h"
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 18:56:37 +08:00
#include "llvm/Support/Regex.h"
Unbreak build with gcc 4.3: provide missed includes and silence most annoying warnings. llvm-svn: 47367
2008-02-20 19:08:44 +08:00
#include <cstring>
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
using namespace llvm;
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 23:51:42 +08:00
static void rename(GlobalValue *GV) { GV->setName(GV->getName() + ".old"); }
[IR][X86] Rename an AutoUpgrade helper function to more accurately match what intrinsics it handles. NFC llvm-svn: 295425
2017-02-17 15:07:21 +08:00
// Upgrade the declarations of the SSE4.1 ptest intrinsics whose arguments have
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-11 02:42:51 +08:00
// changed their type from v4f32 to v2i64.
[IR][X86] Rename an AutoUpgrade helper function to more accurately match what intrinsics it handles. NFC llvm-svn: 295425
2017-02-17 15:07:21 +08:00
static bool UpgradePTESTIntrinsic(Function* F, Intrinsic::ID IID,
Function *&NewFn) {
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-11 02:42:51 +08:00
// Check whether this is an old version of the function, which received
// v4f32 arguments.
Type *Arg0Type = F->getFunctionType()->getParamType(0);
if (Arg0Type != VectorType::get(Type::getFloatTy(F->getContext()), 4))
return false;
// Yes, it's old, replace it with new version.
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 23:51:42 +08:00
rename(F);
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-11 02:42:51 +08:00
NewFn = Intrinsic::getDeclaration(F->getParent(), IID);
return true;
}
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
[x86] Fix a pretty horrible bug and inconsistency in the x86 asm parsing (and latent bug in the instruction definitions). This is effectively a revert of r136287 which tried to address a specific and narrow case of immediate operands failing to be accepted by x86 instructions with a pretty heavy hammer: it introduced a new kind of operand that behaved differently. All of that is removed with this commit, but the test cases are both preserved and enhanced. The core problem that r136287 and this commit are trying to handle is that gas accepts both of the following instructions: insertps $192, %xmm0, %xmm1 insertps $-64, %xmm0, %xmm1 These will encode to the same byte sequence, with the immediate occupying an 8-bit entry. The first form was fixed by r136287 but that broke the prior handling of the second form! =[ Ironically, we would still emit the second form in some cases and then be unable to re-assemble the output. The reason why the first instruction failed to be handled is because prior to r136287 the operands ere marked 'i32i8imm' which forces them to be sign-extenable. Clearly, that won't work for 192 in a single byte. However, making thim zero-extended or "unsigned" doesn't really address the core issue either because it breaks negative immediates. The correct fix is to make these operands 'i8imm' reflecting that they can be either signed or unsigned but must be 8-bit immediates. This patch backs out r136287 and then changes those places as well as some others to use 'i8imm' rather than one of the extended variants. Naturally, this broke something else. The custom DAG nodes had to be updated to have a much more accurate type constraint of an i8 node, and a bunch of Pat immediates needed to be specified as i8 values. The fallout didn't end there though. We also then ceased to be able to match the instruction-specific intrinsics to the instructions so modified. Digging, this is because they too used i32 rather than i8 in their signature. So I've also switched those intrinsics to i8 arguments in line with the instructions. In order to make the intrinsic adjustments of course, I also had to add auto upgrading for the intrinsics. I suspect that the intrinsic argument types may have led everything down this rabbit hole. Pretty happy with the result. llvm-svn: 217310
2014-09-06 18:00:01 +08:00
// Upgrade the declarations of intrinsic functions whose 8-bit immediate mask
// arguments have changed their type from i32 to i8.
static bool UpgradeX86IntrinsicsWith8BitMask(Function *F, Intrinsic::ID IID,
Function *&NewFn) {
// Check that the last argument is an i32.
Type *LastArgType = F->getFunctionType()->getParamType(
F->getFunctionType()->getNumParams() - 1);
if (!LastArgType->isIntegerTy(32))
return false;
// Move this function aside and map down.
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 23:51:42 +08:00
rename(F);
[x86] Fix a pretty horrible bug and inconsistency in the x86 asm parsing (and latent bug in the instruction definitions). This is effectively a revert of r136287 which tried to address a specific and narrow case of immediate operands failing to be accepted by x86 instructions with a pretty heavy hammer: it introduced a new kind of operand that behaved differently. All of that is removed with this commit, but the test cases are both preserved and enhanced. The core problem that r136287 and this commit are trying to handle is that gas accepts both of the following instructions: insertps $192, %xmm0, %xmm1 insertps $-64, %xmm0, %xmm1 These will encode to the same byte sequence, with the immediate occupying an 8-bit entry. The first form was fixed by r136287 but that broke the prior handling of the second form! =[ Ironically, we would still emit the second form in some cases and then be unable to re-assemble the output. The reason why the first instruction failed to be handled is because prior to r136287 the operands ere marked 'i32i8imm' which forces them to be sign-extenable. Clearly, that won't work for 192 in a single byte. However, making thim zero-extended or "unsigned" doesn't really address the core issue either because it breaks negative immediates. The correct fix is to make these operands 'i8imm' reflecting that they can be either signed or unsigned but must be 8-bit immediates. This patch backs out r136287 and then changes those places as well as some others to use 'i8imm' rather than one of the extended variants. Naturally, this broke something else. The custom DAG nodes had to be updated to have a much more accurate type constraint of an i8 node, and a bunch of Pat immediates needed to be specified as i8 values. The fallout didn't end there though. We also then ceased to be able to match the instruction-specific intrinsics to the instructions so modified. Digging, this is because they too used i32 rather than i8 in their signature. So I've also switched those intrinsics to i8 arguments in line with the instructions. In order to make the intrinsic adjustments of course, I also had to add auto upgrading for the intrinsics. I suspect that the intrinsic argument types may have led everything down this rabbit hole. Pretty happy with the result. llvm-svn: 217310
2014-09-06 18:00:01 +08:00
NewFn = Intrinsic::getDeclaration(F->getParent(), IID);
return true;
}
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
static bool ShouldUpgradeX86Intrinsic(Function *F, StringRef Name) {
// All of the intrinsics matches below should be marked with which llvm
// version started autoupgrading them. At some point in the future we would
// like to use this information to remove upgrade code for some older
// intrinsics. It is currently undecided how we will determine that future
// point.
if (Name.startswith("sse2.pcmpeq.") || // Added in 3.1
Name.startswith("sse2.pcmpgt.") || // Added in 3.1
Name.startswith("avx2.pcmpeq.") || // Added in 3.1
Name.startswith("avx2.pcmpgt.") || // Added in 3.1
Name.startswith("avx512.mask.pcmpeq.") || // Added in 3.9
Name.startswith("avx512.mask.pcmpgt.") || // Added in 3.9
Name == "sse.add.ss" || // Added in 4.0
Name == "sse2.add.sd" || // Added in 4.0
Name == "sse.sub.ss" || // Added in 4.0
Name == "sse2.sub.sd" || // Added in 4.0
Name == "sse.mul.ss" || // Added in 4.0
Name == "sse2.mul.sd" || // Added in 4.0
Name == "sse.div.ss" || // Added in 4.0
Name == "sse2.div.sd" || // Added in 4.0
Name == "sse41.pmaxsb" || // Added in 3.9
Name == "sse2.pmaxs.w" || // Added in 3.9
Name == "sse41.pmaxsd" || // Added in 3.9
Name == "sse2.pmaxu.b" || // Added in 3.9
Name == "sse41.pmaxuw" || // Added in 3.9
Name == "sse41.pmaxud" || // Added in 3.9
Name == "sse41.pminsb" || // Added in 3.9
Name == "sse2.pmins.w" || // Added in 3.9
Name == "sse41.pminsd" || // Added in 3.9
Name == "sse2.pminu.b" || // Added in 3.9
Name == "sse41.pminuw" || // Added in 3.9
Name == "sse41.pminud" || // Added in 3.9
Name.startswith("avx512.mask.pshuf.b.") || // Added in 4.0
Name.startswith("avx2.pmax") || // Added in 3.9
Name.startswith("avx2.pmin") || // Added in 3.9
Name.startswith("avx512.mask.pmax") || // Added in 4.0
Name.startswith("avx512.mask.pmin") || // Added in 4.0
Name.startswith("avx2.vbroadcast") || // Added in 3.8
Name.startswith("avx2.pbroadcast") || // Added in 3.8
Name.startswith("avx.vpermil.") || // Added in 3.1
Name.startswith("sse2.pshuf") || // Added in 3.9
Name.startswith("avx512.pbroadcast") || // Added in 3.9
Name.startswith("avx512.mask.broadcast.s") || // Added in 3.9
Name.startswith("avx512.mask.movddup") || // Added in 3.9
Name.startswith("avx512.mask.movshdup") || // Added in 3.9
Name.startswith("avx512.mask.movsldup") || // Added in 3.9
Name.startswith("avx512.mask.pshuf.d.") || // Added in 3.9
Name.startswith("avx512.mask.pshufl.w.") || // Added in 3.9
Name.startswith("avx512.mask.pshufh.w.") || // Added in 3.9
Name.startswith("avx512.mask.shuf.p") || // Added in 4.0
Name.startswith("avx512.mask.vpermil.p") || // Added in 3.9
Name.startswith("avx512.mask.perm.df.") || // Added in 3.9
Name.startswith("avx512.mask.perm.di.") || // Added in 3.9
Name.startswith("avx512.mask.punpckl") || // Added in 3.9
Name.startswith("avx512.mask.punpckh") || // Added in 3.9
Name.startswith("avx512.mask.unpckl.") || // Added in 3.9
Name.startswith("avx512.mask.unpckh.") || // Added in 3.9
Name.startswith("avx512.mask.pand.") || // Added in 3.9
Name.startswith("avx512.mask.pandn.") || // Added in 3.9
Name.startswith("avx512.mask.por.") || // Added in 3.9
Name.startswith("avx512.mask.pxor.") || // Added in 3.9
Name.startswith("avx512.mask.and.") || // Added in 3.9
Name.startswith("avx512.mask.andn.") || // Added in 3.9
Name.startswith("avx512.mask.or.") || // Added in 3.9
Name.startswith("avx512.mask.xor.") || // Added in 3.9
Name.startswith("avx512.mask.padd.") || // Added in 4.0
Name.startswith("avx512.mask.psub.") || // Added in 4.0
Name.startswith("avx512.mask.pmull.") || // Added in 4.0
Name.startswith("avx512.mask.cvtdq2pd.") || // Added in 4.0
Name.startswith("avx512.mask.cvtudq2pd.") || // Added in 4.0
Name.startswith("avx512.mask.pmul.dq.") || // Added in 4.0
Name.startswith("avx512.mask.pmulu.dq.") || // Added in 4.0
[IR][X86] Fix llvm version number in comments in AutoUpgrade. Forgot the next release is 5.0 not 4.1 llvm-svn: 296092
2017-02-24 13:35:07 +08:00
Name.startswith("avx512.mask.packsswb.") || // Added in 5.0
Name.startswith("avx512.mask.packssdw.") || // Added in 5.0
Name.startswith("avx512.mask.packuswb.") || // Added in 5.0
Name.startswith("avx512.mask.packusdw.") || // Added in 5.0
[AVX-512] Remove and autoupgrade the masked integer compare intrinsics Summary: These intrinsics aren't used by clang and haven't been for a while. There's some really terrible codegen in the 32-bit target for avx512bw due to i64 not being legal. But as I said these intrinsics aren't used by clang even before this patch so this codegen reflects our clang behavior today. Reviewers: spatel, RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34389 llvm-svn: 306047
2017-06-23 04:11:01 +08:00
Name.startswith("avx512.mask.cmp.b") || // Added in 5.0
Name.startswith("avx512.mask.cmp.d") || // Added in 5.0
Name.startswith("avx512.mask.cmp.q") || // Added in 5.0
Name.startswith("avx512.mask.cmp.w") || // Added in 5.0
Name.startswith("avx512.mask.ucmp.") || // Added in 5.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
Name == "avx512.mask.add.pd.128" || // Added in 4.0
Name == "avx512.mask.add.pd.256" || // Added in 4.0
Name == "avx512.mask.add.ps.128" || // Added in 4.0
Name == "avx512.mask.add.ps.256" || // Added in 4.0
Name == "avx512.mask.div.pd.128" || // Added in 4.0
Name == "avx512.mask.div.pd.256" || // Added in 4.0
Name == "avx512.mask.div.ps.128" || // Added in 4.0
Name == "avx512.mask.div.ps.256" || // Added in 4.0
Name == "avx512.mask.mul.pd.128" || // Added in 4.0
Name == "avx512.mask.mul.pd.256" || // Added in 4.0
Name == "avx512.mask.mul.ps.128" || // Added in 4.0
Name == "avx512.mask.mul.ps.256" || // Added in 4.0
Name == "avx512.mask.sub.pd.128" || // Added in 4.0
Name == "avx512.mask.sub.pd.256" || // Added in 4.0
Name == "avx512.mask.sub.ps.128" || // Added in 4.0
Name == "avx512.mask.sub.ps.256" || // Added in 4.0
[IR][X86] Fix llvm version number in comments in AutoUpgrade. Forgot the next release is 5.0 not 4.1 llvm-svn: 296092
2017-02-24 13:35:07 +08:00
Name == "avx512.mask.max.pd.128" || // Added in 5.0
Name == "avx512.mask.max.pd.256" || // Added in 5.0
Name == "avx512.mask.max.ps.128" || // Added in 5.0
Name == "avx512.mask.max.ps.256" || // Added in 5.0
Name == "avx512.mask.min.pd.128" || // Added in 5.0
Name == "avx512.mask.min.pd.256" || // Added in 5.0
Name == "avx512.mask.min.ps.128" || // Added in 5.0
Name == "avx512.mask.min.ps.256" || // Added in 5.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
Name.startswith("avx512.mask.vpermilvar.") || // Added in 4.0
Name.startswith("avx512.mask.psll.d") || // Added in 4.0
Name.startswith("avx512.mask.psll.q") || // Added in 4.0
Name.startswith("avx512.mask.psll.w") || // Added in 4.0
Name.startswith("avx512.mask.psra.d") || // Added in 4.0
Name.startswith("avx512.mask.psra.q") || // Added in 4.0
Name.startswith("avx512.mask.psra.w") || // Added in 4.0
Name.startswith("avx512.mask.psrl.d") || // Added in 4.0
Name.startswith("avx512.mask.psrl.q") || // Added in 4.0
Name.startswith("avx512.mask.psrl.w") || // Added in 4.0
Name.startswith("avx512.mask.pslli") || // Added in 4.0
Name.startswith("avx512.mask.psrai") || // Added in 4.0
Name.startswith("avx512.mask.psrli") || // Added in 4.0
Name.startswith("avx512.mask.psllv") || // Added in 4.0
Name.startswith("avx512.mask.psrav") || // Added in 4.0
Name.startswith("avx512.mask.psrlv") || // Added in 4.0
Name.startswith("sse41.pmovsx") || // Added in 3.8
Name.startswith("sse41.pmovzx") || // Added in 3.9
Name.startswith("avx2.pmovsx") || // Added in 3.9
Name.startswith("avx2.pmovzx") || // Added in 3.9
Name.startswith("avx512.mask.pmovsx") || // Added in 4.0
Name.startswith("avx512.mask.pmovzx") || // Added in 4.0
[AVX-512] Remove lzcnt intrinsics and autoupgrade them to generic ctlz intrinsics with select. Clang has been emitting cltz intrinsics for a while now. llvm-svn: 296091
2017-02-24 13:35:04 +08:00
Name.startswith("avx512.mask.lzcnt.") || // Added in 5.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
Name == "sse2.cvtdq2pd" || // Added in 3.9
Name == "sse2.cvtps2pd" || // Added in 3.9
Name == "avx.cvtdq2.pd.256" || // Added in 3.9
Name == "avx.cvt.ps2.pd.256" || // Added in 3.9
Name.startswith("avx.vinsertf128.") || // Added in 3.7
Name == "avx2.vinserti128" || // Added in 3.7
Name.startswith("avx512.mask.insert") || // Added in 4.0
Name.startswith("avx.vextractf128.") || // Added in 3.7
Name == "avx2.vextracti128" || // Added in 3.7
Name.startswith("avx512.mask.vextract") || // Added in 4.0
Name.startswith("sse4a.movnt.") || // Added in 3.9
Name.startswith("avx.movnt.") || // Added in 3.2
Name.startswith("avx512.storent.") || // Added in 3.9
[X86][SSE] Update MOVNTDQA non-temporal loads to generic implementation (LLVM) MOVNTDQA non-temporal aligned vector loads can be correctly represented using generic builtin loads, allowing us to remove the existing x86 intrinsics. Clang companion patch: D31766. Differential Revision: https://reviews.llvm.org/D31767 llvm-svn: 300325
2017-04-14 23:05:35 +08:00
Name == "sse41.movntdqa" || // Added in 5.0
Name == "avx2.movntdqa" || // Added in 5.0
Name == "avx512.movntdqa" || // Added in 5.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
Name == "sse2.storel.dq" || // Added in 3.9
Name.startswith("sse.storeu.") || // Added in 3.9
Name.startswith("sse2.storeu.") || // Added in 3.9
Name.startswith("avx.storeu.") || // Added in 3.9
Name.startswith("avx512.mask.storeu.") || // Added in 3.9
Name.startswith("avx512.mask.store.p") || // Added in 3.9
Name.startswith("avx512.mask.store.b.") || // Added in 3.9
Name.startswith("avx512.mask.store.w.") || // Added in 3.9
Name.startswith("avx512.mask.store.d.") || // Added in 3.9
Name.startswith("avx512.mask.store.q.") || // Added in 3.9
Name.startswith("avx512.mask.loadu.") || // Added in 3.9
Name.startswith("avx512.mask.load.") || // Added in 3.9
Name == "sse42.crc32.64.8" || // Added in 3.4
Name.startswith("avx.vbroadcast.s") || // Added in 3.5
Name.startswith("avx512.mask.palignr.") || // Added in 3.9
Name.startswith("avx512.mask.valign.") || // Added in 4.0
Name.startswith("sse2.psll.dq") || // Added in 3.7
Name.startswith("sse2.psrl.dq") || // Added in 3.7
Name.startswith("avx2.psll.dq") || // Added in 3.7
Name.startswith("avx2.psrl.dq") || // Added in 3.7
Name.startswith("avx512.psll.dq") || // Added in 3.9
Name.startswith("avx512.psrl.dq") || // Added in 3.9
Name == "sse41.pblendw" || // Added in 3.7
Name.startswith("sse41.blendp") || // Added in 3.7
Name.startswith("avx.blend.p") || // Added in 3.7
Name == "avx2.pblendw" || // Added in 3.7
Name.startswith("avx2.pblendd.") || // Added in 3.7
Name.startswith("avx.vbroadcastf128") || // Added in 4.0
Name == "avx2.vbroadcasti128" || // Added in 3.7
[X86][IR] In AutoUpgrade, check explicitly for xop.vpcmov and xop.vpcmov.256 instead of anything starting with xop.vpcmov There were some older intrinsics that only existed for less than a month in 2012 that still exist in some out of tree test files that start with this string, but aren't able to be handled by the current upgrade code and fire an assert. Now we'll go back to treating them as not intrinsics at all and just passing them through to output. Fixes PR32041, sort of. llvm-svn: 295930
2017-02-23 11:22:14 +08:00
Name == "xop.vpcmov" || // Added in 3.8
[IR][X86] Fix llvm version number in comments in AutoUpgrade. Forgot the next release is 5.0 not 4.1 llvm-svn: 296092
2017-02-24 13:35:07 +08:00
Name == "xop.vpcmov.256" || // Added in 5.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
Name.startswith("avx512.mask.move.s") || // Added in 4.0
[X86][LLVM] Converting __mm{|256|512}_movm_epi{8|16|32|64} LLVMIR call into generic intrinsics. This patch is a part one of two reviews, one for the clang and the other for LLVM. The patch deletes the back-end intrinsics and adds support for them in the auto upgrade. Differential Revision: https://reviews.llvm.org/D31393 llvm-svn: 299432
2017-04-04 21:32:14 +08:00
Name.startswith("avx512.cvtmask2") || // Added in 5.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
(Name.startswith("xop.vpcom") && // Added in 3.2
F->arg_size() == 2))
return true;
return false;
}
static bool UpgradeX86IntrinsicFunction(Function *F, StringRef Name,
Function *&NewFn) {
// Only handle intrinsics that start with "x86.".
if (!Name.startswith("x86."))
return false;
// Remove "x86." prefix.
Name = Name.substr(4);
if (ShouldUpgradeX86Intrinsic(F, Name)) {
NewFn = nullptr;
return true;
}
// SSE4.1 ptest functions may have an old signature.
if (Name.startswith("sse41.ptest")) { // Added in 3.2
if (Name.substr(11) == "c")
[IR][X86] Rename an AutoUpgrade helper function to more accurately match what intrinsics it handles. NFC llvm-svn: 295425
2017-02-17 15:07:21 +08:00
return UpgradePTESTIntrinsic(F, Intrinsic::x86_sse41_ptestc, NewFn);
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
if (Name.substr(11) == "z")
[IR][X86] Rename an AutoUpgrade helper function to more accurately match what intrinsics it handles. NFC llvm-svn: 295425
2017-02-17 15:07:21 +08:00
return UpgradePTESTIntrinsic(F, Intrinsic::x86_sse41_ptestz, NewFn);
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
if (Name.substr(11) == "nzc")
[IR][X86] Rename an AutoUpgrade helper function to more accurately match what intrinsics it handles. NFC llvm-svn: 295425
2017-02-17 15:07:21 +08:00
return UpgradePTESTIntrinsic(F, Intrinsic::x86_sse41_ptestnzc, NewFn);
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
}
// Several blend and other instructions with masks used the wrong number of
// bits.
if (Name == "sse41.insertps") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_sse41_insertps,
NewFn);
if (Name == "sse41.dppd") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_sse41_dppd,
NewFn);
if (Name == "sse41.dpps") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_sse41_dpps,
NewFn);
if (Name == "sse41.mpsadbw") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_sse41_mpsadbw,
NewFn);
if (Name == "avx.dp.ps.256") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_avx_dp_ps_256,
NewFn);
if (Name == "avx2.mpsadbw") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_avx2_mpsadbw,
NewFn);
// frcz.ss/sd may need to have an argument dropped. Added in 3.2
if (Name.startswith("xop.vfrcz.ss") && F->arg_size() == 2) {
rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::x86_xop_vfrcz_ss);
return true;
}
if (Name.startswith("xop.vfrcz.sd") && F->arg_size() == 2) {
rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::x86_xop_vfrcz_sd);
return true;
}
// Upgrade any XOP PERMIL2 index operand still using a float/double vector.
if (Name.startswith("xop.vpermil2")) { // Added in 3.9
[IR][X86] Simplify some AutoUpgrade code slightly. NFC llvm-svn: 295426
2017-02-17 15:07:24 +08:00
auto Idx = F->getFunctionType()->getParamType(2);
if (Idx->isFPOrFPVectorTy()) {
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
rename(F);
unsigned IdxSize = Idx->getPrimitiveSizeInBits();
unsigned EltSize = Idx->getScalarSizeInBits();
Intrinsic::ID Permil2ID;
if (EltSize == 64 && IdxSize == 128)
Permil2ID = Intrinsic::x86_xop_vpermil2pd;
else if (EltSize == 32 && IdxSize == 128)
Permil2ID = Intrinsic::x86_xop_vpermil2ps;
else if (EltSize == 64 && IdxSize == 256)
Permil2ID = Intrinsic::x86_xop_vpermil2pd_256;
else
Permil2ID = Intrinsic::x86_xop_vpermil2ps_256;
NewFn = Intrinsic::getDeclaration(F->getParent(), Permil2ID);
return true;
}
}
return false;
}
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-18 06:33:23 +08:00
static bool UpgradeIntrinsicFunction1(Function *F, Function *&NewFn) {
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
assert(F && "Illegal to upgrade a non-existent Function.");
// Quickly eliminate it, if it's not a candidate.
Move setName after accessing Name llvm-svn: 274862
2016-07-09 00:09:51 +08:00
StringRef Name = F->getName();
rework the remaining autoupgrade logic to use a StringRef instead of creating a temporary std::string for every function being checked. llvm-svn: 133355
2011-06-19 02:56:39 +08:00
if (Name.size() <= 8 || !Name.startswith("llvm."))
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-18 06:33:23 +08:00
return false;
rework the remaining autoupgrade logic to use a StringRef instead of creating a temporary std::string for every function being checked. llvm-svn: 133355
2011-06-19 02:56:39 +08:00
Name = Name.substr(5); // Strip off "llvm."
Eli managed to kill off llvm.membarrier in llvm 3.0 also, this means that mainline needs no autoupgrade logic for intrinsics yet, woohoo! llvm-svn: 145178
2011-11-27 16:42:07 +08:00
rework the remaining autoupgrade logic to use a StringRef instead of creating a temporary std::string for every function being checked. llvm-svn: 133355
2011-06-19 02:56:39 +08:00
switch (Name[0]) {
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
default: break;
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 07:25:25 +08:00
case 'a': {
[ARM] Remove rbit intrinsics and autoupgrade to generic bitreverse. Testing already covered by CodeGen/ARM/rbit.ll llvm-svn: 291587
2017-01-11 03:23:51 +08:00
if (Name.startswith("arm.rbit") || Name.startswith("aarch64.rbit")) {
[AArch64] Add support for lowering bitreverse to the rbit instruction. Differential Revision: https://reviews.llvm.org/D28379 llvm-svn: 291575
2017-01-11 01:20:33 +08:00
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::bitreverse,
F->arg_begin()->getType());
return true;
}
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 07:25:25 +08:00
if (Name.startswith("arm.neon.vclz")) {
Type* args[2] = {
Remove trailing whitespace, fix file path in comment llvm-svn: 186766
2013-07-21 01:46:00 +08:00
F->arg_begin()->getType(),
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 07:25:25 +08:00
Type::getInt1Ty(F->getContext())
};
// Can't use Intrinsic::getDeclaration here as it adds a ".i1" to
// the end of the name. Change name from llvm.arm.neon.vclz.* to
// llvm.ctlz.*
FunctionType* fType = FunctionType::get(F->getReturnType(), args, false);
Remove trailing whitespace, fix file path in comment llvm-svn: 186766
2013-07-21 01:46:00 +08:00
NewFn = Function::Create(fType, F->getLinkage(),
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 07:25:25 +08:00
"llvm.ctlz." + Name.substr(14), F->getParent());
return true;
}
More replacing of target-dependent intrinsics with target-indepdent intrinsics. The second instruction(s) to be handled are the vector versions of count set bits (ctpop). The changes here are to clang so that it generates a target independent vector ctpop when it sees an ARM dependent vector bits set count. The changes in llvm are to match the target independent vector ctpop and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector pop counts with target-independent ctpops. There are also changes to an existing test case in llvm for ARM vector count instructions and to a test for the bitcode upgrade. <rdar://problem/11892519> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160410
2012-07-18 08:02:16 +08:00
if (Name.startswith("arm.neon.vcnt")) {
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::ctpop,
F->arg_begin()->getType());
return true;
}
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 18:56:37 +08:00
Regex vldRegex("^arm\\.neon\\.vld([1234]|[234]lane)\\.v[a-z0-9]*$");
if (vldRegex.match(Name)) {
auto fArgs = F->getFunctionType()->params();
SmallVector<Type *, 4> Tys(fArgs.begin(), fArgs.end());
// Can't use Intrinsic::getDeclaration here as the return types might
// then only be structurally equal.
FunctionType* fType = FunctionType::get(F->getReturnType(), Tys, false);
NewFn = Function::Create(fType, F->getLinkage(),
"llvm." + Name + ".p0i8", F->getParent());
return true;
}
Regex vstRegex("^arm\\.neon\\.vst([1234]|[234]lane)\\.v[a-z0-9]*$");
if (vstRegex.match(Name)) {
Make a bunch of static arrays const. llvm-svn: 250642
2015-10-18 13:15:34 +08:00
static const Intrinsic::ID StoreInts[] = {Intrinsic::arm_neon_vst1,
Intrinsic::arm_neon_vst2,
Intrinsic::arm_neon_vst3,
Intrinsic::arm_neon_vst4};
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 18:56:37 +08:00
Make a bunch of static arrays const. llvm-svn: 250642
2015-10-18 13:15:34 +08:00
static const Intrinsic::ID StoreLaneInts[] = {
Intrinsic::arm_neon_vst2lane, Intrinsic::arm_neon_vst3lane,
Intrinsic::arm_neon_vst4lane
};
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 18:56:37 +08:00
auto fArgs = F->getFunctionType()->params();
Type *Tys[] = {fArgs[0], fArgs[1]};
if (Name.find("lane") == StringRef::npos)
NewFn = Intrinsic::getDeclaration(F->getParent(),
StoreInts[fArgs.size() - 3], Tys);
else
NewFn = Intrinsic::getDeclaration(F->getParent(),
StoreLaneInts[fArgs.size() - 5], Tys);
return true;
}
[AArch64] [ARM] Make a target-independent llvm.thread.pointer intrinsic. Both AArch64 and ARM support llvm.<arch>.thread.pointer intrinsics that just return the thread pointer. I have a pending patch that does the same for SystemZ (D19054), and there are many more targets that could benefit from one. This patch merges the ARM and AArch64 intrinsics into a single target independent one that will also be used by subsequent targets. Differential Revision: http://reviews.llvm.org/D19098 llvm-svn: 266818
2016-04-20 04:51:05 +08:00
if (Name == "aarch64.thread.pointer" || Name == "arm.thread.pointer") {
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::thread_pointer);
return true;
}
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 07:25:25 +08:00
break;
}
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 18:56:37 +08:00
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 12:26:04 +08:00
case 'c': {
if (Name.startswith("ctlz.") && F->arg_size() == 1) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 23:51:42 +08:00
rename(F);
Don't rely in there being one argument before we've actually identified a function to upgrade. Also, simplify the code a bit at the expense of one line. llvm-svn: 146368
2011-12-12 18:57:20 +08:00
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::ctlz,
F->arg_begin()->getType());
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 12:26:04 +08:00
return true;
}
if (Name.startswith("cttz.") && F->arg_size() == 1) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 23:51:42 +08:00
rename(F);
Don't rely in there being one argument before we've actually identified a function to upgrade. Also, simplify the code a bit at the expense of one line. llvm-svn: 146368
2011-12-12 18:57:20 +08:00
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::cttz,
F->arg_begin()->getType());
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 12:26:04 +08:00
return true;
}
break;
}
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-11 04:18:21 +08:00
case 'i':
case 'l': {
bool IsLifetimeStart = Name.startswith("lifetime.start");
if (IsLifetimeStart || Name.startswith("invariant.start")) {
Intrinsic::ID ID = IsLifetimeStart ?
Intrinsic::lifetime_start : Intrinsic::invariant_start;
Revert "Revert "Invariant start/end intrinsics overloaded for address space"" This reverts commit 32fc6488e48eafc0ca1bac1bd9cbf0008224d530. llvm-svn: 278609
2016-08-14 07:31:24 +08:00
auto Args = F->getFunctionType()->params();
Type* ObjectPtr[1] = {Args[1]};
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-11 04:18:21 +08:00
if (F->getName() != Intrinsic::getName(ID, ObjectPtr)) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 23:51:42 +08:00
rename(F);
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-11 04:18:21 +08:00
NewFn = Intrinsic::getDeclaration(F->getParent(), ID, ObjectPtr);
Revert "Revert "Invariant start/end intrinsics overloaded for address space"" This reverts commit 32fc6488e48eafc0ca1bac1bd9cbf0008224d530. llvm-svn: 278609
2016-08-14 07:31:24 +08:00
return true;
}
}
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-11 04:18:21 +08:00
bool IsLifetimeEnd = Name.startswith("lifetime.end");
if (IsLifetimeEnd || Name.startswith("invariant.end")) {
Intrinsic::ID ID = IsLifetimeEnd ?
Intrinsic::lifetime_end : Intrinsic::invariant_end;
Revert "Revert "Invariant start/end intrinsics overloaded for address space"" This reverts commit 32fc6488e48eafc0ca1bac1bd9cbf0008224d530. llvm-svn: 278609
2016-08-14 07:31:24 +08:00
auto Args = F->getFunctionType()->params();
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-11 04:18:21 +08:00
Type* ObjectPtr[1] = {Args[IsLifetimeEnd ? 1 : 2]};
if (F->getName() != Intrinsic::getName(ID, ObjectPtr)) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 23:51:42 +08:00
rename(F);
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-11 04:18:21 +08:00
NewFn = Intrinsic::getDeclaration(F->getParent(), ID, ObjectPtr);
Revert "Revert "Invariant start/end intrinsics overloaded for address space"" This reverts commit 32fc6488e48eafc0ca1bac1bd9cbf0008224d530. llvm-svn: 278609
2016-08-14 07:31:24 +08:00
return true;
}
}
break;
}
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-29 02:27:25 +08:00
case 'm': {
if (Name.startswith("masked.load.")) {
Type *Tys[] = { F->getReturnType(), F->arg_begin()->getType() };
if (F->getName() != Intrinsic::getName(Intrinsic::masked_load, Tys)) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 23:51:42 +08:00
rename(F);
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-29 02:27:25 +08:00
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::masked_load,
Tys);
return true;
}
}
if (Name.startswith("masked.store.")) {
auto Args = F->getFunctionType()->params();
Type *Tys[] = { Args[0], Args[1] };
if (F->getName() != Intrinsic::getName(Intrinsic::masked_store, Tys)) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 23:51:42 +08:00
rename(F);
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-29 02:27:25 +08:00
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::masked_store,
Tys);
return true;
}
}
Support arbitrary address space pointers in masked gather/scatter intrinsics. Fixes PR31789 - When loop-vectorize tries to use these intrinsics for a non-default address space pointer we fail with a "Calling a function with a bad singature!" assertion. This patch solves this by adding the 'vector of pointers' argument as an overloaded type which will determine the address space. Differential revision: https://reviews.llvm.org/D31490 llvm-svn: 302018
2017-05-03 20:28:54 +08:00
// Renaming gather/scatter intrinsics with no address space overloading
// to the new overload which includes an address space
if (Name.startswith("masked.gather.")) {
Type *Tys[] = {F->getReturnType(), F->arg_begin()->getType()};
if (F->getName() != Intrinsic::getName(Intrinsic::masked_gather, Tys)) {
rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::masked_gather, Tys);
return true;
}
}
if (Name.startswith("masked.scatter.")) {
auto Args = F->getFunctionType()->params();
Type *Tys[] = {Args[0], Args[1]};
if (F->getName() != Intrinsic::getName(Intrinsic::masked_scatter, Tys)) {
rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::masked_scatter, Tys);
return true;
}
}
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-29 02:27:25 +08:00
break;
}
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 09:00:32 +08:00
case 'n': {
if (Name.startswith("nvvm.")) {
Name = Name.substr(5);
// The following nvvm intrinsics correspond exactly to an LLVM intrinsic.
Intrinsic::ID IID = StringSwitch<Intrinsic::ID>(Name)
.Cases("brev32", "brev64", Intrinsic::bitreverse)
.Case("clz.i", Intrinsic::ctlz)
.Case("popc.i", Intrinsic::ctpop)
.Default(Intrinsic::not_intrinsic);
if (IID != Intrinsic::not_intrinsic && F->arg_size() == 1) {
NewFn = Intrinsic::getDeclaration(F->getParent(), IID,
{F->getReturnType()});
return true;
}
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-29 02:27:25 +08:00
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 09:00:32 +08:00
// The following nvvm intrinsics correspond exactly to an LLVM idiom, but
// not to an intrinsic alone. We expand them in UpgradeIntrinsicCall.
//
// TODO: We could add lohi.i2d.
bool Expand = StringSwitch<bool>(Name)
.Cases("abs.i", "abs.ll", true)
.Cases("clz.ll", "popc.ll", "h2f", true)
.Cases("max.i", "max.ll", "max.ui", "max.ull", true)
.Cases("min.i", "min.ll", "min.ui", "min.ull", true)
.Default(false);
if (Expand) {
NewFn = nullptr;
return true;
}
}
Added missing break. llvm-svn: 303454
2017-05-20 04:31:51 +08:00
break;
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 09:00:32 +08:00
}
Change objectsize intrinsic to accept different address spaces. Bitcasting everything to i8* won't work. Autoupgrade the old intrinsic declarations to use the new mangling. llvm-svn: 192117
2013-10-08 02:06:48 +08:00
case 'o':
// We only need to change the name to match the mangling including the
// address space.
Let llvm.objectsize be conservative with null pointers This adds a parameter to @llvm.objectsize that makes it return conservative values if it's given null. This fixes PR23277. Differential Revision: https://reviews.llvm.org/D28494 llvm-svn: 298430
2017-03-22 04:08:59 +08:00
if (Name.startswith("objectsize.")) {
Change objectsize intrinsic to accept different address spaces. Bitcasting everything to i8* won't work. Autoupgrade the old intrinsic declarations to use the new mangling. llvm-svn: 192117
2013-10-08 02:06:48 +08:00
Type *Tys[2] = { F->getReturnType(), F->arg_begin()->getType() };
Let llvm.objectsize be conservative with null pointers This adds a parameter to @llvm.objectsize that makes it return conservative values if it's given null. This fixes PR23277. Differential Revision: https://reviews.llvm.org/D28494 llvm-svn: 298430
2017-03-22 04:08:59 +08:00
if (F->arg_size() == 2 ||
F->getName() != Intrinsic::getName(Intrinsic::objectsize, Tys)) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 23:51:42 +08:00
rename(F);
Let llvm.objectsize be conservative with null pointers This adds a parameter to @llvm.objectsize that makes it return conservative values if it's given null. This fixes PR23277. Differential Revision: https://reviews.llvm.org/D28494 llvm-svn: 298430
2017-03-22 04:08:59 +08:00
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::objectsize,
Tys);
Change objectsize intrinsic to accept different address spaces. Bitcasting everything to i8* won't work. Autoupgrade the old intrinsic declarations to use the new mangling. llvm-svn: 192117
2013-10-08 02:06:48 +08:00
return true;
}
}
break;
[SSP] Remove llvm.stackprotectorcheck. This is a cleanup patch for SSP support in LLVM. There is no functional change. llvm.stackprotectorcheck is not needed, because SelectionDAG isn't actually lowering it in SelectBasicBlock; rather, it adds check code in FinishBasicBlock, ignoring the position where the intrinsic is inserted (See FindSplitPointForStackProtector()). llvm-svn: 265851
2016-04-09 05:26:31 +08:00
case 's':
if (Name == "stackprotectorcheck") {
NewFn = nullptr;
return true;
}
Add a missing break statement. NFC. llvm-svn: 286203
2016-11-08 12:01:50 +08:00
break;
[SSP] Remove llvm.stackprotectorcheck. This is a cleanup patch for SSP support in LLVM. There is no functional change. llvm.stackprotectorcheck is not needed, because SelectionDAG isn't actually lowering it in SelectBasicBlock; rather, it adds check code in FinishBasicBlock, ignoring the position where the intrinsic is inserted (See FindSplitPointForStackProtector()). llvm-svn: 265851
2016-04-09 05:26:31 +08:00
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 15:07:19 +08:00
case 'x':
if (UpgradeX86IntrinsicFunction(F, Name, NewFn))
Fix intrinsics for XOP frczss/sd instructions. These instructions only take one source register and zero the upper bits of the destination rather than preserving them. llvm-svn: 158396
2012-06-13 15:18:53 +08:00
return true;
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
}
Implement intrinsic mangling for literal struct types. Fixes PR 31921 Summary: Predicateinfo requires an ugly workaround to try to avoid literal struct types due to the intrinsic mangling not being implemented. This workaround actually does not work in all cases (you can hit the assert by bootstrapping with -print-predicateinfo), and can't be made to work without DFS'ing the type (IE copying getMangledStr and using a version that detects if it would crash). Rather than do that, i just implemented the mangling. It seems simple, since they are unified structurally. Looking at the overloaded-mangling testcase we have, it actually turns out the gc intrinsics will *also* crash if you try to use a literal struct. Thus, the testcase added fails before this patch, and works after, without needing to resort to predicateinfo. Reviewers: chandlerc, davide Subscribers: llvm-commits, sanjoy Differential Revision: https://reviews.llvm.org/D29925 llvm-svn: 295253
2017-02-16 07:16:20 +08:00
// Remangle our intrinsic since we upgrade the mangling
auto Result = llvm::Intrinsic::remangleIntrinsicFunction(F);
if (Result != None) {
NewFn = Result.getValue();
return true;
}
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-11 02:42:51 +08:00
// This may not belong here. This function is effectively being overloaded
// to both detect an intrinsic which needs upgrading, and to provide the
// upgraded form of the intrinsic. We should perhaps have two separate
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
// functions for this.
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-18 06:33:23 +08:00
return false;
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
}
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-18 06:33:23 +08:00
bool llvm::UpgradeIntrinsicFunction(Function *F, Function *&NewFn) {
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 14:08:46 +08:00
NewFn = nullptr;
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-18 06:33:23 +08:00
bool Upgraded = UpgradeIntrinsicFunction1(F, NewFn);
Remove always-true comparison, NFC. Summary: Looking at r241279, I noticed that UpgradedIntrinsics only gets written to in the following code: if (UpgradeIntrinsicFunction(&F, NewFn)) UpgradedIntrinsics[&F] = NewFn; Looking through UpgradeIntrinsicFunction, we always return false OR NewFn will be set to a different function from our source. This patch pulls the F != NewFn into UpgradeIntrinsicFunction as an assert, and removes the check from callers of UpgradeIntrinsicFunction. Reviewers: rafael, chandlerc Subscribers: llvm-commits-list Differential Revision: http://reviews.llvm.org/D10915 llvm-svn: 241369
2015-07-04 04:12:01 +08:00
assert(F != NewFn && "Intrinsic function upgraded to the same function");
Rather than having special rules like "intrinsics cannot throw exceptions", just mark intrinsics with the nounwind attribute. Likewise, mark intrinsics as readnone/readonly and get rid of special aliasing logic (which didn't use anything more than this anyway). llvm-svn: 44544
2007-12-04 04:06:50 +08:00
// Upgrade intrinsic attributes. This does not change the function.
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-18 06:33:23 +08:00
if (NewFn)
F = NewFn;
Change Function::getIntrinsicID() to return an Intrinsic::ID. NFC. Now that Intrinsic::ID is a typed enum, we can forward declare it and so return it from this method. This updates all users which were either using an unsigned to store it, or had a now unnecessary cast. llvm-svn: 237810
2015-05-21 01:16:39 +08:00
if (Intrinsic::ID id = F->getIntrinsicID())
F->setAttributes(Intrinsic::getAttributes(F->getContext(), id));
Rather than having special rules like "intrinsics cannot throw exceptions", just mark intrinsics with the nounwind attribute. Likewise, mark intrinsics as readnone/readonly and get rid of special aliasing logic (which didn't use anything more than this anyway). llvm-svn: 44544
2007-12-04 04:06:50 +08:00
return Upgraded;
}
Auto-upgrade the magic ".llvm.eh.catch.all.value" global to "llvm.eh.catch.all.value". Only the name needs to be changed. llvm-svn: 113600
2010-09-11 02:51:56 +08:00
bool llvm::UpgradeGlobalVariable(GlobalVariable *GV) {
rip out a ton of intrinsic modernization logic from AutoUpgrade.cpp, which is for pre-2.9 bitcode files. We keep x86 unaligned loads, movnt, crc32, and the target indep prefetch change. As usual, updating the testsuite is a PITA. llvm-svn: 133337
2011-06-18 14:05:24 +08:00
// Nothing to do yet.
Auto-upgrade the magic ".llvm.eh.catch.all.value" global to "llvm.eh.catch.all.value". Only the name needs to be changed. llvm-svn: 113600
2010-09-11 02:51:56 +08:00
return false;
}
[X86][AVX512] Dropped avx512 VPSLLDQ/VPSRLDQ intrinsics Auto-upgrade to generic shuffles like sse/avx2 implementations now that we can lower to VPSLLDQ/VPSRLDQ llvm-svn: 272308
2016-06-10 05:09:03 +08:00
// Handles upgrading SSE2/AVX2/AVX512BW PSLLDQ intrinsics by converting them
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
// to byte shuffles.
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
static Value *UpgradeX86PSLLDQIntrinsics(IRBuilder<> &Builder,
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 14:37:33 +08:00
Value *Op, unsigned Shift) {
Type *ResultTy = Op->getType();
unsigned NumElts = ResultTy->getVectorNumElements() * 8;
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
// Bitcast from a 64-bit element type to a byte element type.
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
Type *VecTy = VectorType::get(Builder.getInt8Ty(), NumElts);
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 14:37:33 +08:00
Op = Builder.CreateBitCast(Op, VecTy, "cast");
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
// We'll be shuffling in zeroes.
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 14:37:33 +08:00
Value *Res = Constant::getNullValue(VecTy);
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
// If shift is less than 16, emit a shuffle to move the bytes. Otherwise,
// we'll just return the zero vector.
if (Shift < 16) {
[IR] Require ArrayRef of 'uint32_t' instead of 'int' for the mask argument for one of the signatures of CreateShuffleVector. This better emphasises that you can't use it for the -1 as undef behavior. llvm-svn: 272491
2016-06-12 08:41:19 +08:00
uint32_t Idxs[64];
[X86][AVX512] Dropped avx512 VPSLLDQ/VPSRLDQ intrinsics Auto-upgrade to generic shuffles like sse/avx2 implementations now that we can lower to VPSLLDQ/VPSRLDQ llvm-svn: 272308
2016-06-10 05:09:03 +08:00
// 256/512-bit version is split into 2/4 16-byte lanes.
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
for (unsigned l = 0; l != NumElts; l += 16)
for (unsigned i = 0; i != 16; ++i) {
unsigned Idx = NumElts + i - Shift;
if (Idx < NumElts)
Idx -= NumElts - 16; // end of lane, switch operand.
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 14:37:33 +08:00
Idxs[l + i] = Idx + l;
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
}
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 14:37:33 +08:00
Res = Builder.CreateShuffleVector(Res, Op, makeArrayRef(Idxs, NumElts));
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
}
// Bitcast back to a 64-bit element type.
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 14:37:33 +08:00
return Builder.CreateBitCast(Res, ResultTy, "cast");
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
}
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 10:36:42 +08:00
// Handles upgrading SSE2/AVX2/AVX512BW PSRLDQ intrinsics by converting them
// to byte shuffles.
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
static Value *UpgradeX86PSRLDQIntrinsics(IRBuilder<> &Builder, Value *Op,
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 10:36:42 +08:00
unsigned Shift) {
Type *ResultTy = Op->getType();
unsigned NumElts = ResultTy->getVectorNumElements() * 8;
// Bitcast from a 64-bit element type to a byte element type.
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
Type *VecTy = VectorType::get(Builder.getInt8Ty(), NumElts);
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 10:36:42 +08:00
Op = Builder.CreateBitCast(Op, VecTy, "cast");
// We'll be shuffling in zeroes.
Value *Res = Constant::getNullValue(VecTy);
// If shift is less than 16, emit a shuffle to move the bytes. Otherwise,
// we'll just return the zero vector.
if (Shift < 16) {
uint32_t Idxs[64];
// 256/512-bit version is split into 2/4 16-byte lanes.
for (unsigned l = 0; l != NumElts; l += 16)
for (unsigned i = 0; i != 16; ++i) {
unsigned Idx = i + Shift;
if (Idx >= 16)
Idx += NumElts - 16; // end of lane, switch operand.
Idxs[l + i] = Idx + l;
}
Res = Builder.CreateShuffleVector(Op, Res, makeArrayRef(Idxs, NumElts));
}
// Bitcast back to a 64-bit element type.
return Builder.CreateBitCast(Res, ResultTy, "cast");
}
static Value *getX86MaskVec(IRBuilder<> &Builder, Value *Mask,
unsigned NumElts) {
llvm::VectorType *MaskTy = llvm::VectorType::get(Builder.getInt1Ty(),
cast<IntegerType>(Mask->getType())->getBitWidth());
Mask = Builder.CreateBitCast(Mask, MaskTy);
// If we have less than 8 elements, then the starting mask was an i8 and
// we need to extract down to the right number of elements.
if (NumElts < 8) {
uint32_t Indices[4];
for (unsigned i = 0; i != NumElts; ++i)
Indices[i] = i;
Mask = Builder.CreateShuffleVector(Mask, Mask,
makeArrayRef(Indices, NumElts),
"extract");
}
return Mask;
}
static Value *EmitX86Select(IRBuilder<> &Builder, Value *Mask,
Value *Op0, Value *Op1) {
// If the mask is all ones just emit the align operation.
if (const auto *C = dyn_cast<Constant>(Mask))
if (C->isAllOnesValue())
return Op0;
Mask = getX86MaskVec(Builder, Mask, Op0->getType()->getVectorNumElements());
return Builder.CreateSelect(Mask, Op0, Op1);
}
[AVX-512] Remove intrinsics for valignd/q and autoupgrade them to native shuffles. llvm-svn: 287744
2016-11-23 14:54:55 +08:00
// Handle autoupgrade for masked PALIGNR and VALIGND/Q intrinsics.
// PALIGNR handles large immediates by shifting while VALIGN masks the immediate
// so we need to handle both cases. VALIGN also doesn't have 128-bit lanes.
static Value *UpgradeX86ALIGNIntrinsics(IRBuilder<> &Builder, Value *Op0,
Value *Op1, Value *Shift,
Value *Passthru, Value *Mask,
bool IsVALIGN) {
[AVX512] Remove masked palignr intrinsics and auto-upgrade them to native IR of vector shuffle and select. llvm-svn: 271872
2016-06-06 14:12:54 +08:00
unsigned ShiftVal = cast<llvm::ConstantInt>(Shift)->getZExtValue();
unsigned NumElts = Op0->getType()->getVectorNumElements();
[AVX-512] Remove intrinsics for valignd/q and autoupgrade them to native shuffles. llvm-svn: 287744
2016-11-23 14:54:55 +08:00
assert((IsVALIGN || NumElts % 16 == 0) && "Illegal NumElts for PALIGNR!");
assert((!IsVALIGN || NumElts <= 16) && "NumElts too large for VALIGN!");
assert(isPowerOf2_32(NumElts) && "NumElts not a power of 2!");
// Mask the immediate for VALIGN.
if (IsVALIGN)
ShiftVal &= (NumElts - 1);
[AVX512] Remove masked palignr intrinsics and auto-upgrade them to native IR of vector shuffle and select. llvm-svn: 271872
2016-06-06 14:12:54 +08:00
// If palignr is shifting the pair of vectors more than the size of two
// lanes, emit zero.
if (ShiftVal >= 32)
return llvm::Constant::getNullValue(Op0->getType());
// If palignr is shifting the pair of input vectors more than one lane,
// but less than two lanes, convert to shifting in zeroes.
if (ShiftVal > 16) {
ShiftVal -= 16;
Op1 = Op0;
Op0 = llvm::Constant::getNullValue(Op0->getType());
}
[IR] Require ArrayRef of 'uint32_t' instead of 'int' for the mask argument for one of the signatures of CreateShuffleVector. This better emphasises that you can't use it for the -1 as undef behavior. llvm-svn: 272491
2016-06-12 08:41:19 +08:00
uint32_t Indices[64];
[AVX512] Remove masked palignr intrinsics and auto-upgrade them to native IR of vector shuffle and select. llvm-svn: 271872
2016-06-06 14:12:54 +08:00
// 256-bit palignr operates on 128-bit lanes so we need to handle that
[AVX-512] Remove intrinsics for valignd/q and autoupgrade them to native shuffles. llvm-svn: 287744
2016-11-23 14:54:55 +08:00
for (unsigned l = 0; l < NumElts; l += 16) {
[AVX512] Remove masked palignr intrinsics and auto-upgrade them to native IR of vector shuffle and select. llvm-svn: 271872
2016-06-06 14:12:54 +08:00
for (unsigned i = 0; i != 16; ++i) {
unsigned Idx = ShiftVal + i;
[AVX-512] Remove intrinsics for valignd/q and autoupgrade them to native shuffles. llvm-svn: 287744
2016-11-23 14:54:55 +08:00
if (!IsVALIGN && Idx >= 16) // Disable wrap for VALIGN.
[AVX512] Remove masked palignr intrinsics and auto-upgrade them to native IR of vector shuffle and select. llvm-svn: 271872
2016-06-06 14:12:54 +08:00
Idx += NumElts - 16; // End of lane, switch operand.
Indices[l + i] = Idx + l;
}
}
Value *Align = Builder.CreateShuffleVector(Op1, Op0,
makeArrayRef(Indices, NumElts),
"palignr");
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 10:36:42 +08:00
return EmitX86Select(Builder, Mask, Align, Passthru);
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
}
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
static Value *UpgradeMaskedStore(IRBuilder<> &Builder,
[AVX512] Remove masked store intrinsics. Clang now emits generic masked store intrinsics instead. The intrinsics will be autoupgraded to the same generic masked stores. llvm-svn: 271245
2016-05-31 09:50:02 +08:00
Value *Ptr, Value *Data, Value *Mask,
bool Aligned) {
// Cast the pointer to the right type.
Ptr = Builder.CreateBitCast(Ptr,
llvm::PointerType::getUnqual(Data->getType()));
unsigned Align =
Aligned ? cast<VectorType>(Data->getType())->getBitWidth() / 8 : 1;
// If the mask is all ones just emit a regular store.
if (const auto *C = dyn_cast<Constant>(Mask))
if (C->isAllOnesValue())
return Builder.CreateAlignedStore(Data, Ptr, Align);
// Convert the mask from an integer type to a vector of i1.
unsigned NumElts = Data->getType()->getVectorNumElements();
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 10:36:42 +08:00
Mask = getX86MaskVec(Builder, Mask, NumElts);
[AVX512] Remove masked store intrinsics. Clang now emits generic masked store intrinsics instead. The intrinsics will be autoupgraded to the same generic masked stores. llvm-svn: 271245
2016-05-31 09:50:02 +08:00
return Builder.CreateMaskedStore(Data, Ptr, Align, Mask);
}
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
static Value *UpgradeMaskedLoad(IRBuilder<> &Builder,
[AVX512] Remove masked load intrinsics. Clang now emits generic masked load intrinsics instead. The intrinsics will be autoupgraded to the same generic masked loads. llvm-svn: 271478
2016-06-02 12:19:36 +08:00
Value *Ptr, Value *Passthru, Value *Mask,
bool Aligned) {
// Cast the pointer to the right type.
Ptr = Builder.CreateBitCast(Ptr,
llvm::PointerType::getUnqual(Passthru->getType()));
unsigned Align =
Aligned ? cast<VectorType>(Passthru->getType())->getBitWidth() / 8 : 1;
// If the mask is all ones just emit a regular store.
if (const auto *C = dyn_cast<Constant>(Mask))
if (C->isAllOnesValue())
return Builder.CreateAlignedLoad(Ptr, Align);
// Convert the mask from an integer type to a vector of i1.
unsigned NumElts = Passthru->getType()->getVectorNumElements();
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 10:36:42 +08:00
Mask = getX86MaskVec(Builder, Mask, NumElts);
[AVX512] Remove masked load intrinsics. Clang now emits generic masked load intrinsics instead. The intrinsics will be autoupgraded to the same generic masked loads. llvm-svn: 271478
2016-06-02 12:19:36 +08:00
return Builder.CreateMaskedLoad(Ptr, Align, Mask, Passthru);
}
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 23:48:30 +08:00
static Value *upgradeIntMinMax(IRBuilder<> &Builder, CallInst &CI,
ICmpInst::Predicate Pred) {
Value *Op0 = CI.getArgOperand(0);
Value *Op1 = CI.getArgOperand(1);
Value *Cmp = Builder.CreateICmp(Pred, Op0, Op1);
[AVX-512] Remove masked pmin/pmax intrinsics and autoupgrade to native IR. Clang patch to replace 512-bit vector and 64-bit element versions with native IR will follow. llvm-svn: 284955
2016-10-24 12:04:16 +08:00
Value *Res = Builder.CreateSelect(Cmp, Op0, Op1);
if (CI.getNumArgOperands() == 4)
Res = EmitX86Select(Builder, CI.getArgOperand(3), Res, CI.getArgOperand(2));
return Res;
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 23:48:30 +08:00
}
[AVX512] Remove the masked vpcmpeq/vcmpgt intrinsics and autoupgrade them to native icmps. llvm-svn: 273240
2016-06-21 11:53:24 +08:00
static Value *upgradeMaskedCompare(IRBuilder<> &Builder, CallInst &CI,
[AVX-512] Remove and autoupgrade the masked integer compare intrinsics Summary: These intrinsics aren't used by clang and haven't been for a while. There's some really terrible codegen in the 32-bit target for avx512bw due to i64 not being legal. But as I said these intrinsics aren't used by clang even before this patch so this codegen reflects our clang behavior today. Reviewers: spatel, RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34389 llvm-svn: 306047
2017-06-23 04:11:01 +08:00
unsigned CC, bool Signed) {
[AVX512] Remove the masked vpcmpeq/vcmpgt intrinsics and autoupgrade them to native icmps. llvm-svn: 273240
2016-06-21 11:53:24 +08:00
Value *Op0 = CI.getArgOperand(0);
unsigned NumElts = Op0->getType()->getVectorNumElements();
[AVX-512] Remove and autoupgrade the masked integer compare intrinsics Summary: These intrinsics aren't used by clang and haven't been for a while. There's some really terrible codegen in the 32-bit target for avx512bw due to i64 not being legal. But as I said these intrinsics aren't used by clang even before this patch so this codegen reflects our clang behavior today. Reviewers: spatel, RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34389 llvm-svn: 306047
2017-06-23 04:11:01 +08:00
Value *Cmp;
if (CC == 3) {
Cmp = Constant::getNullValue(llvm::VectorType::get(Builder.getInt1Ty(), NumElts));
} else if (CC == 7) {
Cmp = Constant::getAllOnesValue(llvm::VectorType::get(Builder.getInt1Ty(), NumElts));
} else {
ICmpInst::Predicate Pred;
switch (CC) {
default: llvm_unreachable("Unknown condition code");
case 0: Pred = ICmpInst::ICMP_EQ; break;
case 1: Pred = Signed ? ICmpInst::ICMP_SLT : ICmpInst::ICMP_ULT; break;
case 2: Pred = Signed ? ICmpInst::ICMP_SLE : ICmpInst::ICMP_ULE; break;
case 4: Pred = ICmpInst::ICMP_NE; break;
case 5: Pred = Signed ? ICmpInst::ICMP_SGE : ICmpInst::ICMP_UGE; break;
case 6: Pred = Signed ? ICmpInst::ICMP_SGT : ICmpInst::ICMP_UGT; break;
}
Cmp = Builder.CreateICmp(Pred, Op0, CI.getArgOperand(1));
}
Value *Mask = CI.getArgOperand(CI.getNumArgOperands() - 1);
[AVX512] Remove the masked vpcmpeq/vcmpgt intrinsics and autoupgrade them to native icmps. llvm-svn: 273240
2016-06-21 11:53:24 +08:00
const auto *C = dyn_cast<Constant>(Mask);
if (!C || !C->isAllOnesValue())
Cmp = Builder.CreateAnd(Cmp, getX86MaskVec(Builder, Mask, NumElts));
if (NumElts < 8) {
uint32_t Indices[8];
for (unsigned i = 0; i != NumElts; ++i)
Indices[i] = i;
for (unsigned i = NumElts; i != 8; ++i)
[AVX512] Zero extend the result of vpcmpeq/vpcmpgt and similar intrinsics in the autoupgrade code. This currently results in worse codegen but is needed for correctness. llvm-svn: 274736
2016-07-07 14:11:07 +08:00
Indices[i] = NumElts + i % NumElts;
Cmp = Builder.CreateShuffleVector(Cmp,
Constant::getNullValue(Cmp->getType()),
[AVX512] Remove the masked vpcmpeq/vcmpgt intrinsics and autoupgrade them to native icmps. llvm-svn: 273240
2016-06-21 11:53:24 +08:00
Indices);
}
return Builder.CreateBitCast(Cmp, IntegerType::get(CI.getContext(),
std::max(NumElts, 8U)));
}
[AVX-512] Remove intrinsics for 128/256-bit masked shift by single element in xmm. Instead upgrade them to a select and the older SSE/AVX2 intrinsic. llvm-svn: 286070
2016-11-07 00:29:08 +08:00
// Replace a masked intrinsic with an older unmasked intrinsic.
static Value *UpgradeX86MaskedShift(IRBuilder<> &Builder, CallInst &CI,
Intrinsic::ID IID) {
Function *F = CI.getCalledFunction();
Function *Intrin = Intrinsic::getDeclaration(F->getParent(), IID);
Value *Rep = Builder.CreateCall(Intrin,
{ CI.getArgOperand(0), CI.getArgOperand(1) });
return EmitX86Select(Builder, CI.getArgOperand(3), Rep, CI.getArgOperand(2));
}
[X86][AVX512] Removing llvm x86 intrinsics for _mm_mask_move_{ss|sd} intrinsics. Differential Revision: https://reviews.llvm.org/D26128 llvm-svn: 287087
2016-11-16 17:00:28 +08:00
static Value* upgradeMaskedMove(IRBuilder<> &Builder, CallInst &CI) {
Value* A = CI.getArgOperand(0);
Value* B = CI.getArgOperand(1);
Value* Src = CI.getArgOperand(2);
Value* Mask = CI.getArgOperand(3);
Value* AndNode = Builder.CreateAnd(Mask, APInt(8, 1));
Value* Cmp = Builder.CreateIsNotNull(AndNode);
Value* Extract1 = Builder.CreateExtractElement(B, (uint64_t)0);
Value* Extract2 = Builder.CreateExtractElement(Src, (uint64_t)0);
Value* Select = Builder.CreateSelect(Cmp, Extract1, Extract2);
return Builder.CreateInsertElement(A, Select, (uint64_t)0);
}
[AVX-512] Remove intrinsics for 128/256-bit masked shift by single element in xmm. Instead upgrade them to a select and the older SSE/AVX2 intrinsic. llvm-svn: 286070
2016-11-07 00:29:08 +08:00
[X86][LLVM] Converting __mm{|256|512}_movm_epi{8|16|32|64} LLVMIR call into generic intrinsics. This patch is a part one of two reviews, one for the clang and the other for LLVM. The patch deletes the back-end intrinsics and adds support for them in the auto upgrade. Differential Revision: https://reviews.llvm.org/D31393 llvm-svn: 299432
2017-04-04 21:32:14 +08:00
static Value* UpgradeMaskToInt(IRBuilder<> &Builder, CallInst &CI) {
Value* Op = CI.getArgOperand(0);
Type* ReturnOp = CI.getType();
unsigned NumElts = CI.getType()->getVectorNumElements();
Value *Mask = getX86MaskVec(Builder, Op, NumElts);
return Builder.CreateSExt(Mask, ReturnOp, "vpmovm2");
}
fix comments; NFC llvm-svn: 272848
2016-06-16 06:01:28 +08:00
/// Upgrade a call to an old intrinsic. All argument and return casting must be
/// provided to seamlessly integrate with existing context.
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
void llvm::UpgradeIntrinsicCall(CallInst *CI, Function *NewFn) {
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 14:10:55 +08:00
Function *F = CI->getCalledFunction();
Fix unused value warning for value used only in assert. llvm-svn: 146440
2011-12-13 06:59:34 +08:00
LLVMContext &C = CI->getContext();
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 12:26:04 +08:00
IRBuilder<> Builder(C);
IR: Remove implicit iterator conversions from lib/IR, NFC Stop converting implicitly between iterators and pointers/references in lib/IR. For convenience, I've added a `getIterator()` accessor to `ilist_node` so that callers don't need to know how to spell the iterator class (i.e., they can use `X.getIterator()` instead of `Function::iterator(X)`). I'll eventually disallow these implicit conversions entirely, but there's a lot of code, so it doesn't make sense to do it all in one patch. One library or so at a time. Why? To root out cases of `getNextNode()` and `getPrevNode()` being used in iterator logic. The design of `ilist` makes that invalid when the current node could be at the back of the list, but it happens to "work" right now because of a bug where those functions never return `nullptr` if you're using a half-node sentinel. Before I can fix the function, I have to remove uses of it that rely on it misbehaving. (Maybe the function should just be deleted anyway? But I don't want deleting it -- potentially a huge project -- to block fixing ilist/iplist.) llvm-svn: 249782
2015-10-09 07:49:46 +08:00
Builder.SetInsertPoint(CI->getParent(), CI->getIterator());
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 12:26:04 +08:00
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 14:10:55 +08:00
assert(F && "Intrinsic call is not direct?");
if (!NewFn) {
// Get the Function's name.
StringRef Name = F->getName();
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
assert(Name.startswith("llvm.") && "Intrinsic doesn't start with 'llvm.'");
Name = Name.substr(5);
bool IsX86 = Name.startswith("x86.");
if (IsX86)
Name = Name.substr(4);
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 09:00:32 +08:00
bool IsNVVM = Name.startswith("nvvm.");
if (IsNVVM)
Name = Name.substr(5);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
[X86][IR] Move the autoupgrading of store intrinsics out of the main nested if/else chain. This should buy a little more time against the MSVC limit mentioned in PR31034. The handlers for stores all return at the end of their block so they can be picked off early. llvm-svn: 289339
2016-12-11 05:15:48 +08:00
if (IsX86 && Name.startswith("sse4a.movnt.")) {
Module *M = F->getParent();
SmallVector<Metadata *, 1> Elts;
Elts.push_back(
ConstantAsMetadata::get(ConstantInt::get(Type::getInt32Ty(C), 1)));
MDNode *Node = MDNode::get(C, Elts);
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
// Nontemporal (unaligned) store of the 0'th element of the float/double
// vector.
Type *SrcEltTy = cast<VectorType>(Arg1->getType())->getElementType();
PointerType *EltPtrTy = PointerType::getUnqual(SrcEltTy);
Value *Addr = Builder.CreateBitCast(Arg0, EltPtrTy, "cast");
Value *Extract =
Builder.CreateExtractElement(Arg1, (uint64_t)0, "extractelement");
StoreInst *SI = Builder.CreateAlignedStore(Extract, Addr, 1);
SI->setMetadata(M->getMDKindID("nontemporal"), Node);
// Remove intrinsic.
CI->eraseFromParent();
return;
}
if (IsX86 && (Name.startswith("avx.movnt.") ||
Name.startswith("avx512.storent."))) {
Module *M = F->getParent();
SmallVector<Metadata *, 1> Elts;
Elts.push_back(
ConstantAsMetadata::get(ConstantInt::get(Type::getInt32Ty(C), 1)));
MDNode *Node = MDNode::get(C, Elts);
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
// Convert the type of the pointer to a pointer to the stored type.
Value *BC = Builder.CreateBitCast(Arg0,
PointerType::getUnqual(Arg1->getType()),
"cast");
VectorType *VTy = cast<VectorType>(Arg1->getType());
StoreInst *SI = Builder.CreateAlignedStore(Arg1, BC,
VTy->getBitWidth() / 8);
SI->setMetadata(M->getMDKindID("nontemporal"), Node);
// Remove intrinsic.
CI->eraseFromParent();
return;
}
if (IsX86 && Name == "sse2.storel.dq") {
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
Type *NewVecTy = VectorType::get(Type::getInt64Ty(C), 2);
Value *BC0 = Builder.CreateBitCast(Arg1, NewVecTy, "cast");
Value *Elt = Builder.CreateExtractElement(BC0, (uint64_t)0);
Value *BC = Builder.CreateBitCast(Arg0,
PointerType::getUnqual(Elt->getType()),
"cast");
Builder.CreateAlignedStore(Elt, BC, 1);
// Remove intrinsic.
CI->eraseFromParent();
return;
}
if (IsX86 && (Name.startswith("sse.storeu.") ||
Name.startswith("sse2.storeu.") ||
Name.startswith("avx.storeu."))) {
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
Arg0 = Builder.CreateBitCast(Arg0,
PointerType::getUnqual(Arg1->getType()),
"cast");
Builder.CreateAlignedStore(Arg1, Arg0, 1);
// Remove intrinsic.
CI->eraseFromParent();
return;
}
[X86][IR] Merge together some very similar AutoUpgrade handling. NFC llvm-svn: 295562
2017-02-19 03:51:14 +08:00
if (IsX86 && (Name.startswith("avx512.mask.store"))) {
// "avx512.mask.storeu." or "avx512.mask.store."
bool Aligned = Name[17] != 'u'; // "avx512.mask.storeu".
[X86][IR] Move the autoupgrading of store intrinsics out of the main nested if/else chain. This should buy a little more time against the MSVC limit mentioned in PR31034. The handlers for stores all return at the end of their block so they can be picked off early. llvm-svn: 289339
2016-12-11 05:15:48 +08:00
UpgradeMaskedStore(Builder, CI->getArgOperand(0), CI->getArgOperand(1),
[X86][IR] Merge together some very similar AutoUpgrade handling. NFC llvm-svn: 295562
2017-02-19 03:51:14 +08:00
CI->getArgOperand(2), Aligned);
[X86][IR] Move the autoupgrading of store intrinsics out of the main nested if/else chain. This should buy a little more time against the MSVC limit mentioned in PR31034. The handlers for stores all return at the end of their block so they can be picked off early. llvm-svn: 289339
2016-12-11 05:15:48 +08:00
// Remove intrinsic.
CI->eraseFromParent();
return;
}
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 14:10:55 +08:00
Value *Rep;
fix comments; NFC llvm-svn: 272848
2016-06-16 06:01:28 +08:00
// Upgrade packed integer vector compare intrinsics to compare instructions.
[X86][IR] Merge together some very similar AutoUpgrade handling. NFC llvm-svn: 295562
2017-02-19 03:51:14 +08:00
if (IsX86 && (Name.startswith("sse2.pcmp") ||
Name.startswith("avx2.pcmp"))) {
// "sse2.pcpmpeq." "sse2.pcmpgt." "avx2.pcmpeq." or "avx2.pcmpgt."
bool CmpEq = Name[9] == 'e';
Rep = Builder.CreateICmp(CmpEq ? ICmpInst::ICMP_EQ : ICmpInst::ICMP_SGT,
CI->getArgOperand(0), CI->getArgOperand(1));
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 14:10:55 +08:00
Rep = Builder.CreateSExt(Rep, CI->getType(), "");
[X86] Remove the scalar intrinsics for fadd/fsub/fdiv/fmul Summary: These intrinsics have been unused for clang for a while. This patch removes them. We auto upgrade them to extractelements, a scalar operation and then an insertelement. This matches the sequence used by clangs intrinsic file. Reviewers: zvi, delena, RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D26660 llvm-svn: 287083
2016-11-16 13:24:10 +08:00
} else if (IsX86 && (Name == "sse.add.ss" || Name == "sse2.add.sd")) {
Type *I32Ty = Type::getInt32Ty(C);
Value *Elt0 = Builder.CreateExtractElement(CI->getArgOperand(0),
ConstantInt::get(I32Ty, 0));
Value *Elt1 = Builder.CreateExtractElement(CI->getArgOperand(1),
ConstantInt::get(I32Ty, 0));
Rep = Builder.CreateInsertElement(CI->getArgOperand(0),
Builder.CreateFAdd(Elt0, Elt1),
ConstantInt::get(I32Ty, 0));
} else if (IsX86 && (Name == "sse.sub.ss" || Name == "sse2.sub.sd")) {
Type *I32Ty = Type::getInt32Ty(C);
Value *Elt0 = Builder.CreateExtractElement(CI->getArgOperand(0),
ConstantInt::get(I32Ty, 0));
Value *Elt1 = Builder.CreateExtractElement(CI->getArgOperand(1),
ConstantInt::get(I32Ty, 0));
Rep = Builder.CreateInsertElement(CI->getArgOperand(0),
Builder.CreateFSub(Elt0, Elt1),
ConstantInt::get(I32Ty, 0));
} else if (IsX86 && (Name == "sse.mul.ss" || Name == "sse2.mul.sd")) {
Type *I32Ty = Type::getInt32Ty(C);
Value *Elt0 = Builder.CreateExtractElement(CI->getArgOperand(0),
ConstantInt::get(I32Ty, 0));
Value *Elt1 = Builder.CreateExtractElement(CI->getArgOperand(1),
ConstantInt::get(I32Ty, 0));
Rep = Builder.CreateInsertElement(CI->getArgOperand(0),
Builder.CreateFMul(Elt0, Elt1),
ConstantInt::get(I32Ty, 0));
} else if (IsX86 && (Name == "sse.div.ss" || Name == "sse2.div.sd")) {
Type *I32Ty = Type::getInt32Ty(C);
Value *Elt0 = Builder.CreateExtractElement(CI->getArgOperand(0),
ConstantInt::get(I32Ty, 0));
Value *Elt1 = Builder.CreateExtractElement(CI->getArgOperand(1),
ConstantInt::get(I32Ty, 0));
Rep = Builder.CreateInsertElement(CI->getArgOperand(0),
Builder.CreateFDiv(Elt0, Elt1),
ConstantInt::get(I32Ty, 0));
[X86][IR] Merge together some very similar AutoUpgrade handling. NFC llvm-svn: 295562
2017-02-19 03:51:14 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.pcmp")) {
// "avx512.mask.pcmpeq." or "avx512.mask.pcmpgt."
bool CmpEq = Name[16] == 'e';
[AVX-512] Remove and autoupgrade the masked integer compare intrinsics Summary: These intrinsics aren't used by clang and haven't been for a while. There's some really terrible codegen in the 32-bit target for avx512bw due to i64 not being legal. But as I said these intrinsics aren't used by clang even before this patch so this codegen reflects our clang behavior today. Reviewers: spatel, RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34389 llvm-svn: 306047
2017-06-23 04:11:01 +08:00
Rep = upgradeMaskedCompare(Builder, *CI, CmpEq ? 0 : 6, true);
} else if (IsX86 && Name.startswith("avx512.mask.cmp")) {
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
Rep = upgradeMaskedCompare(Builder, *CI, Imm, true);
} else if (IsX86 && Name.startswith("avx512.mask.ucmp")) {
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
Rep = upgradeMaskedCompare(Builder, *CI, Imm, false);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse41.pmaxsb" ||
Name == "sse2.pmaxs.w" ||
Name == "sse41.pmaxsd" ||
[AVX-512] Remove masked pmin/pmax intrinsics and autoupgrade to native IR. Clang patch to replace 512-bit vector and 64-bit element versions with native IR will follow. llvm-svn: 284955
2016-10-24 12:04:16 +08:00
Name.startswith("avx2.pmaxs") ||
Name.startswith("avx512.mask.pmaxs"))) {
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 23:48:30 +08:00
Rep = upgradeIntMinMax(Builder, *CI, ICmpInst::ICMP_SGT);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse2.pmaxu.b" ||
Name == "sse41.pmaxuw" ||
Name == "sse41.pmaxud" ||
[AVX-512] Remove masked pmin/pmax intrinsics and autoupgrade to native IR. Clang patch to replace 512-bit vector and 64-bit element versions with native IR will follow. llvm-svn: 284955
2016-10-24 12:04:16 +08:00
Name.startswith("avx2.pmaxu") ||
Name.startswith("avx512.mask.pmaxu"))) {
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 23:48:30 +08:00
Rep = upgradeIntMinMax(Builder, *CI, ICmpInst::ICMP_UGT);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse41.pminsb" ||
Name == "sse2.pmins.w" ||
Name == "sse41.pminsd" ||
[AVX-512] Remove masked pmin/pmax intrinsics and autoupgrade to native IR. Clang patch to replace 512-bit vector and 64-bit element versions with native IR will follow. llvm-svn: 284955
2016-10-24 12:04:16 +08:00
Name.startswith("avx2.pmins") ||
Name.startswith("avx512.mask.pmins"))) {
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 23:48:30 +08:00
Rep = upgradeIntMinMax(Builder, *CI, ICmpInst::ICMP_SLT);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse2.pminu.b" ||
Name == "sse41.pminuw" ||
Name == "sse41.pminud" ||
[AVX-512] Remove masked pmin/pmax intrinsics and autoupgrade to native IR. Clang patch to replace 512-bit vector and 64-bit element versions with native IR will follow. llvm-svn: 284955
2016-10-24 12:04:16 +08:00
Name.startswith("avx2.pminu") ||
Name.startswith("avx512.mask.pminu"))) {
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 23:48:30 +08:00
Rep = upgradeIntMinMax(Builder, *CI, ICmpInst::ICMP_ULT);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse2.cvtdq2pd" ||
Name == "sse2.cvtps2pd" ||
Name == "avx.cvtdq2.pd.256" ||
[X86][AVX512] Autoupgrade lossless i32/u32 to f64 conversion intrinsics with generic IR Both the (V)CVTDQ2PD (i32 to f64) and (V)CVTUDQ2PD (u32 to f64) conversion instructions are lossless and can be safely represented as generic SINT_TO_FP/UINT_TO_FP calls instead of x86 intrinsics without affecting final codegen. LLVM counterpart to D26686 Differential Revision: https://reviews.llvm.org/D26736 llvm-svn: 287108
2016-11-16 22:48:32 +08:00
Name == "avx.cvt.ps2.pd.256" ||
Name.startswith("avx512.mask.cvtdq2pd.") ||
Name.startswith("avx512.mask.cvtudq2pd."))) {
[X86][SSE] Replace (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) lossless conversion intrinsics with generic IR Followup to D20528 clang patch, this removes the (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) llvm intrinsics and auto-upgrades to sitofp/fpext instead. Differential Revision: http://reviews.llvm.org/D20568 llvm-svn: 270678
2016-05-25 16:59:18 +08:00
// Lossless i32/float to double conversion.
// Extract the bottom elements if necessary and convert to double vector.
Value *Src = CI->getArgOperand(0);
VectorType *SrcTy = cast<VectorType>(Src->getType());
VectorType *DstTy = cast<VectorType>(CI->getType());
Rep = CI->getArgOperand(0);
unsigned NumDstElts = DstTy->getNumElements();
if (NumDstElts < SrcTy->getNumElements()) {
assert(NumDstElts == 2 && "Unexpected vector size");
[IR] Require ArrayRef of 'uint32_t' instead of 'int' for the mask argument for one of the signatures of CreateShuffleVector. This better emphasises that you can't use it for the -1 as undef behavior. llvm-svn: 272491
2016-06-12 08:41:19 +08:00
uint32_t ShuffleMask[2] = { 0, 1 };
Rep = Builder.CreateShuffleVector(Rep, UndefValue::get(SrcTy),
ShuffleMask);
[X86][SSE] Replace (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) lossless conversion intrinsics with generic IR Followup to D20528 clang patch, this removes the (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) llvm intrinsics and auto-upgrades to sitofp/fpext instead. Differential Revision: http://reviews.llvm.org/D20568 llvm-svn: 270678
2016-05-25 16:59:18 +08:00
}
[X86][AVX512] Autoupgrade lossless i32/u32 to f64 conversion intrinsics with generic IR Both the (V)CVTDQ2PD (i32 to f64) and (V)CVTUDQ2PD (u32 to f64) conversion instructions are lossless and can be safely represented as generic SINT_TO_FP/UINT_TO_FP calls instead of x86 intrinsics without affecting final codegen. LLVM counterpart to D26686 Differential Revision: https://reviews.llvm.org/D26736 llvm-svn: 287108
2016-11-16 22:48:32 +08:00
bool SInt2Double = (StringRef::npos != Name.find("cvtdq2"));
bool UInt2Double = (StringRef::npos != Name.find("cvtudq2"));
if (SInt2Double)
[X86][SSE] Replace (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) lossless conversion intrinsics with generic IR Followup to D20528 clang patch, this removes the (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) llvm intrinsics and auto-upgrades to sitofp/fpext instead. Differential Revision: http://reviews.llvm.org/D20568 llvm-svn: 270678
2016-05-25 16:59:18 +08:00
Rep = Builder.CreateSIToFP(Rep, DstTy, "cvtdq2pd");
[X86][AVX512] Autoupgrade lossless i32/u32 to f64 conversion intrinsics with generic IR Both the (V)CVTDQ2PD (i32 to f64) and (V)CVTUDQ2PD (u32 to f64) conversion instructions are lossless and can be safely represented as generic SINT_TO_FP/UINT_TO_FP calls instead of x86 intrinsics without affecting final codegen. LLVM counterpart to D26686 Differential Revision: https://reviews.llvm.org/D26736 llvm-svn: 287108
2016-11-16 22:48:32 +08:00
else if (UInt2Double)
Rep = Builder.CreateUIToFP(Rep, DstTy, "cvtudq2pd");
[X86][SSE] Replace (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) lossless conversion intrinsics with generic IR Followup to D20528 clang patch, this removes the (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) llvm intrinsics and auto-upgrades to sitofp/fpext instead. Differential Revision: http://reviews.llvm.org/D20568 llvm-svn: 270678
2016-05-25 16:59:18 +08:00
else
Rep = Builder.CreateFPExt(Rep, DstTy, "cvtps2pd");
[X86][AVX512] Autoupgrade lossless i32/u32 to f64 conversion intrinsics with generic IR Both the (V)CVTDQ2PD (i32 to f64) and (V)CVTUDQ2PD (u32 to f64) conversion instructions are lossless and can be safely represented as generic SINT_TO_FP/UINT_TO_FP calls instead of x86 intrinsics without affecting final codegen. LLVM counterpart to D26686 Differential Revision: https://reviews.llvm.org/D26736 llvm-svn: 287108
2016-11-16 22:48:32 +08:00
if (CI->getNumArgOperands() == 3)
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[X86] Combine some of the strings in autoupgrade code. llvm-svn: 280603
2016-09-04 07:55:13 +08:00
} else if (IsX86 && (Name.startswith("avx512.mask.loadu."))) {
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
Rep = UpgradeMaskedLoad(Builder, CI->getArgOperand(0),
[AVX512] Remove masked load intrinsics. Clang now emits generic masked load intrinsics instead. The intrinsics will be autoupgraded to the same generic masked loads. llvm-svn: 271478
2016-06-02 12:19:36 +08:00
CI->getArgOperand(1), CI->getArgOperand(2),
/*Aligned*/false);
[X86] Combine some of the strings in autoupgrade code. llvm-svn: 280603
2016-09-04 07:55:13 +08:00
} else if (IsX86 && (Name.startswith("avx512.mask.load."))) {
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
Rep = UpgradeMaskedLoad(Builder, CI->getArgOperand(0),
[AVX512] Remove masked load intrinsics. Clang now emits generic masked load intrinsics instead. The intrinsics will be autoupgraded to the same generic masked loads. llvm-svn: 271478
2016-06-02 12:19:36 +08:00
CI->getArgOperand(1),CI->getArgOperand(2),
/*Aligned*/true);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && Name.startswith("xop.vpcom")) {
Replace XOP vpcom intrinsics with fewer intrinsics that take the immediate as an argument. llvm-svn: 158278
2012-06-10 00:46:13 +08:00
Intrinsic::ID intID;
if (Name.endswith("ub"))
intID = Intrinsic::x86_xop_vpcomub;
else if (Name.endswith("uw"))
intID = Intrinsic::x86_xop_vpcomuw;
else if (Name.endswith("ud"))
intID = Intrinsic::x86_xop_vpcomud;
else if (Name.endswith("uq"))
intID = Intrinsic::x86_xop_vpcomuq;
else if (Name.endswith("b"))
intID = Intrinsic::x86_xop_vpcomb;
else if (Name.endswith("w"))
intID = Intrinsic::x86_xop_vpcomw;
else if (Name.endswith("d"))
intID = Intrinsic::x86_xop_vpcomd;
else if (Name.endswith("q"))
intID = Intrinsic::x86_xop_vpcomq;
else
llvm_unreachable("Unknown suffix");
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
Name = Name.substr(9); // strip off "xop.vpcom"
Replace XOP vpcom intrinsics with fewer intrinsics that take the immediate as an argument. llvm-svn: 158278
2012-06-10 00:46:13 +08:00
unsigned Imm;
if (Name.startswith("lt"))
Imm = 0;
else if (Name.startswith("le"))
Imm = 1;
else if (Name.startswith("gt"))
Imm = 2;
else if (Name.startswith("ge"))
Imm = 3;
else if (Name.startswith("eq"))
Imm = 4;
else if (Name.startswith("ne"))
Imm = 5;
else if (Name.startswith("false"))
[X86] Fix XOP vpcom intrinsic autoupgrade to map 'true' and 'false' to the correct immediates. Seems they were swapped. llvm-svn: 229077
2015-02-13 15:42:15 +08:00
Imm = 6;
else if (Name.startswith("true"))
Replace XOP vpcom intrinsics with fewer intrinsics that take the immediate as an argument. llvm-svn: 158278
2012-06-10 00:46:13 +08:00
Imm = 7;
else
llvm_unreachable("Unknown condition");
Function *VPCOM = Intrinsic::getDeclaration(F->getParent(), intID);
Simplify IRBuilder::CreateCall* by using ArrayRef+initializer_list/braced init only llvm-svn: 237624
2015-05-19 06:13:54 +08:00
Rep =
Builder.CreateCall(VPCOM, {CI->getArgOperand(0), CI->getArgOperand(1),
Builder.getInt8(Imm)});
Recommit "[X86] Remove XOP VPCMOV intrinsics and autoupgrade them to native IR." Clang has now been fixed to not use these intrinsics. llvm-svn: 295571
2017-02-19 05:50:58 +08:00
} else if (IsX86 && Name.startswith("xop.vpcmov")) {
[X86][XOP] Add support for the matching of the VPCMOV bit select instruction XOP has the VPCMOV instruction that performs the common vector bit select operation OR( AND( SRC1, SRC3 ), AND( SRC2, ~SRC3 ) ) This patch adds tablegen pattern matching for this instruction. Differential Revision: http://reviews.llvm.org/D8841 llvm-svn: 251975
2015-11-04 04:27:01 +08:00
Value *Sel = CI->getArgOperand(2);
[X86][IR] Simplify the XOP vpcmov autoupgrade code. NFC llvm-svn: 295563
2017-02-19 03:51:19 +08:00
Value *NotSel = Builder.CreateNot(Sel);
Value *Sel0 = Builder.CreateAnd(CI->getArgOperand(0), Sel);
Value *Sel1 = Builder.CreateAnd(CI->getArgOperand(1), NotSel);
[X86][XOP] Add support for the matching of the VPCMOV bit select instruction XOP has the VPCMOV instruction that performs the common vector bit select operation OR( AND( SRC1, SRC3 ), AND( SRC2, ~SRC3 ) ) This patch adds tablegen pattern matching for this instruction. Differential Revision: http://reviews.llvm.org/D8841 llvm-svn: 251975
2015-11-04 04:27:01 +08:00
Rep = Builder.CreateOr(Sel0, Sel1);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && Name == "sse42.crc32.64.8") {
Remove x86_sse42_crc32_64_8 intrinsic. It has no functional difference from x86_sse42_crc32_32_8 and was not mapped to a clang builtin. I'm not even sure why this form of the instruction is even called out explicitly in the docs. Also add AutoUpgrade support to convert it into the other intrinsic with appropriate trunc and zext. llvm-svn: 192672
2013-10-15 13:20:47 +08:00
Function *CRC32 = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::x86_sse42_crc32_32_8);
Value *Trunc0 = Builder.CreateTrunc(CI->getArgOperand(0), Type::getInt32Ty(C));
Simplify IRBuilder::CreateCall* by using ArrayRef+initializer_list/braced init only llvm-svn: 237624
2015-05-19 06:13:54 +08:00
Rep = Builder.CreateCall(CRC32, {Trunc0, CI->getArgOperand(1)});
Remove x86_sse42_crc32_64_8 intrinsic. It has no functional difference from x86_sse42_crc32_32_8 and was not mapped to a clang builtin. I'm not even sure why this form of the instruction is even called out explicitly in the docs. Also add AutoUpgrade support to convert it into the other intrinsic with appropriate trunc and zext. llvm-svn: 192672
2013-10-15 13:20:47 +08:00
Rep = Builder.CreateZExt(Rep, CI->getType(), "");
[X86][AVX] Added support for lowering to VBROADCASTF128/VBROADCASTI128 (reapplied) As reported on PR26235, we don't currently make use of the VBROADCASTF128/VBROADCASTI128 instructions (or the AVX512 equivalents) to load+splat a 128-bit vector to both lanes of a 256-bit vector. This patch enables lowering from subvector insertion/concatenation patterns and auto-upgrades the llvm.x86.avx.vbroadcastf128.pd.256 / llvm.x86.avx.vbroadcastf128.ps.256 intrinsics to match. We could possibly investigate using VBROADCASTF128/VBROADCASTI128 to load repeated constants as well (similar to how we already do for scalar broadcasts). Reapplied with fix for PR28657 - removed intrinsic definitions (clang companion patch to be be submitted shortly). Differential Revision: https://reviews.llvm.org/D22460 llvm-svn: 276416
2016-07-22 21:58:44 +08:00
} else if (IsX86 && Name.startswith("avx.vbroadcast.s")) {
[X86] Auto-upgrade AVX1 vbroadcast intrinsics They are replaced with the same IR that is generated for the vector-initializers in avxintrin.h. The test verifies that we get back the original instruction. I haven't seen this approach to be used in other auto-upgrade tests (i.e. llc + FileCheck) but I think it's the most direct way to test this case. I believe this should work because llc upgrades calls during parsing. (Other tests mostly check that assembling and disassembling yields the upgraded IR.) llvm-svn: 209863
2014-05-30 07:35:33 +08:00
// Replace broadcasts with a series of insertelements.
Type *VecTy = CI->getType();
Type *EltTy = VecTy->getVectorElementType();
unsigned EltNum = VecTy->getVectorNumElements();
Value *Cast = Builder.CreateBitCast(CI->getArgOperand(0),
EltTy->getPointerTo());
[opaque pointer type] Pass explicit type to Load instruction creation in AutoUpgrade llvm-svn: 237838
2015-05-21 05:46:30 +08:00
Value *Load = Builder.CreateLoad(EltTy, Cast);
[X86] Auto-upgrade AVX1 vbroadcast intrinsics They are replaced with the same IR that is generated for the vector-initializers in avxintrin.h. The test verifies that we get back the original instruction. I haven't seen this approach to be used in other auto-upgrade tests (i.e. llc + FileCheck) but I think it's the most direct way to test this case. I believe this should work because llc upgrades calls during parsing. (Other tests mostly check that assembling and disassembling yields the upgraded IR.) llvm-svn: 209863
2014-05-30 07:35:33 +08:00
Type *I32Ty = Type::getInt32Ty(C);
Rep = UndefValue::get(VecTy);
for (unsigned I = 0; I < EltNum; ++I)
Rep = Builder.CreateInsertElement(Rep, Load,
ConstantInt::get(I32Ty, I));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name.startswith("sse41.pmovsx") ||
Name.startswith("sse41.pmovzx") ||
Name.startswith("avx2.pmovsx") ||
[AVX-512] Remove masked pmovzx/pmovsx builtins and autoupgrade them to selects and native zext/sext. This mostly reuses earlier autoupgrade support for the sse and avx equivalents. Just needed to add the code to add the select. llvm-svn: 286092
2016-11-07 10:12:57 +08:00
Name.startswith("avx2.pmovzx") ||
Name.startswith("avx512.mask.pmovsx") ||
Name.startswith("avx512.mask.pmovzx"))) {
[X86][SSE] Replace 128-bit SSE41 PMOVSX intrinsics with native IR This patches removes the x86.sse41.pmovsx* intrinsics, provides a suitable upgrade path and updates relevant tests to sign extend a subvector instead. LLVM counterpart to D12835 Differential Revision: http://reviews.llvm.org/D13002 llvm-svn: 248368
2015-09-23 16:48:33 +08:00
VectorType *SrcTy = cast<VectorType>(CI->getArgOperand(0)->getType());
VectorType *DstTy = cast<VectorType>(CI->getType());
unsigned NumDstElts = DstTy->getNumElements();
[X86][SSE] (Reapplied) Replace (V)PMOVSX and (V)PMOVZX integer extension intrinsics with generic IR (llvm) This patch removes the llvm intrinsics VPMOVSX and (V)PMOVZX sign/zero extension intrinsics and auto-upgrades to SEXT/ZEXT calls instead. We already did this for SSE41 PMOVSX sometime ago so much of that implementation can be reused. Reapplied now that the the companion patch (D20684) removes/auto-upgrade the clang intrinsics has been committed. Differential Revision: http://reviews.llvm.org/D20686 llvm-svn: 271131
2016-05-29 02:03:41 +08:00
// Extract a subvector of the first NumDstElts lanes and sign/zero extend.
[X86] Pre-allocate some of the shuffle mask SmallVectors in the auto upgrade code instead of calling push_back in a loop. This removes the need to check if the vector needs to grow on each iteration. llvm-svn: 272501
2016-06-12 12:48:00 +08:00
SmallVector<uint32_t, 8> ShuffleMask(NumDstElts);
[IR] Require ArrayRef of 'uint32_t' instead of 'int' for the mask argument for one of the signatures of CreateShuffleVector. This better emphasises that you can't use it for the -1 as undef behavior. llvm-svn: 272491
2016-06-12 08:41:19 +08:00
for (unsigned i = 0; i != NumDstElts; ++i)
[X86] Pre-allocate some of the shuffle mask SmallVectors in the auto upgrade code instead of calling push_back in a loop. This removes the need to check if the vector needs to grow on each iteration. llvm-svn: 272501
2016-06-12 12:48:00 +08:00
ShuffleMask[i] = i;
[X86][SSE] Replace 128-bit SSE41 PMOVSX intrinsics with native IR This patches removes the x86.sse41.pmovsx* intrinsics, provides a suitable upgrade path and updates relevant tests to sign extend a subvector instead. LLVM counterpart to D12835 Differential Revision: http://reviews.llvm.org/D13002 llvm-svn: 248368
2015-09-23 16:48:33 +08:00
Value *SV = Builder.CreateShuffleVector(
CI->getArgOperand(0), UndefValue::get(SrcTy), ShuffleMask);
[X86][SSE] (Reapplied) Replace (V)PMOVSX and (V)PMOVZX integer extension intrinsics with generic IR (llvm) This patch removes the llvm intrinsics VPMOVSX and (V)PMOVZX sign/zero extension intrinsics and auto-upgrades to SEXT/ZEXT calls instead. We already did this for SSE41 PMOVSX sometime ago so much of that implementation can be reused. Reapplied now that the the companion patch (D20684) removes/auto-upgrade the clang intrinsics has been committed. Differential Revision: http://reviews.llvm.org/D20686 llvm-svn: 271131
2016-05-29 02:03:41 +08:00
bool DoSext = (StringRef::npos != Name.find("pmovsx"));
Rep = DoSext ? Builder.CreateSExt(SV, DstTy)
: Builder.CreateZExt(SV, DstTy);
[AVX-512] Remove masked pmovzx/pmovsx builtins and autoupgrade them to selects and native zext/sext. This mostly reuses earlier autoupgrade support for the sse and avx equivalents. Just needed to add the code to add the select. llvm-svn: 286092
2016-11-07 10:12:57 +08:00
// If there are 3 arguments, it's a masked intrinsic so we need a select.
if (CI->getNumArgOperands() == 3)
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[X86][AVX] Added support for lowering to VBROADCASTF128/VBROADCASTI128 (reapplied) As reported on PR26235, we don't currently make use of the VBROADCASTF128/VBROADCASTI128 instructions (or the AVX512 equivalents) to load+splat a 128-bit vector to both lanes of a 256-bit vector. This patch enables lowering from subvector insertion/concatenation patterns and auto-upgrades the llvm.x86.avx.vbroadcastf128.pd.256 / llvm.x86.avx.vbroadcastf128.ps.256 intrinsics to match. We could possibly investigate using VBROADCASTF128/VBROADCASTI128 to load repeated constants as well (similar to how we already do for scalar broadcasts). Reapplied with fix for PR28657 - removed intrinsic definitions (clang companion patch to be be submitted shortly). Differential Revision: https://reviews.llvm.org/D22460 llvm-svn: 276416
2016-07-22 21:58:44 +08:00
} else if (IsX86 && (Name.startswith("avx.vbroadcastf128") ||
Name == "avx2.vbroadcasti128")) {
// Replace vbroadcastf128/vbroadcasti128 with a vector load+shuffle.
Type *EltTy = CI->getType()->getVectorElementType();
unsigned NumSrcElts = 128 / EltTy->getPrimitiveSizeInBits();
Type *VT = VectorType::get(EltTy, NumSrcElts);
[opaque pointer type] Pass explicit type to Load instruction creation in AutoUpgrade llvm-svn: 237838
2015-05-21 05:46:30 +08:00
Value *Op = Builder.CreatePointerCast(CI->getArgOperand(0),
PointerType::getUnqual(VT));
[x86] Fix a bug in the auto-upgrade from r276416 where we failed to give a sufficiently low alignment for the IR load created. There is no test case because we don't have any test cases for the *IR* produced by the autoupgrade, only the x86 assembly, and it happens that the x86 assembly for this intrinsic as it is tested in the autoupgrade path just happens to not produce a separate load instruction where we might have observed the alignment. I'm going to follow up on the original commit to suggest getting IR-level testing in addition to the asm level testing here so that we can see and test these kinds of issues. We might never get an x86 instruction out with an alignment constraint, but we could stil miscompile code by folding against the alignment marked on (or inferred for in this case) the load. llvm-svn: 278203
2016-08-10 15:41:26 +08:00
Value *Load = Builder.CreateAlignedLoad(Op, 1);
[X86][AVX] Added support for lowering to VBROADCASTF128/VBROADCASTI128 (reapplied) As reported on PR26235, we don't currently make use of the VBROADCASTF128/VBROADCASTI128 instructions (or the AVX512 equivalents) to load+splat a 128-bit vector to both lanes of a 256-bit vector. This patch enables lowering from subvector insertion/concatenation patterns and auto-upgrades the llvm.x86.avx.vbroadcastf128.pd.256 / llvm.x86.avx.vbroadcastf128.ps.256 intrinsics to match. We could possibly investigate using VBROADCASTF128/VBROADCASTI128 to load repeated constants as well (similar to how we already do for scalar broadcasts). Reapplied with fix for PR28657 - removed intrinsic definitions (clang companion patch to be be submitted shortly). Differential Revision: https://reviews.llvm.org/D22460 llvm-svn: 276416
2016-07-22 21:58:44 +08:00
if (NumSrcElts == 2)
Rep = Builder.CreateShuffleVector(Load, UndefValue::get(Load->getType()),
{ 0, 1, 0, 1 });
else
Rep = Builder.CreateShuffleVector(Load, UndefValue::get(Load->getType()),
{ 0, 1, 2, 3, 0, 1, 2, 3 });
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name.startswith("avx2.pbroadcast") ||
[X86][AVX512] Autoupgrade the BROADCAST intrinsics llvm-svn: 274550
2016-07-05 21:58:47 +08:00
Name.startswith("avx2.vbroadcast") ||
Name.startswith("avx512.pbroadcast") ||
Name.startswith("avx512.mask.broadcast.s"))) {
[X86] Replace avx2 broadcast intrinsics with native IR. Since r245605, the clang headers don't use these anymore. r245165 updated some of the tests already; update the others, add an autoupgrade, remove the intrinsics, and cleanup the definitions. Differential Revision: http://reviews.llvm.org/D10555 llvm-svn: 245606
2015-08-21 04:36:19 +08:00
// Replace vp?broadcasts with a vector shuffle.
Value *Op = CI->getArgOperand(0);
unsigned NumElts = CI->getType()->getVectorNumElements();
Type *MaskTy = VectorType::get(Type::getInt32Ty(C), NumElts);
Rep = Builder.CreateShuffleVector(Op, UndefValue::get(Op->getType()),
Constant::getNullValue(MaskTy));
[X86][AVX512] Autoupgrade the BROADCAST intrinsics llvm-svn: 274550
2016-07-05 21:58:47 +08:00
if (CI->getNumArgOperands() == 3)
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.palignr.")) {
[AVX-512] Remove intrinsics for valignd/q and autoupgrade them to native shuffles. llvm-svn: 287744
2016-11-23 14:54:55 +08:00
Rep = UpgradeX86ALIGNIntrinsics(Builder, CI->getArgOperand(0),
CI->getArgOperand(1),
CI->getArgOperand(2),
CI->getArgOperand(3),
CI->getArgOperand(4),
false);
} else if (IsX86 && Name.startswith("avx512.mask.valign.")) {
Rep = UpgradeX86ALIGNIntrinsics(Builder, CI->getArgOperand(0),
CI->getArgOperand(1),
CI->getArgOperand(2),
CI->getArgOperand(3),
CI->getArgOperand(4),
true);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse2.psll.dq" ||
Name == "avx2.psll.dq")) {
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 14:37:33 +08:00
// 128/256-bit shift left specified in bits.
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
unsigned Shift = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
Rep = UpgradeX86PSLLDQIntrinsics(Builder, CI->getArgOperand(0),
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
Shift / 8); // Shift is in bits.
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse2.psrl.dq" ||
Name == "avx2.psrl.dq")) {
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 14:37:33 +08:00
// 128/256-bit shift right specified in bits.
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
unsigned Shift = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
Rep = UpgradeX86PSRLDQIntrinsics(Builder, CI->getArgOperand(0),
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 14:24:44 +08:00
Shift / 8); // Shift is in bits.
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse2.psll.dq.bs" ||
Name == "avx2.psll.dq.bs" ||
Name == "avx512.psll.dq.512")) {
[X86][AVX512] Dropped avx512 VPSLLDQ/VPSRLDQ intrinsics Auto-upgrade to generic shuffles like sse/avx2 implementations now that we can lower to VPSLLDQ/VPSRLDQ llvm-svn: 272308
2016-06-10 05:09:03 +08:00
// 128/256/512-bit shift left specified in bytes.
[X86] Remove x86.avx2.psll.dq.bs and x86.avx2.psrl.dq.bs intrinsics. llvm-svn: 229430
2015-02-17 04:51:59 +08:00
unsigned Shift = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
Rep = UpgradeX86PSLLDQIntrinsics(Builder, CI->getArgOperand(0), Shift);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse2.psrl.dq.bs" ||
Name == "avx2.psrl.dq.bs" ||
Name == "avx512.psrl.dq.512")) {
[X86][AVX512] Dropped avx512 VPSLLDQ/VPSRLDQ intrinsics Auto-upgrade to generic shuffles like sse/avx2 implementations now that we can lower to VPSLLDQ/VPSRLDQ llvm-svn: 272308
2016-06-10 05:09:03 +08:00
// 128/256/512-bit shift right specified in bytes.
[X86] Remove x86.avx2.psll.dq.bs and x86.avx2.psrl.dq.bs intrinsics. llvm-svn: 229430
2015-02-17 04:51:59 +08:00
unsigned Shift = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 09:42:33 +08:00
Rep = UpgradeX86PSRLDQIntrinsics(Builder, CI->getArgOperand(0), Shift);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse41.pblendw" ||
Name.startswith("sse41.blendp") ||
Name.startswith("avx.blend.p") ||
Name == "avx2.pblendw" ||
Name.startswith("avx2.pblendd."))) {
[X86] Remove the blendpd/blendps/pblendw/pblendd intrinsics. They can represented by shuffle_vector instructions. llvm-svn: 230860
2015-03-01 03:33:17 +08:00
Value *Op0 = CI->getArgOperand(0);
Value *Op1 = CI->getArgOperand(1);
unsigned Imm = cast <ConstantInt>(CI->getArgOperand(2))->getZExtValue();
VectorType *VecTy = cast<VectorType>(CI->getType());
unsigned NumElts = VecTy->getNumElements();
[X86] Pre-allocate some of the shuffle mask SmallVectors in the auto upgrade code instead of calling push_back in a loop. This removes the need to check if the vector needs to grow on each iteration. llvm-svn: 272501
2016-06-12 12:48:00 +08:00
SmallVector<uint32_t, 16> Idxs(NumElts);
for (unsigned i = 0; i != NumElts; ++i)
Idxs[i] = ((Imm >> (i%8)) & 1) ? i + NumElts : i;
[X86] Remove the blendpd/blendps/pblendw/pblendd intrinsics. They can represented by shuffle_vector instructions. llvm-svn: 230860
2015-03-01 03:33:17 +08:00
[X86,IR] Make use of the CreateShuffleVector form that takes an ArrayRef<uint32_t> to avoid the need to manually create a bunch of Constants and a ConstantVector. NFC llvm-svn: 272493
2016-06-12 09:05:59 +08:00
Rep = Builder.CreateShuffleVector(Op0, Op1, Idxs);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name.startswith("avx.vinsertf128.") ||
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 13:45:57 +08:00
Name == "avx2.vinserti128" ||
Name.startswith("avx512.mask.insert"))) {
[X86, AVX] replace vinsertf128 intrinsics with generic shuffles We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is the sibling patch for the Clang half of this change: http://reviews.llvm.org/D8088 Differential Revision: http://reviews.llvm.org/D8086 llvm-svn: 231794
2015-03-11 00:08:36 +08:00
Value *Op0 = CI->getArgOperand(0);
Value *Op1 = CI->getArgOperand(1);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 13:45:57 +08:00
unsigned DstNumElts = CI->getType()->getVectorNumElements();
unsigned SrcNumElts = Op1->getType()->getVectorNumElements();
unsigned Scale = DstNumElts / SrcNumElts;
[X86][SSE] Replace 128-bit SSE41 PMOVSX intrinsics with native IR This patches removes the x86.sse41.pmovsx* intrinsics, provides a suitable upgrade path and updates relevant tests to sign extend a subvector instead. LLVM counterpart to D12835 Differential Revision: http://reviews.llvm.org/D13002 llvm-svn: 248368
2015-09-23 16:48:33 +08:00
[X86, AVX] replace vinsertf128 intrinsics with generic shuffles We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is the sibling patch for the Clang half of this change: http://reviews.llvm.org/D8088 Differential Revision: http://reviews.llvm.org/D8086 llvm-svn: 231794
2015-03-11 00:08:36 +08:00
// Mask off the high bits of the immediate value; hardware ignores those.
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 13:45:57 +08:00
Imm = Imm % Scale;
[X86][SSE] Replace 128-bit SSE41 PMOVSX intrinsics with native IR This patches removes the x86.sse41.pmovsx* intrinsics, provides a suitable upgrade path and updates relevant tests to sign extend a subvector instead. LLVM counterpart to D12835 Differential Revision: http://reviews.llvm.org/D13002 llvm-svn: 248368
2015-09-23 16:48:33 +08:00
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 13:45:57 +08:00
// Extend the second operand into a vector the size of the destination.
[X86, AVX] replace vinsertf128 intrinsics with generic shuffles We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is the sibling patch for the Clang half of this change: http://reviews.llvm.org/D8088 Differential Revision: http://reviews.llvm.org/D8086 llvm-svn: 231794
2015-03-11 00:08:36 +08:00
Value *UndefV = UndefValue::get(Op1->getType());
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 13:45:57 +08:00
SmallVector<uint32_t, 8> Idxs(DstNumElts);
for (unsigned i = 0; i != SrcNumElts; ++i)
[X86] Pre-allocate some of the shuffle mask SmallVectors in the auto upgrade code instead of calling push_back in a loop. This removes the need to check if the vector needs to grow on each iteration. llvm-svn: 272501
2016-06-12 12:48:00 +08:00
Idxs[i] = i;
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 13:45:57 +08:00
for (unsigned i = SrcNumElts; i != DstNumElts; ++i)
Idxs[i] = SrcNumElts;
[X86,IR] Make use of the CreateShuffleVector form that takes an ArrayRef<uint32_t> to avoid the need to manually create a bunch of Constants and a ConstantVector. NFC llvm-svn: 272493
2016-06-12 09:05:59 +08:00
Rep = Builder.CreateShuffleVector(Op1, UndefV, Idxs);
[X86, AVX] replace vinsertf128 intrinsics with generic shuffles We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is the sibling patch for the Clang half of this change: http://reviews.llvm.org/D8088 Differential Revision: http://reviews.llvm.org/D8086 llvm-svn: 231794
2015-03-11 00:08:36 +08:00
// Insert the second operand into the first operand.
// Note that there is no guarantee that instruction lowering will actually
// produce a vinsertf128 instruction for the created shuffles. In
// particular, the 0 immediate case involves no lane changes, so it can
// be handled as a blend.
// Example of shuffle mask for 32-bit elements:
// Imm = 1 <i32 0, i32 1, i32 2, i32 3, i32 8, i32 9, i32 10, i32 11>
// Imm = 0 <i32 8, i32 9, i32 10, i32 11, i32 4, i32 5, i32 6, i32 7 >
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 13:45:57 +08:00
// First fill with identify mask.
for (unsigned i = 0; i != DstNumElts; ++i)
Idxs[i] = i;
// Then replace the elements where we need to insert.
for (unsigned i = 0; i != SrcNumElts; ++i)
Idxs[i + Imm * SrcNumElts] = i + DstNumElts;
[X86,IR] Make use of the CreateShuffleVector form that takes an ArrayRef<uint32_t> to avoid the need to manually create a bunch of Constants and a ConstantVector. NFC llvm-svn: 272493
2016-06-12 09:05:59 +08:00
Rep = Builder.CreateShuffleVector(Op0, Rep, Idxs);
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 13:45:57 +08:00
// If the intrinsic has a mask operand, handle that.
if (CI->getNumArgOperands() == 5)
Rep = EmitX86Select(Builder, CI->getArgOperand(4), Rep,
CI->getArgOperand(3));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name.startswith("avx.vextractf128.") ||
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 13:45:46 +08:00
Name == "avx2.vextracti128" ||
Name.startswith("avx512.mask.vextract"))) {
[X86, AVX] replace vextractf128 intrinsics with generic shuffles Now that we've replaced the vinsertf128 intrinsics, do the same for their extract twins. This is very much like D8086 (checked in at r231794): We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is also the LLVM sibling to the cfe D8275 patch. Differential Revision: http://reviews.llvm.org/D8276 llvm-svn: 232045
2015-03-12 23:15:19 +08:00
Value *Op0 = CI->getArgOperand(0);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 13:45:46 +08:00
unsigned DstNumElts = CI->getType()->getVectorNumElements();
unsigned SrcNumElts = Op0->getType()->getVectorNumElements();
unsigned Scale = SrcNumElts / DstNumElts;
[X86][SSE] Replace 128-bit SSE41 PMOVSX intrinsics with native IR This patches removes the x86.sse41.pmovsx* intrinsics, provides a suitable upgrade path and updates relevant tests to sign extend a subvector instead. LLVM counterpart to D12835 Differential Revision: http://reviews.llvm.org/D13002 llvm-svn: 248368
2015-09-23 16:48:33 +08:00
[X86, AVX] replace vextractf128 intrinsics with generic shuffles Now that we've replaced the vinsertf128 intrinsics, do the same for their extract twins. This is very much like D8086 (checked in at r231794): We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is also the LLVM sibling to the cfe D8275 patch. Differential Revision: http://reviews.llvm.org/D8276 llvm-svn: 232045
2015-03-12 23:15:19 +08:00
// Mask off the high bits of the immediate value; hardware ignores those.
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 13:45:46 +08:00
Imm = Imm % Scale;
[X86, AVX] replace vextractf128 intrinsics with generic shuffles Now that we've replaced the vinsertf128 intrinsics, do the same for their extract twins. This is very much like D8086 (checked in at r231794): We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is also the LLVM sibling to the cfe D8275 patch. Differential Revision: http://reviews.llvm.org/D8276 llvm-svn: 232045
2015-03-12 23:15:19 +08:00
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 13:45:46 +08:00
// Get indexes for the subvector of the input vector.
SmallVector<uint32_t, 8> Idxs(DstNumElts);
for (unsigned i = 0; i != DstNumElts; ++i) {
Idxs[i] = i + (Imm * DstNumElts);
[X86, AVX] replace vextractf128 intrinsics with generic shuffles Now that we've replaced the vinsertf128 intrinsics, do the same for their extract twins. This is very much like D8086 (checked in at r231794): We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is also the LLVM sibling to the cfe D8275 patch. Differential Revision: http://reviews.llvm.org/D8276 llvm-svn: 232045
2015-03-12 23:15:19 +08:00
}
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 13:45:46 +08:00
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
[X86, AVX] replace vextractf128 intrinsics with generic shuffles Now that we've replaced the vinsertf128 intrinsics, do the same for their extract twins. This is very much like D8086 (checked in at r231794): We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is also the LLVM sibling to the cfe D8275 patch. Differential Revision: http://reviews.llvm.org/D8276 llvm-svn: 232045
2015-03-12 23:15:19 +08:00
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 13:45:46 +08:00
// If the intrinsic has a mask operand, handle that.
if (CI->getNumArgOperands() == 4)
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (!IsX86 && Name == "stackprotectorcheck") {
[SSP] Remove llvm.stackprotectorcheck. This is a cleanup patch for SSP support in LLVM. There is no functional change. llvm.stackprotectorcheck is not needed, because SelectionDAG isn't actually lowering it in SelectBasicBlock; rather, it adds check code in FinishBasicBlock, ignoring the position where the intrinsic is inserted (See FindSplitPointForStackProtector()). llvm-svn: 265851
2016-04-09 05:26:31 +08:00
Rep = nullptr;
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name.startswith("avx512.mask.perm.df.") ||
Name.startswith("avx512.mask.perm.di."))) {
[X86][AVX512] Autoupgrade the VPERMPD/VPERMQ intrinsics llvm-svn: 274506
2016-07-04 22:19:05 +08:00
Value *Op0 = CI->getArgOperand(0);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
VectorType *VecTy = cast<VectorType>(CI->getType());
unsigned NumElts = VecTy->getNumElements();
SmallVector<uint32_t, 8> Idxs(NumElts);
for (unsigned i = 0; i != NumElts; ++i)
Idxs[i] = (i & ~0x3) + ((Imm >> (2 * (i & 0x3))) & 3);
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
if (CI->getNumArgOperands() == 4)
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name.startswith("avx.vpermil.") ||
Name == "sse2.pshuf.d" ||
Name.startswith("avx512.mask.vpermil.p") ||
Name.startswith("avx512.mask.pshuf.d."))) {
[X86] Greatly simplify the llvm.x86.avx.vpermil.* auto-upgrade code. We can fully derive everything using types of the intrinsic arguments rather than writing separate loops for each intrinsic. NFC llvm-svn: 272496
2016-06-12 11:10:47 +08:00
Value *Op0 = CI->getArgOperand(0);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
VectorType *VecTy = cast<VectorType>(CI->getType());
unsigned NumElts = VecTy->getNumElements();
[X86][AVX512] Autoupgrade the VPERMILPD/VPERMILPS intrinsics llvm-svn: 274498
2016-07-04 20:40:54 +08:00
// Calculate the size of each index in the immediate.
[X86] Greatly simplify the llvm.x86.avx.vpermil.* auto-upgrade code. We can fully derive everything using types of the intrinsic arguments rather than writing separate loops for each intrinsic. NFC llvm-svn: 272496
2016-06-12 11:10:47 +08:00
unsigned IdxSize = 64 / VecTy->getScalarSizeInBits();
unsigned IdxMask = ((1 << IdxSize) - 1);
SmallVector<uint32_t, 8> Idxs(NumElts);
// Lookup the bits for this element, wrapping around the immediate every
// 8-bits. Elements are grouped into sets of 2 or 4 elements so we need
// to offset by the first index of each group.
for (unsigned i = 0; i != NumElts; ++i)
Idxs[i] = ((Imm >> ((i * IdxSize) % 8)) & IdxMask) | (i & ~IdxMask);
[X86] Remove sse2 pshufd/pshuflw/pshufhw intrinsics and upgrade them to shufflevector. llvm-svn: 272510
2016-06-12 22:11:32 +08:00
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
[AVX512] Remove maksed pshufd, pshuflw, and phufhw intrinsics and autoupgrade them to selects and shufflevector. llvm-svn: 272527
2016-06-13 10:36:48 +08:00
if (CI->getNumArgOperands() == 4)
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse2.pshufl.w" ||
Name.startswith("avx512.mask.pshufl.w."))) {
[X86] Remove sse2 pshufd/pshuflw/pshufhw intrinsics and upgrade them to shufflevector. llvm-svn: 272510
2016-06-12 22:11:32 +08:00
Value *Op0 = CI->getArgOperand(0);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
unsigned NumElts = CI->getType()->getVectorNumElements();
SmallVector<uint32_t, 16> Idxs(NumElts);
for (unsigned l = 0; l != NumElts; l += 8) {
for (unsigned i = 0; i != 4; ++i)
Idxs[i + l] = ((Imm >> (2 * i)) & 0x3) + l;
for (unsigned i = 4; i != 8; ++i)
Idxs[i + l] = i + l;
}
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
[AVX512] Remove maksed pshufd, pshuflw, and phufhw intrinsics and autoupgrade them to selects and shufflevector. llvm-svn: 272527
2016-06-13 10:36:48 +08:00
if (CI->getNumArgOperands() == 4)
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name == "sse2.pshufh.w" ||
Name.startswith("avx512.mask.pshufh.w."))) {
[X86] Remove sse2 pshufd/pshuflw/pshufhw intrinsics and upgrade them to shufflevector. llvm-svn: 272510
2016-06-12 22:11:32 +08:00
Value *Op0 = CI->getArgOperand(0);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
unsigned NumElts = CI->getType()->getVectorNumElements();
SmallVector<uint32_t, 16> Idxs(NumElts);
for (unsigned l = 0; l != NumElts; l += 8) {
for (unsigned i = 0; i != 4; ++i)
Idxs[i + l] = i + l;
for (unsigned i = 0; i != 4; ++i)
Idxs[i + l + 4] = ((Imm >> (2 * i)) & 0x3) + 4 + l;
}
[X86] Greatly simplify the llvm.x86.avx.vpermil.* auto-upgrade code. We can fully derive everything using types of the intrinsic arguments rather than writing separate loops for each intrinsic. NFC llvm-svn: 272496
2016-06-12 11:10:47 +08:00
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
[AVX512] Remove maksed pshufd, pshuflw, and phufhw intrinsics and autoupgrade them to selects and shufflevector. llvm-svn: 272527
2016-06-13 10:36:48 +08:00
if (CI->getNumArgOperands() == 4)
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86] Remove masked shufpd/shufps intrinsics and autoupgrade to native vector shuffles. They were removed from clang previously but accidentally left in the backend. llvm-svn: 281300
2016-09-13 15:40:53 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.shuf.p")) {
Value *Op0 = CI->getArgOperand(0);
Value *Op1 = CI->getArgOperand(1);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
unsigned NumElts = CI->getType()->getVectorNumElements();
unsigned NumLaneElts = 128/CI->getType()->getScalarSizeInBits();
unsigned HalfLaneElts = NumLaneElts / 2;
SmallVector<uint32_t, 16> Idxs(NumElts);
for (unsigned i = 0; i != NumElts; ++i) {
// Base index is the starting element of the lane.
Idxs[i] = i - (i % NumLaneElts);
// If we are half way through the lane switch to the other source.
if ((i % NumLaneElts) >= HalfLaneElts)
Idxs[i] += NumElts;
// Now select the specific element. By adding HalfLaneElts bits from
// the immediate. Wrapping around the immediate every 8-bits.
Idxs[i] += (Imm >> ((i * HalfLaneElts) % 8)) & ((1 << HalfLaneElts) - 1);
}
Rep = Builder.CreateShuffleVector(Op0, Op1, Idxs);
Rep = EmitX86Select(Builder, CI->getArgOperand(4), Rep,
CI->getArgOperand(3));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name.startswith("avx512.mask.movddup") ||
Name.startswith("avx512.mask.movshdup") ||
Name.startswith("avx512.mask.movsldup"))) {
[X86][AVX512] Autoupgrade the MOVDDUP/MOVSLDUP/MOVSHDUP intrinsics llvm-svn: 274439
2016-07-02 22:42:35 +08:00
Value *Op0 = CI->getArgOperand(0);
unsigned NumElts = CI->getType()->getVectorNumElements();
unsigned NumLaneElts = 128/CI->getType()->getScalarSizeInBits();
unsigned Offset = 0;
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
if (Name.startswith("avx512.mask.movshdup."))
[X86][AVX512] Autoupgrade the MOVDDUP/MOVSLDUP/MOVSHDUP intrinsics llvm-svn: 274439
2016-07-02 22:42:35 +08:00
Offset = 1;
SmallVector<uint32_t, 16> Idxs(NumElts);
for (unsigned l = 0; l != NumElts; l += NumLaneElts)
for (unsigned i = 0; i != NumLaneElts; i += 2) {
Idxs[i + l + 0] = i + l + Offset;
Idxs[i + l + 1] = i + l + Offset;
}
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name.startswith("avx512.mask.punpckl") ||
Name.startswith("avx512.mask.unpckl."))) {
[AVX512] Remove masked unpack intrinsics and autoupgrade to vectorshuffle and selects. llvm-svn: 273543
2016-06-23 15:37:33 +08:00
Value *Op0 = CI->getArgOperand(0);
Value *Op1 = CI->getArgOperand(1);
int NumElts = CI->getType()->getVectorNumElements();
int NumLaneElts = 128/CI->getType()->getScalarSizeInBits();
SmallVector<uint32_t, 64> Idxs(NumElts);
for (int l = 0; l != NumElts; l += NumLaneElts)
for (int i = 0; i != NumLaneElts; ++i)
Idxs[i + l] = l + (i / 2) + NumElts * (i % 2);
Rep = Builder.CreateShuffleVector(Op0, Op1, Idxs);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-05 04:56:38 +08:00
} else if (IsX86 && (Name.startswith("avx512.mask.punpckh") ||
Name.startswith("avx512.mask.unpckh."))) {
[AVX512] Remove masked unpack intrinsics and autoupgrade to vectorshuffle and selects. llvm-svn: 273543
2016-06-23 15:37:33 +08:00
Value *Op0 = CI->getArgOperand(0);
Value *Op1 = CI->getArgOperand(1);
int NumElts = CI->getType()->getVectorNumElements();
int NumLaneElts = 128/CI->getType()->getScalarSizeInBits();
SmallVector<uint32_t, 64> Idxs(NumElts);
for (int l = 0; l != NumElts; l += NumLaneElts)
for (int i = 0; i != NumLaneElts; ++i)
Idxs[i + l] = (NumLaneElts / 2) + l + (i / 2) + NumElts * (i % 2);
Rep = Builder.CreateShuffleVector(Op0, Op1, Idxs);
[AVX512] Remove masked logic op intrinsics and autoupgrade them to native IR. llvm-svn: 275155
2016-07-12 13:27:53 +08:00
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.pand.")) {
Rep = Builder.CreateAnd(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.pandn.")) {
Rep = Builder.CreateAnd(Builder.CreateNot(CI->getArgOperand(0)),
CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.por.")) {
Rep = Builder.CreateOr(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.pxor.")) {
Rep = Builder.CreateXor(CI->getArgOperand(0), CI->getArgOperand(1));
[AVX512] Remove masked unpack intrinsics and autoupgrade to vectorshuffle and selects. llvm-svn: 273543
2016-06-23 15:37:33 +08:00
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[AVX-512] Remove floating point logical operation instrinsics and replace them with native IR. llvm-svn: 280466
2016-09-02 13:29:17 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.and.")) {
VectorType *FTy = cast<VectorType>(CI->getType());
VectorType *ITy = VectorType::getInteger(FTy);
Rep = Builder.CreateAnd(Builder.CreateBitCast(CI->getArgOperand(0), ITy),
Builder.CreateBitCast(CI->getArgOperand(1), ITy));
Rep = Builder.CreateBitCast(Rep, FTy);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.andn.")) {
VectorType *FTy = cast<VectorType>(CI->getType());
VectorType *ITy = VectorType::getInteger(FTy);
Rep = Builder.CreateNot(Builder.CreateBitCast(CI->getArgOperand(0), ITy));
Rep = Builder.CreateAnd(Rep,
Builder.CreateBitCast(CI->getArgOperand(1), ITy));
Rep = Builder.CreateBitCast(Rep, FTy);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.or.")) {
VectorType *FTy = cast<VectorType>(CI->getType());
VectorType *ITy = VectorType::getInteger(FTy);
Rep = Builder.CreateOr(Builder.CreateBitCast(CI->getArgOperand(0), ITy),
Builder.CreateBitCast(CI->getArgOperand(1), ITy));
Rep = Builder.CreateBitCast(Rep, FTy);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.xor.")) {
VectorType *FTy = cast<VectorType>(CI->getType());
VectorType *ITy = VectorType::getInteger(FTy);
Rep = Builder.CreateXor(Builder.CreateBitCast(CI->getArgOperand(0), ITy),
Builder.CreateBitCast(CI->getArgOperand(1), ITy));
Rep = Builder.CreateBitCast(Rep, FTy);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[AVX-512] Remove masked integer add/sub/mull intrinsics and upgrade to native IR. llvm-svn: 280611
2016-09-04 10:09:53 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.padd.")) {
Rep = Builder.CreateAdd(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.psub.")) {
Rep = Builder.CreateSub(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.pmull.")) {
Rep = Builder.CreateMul(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[AVX-512] Use an equality compare instead of StringRef::startswith in a few places in auto upgrade that were looking for the complete intrinsic name anyway. llvm-svn: 286033
2016-11-05 13:35:23 +08:00
} else if (IsX86 && (Name.startswith("avx512.mask.add.p"))) {
[AVX-512] Remove 128-bit and 256-bit masked floating point add/sub/mul/div intrinsics and upgrade to native IR. llvm-svn: 280633
2016-09-05 02:13:33 +08:00
Rep = Builder.CreateFAdd(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86] Use StringRef::startswith to reduce a few compares in the intrinsic autoupgrade code. llvm-svn: 286090
2016-11-07 08:13:42 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.div.p")) {
[AVX-512] Remove 128-bit and 256-bit masked floating point add/sub/mul/div intrinsics and upgrade to native IR. llvm-svn: 280633
2016-09-05 02:13:33 +08:00
Rep = Builder.CreateFDiv(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86] Use StringRef::startswith to reduce a few compares in the intrinsic autoupgrade code. llvm-svn: 286090
2016-11-07 08:13:42 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.mul.p")) {
[AVX-512] Remove 128-bit and 256-bit masked floating point add/sub/mul/div intrinsics and upgrade to native IR. llvm-svn: 280633
2016-09-05 02:13:33 +08:00
Rep = Builder.CreateFMul(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86] Use StringRef::startswith to reduce a few compares in the intrinsic autoupgrade code. llvm-svn: 286090
2016-11-07 08:13:42 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.sub.p")) {
[AVX-512] Remove 128-bit and 256-bit masked floating point add/sub/mul/div intrinsics and upgrade to native IR. llvm-svn: 280633
2016-09-05 02:13:33 +08:00
Rep = Builder.CreateFSub(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[AVX-512] Remove lzcnt intrinsics and autoupgrade them to generic ctlz intrinsics with select. Clang has been emitting cltz intrinsics for a while now. llvm-svn: 296091
2017-02-24 13:35:04 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.lzcnt.")) {
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(),
Intrinsic::ctlz,
CI->getType()),
{ CI->getArgOperand(0), Builder.getInt1(false) });
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[AVX-512] Remove 128/256-bit masked fp max/min intrinsics. Upgrade them to legacy unmasked intrinsics and select instructions. llvm-svn: 295543
2017-02-18 15:07:50 +08:00
} else if (IsX86 && (Name.startswith("avx512.mask.max.p") ||
Name.startswith("avx512.mask.min.p"))) {
bool IsMin = Name[13] == 'i';
VectorType *VecTy = cast<VectorType>(CI->getType());
unsigned VecWidth = VecTy->getPrimitiveSizeInBits();
unsigned EltWidth = VecTy->getScalarSizeInBits();
Intrinsic::ID IID;
if (!IsMin && VecWidth == 128 && EltWidth == 32)
IID = Intrinsic::x86_sse_max_ps;
else if (!IsMin && VecWidth == 128 && EltWidth == 64)
IID = Intrinsic::x86_sse2_max_pd;
else if (!IsMin && VecWidth == 256 && EltWidth == 32)
IID = Intrinsic::x86_avx_max_ps_256;
else if (!IsMin && VecWidth == 256 && EltWidth == 64)
IID = Intrinsic::x86_avx_max_pd_256;
else if (IsMin && VecWidth == 128 && EltWidth == 32)
IID = Intrinsic::x86_sse_min_ps;
else if (IsMin && VecWidth == 128 && EltWidth == 64)
IID = Intrinsic::x86_sse2_min_pd;
else if (IsMin && VecWidth == 256 && EltWidth == 32)
IID = Intrinsic::x86_avx_min_ps_256;
else if (IsMin && VecWidth == 256 && EltWidth == 64)
IID = Intrinsic::x86_avx_min_pd_256;
else
llvm_unreachable("Unexpected intrinsic");
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{ CI->getArgOperand(0), CI->getArgOperand(1) });
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[AVX-512] Remove 128/256 masked pshufb intrinsics. Autoupgrade them to legacy intrinsics and a select. llvm-svn: 286089
2016-11-07 08:13:39 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.pshuf.b.")) {
VectorType *VecTy = cast<VectorType>(CI->getType());
Intrinsic::ID IID;
if (VecTy->getPrimitiveSizeInBits() == 128)
IID = Intrinsic::x86_ssse3_pshuf_b_128;
else if (VecTy->getPrimitiveSizeInBits() == 256)
IID = Intrinsic::x86_avx2_pshuf_b;
[X86] Remove masking from 512-bit PSHUFB intrinsics in preparation for being able to constant fold it in InstCombineCalls like we do for 128/256-bit. llvm-svn: 289344
2016-12-11 07:09:43 +08:00
else if (VecTy->getPrimitiveSizeInBits() == 512)
IID = Intrinsic::x86_avx512_pshuf_b_512;
[AVX-512] Remove 128/256 masked pshufb intrinsics. Autoupgrade them to legacy intrinsics and a select. llvm-svn: 286089
2016-11-07 08:13:39 +08:00
else
llvm_unreachable("Unexpected intrinsic");
[AVX-512] Remove masked pmuldq and pmuludq intrinsics and autoupgrade them to unmasked intrinsics plus a select. llvm-svn: 290583
2016-12-27 13:30:14 +08:00
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{ CI->getArgOperand(0), CI->getArgOperand(1) });
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && (Name.startswith("avx512.mask.pmul.dq.") ||
Name.startswith("avx512.mask.pmulu.dq."))) {
bool IsUnsigned = Name[16] == 'u';
VectorType *VecTy = cast<VectorType>(CI->getType());
Intrinsic::ID IID;
if (!IsUnsigned && VecTy->getPrimitiveSizeInBits() == 128)
IID = Intrinsic::x86_sse41_pmuldq;
else if (!IsUnsigned && VecTy->getPrimitiveSizeInBits() == 256)
IID = Intrinsic::x86_avx2_pmul_dq;
else if (!IsUnsigned && VecTy->getPrimitiveSizeInBits() == 512)
IID = Intrinsic::x86_avx512_pmul_dq_512;
else if (IsUnsigned && VecTy->getPrimitiveSizeInBits() == 128)
IID = Intrinsic::x86_sse2_pmulu_dq;
else if (IsUnsigned && VecTy->getPrimitiveSizeInBits() == 256)
IID = Intrinsic::x86_avx2_pmulu_dq;
else if (IsUnsigned && VecTy->getPrimitiveSizeInBits() == 512)
IID = Intrinsic::x86_avx512_pmulu_dq_512;
else
llvm_unreachable("Unexpected intrinsic");
[AVX-512] Remove masked packss/packus intrinsics and autoupgrade to unmasked intrinsics with select instructions. For 512-bit add new unmasked intrinsics. The new 512-bit unmasked intrinsics will make it easy to handle these with the SSE/AVX intrinsics in InstCombine where we currently have a TODO. llvm-svn: 295290
2017-02-16 14:31:54 +08:00
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{ CI->getArgOperand(0), CI->getArgOperand(1) });
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.pack")) {
bool IsUnsigned = Name[16] == 'u';
bool IsDW = Name[18] == 'd';
VectorType *VecTy = cast<VectorType>(CI->getType());
Intrinsic::ID IID;
if (!IsUnsigned && !IsDW && VecTy->getPrimitiveSizeInBits() == 128)
IID = Intrinsic::x86_sse2_packsswb_128;
else if (!IsUnsigned && !IsDW && VecTy->getPrimitiveSizeInBits() == 256)
IID = Intrinsic::x86_avx2_packsswb;
else if (!IsUnsigned && !IsDW && VecTy->getPrimitiveSizeInBits() == 512)
IID = Intrinsic::x86_avx512_packsswb_512;
else if (!IsUnsigned && IsDW && VecTy->getPrimitiveSizeInBits() == 128)
IID = Intrinsic::x86_sse2_packssdw_128;
else if (!IsUnsigned && IsDW && VecTy->getPrimitiveSizeInBits() == 256)
IID = Intrinsic::x86_avx2_packssdw;
else if (!IsUnsigned && IsDW && VecTy->getPrimitiveSizeInBits() == 512)
IID = Intrinsic::x86_avx512_packssdw_512;
else if (IsUnsigned && !IsDW && VecTy->getPrimitiveSizeInBits() == 128)
IID = Intrinsic::x86_sse2_packuswb_128;
else if (IsUnsigned && !IsDW && VecTy->getPrimitiveSizeInBits() == 256)
IID = Intrinsic::x86_avx2_packuswb;
else if (IsUnsigned && !IsDW && VecTy->getPrimitiveSizeInBits() == 512)
IID = Intrinsic::x86_avx512_packuswb_512;
else if (IsUnsigned && IsDW && VecTy->getPrimitiveSizeInBits() == 128)
IID = Intrinsic::x86_sse41_packusdw;
else if (IsUnsigned && IsDW && VecTy->getPrimitiveSizeInBits() == 256)
IID = Intrinsic::x86_avx2_packusdw;
else if (IsUnsigned && IsDW && VecTy->getPrimitiveSizeInBits() == 512)
IID = Intrinsic::x86_avx512_packusdw_512;
else
llvm_unreachable("Unexpected intrinsic");
[AVX-512] Remove 128/256 masked pshufb intrinsics. Autoupgrade them to legacy intrinsics and a select. llvm-svn: 286089
2016-11-07 08:13:39 +08:00
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{ CI->getArgOperand(0), CI->getArgOperand(1) });
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.psll")) {
bool IsImmediate = Name[16] == 'i' ||
(Name.size() > 18 && Name[18] == 'i');
bool IsVariable = Name[16] == 'v';
char Size = Name[16] == '.' ? Name[17] :
Name[17] == '.' ? Name[18] :
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 13:04:44 +08:00
Name[18] == '.' ? Name[19] :
Name[20];
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
Intrinsic::ID IID;
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 09:53:22 +08:00
if (IsVariable && Name[17] != '.') {
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
if (Size == 'd' && Name[17] == '2') // avx512.mask.psllv2.di
IID = Intrinsic::x86_avx2_psllv_q;
else if (Size == 'd' && Name[17] == '4') // avx512.mask.psllv4.di
IID = Intrinsic::x86_avx2_psllv_q_256;
else if (Size == 's' && Name[17] == '4') // avx512.mask.psllv4.si
IID = Intrinsic::x86_avx2_psllv_d;
else if (Size == 's' && Name[17] == '8') // avx512.mask.psllv8.si
IID = Intrinsic::x86_avx2_psllv_d_256;
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 13:04:44 +08:00
else if (Size == 'h' && Name[17] == '8') // avx512.mask.psllv8.hi
IID = Intrinsic::x86_avx512_psllv_w_128;
else if (Size == 'h' && Name[17] == '1') // avx512.mask.psllv16.hi
IID = Intrinsic::x86_avx512_psllv_w_256;
else if (Name[17] == '3' && Name[18] == '2') // avx512.mask.psllv32hi
IID = Intrinsic::x86_avx512_psllv_w_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".128")) {
if (Size == 'd') // avx512.mask.psll.d.128, avx512.mask.psll.di.128
IID = IsImmediate ? Intrinsic::x86_sse2_pslli_d
: Intrinsic::x86_sse2_psll_d;
else if (Size == 'q') // avx512.mask.psll.q.128, avx512.mask.psll.qi.128
IID = IsImmediate ? Intrinsic::x86_sse2_pslli_q
: Intrinsic::x86_sse2_psll_q;
else if (Size == 'w') // avx512.mask.psll.w.128, avx512.mask.psll.wi.128
IID = IsImmediate ? Intrinsic::x86_sse2_pslli_w
: Intrinsic::x86_sse2_psll_w;
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".256")) {
if (Size == 'd') // avx512.mask.psll.d.256, avx512.mask.psll.di.256
IID = IsImmediate ? Intrinsic::x86_avx2_pslli_d
: Intrinsic::x86_avx2_psll_d;
else if (Size == 'q') // avx512.mask.psll.q.256, avx512.mask.psll.qi.256
IID = IsImmediate ? Intrinsic::x86_avx2_pslli_q
: Intrinsic::x86_avx2_psll_q;
else if (Size == 'w') // avx512.mask.psll.w.256, avx512.mask.psll.wi.256
IID = IsImmediate ? Intrinsic::x86_avx2_pslli_w
: Intrinsic::x86_avx2_psll_w;
else
llvm_unreachable("Unexpected size");
} else {
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 09:53:22 +08:00
if (Size == 'd') // psll.di.512, pslli.d, psll.d, psllv.d.512
IID = IsImmediate ? Intrinsic::x86_avx512_pslli_d_512 :
IsVariable ? Intrinsic::x86_avx512_psllv_d_512 :
Intrinsic::x86_avx512_psll_d_512;
else if (Size == 'q') // psll.qi.512, pslli.q, psll.q, psllv.q.512
IID = IsImmediate ? Intrinsic::x86_avx512_pslli_q_512 :
IsVariable ? Intrinsic::x86_avx512_psllv_q_512 :
Intrinsic::x86_avx512_psll_q_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
else if (Size == 'w') // psll.wi.512, pslli.w, psll.w
IID = IsImmediate ? Intrinsic::x86_avx512_pslli_w_512
: Intrinsic::x86_avx512_psll_w_512;
else
llvm_unreachable("Unexpected size");
}
Rep = UpgradeX86MaskedShift(Builder, *CI, IID);
} else if (IsX86 && Name.startswith("avx512.mask.psrl")) {
bool IsImmediate = Name[16] == 'i' ||
(Name.size() > 18 && Name[18] == 'i');
bool IsVariable = Name[16] == 'v';
char Size = Name[16] == '.' ? Name[17] :
Name[17] == '.' ? Name[18] :
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 13:04:44 +08:00
Name[18] == '.' ? Name[19] :
Name[20];
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
Intrinsic::ID IID;
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 09:53:22 +08:00
if (IsVariable && Name[17] != '.') {
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
if (Size == 'd' && Name[17] == '2') // avx512.mask.psrlv2.di
IID = Intrinsic::x86_avx2_psrlv_q;
else if (Size == 'd' && Name[17] == '4') // avx512.mask.psrlv4.di
IID = Intrinsic::x86_avx2_psrlv_q_256;
else if (Size == 's' && Name[17] == '4') // avx512.mask.psrlv4.si
IID = Intrinsic::x86_avx2_psrlv_d;
else if (Size == 's' && Name[17] == '8') // avx512.mask.psrlv8.si
IID = Intrinsic::x86_avx2_psrlv_d_256;
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 13:04:44 +08:00
else if (Size == 'h' && Name[17] == '8') // avx512.mask.psrlv8.hi
IID = Intrinsic::x86_avx512_psrlv_w_128;
else if (Size == 'h' && Name[17] == '1') // avx512.mask.psrlv16.hi
IID = Intrinsic::x86_avx512_psrlv_w_256;
else if (Name[17] == '3' && Name[18] == '2') // avx512.mask.psrlv32hi
IID = Intrinsic::x86_avx512_psrlv_w_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".128")) {
if (Size == 'd') // avx512.mask.psrl.d.128, avx512.mask.psrl.di.128
IID = IsImmediate ? Intrinsic::x86_sse2_psrli_d
: Intrinsic::x86_sse2_psrl_d;
else if (Size == 'q') // avx512.mask.psrl.q.128, avx512.mask.psrl.qi.128
IID = IsImmediate ? Intrinsic::x86_sse2_psrli_q
: Intrinsic::x86_sse2_psrl_q;
else if (Size == 'w') // avx512.mask.psrl.w.128, avx512.mask.psrl.wi.128
IID = IsImmediate ? Intrinsic::x86_sse2_psrli_w
: Intrinsic::x86_sse2_psrl_w;
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".256")) {
if (Size == 'd') // avx512.mask.psrl.d.256, avx512.mask.psrl.di.256
IID = IsImmediate ? Intrinsic::x86_avx2_psrli_d
: Intrinsic::x86_avx2_psrl_d;
else if (Size == 'q') // avx512.mask.psrl.q.256, avx512.mask.psrl.qi.256
IID = IsImmediate ? Intrinsic::x86_avx2_psrli_q
: Intrinsic::x86_avx2_psrl_q;
else if (Size == 'w') // avx512.mask.psrl.w.256, avx512.mask.psrl.wi.256
IID = IsImmediate ? Intrinsic::x86_avx2_psrli_w
: Intrinsic::x86_avx2_psrl_w;
else
llvm_unreachable("Unexpected size");
} else {
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 09:53:22 +08:00
if (Size == 'd') // psrl.di.512, psrli.d, psrl.d, psrl.d.512
IID = IsImmediate ? Intrinsic::x86_avx512_psrli_d_512 :
IsVariable ? Intrinsic::x86_avx512_psrlv_d_512 :
Intrinsic::x86_avx512_psrl_d_512;
else if (Size == 'q') // psrl.qi.512, psrli.q, psrl.q, psrl.q.512
IID = IsImmediate ? Intrinsic::x86_avx512_psrli_q_512 :
IsVariable ? Intrinsic::x86_avx512_psrlv_q_512 :
Intrinsic::x86_avx512_psrl_q_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
else if (Size == 'w') // psrl.wi.512, psrli.w, psrl.w)
IID = IsImmediate ? Intrinsic::x86_avx512_psrli_w_512
: Intrinsic::x86_avx512_psrl_w_512;
else
llvm_unreachable("Unexpected size");
}
Rep = UpgradeX86MaskedShift(Builder, *CI, IID);
} else if (IsX86 && Name.startswith("avx512.mask.psra")) {
bool IsImmediate = Name[16] == 'i' ||
(Name.size() > 18 && Name[18] == 'i');
bool IsVariable = Name[16] == 'v';
char Size = Name[16] == '.' ? Name[17] :
Name[17] == '.' ? Name[18] :
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 13:04:44 +08:00
Name[18] == '.' ? Name[19] :
Name[20];
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
Intrinsic::ID IID;
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 09:53:22 +08:00
if (IsVariable && Name[17] != '.') {
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
if (Size == 's' && Name[17] == '4') // avx512.mask.psrav4.si
IID = Intrinsic::x86_avx2_psrav_d;
else if (Size == 's' && Name[17] == '8') // avx512.mask.psrav8.si
IID = Intrinsic::x86_avx2_psrav_d_256;
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 13:04:44 +08:00
else if (Size == 'h' && Name[17] == '8') // avx512.mask.psrav8.hi
IID = Intrinsic::x86_avx512_psrav_w_128;
else if (Size == 'h' && Name[17] == '1') // avx512.mask.psrav16.hi
IID = Intrinsic::x86_avx512_psrav_w_256;
else if (Name[17] == '3' && Name[18] == '2') // avx512.mask.psrav32hi
IID = Intrinsic::x86_avx512_psrav_w_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".128")) {
if (Size == 'd') // avx512.mask.psra.d.128, avx512.mask.psra.di.128
IID = IsImmediate ? Intrinsic::x86_sse2_psrai_d
: Intrinsic::x86_sse2_psra_d;
else if (Size == 'q') // avx512.mask.psra.q.128, avx512.mask.psra.qi.128
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 09:53:22 +08:00
IID = IsImmediate ? Intrinsic::x86_avx512_psrai_q_128 :
IsVariable ? Intrinsic::x86_avx512_psrav_q_128 :
Intrinsic::x86_avx512_psra_q_128;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
else if (Size == 'w') // avx512.mask.psra.w.128, avx512.mask.psra.wi.128
IID = IsImmediate ? Intrinsic::x86_sse2_psrai_w
: Intrinsic::x86_sse2_psra_w;
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".256")) {
if (Size == 'd') // avx512.mask.psra.d.256, avx512.mask.psra.di.256
IID = IsImmediate ? Intrinsic::x86_avx2_psrai_d
: Intrinsic::x86_avx2_psra_d;
else if (Size == 'q') // avx512.mask.psra.q.256, avx512.mask.psra.qi.256
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 09:53:22 +08:00
IID = IsImmediate ? Intrinsic::x86_avx512_psrai_q_256 :
IsVariable ? Intrinsic::x86_avx512_psrav_q_256 :
Intrinsic::x86_avx512_psra_q_256;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
else if (Size == 'w') // avx512.mask.psra.w.256, avx512.mask.psra.wi.256
IID = IsImmediate ? Intrinsic::x86_avx2_psrai_w
: Intrinsic::x86_avx2_psra_w;
else
llvm_unreachable("Unexpected size");
} else {
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 09:53:22 +08:00
if (Size == 'd') // psra.di.512, psrai.d, psra.d, psrav.d.512
IID = IsImmediate ? Intrinsic::x86_avx512_psrai_d_512 :
IsVariable ? Intrinsic::x86_avx512_psrav_d_512 :
Intrinsic::x86_avx512_psra_d_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
else if (Size == 'q') // psra.qi.512, psrai.q, psra.q
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 09:53:22 +08:00
IID = IsImmediate ? Intrinsic::x86_avx512_psrai_q_512 :
IsVariable ? Intrinsic::x86_avx512_psrav_q_512 :
Intrinsic::x86_avx512_psra_q_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-14 03:09:56 +08:00
else if (Size == 'w') // psra.wi.512, psrai.w, psra.w
IID = IsImmediate ? Intrinsic::x86_avx512_psrai_w_512
: Intrinsic::x86_avx512_psra_w_512;
else
llvm_unreachable("Unexpected size");
}
Rep = UpgradeX86MaskedShift(Builder, *CI, IID);
[X86][AVX512] Removing llvm x86 intrinsics for _mm_mask_move_{ss|sd} intrinsics. Differential Revision: https://reviews.llvm.org/D26128 llvm-svn: 287087
2016-11-16 17:00:28 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.move.s")) {
Rep = upgradeMaskedMove(Builder, *CI);
[X86][LLVM] Converting __mm{|256|512}_movm_epi{8|16|32|64} LLVMIR call into generic intrinsics. This patch is a part one of two reviews, one for the clang and the other for LLVM. The patch deletes the back-end intrinsics and adds support for them in the auto upgrade. Differential Revision: https://reviews.llvm.org/D31393 llvm-svn: 299432
2017-04-04 21:32:14 +08:00
} else if (IsX86 && Name.startswith("avx512.cvtmask2")) {
Rep = UpgradeMaskToInt(Builder, *CI);
[AVX-512] Remove 128/256 masked vpermil instrinsics and autoupgrade to a select around the unmasked avx1 intrinsics. llvm-svn: 289340
2016-12-11 05:15:52 +08:00
} else if (IsX86 && Name.startswith("avx512.mask.vpermilvar.")) {
Intrinsic::ID IID;
if (Name.endswith("ps.128"))
IID = Intrinsic::x86_avx_vpermilvar_ps;
else if (Name.endswith("pd.128"))
IID = Intrinsic::x86_avx_vpermilvar_pd;
else if (Name.endswith("ps.256"))
IID = Intrinsic::x86_avx_vpermilvar_ps_256;
else if (Name.endswith("pd.256"))
IID = Intrinsic::x86_avx_vpermilvar_pd_256;
[X86] Remove masking from 512-bit VPERMIL intrinsics in preparation for being able to constant fold them in InstCombineCalls like we do for 128/256-bit. llvm-svn: 289350
2016-12-11 09:26:44 +08:00
else if (Name.endswith("ps.512"))
IID = Intrinsic::x86_avx512_vpermilvar_ps_512;
else if (Name.endswith("pd.512"))
IID = Intrinsic::x86_avx512_vpermilvar_pd_512;
[AVX-512] Remove 128/256 masked vpermil instrinsics and autoupgrade to a select around the unmasked avx1 intrinsics. llvm-svn: 289340
2016-12-11 05:15:52 +08:00
else
llvm_unreachable("Unexpected vpermilvar intrinsic");
Function *Intrin = Intrinsic::getDeclaration(F->getParent(), IID);
Rep = Builder.CreateCall(Intrin,
{ CI->getArgOperand(0), CI->getArgOperand(1) });
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86][SSE] Update MOVNTDQA non-temporal loads to generic implementation (LLVM) MOVNTDQA non-temporal aligned vector loads can be correctly represented using generic builtin loads, allowing us to remove the existing x86 intrinsics. Clang companion patch: D31766. Differential Revision: https://reviews.llvm.org/D31767 llvm-svn: 300325
2017-04-14 23:05:35 +08:00
} else if (IsX86 && Name.endswith(".movntdqa")) {
Module *M = F->getParent();
MDNode *Node = MDNode::get(
C, ConstantAsMetadata::get(ConstantInt::get(Type::getInt32Ty(C), 1)));
Value *Ptr = CI->getArgOperand(0);
VectorType *VTy = cast<VectorType>(CI->getType());
// Convert the type of the pointer to a pointer to the stored type.
Value *BC =
Builder.CreateBitCast(Ptr, PointerType::getUnqual(VTy), "cast");
LoadInst *LI = Builder.CreateAlignedLoad(BC, VTy->getBitWidth() / 8);
LI->setMetadata(M->getMDKindID("nontemporal"), Node);
Rep = LI;
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 09:00:32 +08:00
} else if (IsNVVM && (Name == "abs.i" || Name == "abs.ll")) {
Value *Arg = CI->getArgOperand(0);
Value *Neg = Builder.CreateNeg(Arg, "neg");
Value *Cmp = Builder.CreateICmpSGE(
Arg, llvm::Constant::getNullValue(Arg->getType()), "abs.cond");
Rep = Builder.CreateSelect(Cmp, Arg, Neg, "abs");
} else if (IsNVVM && (Name == "max.i" || Name == "max.ll" ||
Name == "max.ui" || Name == "max.ull")) {
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
Value *Cmp = Name.endswith(".ui") || Name.endswith(".ull")
? Builder.CreateICmpUGE(Arg0, Arg1, "max.cond")
: Builder.CreateICmpSGE(Arg0, Arg1, "max.cond");
Rep = Builder.CreateSelect(Cmp, Arg0, Arg1, "max");
} else if (IsNVVM && (Name == "min.i" || Name == "min.ll" ||
Name == "min.ui" || Name == "min.ull")) {
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
Value *Cmp = Name.endswith(".ui") || Name.endswith(".ull")
? Builder.CreateICmpULE(Arg0, Arg1, "min.cond")
: Builder.CreateICmpSLE(Arg0, Arg1, "min.cond");
Rep = Builder.CreateSelect(Cmp, Arg0, Arg1, "min");
} else if (IsNVVM && Name == "clz.ll") {
// llvm.nvvm.clz.ll returns an i32, but llvm.ctlz.i64 and returns an i64.
Value *Arg = CI->getArgOperand(0);
Value *Ctlz = Builder.CreateCall(
Intrinsic::getDeclaration(F->getParent(), Intrinsic::ctlz,
{Arg->getType()}),
{Arg, Builder.getFalse()}, "ctlz");
Rep = Builder.CreateTrunc(Ctlz, Builder.getInt32Ty(), "ctlz.trunc");
} else if (IsNVVM && Name == "popc.ll") {
// llvm.nvvm.popc.ll returns an i32, but llvm.ctpop.i64 and returns an
// i64.
Value *Arg = CI->getArgOperand(0);
Value *Popc = Builder.CreateCall(
Intrinsic::getDeclaration(F->getParent(), Intrinsic::ctpop,
{Arg->getType()}),
Arg, "ctpop");
Rep = Builder.CreateTrunc(Popc, Builder.getInt32Ty(), "ctpop.trunc");
} else if (IsNVVM && Name == "h2f") {
Rep = Builder.CreateCall(Intrinsic::getDeclaration(
F->getParent(), Intrinsic::convert_from_fp16,
{Builder.getFloatTy()}),
CI->getArgOperand(0), "h2f");
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 14:10:55 +08:00
} else {
[X86] Greatly simplify the llvm.x86.avx.vpermil.* auto-upgrade code. We can fully derive everything using types of the intrinsic arguments rather than writing separate loops for each intrinsic. NFC llvm-svn: 272496
2016-06-12 11:10:47 +08:00
llvm_unreachable("Unknown function for CallInst upgrade.");
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 14:10:55 +08:00
}
[SSP] Remove llvm.stackprotectorcheck. This is a cleanup patch for SSP support in LLVM. There is no functional change. llvm.stackprotectorcheck is not needed, because SelectionDAG isn't actually lowering it in SelectBasicBlock; rather, it adds check code in FinishBasicBlock, ignoring the position where the intrinsic is inserted (See FindSplitPointForStackProtector()). llvm-svn: 265851
2016-04-09 05:26:31 +08:00
if (Rep)
CI->replaceAllUsesWith(Rep);
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 14:10:55 +08:00
CI->eraseFromParent();
return;
}
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
CallInst *NewCall = nullptr;
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 12:26:04 +08:00
switch (NewFn->getIntrinsicID()) {
Implement intrinsic mangling for literal struct types. Fixes PR 31921 Summary: Predicateinfo requires an ugly workaround to try to avoid literal struct types due to the intrinsic mangling not being implemented. This workaround actually does not work in all cases (you can hit the assert by bootstrapping with -print-predicateinfo), and can't be made to work without DFS'ing the type (IE copying getMangledStr and using a version that detects if it would crash). Rather than do that, i just implemented the mangling. It seems simple, since they are unified structurally. Looking at the overloaded-mangling testcase we have, it actually turns out the gc intrinsics will *also* crash if you try to use a literal struct. Thus, the testcase added fails before this patch, and works after, without needing to resort to predicateinfo. Reviewers: chandlerc, davide Subscribers: llvm-commits, sanjoy Differential Revision: https://reviews.llvm.org/D29925 llvm-svn: 295253
2017-02-16 07:16:20 +08:00
default: {
// Handle generic mangling change, but nothing else
assert(
(CI->getCalledFunction()->getName() != NewFn->getName()) &&
"Unknown function for CallInst upgrade and isn't just a name change");
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
CI->setCalledFunction(NewFn);
Implement intrinsic mangling for literal struct types. Fixes PR 31921 Summary: Predicateinfo requires an ugly workaround to try to avoid literal struct types due to the intrinsic mangling not being implemented. This workaround actually does not work in all cases (you can hit the assert by bootstrapping with -print-predicateinfo), and can't be made to work without DFS'ing the type (IE copying getMangledStr and using a version that detects if it would crash). Rather than do that, i just implemented the mangling. It seems simple, since they are unified structurally. Looking at the overloaded-mangling testcase we have, it actually turns out the gc intrinsics will *also* crash if you try to use a literal struct. Thus, the testcase added fails before this patch, and works after, without needing to resort to predicateinfo. Reviewers: chandlerc, davide Subscribers: llvm-commits, sanjoy Differential Revision: https://reviews.llvm.org/D29925 llvm-svn: 295253
2017-02-16 07:16:20 +08:00
return;
}
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 12:26:04 +08:00
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 18:56:37 +08:00
case Intrinsic::arm_neon_vld1:
case Intrinsic::arm_neon_vld2:
case Intrinsic::arm_neon_vld3:
case Intrinsic::arm_neon_vld4:
case Intrinsic::arm_neon_vld2lane:
case Intrinsic::arm_neon_vld3lane:
case Intrinsic::arm_neon_vld4lane:
case Intrinsic::arm_neon_vst1:
case Intrinsic::arm_neon_vst2:
case Intrinsic::arm_neon_vst3:
case Intrinsic::arm_neon_vst4:
case Intrinsic::arm_neon_vst2lane:
case Intrinsic::arm_neon_vst3lane:
case Intrinsic::arm_neon_vst4lane: {
SmallVector<Value *, 4> Args(CI->arg_operands().begin(),
CI->arg_operands().end());
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
NewCall = Builder.CreateCall(NewFn, Args);
break;
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 18:56:37 +08:00
}
[AArch64] Add support for lowering bitreverse to the rbit instruction. Differential Revision: https://reviews.llvm.org/D28379 llvm-svn: 291575
2017-01-11 01:20:33 +08:00
case Intrinsic::bitreverse:
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
NewCall = Builder.CreateCall(NewFn, {CI->getArgOperand(0)});
break;
[AArch64] Add support for lowering bitreverse to the rbit instruction. Differential Revision: https://reviews.llvm.org/D28379 llvm-svn: 291575
2017-01-11 01:20:33 +08:00
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 12:26:04 +08:00
case Intrinsic::ctlz:
revert my previous patches that introduced an additional parameter to the objectsize intrinsic. After a lot of discussion, we realized it's not the best option for run-time bounds checking llvm-svn: 157255
2012-05-22 23:25:31 +08:00
case Intrinsic::cttz:
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 12:26:04 +08:00
assert(CI->getNumArgOperands() == 1 &&
"Mismatch between function args and call args");
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
NewCall =
Builder.CreateCall(NewFn, {CI->getArgOperand(0), Builder.getFalse()});
break;
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-11 02:42:51 +08:00
Let llvm.objectsize be conservative with null pointers This adds a parameter to @llvm.objectsize that makes it return conservative values if it's given null. This fixes PR23277. Differential Revision: https://reviews.llvm.org/D28494 llvm-svn: 298430
2017-03-22 04:08:59 +08:00
case Intrinsic::objectsize: {
Value *NullIsUnknownSize = CI->getNumArgOperands() == 2
? Builder.getFalse()
: CI->getArgOperand(2);
NewCall = Builder.CreateCall(
NewFn, {CI->getArgOperand(0), CI->getArgOperand(1), NullIsUnknownSize});
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
break;
Let llvm.objectsize be conservative with null pointers This adds a parameter to @llvm.objectsize that makes it return conservative values if it's given null. This fixes PR23277. Differential Revision: https://reviews.llvm.org/D28494 llvm-svn: 298430
2017-03-22 04:08:59 +08:00
}
Change objectsize intrinsic to accept different address spaces. Bitcasting everything to i8* won't work. Autoupgrade the old intrinsic declarations to use the new mangling. llvm-svn: 192117
2013-10-08 02:06:48 +08:00
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 09:00:32 +08:00
case Intrinsic::ctpop:
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
NewCall = Builder.CreateCall(NewFn, {CI->getArgOperand(0)});
break;
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 09:00:32 +08:00
case Intrinsic::convert_from_fp16:
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
NewCall = Builder.CreateCall(NewFn, {CI->getArgOperand(0)});
break;
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 07:25:25 +08:00
Fix intrinsics for XOP frczss/sd instructions. These instructions only take one source register and zero the upper bits of the destination rather than preserving them. llvm-svn: 158396
2012-06-13 15:18:53 +08:00
case Intrinsic::x86_xop_vfrcz_ss:
case Intrinsic::x86_xop_vfrcz_sd:
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
NewCall = Builder.CreateCall(NewFn, {CI->getArgOperand(1)});
break;
Fix intrinsics for XOP frczss/sd instructions. These instructions only take one source register and zero the upper bits of the destination rather than preserving them. llvm-svn: 158396
2012-06-13 15:18:53 +08:00
[X86][XOP] Support for VPERMIL2PD/VPERMIL2PS 2-input shuffle instructions This patch begins adding support for lowering to the XOP VPERMIL2PD/VPERMIL2PS shuffle instructions - adding the X86ISD::VPERMIL2 opcode and cleaning up the usage. The internal llvm intrinsics were assuming the shuffle mask operand was the same type as the float/double input operands (I guess to simplify the intrinsic definitions in X86InstrXOP.td to a single value type). These needed changing to integer types (matching the clang builtin and the AMD intrinsics definitions), an auto upgrade path is added to convert old calls. Mask decoding/target shuffle support will be added in future patches. Differential Revision: http://reviews.llvm.org/D20049 llvm-svn: 271633
2016-06-03 16:06:03 +08:00
case Intrinsic::x86_xop_vpermil2pd:
case Intrinsic::x86_xop_vpermil2ps:
case Intrinsic::x86_xop_vpermil2pd_256:
case Intrinsic::x86_xop_vpermil2ps_256: {
SmallVector<Value *, 4> Args(CI->arg_operands().begin(),
CI->arg_operands().end());
VectorType *FltIdxTy = cast<VectorType>(Args[2]->getType());
VectorType *IntIdxTy = VectorType::getInteger(FltIdxTy);
Args[2] = Builder.CreateBitCast(Args[2], IntIdxTy);
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
NewCall = Builder.CreateCall(NewFn, Args);
break;
[X86][XOP] Support for VPERMIL2PD/VPERMIL2PS 2-input shuffle instructions This patch begins adding support for lowering to the XOP VPERMIL2PD/VPERMIL2PS shuffle instructions - adding the X86ISD::VPERMIL2 opcode and cleaning up the usage. The internal llvm intrinsics were assuming the shuffle mask operand was the same type as the float/double input operands (I guess to simplify the intrinsic definitions in X86InstrXOP.td to a single value type). These needed changing to integer types (matching the clang builtin and the AMD intrinsics definitions), an auto upgrade path is added to convert old calls. Mask decoding/target shuffle support will be added in future patches. Differential Revision: http://reviews.llvm.org/D20049 llvm-svn: 271633
2016-06-03 16:06:03 +08:00
}
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-11 02:42:51 +08:00
case Intrinsic::x86_sse41_ptestc:
case Intrinsic::x86_sse41_ptestz:
Fix intrinsics for XOP frczss/sd instructions. These instructions only take one source register and zero the upper bits of the destination rather than preserving them. llvm-svn: 158396
2012-06-13 15:18:53 +08:00
case Intrinsic::x86_sse41_ptestnzc: {
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-11 02:42:51 +08:00
// The arguments for these intrinsics used to be v4f32, and changed
// to v2i64. This is purely a nop, since those are bitwise intrinsics.
// So, the only thing required is a bitcast for both arguments.
// First, check the arguments have the old type.
Value *Arg0 = CI->getArgOperand(0);
if (Arg0->getType() != VectorType::get(Type::getFloatTy(C), 4))
return;
// Old intrinsic, add bitcasts
Value *Arg1 = CI->getArgOperand(1);
[opaque pointer type] Verifier/AutoUpgrade: Remove a few uses of PointerType::getElementType llvm-svn: 235777
2015-04-25 05:16:07 +08:00
Type *NewVecTy = VectorType::get(Type::getInt64Ty(C), 2);
Value *BC0 = Builder.CreateBitCast(Arg0, NewVecTy, "cast");
Value *BC1 = Builder.CreateBitCast(Arg1, NewVecTy, "cast");
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
NewCall = Builder.CreateCall(NewFn, {BC0, BC1});
break;
[x86] Fix a pretty horrible bug and inconsistency in the x86 asm parsing (and latent bug in the instruction definitions). This is effectively a revert of r136287 which tried to address a specific and narrow case of immediate operands failing to be accepted by x86 instructions with a pretty heavy hammer: it introduced a new kind of operand that behaved differently. All of that is removed with this commit, but the test cases are both preserved and enhanced. The core problem that r136287 and this commit are trying to handle is that gas accepts both of the following instructions: insertps $192, %xmm0, %xmm1 insertps $-64, %xmm0, %xmm1 These will encode to the same byte sequence, with the immediate occupying an 8-bit entry. The first form was fixed by r136287 but that broke the prior handling of the second form! =[ Ironically, we would still emit the second form in some cases and then be unable to re-assemble the output. The reason why the first instruction failed to be handled is because prior to r136287 the operands ere marked 'i32i8imm' which forces them to be sign-extenable. Clearly, that won't work for 192 in a single byte. However, making thim zero-extended or "unsigned" doesn't really address the core issue either because it breaks negative immediates. The correct fix is to make these operands 'i8imm' reflecting that they can be either signed or unsigned but must be 8-bit immediates. This patch backs out r136287 and then changes those places as well as some others to use 'i8imm' rather than one of the extended variants. Naturally, this broke something else. The custom DAG nodes had to be updated to have a much more accurate type constraint of an i8 node, and a bunch of Pat immediates needed to be specified as i8 values. The fallout didn't end there though. We also then ceased to be able to match the instruction-specific intrinsics to the instructions so modified. Digging, this is because they too used i32 rather than i8 in their signature. So I've also switched those intrinsics to i8 arguments in line with the instructions. In order to make the intrinsic adjustments of course, I also had to add auto upgrading for the intrinsics. I suspect that the intrinsic argument types may have led everything down this rabbit hole. Pretty happy with the result. llvm-svn: 217310
2014-09-06 18:00:01 +08:00
}
case Intrinsic::x86_sse41_insertps:
case Intrinsic::x86_sse41_dppd:
case Intrinsic::x86_sse41_dpps:
case Intrinsic::x86_sse41_mpsadbw:
case Intrinsic::x86_avx_dp_ps_256:
case Intrinsic::x86_avx2_mpsadbw: {
// Need to truncate the last argument from i32 to i8 -- this argument models
// an inherently 8-bit immediate operand to these x86 instructions.
SmallVector<Value *, 4> Args(CI->arg_operands().begin(),
CI->arg_operands().end());
// Replace the last argument with a trunc.
Args.back() = Builder.CreateTrunc(Args.back(), Type::getInt8Ty(C), "trunc");
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
NewCall = Builder.CreateCall(NewFn, Args);
break;
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-18 06:33:23 +08:00
}
[AArch64] [ARM] Make a target-independent llvm.thread.pointer intrinsic. Both AArch64 and ARM support llvm.<arch>.thread.pointer intrinsics that just return the thread pointer. I have a pending patch that does the same for SystemZ (D19054), and there are many more targets that could benefit from one. This patch merges the ARM and AArch64 intrinsics into a single target independent one that will also be used by subsequent targets. Differential Revision: http://reviews.llvm.org/D19098 llvm-svn: 266818
2016-04-20 04:51:05 +08:00
case Intrinsic::thread_pointer: {
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
NewCall = Builder.CreateCall(NewFn, {});
break;
[AArch64] [ARM] Make a target-independent llvm.thread.pointer intrinsic. Both AArch64 and ARM support llvm.<arch>.thread.pointer intrinsics that just return the thread pointer. I have a pending patch that does the same for SystemZ (D19054), and there are many more targets that could benefit from one. This patch merges the ARM and AArch64 intrinsics into a single target independent one that will also be used by subsequent targets. Differential Revision: http://reviews.llvm.org/D19098 llvm-svn: 266818
2016-04-20 04:51:05 +08:00
}
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-29 02:27:25 +08:00
Revert "Revert "Invariant start/end intrinsics overloaded for address space"" This reverts commit 32fc6488e48eafc0ca1bac1bd9cbf0008224d530. llvm-svn: 278609
2016-08-14 07:31:24 +08:00
case Intrinsic::invariant_start:
case Intrinsic::invariant_end:
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-29 02:27:25 +08:00
case Intrinsic::masked_load:
Support arbitrary address space pointers in masked gather/scatter intrinsics. Fixes PR31789 - When loop-vectorize tries to use these intrinsics for a non-default address space pointer we fail with a "Calling a function with a bad singature!" assertion. This patch solves this by adding the 'vector of pointers' argument as an overloaded type which will determine the address space. Differential revision: https://reviews.llvm.org/D31490 llvm-svn: 302018
2017-05-03 20:28:54 +08:00
case Intrinsic::masked_store:
case Intrinsic::masked_gather:
case Intrinsic::masked_scatter: {
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-29 02:27:25 +08:00
SmallVector<Value *, 4> Args(CI->arg_operands().begin(),
CI->arg_operands().end());
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
NewCall = Builder.CreateCall(NewFn, Args);
break;
}
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-29 02:27:25 +08:00
}
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
assert(NewCall && "Should have either set this variable or returned through "
"the default case");
std::string Name = CI->getName();
if (!Name.empty()) {
CI->setName(Name + ".old");
NewCall->setName(Name);
Fix intrinsics for XOP frczss/sd instructions. These instructions only take one source register and zero the upper bits of the destination rather than preserving them. llvm-svn: 158396
2012-06-13 15:18:53 +08:00
}
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 09:49:13 +08:00
CI->replaceAllUsesWith(NewCall);
CI->eraseFromParent();
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
}
try to make comments more meaningful; NFC Retry r266541 without the range-based-for-loop-change that was wrong. llvm-svn: 266658
2016-04-19 03:11:57 +08:00
void llvm::UpgradeCallsToIntrinsic(Function *F) {
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
assert(F && "Illegal attempt to upgrade a non-existent intrinsic.");
try to make comments more meaningful; NFC Retry r266541 without the range-based-for-loop-change that was wrong. llvm-svn: 266658
2016-04-19 03:11:57 +08:00
// Check if this function should be upgraded and get the replacement function
// if there is one.
rip out a ton of intrinsic modernization logic from AutoUpgrade.cpp, which is for pre-2.9 bitcode files. We keep x86 unaligned loads, movnt, crc32, and the target indep prefetch change. As usual, updating the testsuite is a PITA. llvm-svn: 133337
2011-06-18 14:05:24 +08:00
Function *NewFn;
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-18 06:33:23 +08:00
if (UpgradeIntrinsicFunction(F, NewFn)) {
try to make comments more meaningful; NFC Retry r266541 without the range-based-for-loop-change that was wrong. llvm-svn: 266658
2016-04-19 03:11:57 +08:00
// Replace all users of the old function with the new function or new
// instructions. This is not a range loop because the call is deleted.
for (auto UI = F->user_begin(), UE = F->user_end(); UI != UE; )
Revert "use range loop, try to make comments more meaningful; NFCI" This reverts commit r266541 since it introduces a use-after-free: http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/11471 llvm-svn: 266550
2016-04-17 11:59:37 +08:00
if (CallInst *CI = dyn_cast<CallInst>(*UI++))
Remove always-true comparison, NFC. Summary: Looking at r241279, I noticed that UpgradedIntrinsics only gets written to in the following code: if (UpgradeIntrinsicFunction(&F, NewFn)) UpgradedIntrinsics[&F] = NewFn; Looking through UpgradeIntrinsicFunction, we always return false OR NewFn will be set to a different function from our source. This patch pulls the F != NewFn into UpgradeIntrinsicFunction as an assert, and removes the check from callers of UpgradeIntrinsicFunction. Reviewers: rafael, chandlerc Subscribers: llvm-commits-list Differential Revision: http://reviews.llvm.org/D10915 llvm-svn: 241369
2015-07-04 04:12:01 +08:00
UpgradeIntrinsicCall(CI, NewFn);
try to make comments more meaningful; NFC Retry r266541 without the range-based-for-loop-change that was wrong. llvm-svn: 266658
2016-04-19 03:11:57 +08:00
Remove always-true comparison, NFC. Summary: Looking at r241279, I noticed that UpgradedIntrinsics only gets written to in the following code: if (UpgradeIntrinsicFunction(&F, NewFn)) UpgradedIntrinsics[&F] = NewFn; Looking through UpgradeIntrinsicFunction, we always return false OR NewFn will be set to a different function from our source. This patch pulls the F != NewFn into UpgradeIntrinsicFunction as an assert, and removes the check from callers of UpgradeIntrinsicFunction. Reviewers: rafael, chandlerc Subscribers: llvm-commits-list Differential Revision: http://reviews.llvm.org/D10915 llvm-svn: 241369
2015-07-04 04:12:01 +08:00
// Remove old function, no longer used, from the module.
F->eraseFromParent();
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 09:51:18 +08:00
}
}
Reapply 79977. Use MDNodes to encode debug info in llvm IR. llvm-svn: 80406
2009-08-29 07:24:31 +08:00
Fix auto-upgrade of TBAA tags in Bitcode Reader If TBAA is on an intrinsic and it gets upgraded, it'll delete the call instruction that we collected in a vector. Even if we were to use WeakVH, it'll drop the TBAA and we'll hit the assert on the upgrade path. r263673 gave a shot to make sure the TBAA upgrade happens before intrinsics upgrade, but failed to account for all cases. Instead of collecting instructions in a vector, this patch makes it just upgrade the TBAA on the fly, because metadata are always already loaded at this point. Differential Revision: https://reviews.llvm.org/D24533 llvm-svn: 281549
2016-09-15 06:29:59 +08:00
MDNode *llvm::UpgradeTBAANode(MDNode &MD) {
AutoUpgrade: upgrade from scalar TBAA format to struct-path aware TBAA format. We treat TBAA tags as struct-path aware TBAA format when the first operand is a MDNode and the tag has 3 or more operands. llvm-svn: 191593
2013-09-28 08:22:27 +08:00
// Check if the tag uses struct-path aware TBAA format.
Fix auto-upgrade of TBAA tags in Bitcode Reader If TBAA is on an intrinsic and it gets upgraded, it'll delete the call instruction that we collected in a vector. Even if we were to use WeakVH, it'll drop the TBAA and we'll hit the assert on the upgrade path. r263673 gave a shot to make sure the TBAA upgrade happens before intrinsics upgrade, but failed to account for all cases. Instead of collecting instructions in a vector, this patch makes it just upgrade the TBAA on the fly, because metadata are always already loaded at this point. Differential Revision: https://reviews.llvm.org/D24533 llvm-svn: 281549
2016-09-15 06:29:59 +08:00
if (isa<MDNode>(MD.getOperand(0)) && MD.getNumOperands() >= 3)
return &MD;
AutoUpgrade: upgrade from scalar TBAA format to struct-path aware TBAA format. We treat TBAA tags as struct-path aware TBAA format when the first operand is a MDNode and the tag has 3 or more operands. llvm-svn: 191593
2013-09-28 08:22:27 +08:00
Fix auto-upgrade of TBAA tags in Bitcode Reader If TBAA is on an intrinsic and it gets upgraded, it'll delete the call instruction that we collected in a vector. Even if we were to use WeakVH, it'll drop the TBAA and we'll hit the assert on the upgrade path. r263673 gave a shot to make sure the TBAA upgrade happens before intrinsics upgrade, but failed to account for all cases. Instead of collecting instructions in a vector, this patch makes it just upgrade the TBAA on the fly, because metadata are always already loaded at this point. Differential Revision: https://reviews.llvm.org/D24533 llvm-svn: 281549
2016-09-15 06:29:59 +08:00
auto &Context = MD.getContext();
if (MD.getNumOperands() == 3) {
Metadata *Elts[] = {MD.getOperand(0), MD.getOperand(1)};
MDNode *ScalarType = MDNode::get(Context, Elts);
AutoUpgrade: upgrade from scalar TBAA format to struct-path aware TBAA format. We treat TBAA tags as struct-path aware TBAA format when the first operand is a MDNode and the tag has 3 or more operands. llvm-svn: 191593
2013-09-28 08:22:27 +08:00
// Create a MDNode <ScalarType, ScalarType, offset 0, const>
IR: Split Metadata from Value Split `Metadata` away from the `Value` class hierarchy, as part of PR21532. Assembly and bitcode changes are in the wings, but this is the bulk of the change for the IR C++ API. I have a follow-up patch prepared for `clang`. If this breaks other sub-projects, I apologize in advance :(. Help me compile it on Darwin I'll try to fix it. FWIW, the errors should be easy to fix, so it may be simpler to just fix it yourself. This breaks the build for all metadata-related code that's out-of-tree. Rest assured the transition is mechanical and the compiler should catch almost all of the problems. Here's a quick guide for updating your code: - `Metadata` is the root of a class hierarchy with three main classes: `MDNode`, `MDString`, and `ValueAsMetadata`. It is distinct from the `Value` class hierarchy. It is typeless -- i.e., instances do *not* have a `Type`. - `MDNode`'s operands are all `Metadata *` (instead of `Value *`). - `TrackingVH<MDNode>` and `WeakVH` referring to metadata can be replaced with `TrackingMDNodeRef` and `TrackingMDRef`, respectively. If you're referring solely to resolved `MDNode`s -- post graph construction -- just use `MDNode*`. - `MDNode` (and the rest of `Metadata`) have only limited support for `replaceAllUsesWith()`. As long as an `MDNode` is pointing at a forward declaration -- the result of `MDNode::getTemporary()` -- it maintains a side map of its uses and can RAUW itself. Once the forward declarations are fully resolved RAUW support is dropped on the ground. This means that uniquing collisions on changing operands cause nodes to become "distinct". (This already happened fairly commonly, whenever an operand went to null.) If you're constructing complex (non self-reference) `MDNode` cycles, you need to call `MDNode::resolveCycles()` on each node (or on a top-level node that somehow references all of the nodes). Also, don't do that. Metadata cycles (and the RAUW machinery needed to construct them) are expensive. - An `MDNode` can only refer to a `Constant` through a bridge called `ConstantAsMetadata` (one of the subclasses of `ValueAsMetadata`). As a side effect, accessing an operand of an `MDNode` that is known to be, e.g., `ConstantInt`, takes three steps: first, cast from `Metadata` to `ConstantAsMetadata`; second, extract the `Constant`; third, cast down to `ConstantInt`. The eventual goal is to introduce `MDInt`/`MDFloat`/etc. and have metadata schema owners transition away from using `Constant`s when the type isn't important (and they don't care about referring to `GlobalValue`s). In the meantime, I've added transitional API to the `mdconst` namespace that matches semantics with the old code, in order to avoid adding the error-prone three-step equivalent to every call site. If your old code was: MDNode *N = foo(); bar(isa <ConstantInt>(N->getOperand(0))); baz(cast <ConstantInt>(N->getOperand(1))); bak(cast_or_null <ConstantInt>(N->getOperand(2))); bat(dyn_cast <ConstantInt>(N->getOperand(3))); bay(dyn_cast_or_null<ConstantInt>(N->getOperand(4))); you can trivially match its semantics with: MDNode *N = foo(); bar(mdconst::hasa <ConstantInt>(N->getOperand(0))); baz(mdconst::extract <ConstantInt>(N->getOperand(1))); bak(mdconst::extract_or_null <ConstantInt>(N->getOperand(2))); bat(mdconst::dyn_extract <ConstantInt>(N->getOperand(3))); bay(mdconst::dyn_extract_or_null<ConstantInt>(N->getOperand(4))); and when you transition your metadata schema to `MDInt`: MDNode *N = foo(); bar(isa <MDInt>(N->getOperand(0))); baz(cast <MDInt>(N->getOperand(1))); bak(cast_or_null <MDInt>(N->getOperand(2))); bat(dyn_cast <MDInt>(N->getOperand(3))); bay(dyn_cast_or_null<MDInt>(N->getOperand(4))); - A `CallInst` -- specifically, intrinsic instructions -- can refer to metadata through a bridge called `MetadataAsValue`. This is a subclass of `Value` where `getType()->isMetadataTy()`. `MetadataAsValue` is the *only* class that can legally refer to a `LocalAsMetadata`, which is a bridged form of non-`Constant` values like `Argument` and `Instruction`. It can also refer to any other `Metadata` subclass. (I'll break all your testcases in a follow-up commit, when I propagate this change to assembly.) llvm-svn: 223802
2014-12-10 02:38:53 +08:00
Metadata *Elts2[] = {ScalarType, ScalarType,
Fix auto-upgrade of TBAA tags in Bitcode Reader If TBAA is on an intrinsic and it gets upgraded, it'll delete the call instruction that we collected in a vector. Even if we were to use WeakVH, it'll drop the TBAA and we'll hit the assert on the upgrade path. r263673 gave a shot to make sure the TBAA upgrade happens before intrinsics upgrade, but failed to account for all cases. Instead of collecting instructions in a vector, this patch makes it just upgrade the TBAA on the fly, because metadata are always already loaded at this point. Differential Revision: https://reviews.llvm.org/D24533 llvm-svn: 281549
2016-09-15 06:29:59 +08:00
ConstantAsMetadata::get(
Constant::getNullValue(Type::getInt64Ty(Context))),
MD.getOperand(2)};
return MDNode::get(Context, Elts2);
AutoUpgrade: upgrade from scalar TBAA format to struct-path aware TBAA format. We treat TBAA tags as struct-path aware TBAA format when the first operand is a MDNode and the tag has 3 or more operands. llvm-svn: 191593
2013-09-28 08:22:27 +08:00
}
Fix auto-upgrade of TBAA tags in Bitcode Reader If TBAA is on an intrinsic and it gets upgraded, it'll delete the call instruction that we collected in a vector. Even if we were to use WeakVH, it'll drop the TBAA and we'll hit the assert on the upgrade path. r263673 gave a shot to make sure the TBAA upgrade happens before intrinsics upgrade, but failed to account for all cases. Instead of collecting instructions in a vector, this patch makes it just upgrade the TBAA on the fly, because metadata are always already loaded at this point. Differential Revision: https://reviews.llvm.org/D24533 llvm-svn: 281549
2016-09-15 06:29:59 +08:00
// Create a MDNode <MD, MD, offset 0>
Metadata *Elts[] = {&MD, &MD, ConstantAsMetadata::get(Constant::getNullValue(
Type::getInt64Ty(Context)))};
return MDNode::get(Context, Elts);
AutoUpgrade: upgrade from scalar TBAA format to struct-path aware TBAA format. We treat TBAA tags as struct-path aware TBAA format when the first operand is a MDNode and the tag has 3 or more operands. llvm-svn: 191593
2013-09-28 08:22:27 +08:00
}
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 09:34:59 +08:00
Instruction *llvm::UpgradeBitCastInst(unsigned Opc, Value *V, Type *DestTy,
Instruction *&Temp) {
if (Opc != Instruction::BitCast)
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 14:08:46 +08:00
return nullptr;
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 09:34:59 +08:00
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 14:08:46 +08:00
Temp = nullptr;
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 09:34:59 +08:00
Type *SrcTy = V->getType();
if (SrcTy->isPtrOrPtrVectorTy() && DestTy->isPtrOrPtrVectorTy() &&
SrcTy->getPointerAddressSpace() != DestTy->getPointerAddressSpace()) {
LLVMContext &Context = V->getContext();
// We have no information about target data layout, so we assume that
// the maximum pointer size is 64bit.
Type *MidTy = Type::getInt64Ty(Context);
Temp = CastInst::Create(Instruction::PtrToInt, V, MidTy);
return CastInst::Create(Instruction::IntToPtr, Temp, DestTy);
}
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 14:08:46 +08:00
return nullptr;
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 09:34:59 +08:00
}
Value *llvm::UpgradeBitCastExpr(unsigned Opc, Constant *C, Type *DestTy) {
if (Opc != Instruction::BitCast)
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 14:08:46 +08:00
return nullptr;
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 09:34:59 +08:00
Type *SrcTy = C->getType();
if (SrcTy->isPtrOrPtrVectorTy() && DestTy->isPtrOrPtrVectorTy() &&
SrcTy->getPointerAddressSpace() != DestTy->getPointerAddressSpace()) {
LLVMContext &Context = C->getContext();
// We have no information about target data layout, so we assume that
// the maximum pointer size is 64bit.
Type *MidTy = Type::getInt64Ty(Context);
return ConstantExpr::getIntToPtr(ConstantExpr::getPtrToInt(C, MidTy),
DestTy);
}
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 14:08:46 +08:00
return nullptr;
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 09:34:59 +08:00
}
Debug Info: drop debug info via upgrading path if version number does not match. Add a helper function getDebugInfoVersionFromModule to return the debug info version number for a module. "Verifier/module-flags-1.ll" checks for verification errors. It will seg fault when calling getDebugInfoVersionFromModule because of the incorrect format for module flags in the testing case. We make getModuleFlagsMetadata more robust by checking for error conditions. PR17982 llvm-svn: 196158
2013-12-03 05:29:56 +08:00
/// Check the debug info version number, if it is out-dated, drop the debug
/// info. Return true if module is modified.
bool llvm::UpgradeDebugInfo(Module &M) {
Report a warning when dropping outdated debug info metadata. Use DiagnosticInfo to emit the warning. llvm-svn: 199346
2014-01-16 09:51:12 +08:00
unsigned Version = getDebugMetadataVersionFromModule(M);
if (Version == DEBUG_METADATA_VERSION)
Debug Info: drop debug info via upgrading path if version number does not match. Add a helper function getDebugInfoVersionFromModule to return the debug info version number for a module. "Verifier/module-flags-1.ll" checks for verification errors. It will seg fault when calling getDebugInfoVersionFromModule because of the incorrect format for module flags in the testing case. We make getModuleFlagsMetadata more robust by checking for error conditions. PR17982 llvm-svn: 196158
2013-12-03 05:29:56 +08:00
return false;
Report a warning when dropping outdated debug info metadata. Use DiagnosticInfo to emit the warning. llvm-svn: 199346
2014-01-16 09:51:12 +08:00
bool RetCode = StripDebugInfo(M);
if (RetCode) {
DiagnosticInfoDebugMetadataVersion DiagVersion(M, Version);
M.getContext().diagnose(DiagVersion);
}
return RetCode;
Debug Info: drop debug info via upgrading path if version number does not match. Add a helper function getDebugInfoVersionFromModule to return the debug info version number for a module. "Verifier/module-flags-1.ll" checks for verification errors. It will seg fault when calling getDebugInfoVersionFromModule because of the incorrect format for module flags in the testing case. We make getModuleFlagsMetadata more robust by checking for error conditions. PR17982 llvm-svn: 196158
2013-12-03 05:29:56 +08:00
}
Rename loop unrolling and loop vectorizer metadata to have a common prefix. [LLVM part] These patches rename the loop unrolling and loop vectorizer metadata such that they have a common 'llvm.loop.' prefix. Metadata name changes: llvm.vectorizer.* => llvm.loop.vectorizer.* llvm.loopunroll.* => llvm.loop.unroll.* This was a suggestion from an earlier review (http://reviews.llvm.org/D4090) which added the loop unrolling metadata. Patch by Mark Heffernan. llvm-svn: 211710
2014-06-25 23:41:00 +08:00
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 07:14:48 +08:00
bool llvm::UpgradeModuleFlags(Module &M) {
const NamedMDNode *ModFlags = M.getModuleFlagsMetadata();
if (!ModFlags)
return false;
bool HasObjCFlag = false, HasClassProperties = false;
for (unsigned I = 0, E = ModFlags->getNumOperands(); I != E; ++I) {
MDNode *Op = ModFlags->getOperand(I);
if (Op->getNumOperands() < 2)
continue;
MDString *ID = dyn_cast_or_null<MDString>(Op->getOperand(1));
if (!ID)
continue;
if (ID->getString() == "Objective-C Image Info Version")
HasObjCFlag = true;
if (ID->getString() == "Objective-C Class Properties")
HasClassProperties = true;
}
// "Objective-C Class Properties" is recently added for Objective-C. We
// upgrade ObjC bitcodes to contain a "Objective-C Class Properties" module
Fix autoupgrade logic for Objective-C class properties module flag Previous we were issuing an error when linking a module containing the new Objective-C metadata structure for class properties with an "old" one. Now instead we downgrade the module flag so that the Objective-C runtime does not expect the new metadata structure. This is consistent with what ld64 is doing on binary files. Differential Revision: https://reviews.llvm.org/D24620 llvm-svn: 281685
2016-09-16 08:38:18 +08:00
// flag of value 0, so we can correclty downgrade this flag when trying to
// link an ObjC bitcode without this module flag with an ObjC bitcode with
// this module flag.
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 07:14:48 +08:00
if (HasObjCFlag && !HasClassProperties) {
Fix autoupgrade logic for Objective-C class properties module flag Previous we were issuing an error when linking a module containing the new Objective-C metadata structure for class properties with an "old" one. Now instead we downgrade the module flag so that the Objective-C runtime does not expect the new metadata structure. This is consistent with what ld64 is doing on binary files. Differential Revision: https://reviews.llvm.org/D24620 llvm-svn: 281685
2016-09-16 08:38:18 +08:00
M.addModuleFlag(llvm::Module::Override, "Objective-C Class Properties",
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 07:14:48 +08:00
(uint32_t)0);
return true;
}
return false;
}
IR: Stop upgrading !llvm.loop attachments via MDString Remove logic to upgrade !llvm.loop by changing the MDString tag directly. This old logic would check (and change) arbitrary strings that had nothing to do with loop metadata. Instead, check !llvm.loop attachments directly, and change which strings get attached. Rather than updating the assembly-based upgrade, drop it entirely. It has been quite a while since we supported upgrading textual IR. llvm-svn: 264373
2016-03-25 08:56:13 +08:00
static bool isOldLoopArgument(Metadata *MD) {
auto *T = dyn_cast_or_null<MDTuple>(MD);
if (!T)
return false;
if (T->getNumOperands() < 1)
return false;
auto *S = dyn_cast_or_null<MDString>(T->getOperand(0));
if (!S)
return false;
return S->getString().startswith("llvm.vectorizer.");
}
static MDString *upgradeLoopTag(LLVMContext &C, StringRef OldTag) {
StringRef OldPrefix = "llvm.vectorizer.";
assert(OldTag.startswith(OldPrefix) && "Expected old prefix");
if (OldTag == "llvm.vectorizer.unroll")
return MDString::get(C, "llvm.loop.interleave.count");
return MDString::get(
C, (Twine("llvm.loop.vectorize.") + OldTag.drop_front(OldPrefix.size()))
.str());
}
static Metadata *upgradeLoopArgument(Metadata *MD) {
auto *T = dyn_cast_or_null<MDTuple>(MD);
if (!T)
return MD;
if (T->getNumOperands() < 1)
return MD;
auto *OldTag = dyn_cast_or_null<MDString>(T->getOperand(0));
if (!OldTag)
return MD;
if (!OldTag->getString().startswith("llvm.vectorizer."))
return MD;
// This has an old tag. Upgrade it.
SmallVector<Metadata *, 8> Ops;
Ops.reserve(T->getNumOperands());
Ops.push_back(upgradeLoopTag(T->getContext(), OldTag->getString()));
for (unsigned I = 1, E = T->getNumOperands(); I != E; ++I)
Ops.push_back(T->getOperand(I));
return MDTuple::get(T->getContext(), Ops);
}
MDNode *llvm::upgradeInstructionLoopAttachment(MDNode &N) {
auto *T = dyn_cast<MDTuple>(&N);
if (!T)
return &N;
Use range algorithms instead of unpacking begin/end No functionality change is intended. llvm-svn: 278417
2016-08-12 05:15:00 +08:00
if (none_of(T->operands(), isOldLoopArgument))
IR: Stop upgrading !llvm.loop attachments via MDString Remove logic to upgrade !llvm.loop by changing the MDString tag directly. This old logic would check (and change) arbitrary strings that had nothing to do with loop metadata. Instead, check !llvm.loop attachments directly, and change which strings get attached. Rather than updating the assembly-based upgrade, drop it entirely. It has been quite a while since we supported upgrading textual IR. llvm-svn: 264373
2016-03-25 08:56:13 +08:00
return &N;
SmallVector<Metadata *, 8> Ops;
Ops.reserve(T->getNumOperands());
for (Metadata *MD : T->operands())
Ops.push_back(upgradeLoopArgument(MD));
return MDTuple::get(T->getContext(), Ops);
Rename loop unrolling and loop vectorizer metadata to have a common prefix. [LLVM part] These patches rename the loop unrolling and loop vectorizer metadata such that they have a common 'llvm.loop.' prefix. Metadata name changes: llvm.vectorizer.* => llvm.loop.vectorizer.* llvm.loopunroll.* => llvm.loop.unroll.* This was a suggestion from an earlier review (http://reviews.llvm.org/D4090) which added the loop unrolling metadata. Patch by Mark Heffernan. llvm-svn: 211710
2014-06-25 23:41:00 +08:00
}