maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/lib/Target/X86/X86InstrFragmentsSIMD.td

744 lines
34 KiB
TableGen
Raw Normal View History

some comment fix for X86 and ARM llvm-svn: 150902
2012-02-19 10:03:36 +08:00
//===-- X86InstrFragmentsSIMD.td - x86 SIMD ISA ------------*- tablegen -*-===//
TableGen fragment refactoring. Move some utility TableGen defs, classes, etc. into a common file so they may be used my multiple pattern files. We will use this for the AVX specification to help with the transition from the current SSE specification. llvm-svn: 95727
2010-02-10 07:52:19 +08:00
//
// The LLVM Compiler Infrastructure
//
// This file is distributed under the University of Illinois Open Source
// License. See LICENSE.TXT for details.
Emacs-tag and some comment fix for all ARM, CellSPU, Hexagon, MBlaze, MSP430, PPC, PTX, Sparc, X86, XCore. llvm-svn: 150878
2012-02-18 20:03:15 +08:00
//
TableGen fragment refactoring. Move some utility TableGen defs, classes, etc. into a common file so they may be used my multiple pattern files. We will use this for the AVX specification to help with the transition from the current SSE specification. llvm-svn: 95727
2010-02-10 07:52:19 +08:00
//===----------------------------------------------------------------------===//
//
// This file provides pattern fragments useful for SIMD instructions.
//
//===----------------------------------------------------------------------===//
[X86][MMX] Move MMX DAG node to proper file llvm-svn: 228291
2015-02-05 21:22:50 +08:00
//===----------------------------------------------------------------------===//
// MMX specific DAG Nodes.
//===----------------------------------------------------------------------===//
// Low word of MMX to GPR.
def MMX_X86movd2w : SDNode<"X86ISD::MMX_MOVD2W", SDTypeProfile<1, 1,
[SDTCisVT<0, i32>, SDTCisVT<1, x86mmx>]>>;
[X86][MMX] Handle i32->mmx conversion using movd Implement a BITCAST dag combine to transform i32->mmx conversion patterns into a X86 specific node (MMX_MOVW2D) and guarantee that moves between i32 and x86mmx are better handled, i.e., don't use store-load to do the conversion.. llvm-svn: 228293
2015-02-05 21:23:07 +08:00
// GPR to low word of MMX.
def MMX_X86movw2d : SDNode<"X86ISD::MMX_MOVW2D", SDTypeProfile<1, 1,
[SDTCisVT<0, x86mmx>, SDTCisVT<1, i32>]>>;
[X86][MMX] Move MMX DAG node to proper file llvm-svn: 228291
2015-02-05 21:22:50 +08:00
TableGen fragment refactoring. Move some utility TableGen defs, classes, etc. into a common file so they may be used my multiple pattern files. We will use this for the AVX specification to help with the transition from the current SSE specification. llvm-svn: 95727
2010-02-10 07:52:19 +08:00
//===----------------------------------------------------------------------===//
// MMX Pattern Fragments
//===----------------------------------------------------------------------===//
Massive rewrite of MMX: The x86_mmx type is used for MMX intrinsics, parameters and return values where these use MMX registers, and is also supported in load, store, and bitcast. Only the above operations generate MMX instructions, and optimizations do not operate on or produce MMX intrinsics. MMX-sized vectors <2 x i32> etc. are lowered to XMM or split into smaller pieces. Optimizations may occur on these forms and the result casted back to x86_mmx, provided the result feeds into a previous existing x86_mmx operation. The point of all this is prevent optimizations from introducing MMX operations, which is unsafe due to the EMMS problem. llvm-svn: 115243
2010-10-01 07:57:10 +08:00
def load_mmx : PatFrag<(ops node:$ptr), (x86mmx (load node:$ptr))>;
[X86][MMX] Support folding loads in psll, psrl and psra intrinsics llvm-svn: 230225
2015-02-23 23:23:14 +08:00
def load_mvmmx : PatFrag<(ops node:$ptr),
(x86mmx (MMX_X86movw2d (load node:$ptr)))>;
Massive rewrite of MMX: The x86_mmx type is used for MMX intrinsics, parameters and return values where these use MMX registers, and is also supported in load, store, and bitcast. Only the above operations generate MMX instructions, and optimizations do not operate on or produce MMX intrinsics. MMX-sized vectors <2 x i32> etc. are lowered to XMM or split into smaller pieces. Optimizations may occur on these forms and the result casted back to x86_mmx, provided the result feeds into a previous existing x86_mmx operation. The point of all this is prevent optimizations from introducing MMX operations, which is unsafe due to the EMMS problem. llvm-svn: 115243
2010-10-01 07:57:10 +08:00
def bc_mmx : PatFrag<(ops node:$in), (x86mmx (bitconvert node:$in))>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
//===----------------------------------------------------------------------===//
// SSE specific DAG Nodes.
//===----------------------------------------------------------------------===//
def SDTX86FPShiftOp : SDTypeProfile<1, 2, [ SDTCisSameAs<0, 1>,
SDTCisFP<0>, SDTCisInt<2> ]>;
def SDTX86VFCMP : SDTypeProfile<1, 3, [SDTCisInt<0>, SDTCisSameAs<1, 2>,
Improve some x86 type constraints. llvm-svn: 200120
2014-01-26 12:59:39 +08:00
SDTCisFP<1>, SDTCisVT<3, i8>,
SDTCisVec<1>]>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
X86: Match the SSE/AVX min/max vector ops using a custom node instead of intrinsics This is very mechanical, no functionality change. Preparation for PR14667. llvm-svn: 170898
2012-12-21 22:04:55 +08:00
def X86umin : SDNode<"X86ISD::UMIN", SDTIntBinOp>;
def X86umax : SDNode<"X86ISD::UMAX", SDTIntBinOp>;
def X86smin : SDNode<"X86ISD::SMIN", SDTIntBinOp>;
def X86smax : SDNode<"X86ISD::SMAX", SDTIntBinOp>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def X86fmin : SDNode<"X86ISD::FMIN", SDTFPBinOp>;
def X86fmax : SDNode<"X86ISD::FMAX", SDTFPBinOp>;
When unsafe math is used, we can use commutative FMAX and FMIN. In some cases this allows for better code generation. Added a new DAGCombine transformation to convert FMAX and FMIN to FMANC and FMINC, which are commutative. For example: movaps %xmm0, %xmm1 movsd LC(%rip), %xmm0 minsd %xmm1, %xmm0 becomes: minsd LC(%rip), %xmm0 llvm-svn: 162187
2012-08-19 21:06:16 +08:00
// Commutative and Associative FMIN and FMAX.
def X86fminc : SDNode<"X86ISD::FMINC", SDTFPBinOp,
[SDNPCommutative, SDNPAssociative]>;
def X86fmaxc : SDNode<"X86ISD::FMAXC", SDTFPBinOp,
[SDNPCommutative, SDNPAssociative]>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def X86fand : SDNode<"X86ISD::FAND", SDTFPBinOp,
[SDNPCommutative, SDNPAssociative]>;
def X86for : SDNode<"X86ISD::FOR", SDTFPBinOp,
[SDNPCommutative, SDNPAssociative]>;
def X86fxor : SDNode<"X86ISD::FXOR", SDTFPBinOp,
[SDNPCommutative, SDNPAssociative]>;
X86: Turn fp selects into mask operations. double test(double a, double b, double c, double d) { return a<b ? c : d; } before: _test: ucomisd %xmm0, %xmm1 ja LBB0_2 movaps %xmm3, %xmm2 LBB0_2: movaps %xmm2, %xmm0 after: _test: cmpltsd %xmm1, %xmm0 andpd %xmm0, %xmm2 andnpd %xmm3, %xmm0 orpd %xmm2, %xmm0 Small speedup on Benchmarks/SmallPT llvm-svn: 187706
2013-08-04 20:05:16 +08:00
def X86fandn : SDNode<"X86ISD::FANDN", SDTFPBinOp,
[SDNPCommutative, SDNPAssociative]>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def X86frsqrt : SDNode<"X86ISD::FRSQRT", SDTFPUnaryOp>;
def X86frcp : SDNode<"X86ISD::FRCP", SDTFPUnaryOp>;
def X86fsrl : SDNode<"X86ISD::FSRL", SDTX86FPShiftOp>;
FGETSIGN support for x86, using movmskps/pd. Will be enabled with a patch to TargetLowering.cpp. rdar://problem/5660695 llvm-svn: 132388
2011-06-01 12:39:42 +08:00
def X86fgetsign: SDNode<"X86ISD::FGETSIGNx86",SDTFPToIntOp>;
Synthesize SSE3/AVX 128 bit horizontal add/sub instructions from floating point add/sub of appropriate shuffle vectors. Does not synthesize the 256 bit AVX versions because they work differently. llvm-svn: 140332
2011-09-23 04:15:48 +08:00
def X86fhadd : SDNode<"X86ISD::FHADD", SDTFPBinOp>;
def X86fhsub : SDNode<"X86ISD::FHSUB", SDTFPBinOp>;
Synthesize SSSE3/AVX 128-bit horizontal integer add/sub instructions from add/sub of appropriate shuffle vectors. llvm-svn: 144989
2011-11-19 17:02:40 +08:00
def X86hadd : SDNode<"X86ISD::HADD", SDTIntBinOp>;
def X86hsub : SDNode<"X86ISD::HSUB", SDTIntBinOp>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def X86comi : SDNode<"X86ISD::COMI", SDTX86CmpTest>;
def X86ucomi : SDNode<"X86ISD::UCOMI", SDTX86CmpTest>;
AVX-512: Added legal type MVT::i1 and VK1 register for it. Added scalar compare VCMPSS, VCMPSD. Implemented LowerSELECT for scalar FP operations. I replaced FSETCCss, FSETCCsd with one node type FSETCCs. Node extract_vector_elt(v16i1/v8i1, idx) returns an element of type i1. llvm-svn: 197384
2013-12-16 21:52:35 +08:00
def X86cmps : SDNode<"X86ISD::FSETCC", SDTX86Cmps>;
//def X86cmpsd : SDNode<"X86ISD::FSETCCsd", SDTX86Cmpsd>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def X86pshufb : SDNode<"X86ISD::PSHUFB",
Custom lower PSIGN and PSHUFB intrinsics to their corresponding target specific nodes so we can remove the isel patterns. llvm-svn: 148933
2012-01-25 14:43:11 +08:00
SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>,
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
SDTCisSameAs<0,2>]>>;
The target specific node PANDN name is misleading. That happens because it's later selected to a ANDNPD/ANDNPS instruction instead of the PANDN instruction. Rename it. llvm-svn: 135087
2011-07-14 05:36:47 +08:00
def X86andnp : SDNode<"X86ISD::ANDNP",
Make X86ISD::ANDNP more general and Codegen 256-bit VANDNP. A more general version of X86ISD::ANDNP also opened the room for a little bit of refactoring. llvm-svn: 135088
2011-07-14 05:36:51 +08:00
SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>,
Add support for matching psign & plendvb to the x86 target Remove unnecessary pandn patterns, 'vnot' patfrag looks through bitcasts llvm-svn: 122098
2010-12-18 06:55:37 +08:00
SDTCisSameAs<0,2>]>>;
Collapse X86 PSIGNB/PSIGNW/PSIGND node types. llvm-svn: 144988
2011-11-19 15:33:10 +08:00
def X86psign : SDNode<"X86ISD::PSIGN",
Extend VPBLENDVB and VPSIGN lowering to work for AVX2. llvm-svn: 144987
2011-11-19 15:07:26 +08:00
SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>,
Add support for matching psign & plendvb to the x86 target Remove unnecessary pandn patterns, 'vnot' patfrag looks through bitcasts llvm-svn: 122098
2010-12-18 06:55:37 +08:00
SDTCisSameAs<0,2>]>>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def X86pextrb : SDNode<"X86ISD::PEXTRB",
SDTypeProfile<1, 2, [SDTCisVT<0, i32>, SDTCisPtrTy<2>]>>;
def X86pextrw : SDNode<"X86ISD::PEXTRW",
SDTypeProfile<1, 2, [SDTCisVT<0, i32>, SDTCisPtrTy<2>]>>;
def X86pinsrb : SDNode<"X86ISD::PINSRB",
SDTypeProfile<1, 3, [SDTCisVT<0, v16i8>, SDTCisSameAs<0,1>,
SDTCisVT<2, i32>, SDTCisPtrTy<3>]>>;
def X86pinsrw : SDNode<"X86ISD::PINSRW",
SDTypeProfile<1, 3, [SDTCisVT<0, v8i16>, SDTCisSameAs<0,1>,
SDTCisVT<2, i32>, SDTCisPtrTy<3>]>>;
Rename X86insrtps to the proper instruction name. Summary: The INSERTPS pattern fragment was called insrtps (mising 'e'), which would make it harder to grep for the patterns related to this instruction. Renaming it to use the proper instruction name. Reviewers: nadav CC: llvm-commits Differential Revision: http://reviews.llvm.org/D3443 llvm-svn: 206779
2014-04-22 04:07:29 +08:00
def X86insertps : SDNode<"X86ISD::INSERTPS",
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
SDTypeProfile<1, 3, [SDTCisVT<0, v4f32>, SDTCisSameAs<0,1>,
[x86] Fix a pretty horrible bug and inconsistency in the x86 asm parsing (and latent bug in the instruction definitions). This is effectively a revert of r136287 which tried to address a specific and narrow case of immediate operands failing to be accepted by x86 instructions with a pretty heavy hammer: it introduced a new kind of operand that behaved differently. All of that is removed with this commit, but the test cases are both preserved and enhanced. The core problem that r136287 and this commit are trying to handle is that gas accepts both of the following instructions: insertps $192, %xmm0, %xmm1 insertps $-64, %xmm0, %xmm1 These will encode to the same byte sequence, with the immediate occupying an 8-bit entry. The first form was fixed by r136287 but that broke the prior handling of the second form! =[ Ironically, we would still emit the second form in some cases and then be unable to re-assemble the output. The reason why the first instruction failed to be handled is because prior to r136287 the operands ere marked 'i32i8imm' which forces them to be sign-extenable. Clearly, that won't work for 192 in a single byte. However, making thim zero-extended or "unsigned" doesn't really address the core issue either because it breaks negative immediates. The correct fix is to make these operands 'i8imm' reflecting that they can be either signed or unsigned but must be 8-bit immediates. This patch backs out r136287 and then changes those places as well as some others to use 'i8imm' rather than one of the extended variants. Naturally, this broke something else. The custom DAG nodes had to be updated to have a much more accurate type constraint of an i8 node, and a bunch of Pat immediates needed to be specified as i8 values. The fallout didn't end there though. We also then ceased to be able to match the instruction-specific intrinsics to the instructions so modified. Digging, this is because they too used i32 rather than i8 in their signature. So I've also switched those intrinsics to i8 arguments in line with the instructions. In order to make the intrinsic adjustments of course, I also had to add auto upgrading for the intrinsics. I suspect that the intrinsic argument types may have led everything down this rabbit hole. Pretty happy with the result. llvm-svn: 217310
2014-09-06 18:00:01 +08:00
SDTCisVT<2, v4f32>, SDTCisVT<3, i8>]>>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def X86vzmovl : SDNode<"X86ISD::VZEXT_MOVL",
SDTypeProfile<1, 1, [SDTCisSameAs<0,1>]>>;
ZERO_EXTEND/SIGN_EXTEND/TRUNCATE optimization for AVX2 llvm-svn: 155309
2012-04-22 17:39:03 +08:00
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def X86vzload : SDNode<"X86ISD::VZEXT_LOAD", SDTLoad,
give VZEXT_LOAD a memory operand, it now works with segment registers. llvm-svn: 114515
2010-09-22 08:34:38 +08:00
[SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
fix PR11334 - FP_EXTEND only support extending from vectors with matching elements. This results in the scalarization of extending to v2f64 from v2f32, which will be legalized to v4f32 not matching with v2f64. - add X86-specific VFPEXT supproting extending from v4f32 to v2f64. - add BUILD_VECTOR lowering helper to recover back the original extending from v4f32 to v2f64. - test case is enhanced to include different vector width. llvm-svn: 161894
2012-08-15 05:24:47 +08:00
Enable lowering ZERO_EXTEND/ANY_EXTEND to PMOVZX from SSE4.1 llvm-svn: 166486
2012-10-24 01:34:00 +08:00
def X86vzext : SDNode<"X86ISD::VZEXT",
SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisVec<1>,
Improve some x86 type constraints. llvm-svn: 200120
2014-01-26 12:59:39 +08:00
SDTCisInt<0>, SDTCisInt<1>,
SDTCisOpSmallerThanOp<1, 0>]>>;
Enable lowering ZERO_EXTEND/ANY_EXTEND to PMOVZX from SSE4.1 llvm-svn: 166486
2012-10-24 01:34:00 +08:00
def X86vsext : SDNode<"X86ISD::VSEXT",
SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisVec<1>,
Improve some x86 type constraints. llvm-svn: 200120
2014-01-26 12:59:39 +08:00
SDTCisInt<0>, SDTCisInt<1>,
SDTCisOpSmallerThanOp<1, 0>]>>;
Enable lowering ZERO_EXTEND/ANY_EXTEND to PMOVZX from SSE4.1 llvm-svn: 166486
2012-10-24 01:34:00 +08:00
AVX-512: added extend and truncate instructions. llvm-svn: 189580
2013-08-29 19:56:53 +08:00
def X86vtrunc : SDNode<"X86ISD::VTRUNC",
SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisVec<1>,
Improve some x86 type constraints. llvm-svn: 200120
2014-01-26 12:59:39 +08:00
SDTCisInt<0>, SDTCisInt<1>,
SDTCisOpSmallerThanOp<0, 1>]>>;
AVX-512: Added implementation of CONCAT_VECTORS for v8i1 vectors (by Alexey Bader). Added implementation of "truncate" from integer type (i64/i32/i16/i8) to i1. llvm-svn: 197482
2013-12-17 16:33:15 +08:00
def X86trunc : SDNode<"X86ISD::TRUNC",
Improve some x86 type constraints. llvm-svn: 200120
2014-01-26 12:59:39 +08:00
SDTypeProfile<1, 1, [SDTCisInt<0>, SDTCisInt<1>,
SDTCisOpSmallerThanOp<0, 1>]>>;
AVX-512: added extend and truncate instructions. llvm-svn: 189580
2013-08-29 19:56:53 +08:00
def X86vtruncm : SDNode<"X86ISD::VTRUNCM",
SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisVec<1>,
SDTCisInt<0>, SDTCisInt<1>,
Improve some x86 type constraints. llvm-svn: 200120
2014-01-26 12:59:39 +08:00
SDTCisVec<2>, SDTCisInt<2>,
SDTCisOpSmallerThanOp<0, 2>]>>;
fix PR11334 - FP_EXTEND only support extending from vectors with matching elements. This results in the scalarization of extending to v2f64 from v2f32, which will be legalized to v4f32 not matching with v2f64. - add X86-specific VFPEXT supproting extending from v4f32 to v2f64. - add BUILD_VECTOR lowering helper to recover back the original extending from v4f32 to v2f64. - test case is enhanced to include different vector width. llvm-svn: 161894
2012-08-15 05:24:47 +08:00
def X86vfpext : SDNode<"X86ISD::VFPEXT",
SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisVec<1>,
Improve some x86 type constraints. llvm-svn: 200120
2014-01-26 12:59:39 +08:00
SDTCisFP<0>, SDTCisFP<1>,
SDTCisOpSmallerThanOp<1, 0>]>>;
Add support for FP_ROUND from v2f64 to v2f32 - Due to the current matching vector elements constraints in ISD::FP_ROUND, rounding from v2f64 to v4f32 (after legalization from v2f32) is scalarized. Add a customized v2f32 widening to convert it into a target-specific X86ISD::VFPROUND to work around this constraints. llvm-svn: 165631
2012-10-11 00:53:28 +08:00
def X86vfpround: SDNode<"X86ISD::VFPROUND",
SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisVec<1>,
Improve some x86 type constraints. llvm-svn: 200120
2014-01-26 12:59:39 +08:00
SDTCisFP<0>, SDTCisFP<1>,
SDTCisOpSmallerThanOp<0, 1>]>>;
fix PR11334 - FP_EXTEND only support extending from vectors with matching elements. This results in the scalarization of extending to v2f64 from v2f32, which will be legalized to v4f32 not matching with v2f64. - add X86-specific VFPEXT supproting extending from v4f32 to v2f64. - add BUILD_VECTOR lowering helper to recover back the original extending from v4f32 to v2f64. - test case is enhanced to include different vector width. llvm-svn: 161894
2012-08-15 05:24:47 +08:00
Add target specific ISD node types for SSE/AVX vector shuffle instructions and change all the code that used to create intrinsic nodes to create the new nodes instead. llvm-svn: 148664
2012-01-23 03:15:14 +08:00
def X86vshldq : SDNode<"X86ISD::VSHLDQ", SDTIntShiftOp>;
def X86vshrdq : SDNode<"X86ISD::VSRLDQ", SDTIntShiftOp>;
Combine X86 CMPPD and CMPPS node types. Simplifies selection code and pattern matching. llvm-svn: 148670
2012-01-23 07:36:02 +08:00
def X86cmpp : SDNode<"X86ISD::CMPP", SDTX86VFCMP>;
Merge PCMPEQB/PCMPEQW/PCMPEQD/PCMPEQQ and PCMPGTB/PCMPGTW/PCMPGTD/PCMPGTQ X86 ISD node types into only two node types. Simplifying opcode selection and pattern matching. llvm-svn: 148667
2012-01-23 06:42:16 +08:00
def X86pcmpeq : SDNode<"X86ISD::PCMPEQ", SDTIntBinOp, [SDNPCommutative]>;
def X86pcmpgt : SDNode<"X86ISD::PCMPGT", SDTIntBinOp>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
AVX-512: Added CMP and BLEND instructions. Lowering for SETCC. llvm-svn: 188265
2013-08-13 21:24:07 +08:00
def X86IntCmpMask : SDTypeProfile<1, 2,
[SDTCisVec<0>, SDTCisSameAs<1, 2>, SDTCisInt<1>]>;
def X86pcmpeqm : SDNode<"X86ISD::PCMPEQM", X86IntCmpMask, [SDNPCommutative]>;
def X86pcmpgtm : SDNode<"X86ISD::PCMPGTM", X86IntCmpMask>;
AVX-512: Added legal type MVT::i1 and VK1 register for it. Added scalar compare VCMPSS, VCMPSD. Implemented LowerSELECT for scalar FP operations. I replaced FSETCCss, FSETCCsd with one node type FSETCCs. Node extract_vector_elt(v16i1/v8i1, idx) returns an element of type i1. llvm-svn: 197384
2013-12-16 21:52:35 +08:00
def X86CmpMaskCC :
AVX-512: Added all forms of FP compare instructions for KNL and SKX. Added intrinsics for the instructions. CC parameter of the intrinsics was changed from i8 to i32 according to the spec. By Igor Breger (igor.breger@intel.com) llvm-svn: 236714
2015-05-07 19:24:42 +08:00
SDTypeProfile<1, 3, [SDTCisVec<0>, SDTCVecEltisVT<0, i1>,
SDTCisVec<1>, SDTCisSameAs<2, 1>,
SDTCisSameNumEltsAs<0, 1>, SDTCisVT<3, i8>]>;
def X86CmpMaskCCRound :
SDTypeProfile<1, 4, [SDTCisVec<0>,SDTCVecEltisVT<0, i1>,
SDTCisVec<1>, SDTCisSameAs<2, 1>,
SDTCisSameNumEltsAs<0, 1>, SDTCisVT<3, i8>,
SDTCisInt<4>]>;
AVX-512: Added legal type MVT::i1 and VK1 register for it. Added scalar compare VCMPSS, VCMPSD. Implemented LowerSELECT for scalar FP operations. I replaced FSETCCss, FSETCCsd with one node type FSETCCs. Node extract_vector_elt(v16i1/v8i1, idx) returns an element of type i1. llvm-svn: 197384
2013-12-16 21:52:35 +08:00
def X86CmpMaskCCScalar :
SDTypeProfile<1, 3, [SDTCisInt<0>, SDTCisSameAs<1, 2>, SDTCisVT<3, i8>]>;
AVX-512: Added all forms of FP compare instructions for KNL and SKX. Added intrinsics for the instructions. CC parameter of the intrinsics was changed from i8 to i32 according to the spec. By Igor Breger (igor.breger@intel.com) llvm-svn: 236714
2015-05-07 19:24:42 +08:00
def X86cmpm : SDNode<"X86ISD::CMPM", X86CmpMaskCC>;
def X86cmpmRnd : SDNode<"X86ISD::CMPM_RND", X86CmpMaskCCRound>;
def X86cmpmu : SDNode<"X86ISD::CMPMU", X86CmpMaskCC>;
def X86cmpms : SDNode<"X86ISD::FSETCC", X86CmpMaskCCScalar>;
AVX-512: Added CMP and BLEND instructions. Lowering for SETCC. llvm-svn: 188265
2013-08-13 21:24:07 +08:00
Add target specific ISD node types for SSE/AVX vector shuffle instructions and change all the code that used to create intrinsic nodes to create the new nodes instead. llvm-svn: 148664
2012-01-23 03:15:14 +08:00
def X86vshl : SDNode<"X86ISD::VSHL",
SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>,
SDTCisVec<2>]>>;
def X86vsrl : SDNode<"X86ISD::VSRL",
SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>,
SDTCisVec<2>]>>;
def X86vsra : SDNode<"X86ISD::VSRA",
SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>,
SDTCisVec<2>]>>;
def X86vshli : SDNode<"X86ISD::VSHLI", SDTIntShiftOp>;
def X86vsrli : SDNode<"X86ISD::VSRLI", SDTIntShiftOp>;
def X86vsrai : SDNode<"X86ISD::VSRAI", SDTIntShiftOp>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def SDTX86CmpPTest : SDTypeProfile<1, 2, [SDTCisVT<0, i32>,
Add AVX matching patterns to Packed Bit Test intrinsics. Apply the same approach of SSE4.1 ptest intrinsics but create a new x86 node "testp" since AVX introduces vtest{ps}{pd} instructions which set ZF and CF depending on sign bit AND and ANDN of packed floating-point sources. This is slightly different from what the "ptest" does. Tests comming with the other 256 intrinsics tests. llvm-svn: 110744
2010-08-11 07:25:42 +08:00
SDTCisVec<1>,
SDTCisSameAs<2, 1>]>;
AVX-512: added integer "add" and "sub" instructions with saturation for SKX with intrinsics and tests by Asaf Badouh (asaf.badouh@intel.com) llvm-svn: 236418
2015-05-04 20:35:55 +08:00
def X86addus : SDNode<"X86ISD::ADDUS", SDTIntBinOp>;
X86: Add a couple of target-specific dag combines that turn VSELECTS into psubus if possible. We match the pattern "x >= y ? x-y : 0" into "subus x, y" and two special cases if y is a constant. DAGCombiner canonicalizes those so we first have to undo the canonicalization for those cases. The pattern occurs in gzip when the loop vectorizer is enabled. Part of PR14613. llvm-svn: 170273
2012-12-16 00:47:44 +08:00
def X86subus : SDNode<"X86ISD::SUBUS", SDTIntBinOp>;
AVX-512: added integer "add" and "sub" instructions with saturation for SKX with intrinsics and tests by Asaf Badouh (asaf.badouh@intel.com) llvm-svn: 236418
2015-05-04 20:35:55 +08:00
def X86adds : SDNode<"X86ISD::ADDS", SDTIntBinOp>;
def X86subs : SDNode<"X86ISD::SUBS", SDTIntBinOp>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def X86ptest : SDNode<"X86ISD::PTEST", SDTX86CmpPTest>;
Add AVX matching patterns to Packed Bit Test intrinsics. Apply the same approach of SSE4.1 ptest intrinsics but create a new x86 node "testp" since AVX introduces vtest{ps}{pd} instructions which set ZF and CF depending on sign bit AND and ANDN of packed floating-point sources. This is slightly different from what the "ptest" does. Tests comming with the other 256 intrinsics tests. llvm-svn: 110744
2010-08-11 07:25:42 +08:00
def X86testp : SDNode<"X86ISD::TESTP", SDTX86CmpPTest>;
AVX-512 set: added mask operations, lowering BUILD_VECTOR for i1 vector types. Added intrinsics and tests. llvm-svn: 187717
2013-08-05 16:52:21 +08:00
def X86kortest : SDNode<"X86ISD::KORTEST", SDTX86CmpPTest>;
AVX-512: Added legal type MVT::i1 and VK1 register for it. Added scalar compare VCMPSS, VCMPSD. Implemented LowerSELECT for scalar FP operations. I replaced FSETCCss, FSETCCsd with one node type FSETCCs. Node extract_vector_elt(v16i1/v8i1, idx) returns an element of type i1. llvm-svn: 197384
2013-12-16 21:52:35 +08:00
def X86testm : SDNode<"X86ISD::TESTM", SDTypeProfile<1, 2, [SDTCisVec<0>,
AVX-512: Added VPTESTM and VPTESTNM instructions for SKX llvm-svn: 235383
2015-04-21 21:13:46 +08:00
SDTCisVec<1>, SDTCisSameAs<2, 1>,
SDTCVecEltisVT<0, i1>,
SDTCisSameNumEltsAs<0, 1>]>>;
AVX-512: Added intrinsic for cvtph2ps. Added VPTESTNM instruction. Added a pattern to vselect (lit tests will follow). llvm-svn: 200823
2014-02-05 15:05:03 +08:00
def X86testnm : SDNode<"X86ISD::TESTNM", SDTypeProfile<1, 2, [SDTCisVec<0>,
AVX-512: Added VPTESTM and VPTESTNM instructions for SKX llvm-svn: 235383
2015-04-21 21:13:46 +08:00
SDTCisVec<1>, SDTCisSameAs<2, 1>,
SDTCVecEltisVT<0, i1>,
SDTCisSameNumEltsAs<0, 1>]>>;
AVX-512: Added legal type MVT::i1 and VK1 register for it. Added scalar compare VCMPSS, VCMPSD. Implemented LowerSELECT for scalar FP operations. I replaced FSETCCss, FSETCCsd with one node type FSETCCs. Node extract_vector_elt(v16i1/v8i1, idx) returns an element of type i1. llvm-svn: 197384
2013-12-16 21:52:35 +08:00
def X86select : SDNode<"X86ISD::SELECT" , SDTSelect>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
Add target specific node for PMULUDQ. Change patterns to use it and custom lower intrinsics to it. Use it instead of intrinsic to handle 64-bit vector multiplies. llvm-svn: 149807
2012-02-05 11:14:49 +08:00
def X86pmuludq : SDNode<"X86ISD::PMULUDQ",
SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisVec<1>,
SDTCisSameAs<1,2>]>>;
X86: Lower SMUL_LOHI of v4i32 to pmuldq when SSE4.1 is available. llvm-svn: 207318
2014-04-26 22:12:19 +08:00
def X86pmuldq : SDNode<"X86ISD::PMULDQ",
SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisVec<1>,
SDTCisSameAs<1,2>]>>;
Add target specific node for PMULUDQ. Change patterns to use it and custom lower intrinsics to it. Use it instead of intrinsic to handle 64-bit vector multiplies. llvm-svn: 149807
2012-02-05 11:14:49 +08:00
This is the first step towards refactoring the x86 vector shuffle code. The general idea here is to have a group of x86 target specific nodes which are going to be selected during lowering and then directly matched in isel. The commit includes the addition of those specific nodes and a *bunch* of patterns, and incrementally we're going to switch between them and what we have right now. Both the patterns and target specific nodes can change as we move forward with this work. llvm-svn: 111691
2010-08-21 06:55:05 +08:00
// Specific shuffle nodes - At some point ISD::VECTOR_SHUFFLE will always get
// translated into one of the target nodes below during lowering.
// Note: this is a work in progress...
def SDTShuff1Op : SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisSameAs<0,1>]>;
def SDTShuff2Op : SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>,
SDTCisSameAs<0,2>]>;
AVX-512: Added VPERM* instructons and MOV* zmm-to-zmm instructions. Added a test for shuffles using VPERM. llvm-svn: 188147
2013-08-11 15:55:09 +08:00
def SDTShuff3Op : SDTypeProfile<1, 3, [SDTCisVec<0>, SDTCisSameAs<0,1>,
SDTCisSameAs<0,2>, SDTCisSameAs<0,3>]>;
This is the first step towards refactoring the x86 vector shuffle code. The general idea here is to have a group of x86 target specific nodes which are going to be selected during lowering and then directly matched in isel. The commit includes the addition of those specific nodes and a *bunch* of patterns, and incrementally we're going to switch between them and what we have right now. Both the patterns and target specific nodes can change as we move forward with this work. llvm-svn: 111691
2010-08-21 06:55:05 +08:00
[x86] Teach the AVX1 path of the new vector shuffle lowering one more trick that I missed. VPERMILPS has a non-immediate memory operand mode that allows it to do asymetric shuffles in the two 128-bit lanes. Use this rather than two shuffles and a blend. However, it turns out the variable shuffle path to VPERMILPS (and VPERMILPD, although that one offers no functional differenc from the immediate operand other than variability) wasn't even plumbed through codegen. Do such plumbing so that we can reasonably emit a variable-masked VPERMILP instruction. Also plumb basic comment parsing and printing through so that the tests are reasonable. There are still a few tests which don't show the shuffle pattern. These are tests with undef lanes. I'll teach the shuffle decoding and printing to handle undef mask entries in a follow-up. I've looked at the masks and they seem reasonable. llvm-svn: 218300
2014-09-23 18:08:29 +08:00
def SDTShuff2OpM : SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>,
SDTCisVec<2>]>;
This is the first step towards refactoring the x86 vector shuffle code. The general idea here is to have a group of x86 target specific nodes which are going to be selected during lowering and then directly matched in isel. The commit includes the addition of those specific nodes and a *bunch* of patterns, and incrementally we're going to switch between them and what we have right now. Both the patterns and target specific nodes can change as we move forward with this work. llvm-svn: 111691
2010-08-21 06:55:05 +08:00
def SDTShuff2OpI : SDTypeProfile<1, 2, [SDTCisVec<0>,
SDTCisSameAs<0,1>, SDTCisInt<2>]>;
def SDTShuff3OpI : SDTypeProfile<1, 3, [SDTCisVec<0>, SDTCisSameAs<0,1>,
SDTCisSameAs<0,2>, SDTCisInt<3>]>;
AVX-512 set: Added BROADCAST instructions with lowering logic and a test. llvm-svn: 187884
2013-08-07 20:34:55 +08:00
def SDTVBroadcast : SDTypeProfile<1, 1, [SDTCisVec<0>]>;
def SDTVBroadcastm : SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisVec<1>]>;
Reapply 154396 after fixing a test. Original message: Modify the code that lowers shuffles to blends from using blendvXX to vblendXX. blendV uses a register for the selection while Vblend uses an immediate. On sandybridge they still have the same latency and execute on the same execution ports. llvm-svn: 154483
2012-04-11 14:40:27 +08:00
def SDTBlend : SDTypeProfile<1, 3, [SDTCisVec<0>, SDTCisSameAs<0,1>,
[x86] Fix a pretty horrible bug and inconsistency in the x86 asm parsing (and latent bug in the instruction definitions). This is effectively a revert of r136287 which tried to address a specific and narrow case of immediate operands failing to be accepted by x86 instructions with a pretty heavy hammer: it introduced a new kind of operand that behaved differently. All of that is removed with this commit, but the test cases are both preserved and enhanced. The core problem that r136287 and this commit are trying to handle is that gas accepts both of the following instructions: insertps $192, %xmm0, %xmm1 insertps $-64, %xmm0, %xmm1 These will encode to the same byte sequence, with the immediate occupying an 8-bit entry. The first form was fixed by r136287 but that broke the prior handling of the second form! =[ Ironically, we would still emit the second form in some cases and then be unable to re-assemble the output. The reason why the first instruction failed to be handled is because prior to r136287 the operands ere marked 'i32i8imm' which forces them to be sign-extenable. Clearly, that won't work for 192 in a single byte. However, making thim zero-extended or "unsigned" doesn't really address the core issue either because it breaks negative immediates. The correct fix is to make these operands 'i8imm' reflecting that they can be either signed or unsigned but must be 8-bit immediates. This patch backs out r136287 and then changes those places as well as some others to use 'i8imm' rather than one of the extended variants. Naturally, this broke something else. The custom DAG nodes had to be updated to have a much more accurate type constraint of an i8 node, and a bunch of Pat immediates needed to be specified as i8 values. The fallout didn't end there though. We also then ceased to be able to match the instruction-specific intrinsics to the instructions so modified. Digging, this is because they too used i32 rather than i8 in their signature. So I've also switched those intrinsics to i8 arguments in line with the instructions. In order to make the intrinsic adjustments of course, I also had to add auto upgrading for the intrinsics. I suspect that the intrinsic argument types may have led everything down this rabbit hole. Pretty happy with the result. llvm-svn: 217310
2014-09-06 18:00:01 +08:00
SDTCisSameAs<1,2>, SDTCisVT<3, i8>]>;
Added FMA functionality to X86 target. llvm-svn: 161110
2012-08-01 20:06:00 +08:00
AVX-512: Added support for FP instructions with embedded rounding mode. By Asaf Badouh <asaf.badouh@intel.com> llvm-svn: 229645
2015-02-18 15:59:20 +08:00
def SDTFPBinOpRound : SDTypeProfile<1, 3, [ // fadd_round, fmul_round, etc.
SDTCisSameAs<0, 1>, SDTCisSameAs<0, 2>, SDTCisFP<0>, SDTCisInt<3>]>;
Added FMA functionality to X86 target. llvm-svn: 161110
2012-08-01 20:06:00 +08:00
def SDTFma : SDTypeProfile<1, 3, [SDTCisSameAs<0,1>,
SDTCisSameAs<1,2>, SDTCisSameAs<1,3>]>;
AVX-512: Added FMA intrinsics with rounding mode By Asaf Badouh and Elena Demikhovsky Added special nodes for rounding: FMADD_RND, FMSUB_RND.. It will prevent merge between nodes with rounding and other standard nodes. llvm-svn: 227303
2015-01-28 18:21:27 +08:00
def SDTFmaRound : SDTypeProfile<1, 4, [SDTCisSameAs<0,1>,
SDTCisSameAs<1,2>, SDTCisSameAs<1,3>, SDTCisInt<4>]>;
AVX-512: Intrinsics for ERI 3 instructions: vrcp28, vrsqrt28, vexp2, only vector forms. Intrinsics include SAE (Suppres All Exceptions) parameter. http://reviews.llvm.org/D6214 llvm-svn: 221774
2014-11-12 15:31:03 +08:00
def STDFp1SrcRm : SDTypeProfile<1, 2, [SDTCisSameAs<0,1>,
SDTCisVec<0>, SDTCisInt<2>]>;
AVX-512: Scalar ERI intrinsics including SAE mode and memory operand. Added AVX512_maskable_scalar template, that should cover all scalar instructions in the future. The main difference between AVX512_maskable_scalar<> and AVX512_maskable<> is using X86select instead of vselect. I need it, because I can't create vselect node for MVT::i1 mask for scalar instruction. http://reviews.llvm.org/D6378 llvm-svn: 222820
2014-11-26 18:46:49 +08:00
def STDFp2SrcRm : SDTypeProfile<1, 3, [SDTCisSameAs<0,1>,
SDTCisVec<0>, SDTCisInt<3>]>;
AVX-512: recommitted 229837 + bugfix + test llvm-svn: 230223
2015-02-23 23:12:31 +08:00
def STDFp3SrcRm : SDTypeProfile<1, 4, [SDTCisSameAs<0,1>,
SDTCisVec<0>, SDTCisInt<3>, SDTCisInt<4>]>;
Introduce matching patterns for vbroadcast AVX instruction. The idea is to match splats in the form (splat (scalar_to_vector (load ...))) whenever the load can be folded. All the logic and instruction emission is working but because of PR8156, there are no ways to match loads, cause they can never be folded for splats. Thus, the tests are XFAILed, but I've tested and exercised all the logic using a relaxed version for checking the foldable loads, as if the bug was already fixed. This should work out of the box once PR8156 gets fixed since MayFoldLoad will work as expected. llvm-svn: 137810
2011-08-17 10:29:19 +08:00
Fix inconsistent usage of PALIGN and PALIGNR when referring to the same instruction. llvm-svn: 173667
2013-01-28 14:48:25 +08:00
def X86PAlignr : SDNode<"X86ISD::PALIGNR", SDTShuff3OpI>;
[X86] Separate DAG node for valign and palignr They have different semantics (valign is interlane while palingr is intralane) and palingr is still needed even in the AVX512 context. According to the latest spec AVX512BW provides these. llvm-svn: 214887
2014-08-06 01:22:55 +08:00
def X86VAlign : SDNode<"X86ISD::VALIGN", SDTShuff3OpI>;
This is the first step towards refactoring the x86 vector shuffle code. The general idea here is to have a group of x86 target specific nodes which are going to be selected during lowering and then directly matched in isel. The commit includes the addition of those specific nodes and a *bunch* of patterns, and incrementally we're going to switch between them and what we have right now. Both the patterns and target specific nodes can change as we move forward with this work. llvm-svn: 111691
2010-08-21 06:55:05 +08:00
def X86PShufd : SDNode<"X86ISD::PSHUFD", SDTShuff2OpI>;
def X86PShufhw : SDNode<"X86ISD::PSHUFHW", SDTShuff2OpI>;
def X86PShuflw : SDNode<"X86ISD::PSHUFLW", SDTShuff2OpI>;
Merge X86 SHUFPS and SHUFPD node types. llvm-svn: 147394
2012-01-01 07:50:21 +08:00
def X86Shufp : SDNode<"X86ISD::SHUFP", SDTShuff3OpI>;
This is the first step towards refactoring the x86 vector shuffle code. The general idea here is to have a group of x86 target specific nodes which are going to be selected during lowering and then directly matched in isel. The commit includes the addition of those specific nodes and a *bunch* of patterns, and incrementally we're going to switch between them and what we have right now. Both the patterns and target specific nodes can change as we move forward with this work. llvm-svn: 111691
2010-08-21 06:55:05 +08:00
def X86Movddup : SDNode<"X86ISD::MOVDDUP", SDTShuff1Op>;
def X86Movshdup : SDNode<"X86ISD::MOVSHDUP", SDTShuff1Op>;
def X86Movsldup : SDNode<"X86ISD::MOVSLDUP", SDTShuff1Op>;
def X86Movsd : SDNode<"X86ISD::MOVSD", SDTShuff2Op>;
def X86Movss : SDNode<"X86ISD::MOVSS", SDTShuff2Op>;
def X86Movlhps : SDNode<"X86ISD::MOVLHPS", SDTShuff2Op>;
def X86Movlhpd : SDNode<"X86ISD::MOVLHPD", SDTShuff2Op>;
Use MOVLHPS and MOVHLPS x86 nodes whenever possible. Also remove some useless nodes llvm-svn: 112642
2010-09-01 05:15:21 +08:00
def X86Movhlps : SDNode<"X86ISD::MOVHLPS", SDTShuff2Op>;
This is the first step towards refactoring the x86 vector shuffle code. The general idea here is to have a group of x86 target specific nodes which are going to be selected during lowering and then directly matched in isel. The commit includes the addition of those specific nodes and a *bunch* of patterns, and incrementally we're going to switch between them and what we have right now. Both the patterns and target specific nodes can change as we move forward with this work. llvm-svn: 111691
2010-08-21 06:55:05 +08:00
Use movlps, movlpd, movss and movsd specific nodes instead of pattern matching with movlp pattern fragment llvm-svn: 112694
2010-09-01 13:08:25 +08:00
def X86Movlps : SDNode<"X86ISD::MOVLPS", SDTShuff2Op>;
def X86Movlpd : SDNode<"X86ISD::MOVLPD", SDTShuff2Op>;
This is the first step towards refactoring the x86 vector shuffle code. The general idea here is to have a group of x86 target specific nodes which are going to be selected during lowering and then directly matched in isel. The commit includes the addition of those specific nodes and a *bunch* of patterns, and incrementally we're going to switch between them and what we have right now. Both the patterns and target specific nodes can change as we move forward with this work. llvm-svn: 111691
2010-08-21 06:55:05 +08:00
[x86] Make the x86 PACKSSWB, PACKSSDW, PACKUSWB, and PACKUSDW instructions available as synthetic SDNodes PACKSS and PACKUS that will select to the correct instruction variants based on the return type. This allows us to use these rather important instructions when lowering vector shuffles. Also moves the relevant instruction definitions to be split out from the fully generic multiclasses to allow them to match these new SDNodes in the same way that the UNPCK instructions do. No functionality should actually be changed here. llvm-svn: 211332
2014-06-20 09:05:28 +08:00
def SDTPack : SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisVec<1>, SDTCisSameAs<2, 1>]>;
def X86Packss : SDNode<"X86ISD::PACKSS", SDTPack>;
def X86Packus : SDNode<"X86ISD::PACKUS", SDTPack>;
Merge floating point and integer UNPCK X86ISD node types. llvm-svn: 145926
2011-12-06 16:21:25 +08:00
def X86Unpckl : SDNode<"X86ISD::UNPCKL", SDTShuff2Op>;
def X86Unpckh : SDNode<"X86ISD::UNPCKH", SDTShuff2Op>;
This is the first step towards refactoring the x86 vector shuffle code. The general idea here is to have a group of x86 target specific nodes which are going to be selected during lowering and then directly matched in isel. The commit includes the addition of those specific nodes and a *bunch* of patterns, and incrementally we're going to switch between them and what we have right now. Both the patterns and target specific nodes can change as we move forward with this work. llvm-svn: 111691
2010-08-21 06:55:05 +08:00
[x86] Teach the AVX1 path of the new vector shuffle lowering one more trick that I missed. VPERMILPS has a non-immediate memory operand mode that allows it to do asymetric shuffles in the two 128-bit lanes. Use this rather than two shuffles and a blend. However, it turns out the variable shuffle path to VPERMILPS (and VPERMILPD, although that one offers no functional differenc from the immediate operand other than variability) wasn't even plumbed through codegen. Do such plumbing so that we can reasonably emit a variable-masked VPERMILP instruction. Also plumb basic comment parsing and printing through so that the tests are reasonable. There are still a few tests which don't show the shuffle pattern. These are tests with undef lanes. I'll teach the shuffle decoding and printing to handle undef mask entries in a follow-up. I've looked at the masks and they seem reasonable. llvm-svn: 218300
2014-09-23 18:08:29 +08:00
def X86VPermilpv : SDNode<"X86ISD::VPERMILPV", SDTShuff2OpM>;
[x86] Rename X86ISD::VPERMILP to X86ISD::VPERMILPI (and the same for the td pattern). Currently we only model the immediate operand variation of VPERMILPS and VPERMILPD, we should make that clear in the pseudos used. Will be adding support for the variable mask variant in my next commit. llvm-svn: 218282
2014-09-23 06:29:42 +08:00
def X86VPermilpi : SDNode<"X86ISD::VPERMILPI", SDTShuff2OpI>;
def X86VPermv : SDNode<"X86ISD::VPERMV", SDTShuff2Op>;
def X86VPermi : SDNode<"X86ISD::VPERMI", SDTShuff2OpI>;
def X86VPermv3 : SDNode<"X86ISD::VPERMV3", SDTShuff3Op>;
AVX-512: added VPERM2D VPERM2Q VPERM2PS VPERM2PD instructions, they give better sequences than VPERMI llvm-svn: 199893
2014-01-23 22:27:26 +08:00
def X86VPermiv3 : SDNode<"X86ISD::VPERMIV3", SDTShuff3Op>;
Add support for 256-bit versions of VPERMIL instruction. This is a new instruction introduced in AVX, which can operate on 128 and 256-bit vectors. It considers a 256-bit vector as two independent 128-bit lanes. It can permute any 32 or 64 elements inside a lane, and restricts the second lane to have the same permutation of the first one. With the improved splat support introduced early today, adding codegen for this instruction enable more efficient 256-bit code: Instead of: vextractf128 $0, %ymm0, %xmm0 punpcklbw %xmm0, %xmm0 punpckhbw %xmm0, %xmm0 vinsertf128 $0, %xmm0, %ymm0, %ymm1 vinsertf128 $1, %xmm0, %ymm1, %ymm0 vextractf128 $1, %ymm0, %xmm1 shufps $1, %xmm1, %xmm1 movss %xmm1, 28(%rsp) movss %xmm1, 24(%rsp) movss %xmm1, 20(%rsp) movss %xmm1, 16(%rsp) vextractf128 $0, %ymm0, %xmm0 shufps $1, %xmm0, %xmm0 movss %xmm0, 12(%rsp) movss %xmm0, 8(%rsp) movss %xmm0, 4(%rsp) movss %xmm0, (%rsp) vmovaps (%rsp), %ymm0 We get: vextractf128 $0, %ymm0, %xmm0 punpcklbw %xmm0, %xmm0 punpckhbw %xmm0, %xmm0 vinsertf128 $0, %xmm0, %ymm0, %ymm1 vinsertf128 $1, %xmm0, %ymm1, %ymm0 vpermilps $85, %ymm0, %ymm0 llvm-svn: 135662
2011-07-21 09:55:47 +08:00
Merge VPERM2F128/VPERM2I128 ISD node types. llvm-svn: 145485
2011-11-30 15:47:51 +08:00
def X86VPerm2x128 : SDNode<"X86ISD::VPERM2X128", SDTShuff3OpI>;
The VPERM2F128 is a AVX instruction which permutes between two 256-bit vectors. It operates on 128-bit elements instead of regular scalar types. Recognize shuffles that are suitable for VPERM2F128 and teach the x86 legalizer how to handle them. llvm-svn: 137519
2011-08-13 05:48:26 +08:00
AVX-512: Added VBROADCASTF64X4, VBROADCASTF64X2, VBROADCASTI32X8, and other instructions from this set Added encoding tests. llvm-svn: 237557
2015-05-18 14:42:57 +08:00
def X86SubVBroadcast : SDNode<"X86ISD::SUBV_BROADCAST",
SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisVec<1>,
SDTCisSubVecOfVec<1, 0>]>, []>;
Introduce matching patterns for vbroadcast AVX instruction. The idea is to match splats in the form (splat (scalar_to_vector (load ...))) whenever the load can be folded. All the logic and instruction emission is working but because of PR8156, there are no ways to match loads, cause they can never be folded for splats. Thus, the tests are XFAILed, but I've tested and exercised all the logic using a relaxed version for checking the foldable loads, as if the bug was already fixed. This should work out of the box once PR8156 gets fixed since MayFoldLoad will work as expected. llvm-svn: 137810
2011-08-17 10:29:19 +08:00
def X86VBroadcast : SDNode<"X86ISD::VBROADCAST", SDTVBroadcast>;
AVX-512: implemented extractelement with variable index. Added parsing of mask register and "zeroing" semantic, like {%k1} {z}. llvm-svn: 190595
2013-09-12 16:55:00 +08:00
def X86Vinsert : SDNode<"X86ISD::VINSERT", SDTypeProfile<1, 3,
[SDTCisSameAs<0, 1>, SDTCisPtrTy<3>]>, []>;
AVX-512: Fixed extract_vector_elt for v16i1 and v8i1 vectors. llvm-svn: 201066
2014-02-10 15:02:39 +08:00
def X86Vextract : SDNode<"X86ISD::VEXTRACT", SDTypeProfile<1, 2,
[SDTCisVec<1>, SDTCisPtrTy<2>]>, []>;
Introduce matching patterns for vbroadcast AVX instruction. The idea is to match splats in the form (splat (scalar_to_vector (load ...))) whenever the load can be folded. All the logic and instruction emission is working but because of PR8156, there are no ways to match loads, cause they can never be folded for splats. Thus, the tests are XFAILed, but I've tested and exercised all the logic using a relaxed version for checking the foldable loads, as if the bug was already fixed. This should work out of the box once PR8156 gets fixed since MayFoldLoad will work as expected. llvm-svn: 137810
2011-08-17 10:29:19 +08:00
Simplified BLEND pattern matching for shuffles. Generate VPBLENDD for AVX2 and VPBLENDW for v16i16 type on AVX2. llvm-svn: 169366
2012-12-05 17:24:57 +08:00
def X86Blendi : SDNode<"X86ISD::BLENDI", SDTBlend>;
[x86] Start fixing our emission of ADDSUBPS and ADDSUBPD instructions by introducing a synthetic X86 ISD node representing this generic operation. The relevant patterns for mapping these nodes into the concrete instructions are also added, and a gnarly bit of C++ code in the target-specific DAG combiner is replaced with simple code emitting this primitive. The next step is to generically combine blends of adds and subs into this node so that we can drop the reliance on an SSE4.1 ISD node (BLENDI) when matching an SSE3 feature (ADDSUB). llvm-svn: 217819
2014-09-16 04:09:47 +08:00
def X86Addsub : SDNode<"X86ISD::ADDSUB", SDTFPBinOp>;
AVX-512: Added support for FP instructions with embedded rounding mode. By Asaf Badouh <asaf.badouh@intel.com> llvm-svn: 229645
2015-02-18 15:59:20 +08:00
def X86faddRnd : SDNode<"X86ISD::FADD_RND", SDTFPBinOpRound>;
def X86fsubRnd : SDNode<"X86ISD::FSUB_RND", SDTFPBinOpRound>;
def X86fmulRnd : SDNode<"X86ISD::FMUL_RND", SDTFPBinOpRound>;
def X86fdivRnd : SDNode<"X86ISD::FDIV_RND", SDTFPBinOpRound>;
AVX-512: Added SKX instructions and intrinsics: {add/sub/mul/div/} x {ps/pd} x {128/256} 2. max/min with sae By Asaf Badouh (asaf.badouh@intel.com) llvm-svn: 236971
2015-05-11 14:05:05 +08:00
def X86fmaxRnd : SDNode<"X86ISD::FMAX_RND", SDTFPBinOpRound>;
def X86fminRnd : SDNode<"X86ISD::FMIN_RND", SDTFPBinOpRound>;
AVX-512: Added support for FP instructions with embedded rounding mode. By Asaf Badouh <asaf.badouh@intel.com> llvm-svn: 229645
2015-02-18 15:59:20 +08:00
Added FMA functionality to X86 target. llvm-svn: 161110
2012-08-01 20:06:00 +08:00
def X86Fmadd : SDNode<"X86ISD::FMADD", SDTFma>;
def X86Fnmadd : SDNode<"X86ISD::FNMADD", SDTFma>;
def X86Fmsub : SDNode<"X86ISD::FMSUB", SDTFma>;
def X86Fnmsub : SDNode<"X86ISD::FNMSUB", SDTFma>;
Convert FMA4 patterns to use target specific nodes instead of intrinsics to align with FMA3. llvm-svn: 162829
2012-08-29 15:18:25 +08:00
def X86Fmaddsub : SDNode<"X86ISD::FMADDSUB", SDTFma>;
def X86Fmsubadd : SDNode<"X86ISD::FMSUBADD", SDTFma>;
Reapply 154396 after fixing a test. Original message: Modify the code that lowers shuffles to blends from using blendvXX to vblendXX. blendV uses a register for the selection while Vblend uses an immediate. On sandybridge they still have the same latency and execute on the same execution ports. llvm-svn: 154483
2012-04-11 14:40:27 +08:00
AVX-512: Added FMA intrinsics with rounding mode By Asaf Badouh and Elena Demikhovsky Added special nodes for rounding: FMADD_RND, FMSUB_RND.. It will prevent merge between nodes with rounding and other standard nodes. llvm-svn: 227303
2015-01-28 18:21:27 +08:00
def X86FmaddRnd : SDNode<"X86ISD::FMADD_RND", SDTFmaRound>;
def X86FnmaddRnd : SDNode<"X86ISD::FNMADD_RND", SDTFmaRound>;
def X86FmsubRnd : SDNode<"X86ISD::FMSUB_RND", SDTFmaRound>;
def X86FnmsubRnd : SDNode<"X86ISD::FNMSUB_RND", SDTFmaRound>;
def X86FmaddsubRnd : SDNode<"X86ISD::FMADDSUB_RND", SDTFmaRound>;
def X86FmsubaddRnd : SDNode<"X86ISD::FMSUBADD_RND", SDTFmaRound>;
AVX-512: Intrinsics for ERI 3 instructions: vrcp28, vrsqrt28, vexp2, only vector forms. Intrinsics include SAE (Suppres All Exceptions) parameter. http://reviews.llvm.org/D6214 llvm-svn: 221774
2014-11-12 15:31:03 +08:00
def X86rsqrt28 : SDNode<"X86ISD::RSQRT28", STDFp1SrcRm>;
def X86rcp28 : SDNode<"X86ISD::RCP28", STDFp1SrcRm>;
AVX-512: Scalar ERI intrinsics including SAE mode and memory operand. Added AVX512_maskable_scalar template, that should cover all scalar instructions in the future. The main difference between AVX512_maskable_scalar<> and AVX512_maskable<> is using X86select instead of vselect. I need it, because I can't create vselect node for MVT::i1 mask for scalar instruction. http://reviews.llvm.org/D6378 llvm-svn: 222820
2014-11-26 18:46:49 +08:00
def X86exp2 : SDNode<"X86ISD::EXP2", STDFp1SrcRm>;
def X86rsqrt28s : SDNode<"X86ISD::RSQRT28", STDFp2SrcRm>;
def X86rcp28s : SDNode<"X86ISD::RCP28", STDFp2SrcRm>;
AVX-512: recommitted 229837 + bugfix + test llvm-svn: 230223
2015-02-23 23:12:31 +08:00
def X86RndScale : SDNode<"X86ISD::RNDSCALE", STDFp3SrcRm>;
AVX-512: Intrinsics for ERI 3 instructions: vrcp28, vrsqrt28, vexp2, only vector forms. Intrinsics include SAE (Suppres All Exceptions) parameter. http://reviews.llvm.org/D6214 llvm-svn: 221774
2014-11-12 15:31:03 +08:00
Implement proper handling for pcmpistri/pcmpestri intrinsics. Requires custom handling in DAGISelToDAG due to limitations in TableGen's implicit def handling. Fixes PR11305. llvm-svn: 161318
2012-08-06 14:22:36 +08:00
def SDT_PCMPISTRI : SDTypeProfile<2, 3, [SDTCisVT<0, i32>, SDTCisVT<1, i32>,
SDTCisVT<2, v16i8>, SDTCisVT<3, v16i8>,
SDTCisVT<4, i8>]>;
def SDT_PCMPESTRI : SDTypeProfile<2, 5, [SDTCisVT<0, i32>, SDTCisVT<1, i32>,
SDTCisVT<2, v16i8>, SDTCisVT<3, i32>,
SDTCisVT<4, v16i8>, SDTCisVT<5, i32>,
SDTCisVT<6, i8>]>;
def X86pcmpistri : SDNode<"X86ISD::PCMPISTRI", SDT_PCMPISTRI>;
def X86pcmpestri : SDNode<"X86ISD::PCMPESTRI", SDT_PCMPESTRI>;
AVX-512: Added all forms of COMPRESS instruction + intrinsics + tests llvm-svn: 224019
2014-12-11 23:02:24 +08:00
def X86compress: SDNode<"X86ISD::COMPRESS", SDTypeProfile<1, 3,
[SDTCisSameAs<0, 2>, SDTCisSameAs<0, 3>,
SDTCisVec<3>, SDTCisVec<1>, SDTCisInt<1>]>, []>;
AVX-512: Added EXPAND instructions and intrinsics. llvm-svn: 224241
2014-12-15 18:03:52 +08:00
def X86expand : SDNode<"X86ISD::EXPAND", SDTypeProfile<1, 3,
[SDTCisSameAs<0, 3>,
SDTCisVec<3>, SDTCisVec<1>, SDTCisInt<1>]>, []>;
AVX-512: Added all forms of COMPRESS instruction + intrinsics + tests llvm-svn: 224019
2014-12-11 23:02:24 +08:00
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
//===----------------------------------------------------------------------===//
// SSE Complex Patterns
//===----------------------------------------------------------------------===//
// These are 'extloads' from a scalar to the low element of a vector, zeroing
// the top elements. These are used for the SSE 'ss' and 'sd' instruction
// forms.
def sse_load_f32 : ComplexPattern<v4f32, 5, "SelectScalarSSELoad", [],
fix a long standing wart: all the ComplexPattern's were being passed the root of the match, even though only a few patterns actually needed this (one in X86, several in ARM [which should be refactored anyway], and some in CellSPU that I don't feel like detangling). Instead of requiring all ComplexPatterns to take the dead root, have targets opt into getting the root by putting SDNPWantRoot on the ComplexPattern. llvm-svn: 114471
2010-09-22 04:31:19 +08:00
[SDNPHasChain, SDNPMayLoad, SDNPMemOperand,
SDNPWantRoot]>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def sse_load_f64 : ComplexPattern<v2f64, 5, "SelectScalarSSELoad", [],
fix a long standing wart: all the ComplexPattern's were being passed the root of the match, even though only a few patterns actually needed this (one in X86, several in ARM [which should be refactored anyway], and some in CellSPU that I don't feel like detangling). Instead of requiring all ComplexPatterns to take the dead root, have targets opt into getting the root by putting SDNPWantRoot on the ComplexPattern. llvm-svn: 114471
2010-09-22 04:31:19 +08:00
[SDNPHasChain, SDNPMayLoad, SDNPMemOperand,
SDNPWantRoot]>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def ssmem : Operand<v4f32> {
let PrintMethod = "printf32mem";
let MIOperandInfo = (ops ptr_rc, i8imm, ptr_rc_nosp, i32imm, i8imm);
Make sure x86 instructions using ssmem/sdmem operand types are only able to parse memory operands of the proper size in Intel syntax. Primarily affects some of sse cvt instructions. llvm-svn: 189206
2013-08-26 08:39:04 +08:00
let ParserMatchClass = X86Mem32AsmOperand;
Port operand types for ARM and X86 over from EDIS to the .td files. llvm-svn: 135198
2011-07-15 05:47:22 +08:00
let OperandType = "OPERAND_MEMORY";
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
}
def sdmem : Operand<v2f64> {
let PrintMethod = "printf64mem";
let MIOperandInfo = (ops ptr_rc, i8imm, ptr_rc_nosp, i32imm, i8imm);
Make sure x86 instructions using ssmem/sdmem operand types are only able to parse memory operands of the proper size in Intel syntax. Primarily affects some of sse cvt instructions. llvm-svn: 189206
2013-08-26 08:39:04 +08:00
let ParserMatchClass = X86Mem64AsmOperand;
Port operand types for ARM and X86 over from EDIS to the .td files. llvm-svn: 135198
2011-07-15 05:47:22 +08:00
let OperandType = "OPERAND_MEMORY";
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
}
//===----------------------------------------------------------------------===//
// SSE pattern fragments
//===----------------------------------------------------------------------===//
Add comments to some pattern fragments in x86 llvm-svn: 111041
2010-08-14 04:39:01 +08:00
// 128-bit load pattern fragments
Add comments near load pattern fragments indicating that all integer vector loads are promoted to v2i64 or v4i64 so that no one tries to reintroduce pattern fragments for other types. llvm-svn: 148771
2012-01-24 11:03:17 +08:00
// NOTE: all 128-bit integer vector loads are promoted to v2i64
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def loadv4f32 : PatFrag<(ops node:$ptr), (v4f32 (load node:$ptr))>;
def loadv2f64 : PatFrag<(ops node:$ptr), (v2f64 (load node:$ptr))>;
def loadv2i64 : PatFrag<(ops node:$ptr), (v2i64 (load node:$ptr))>;
Add comments to some pattern fragments in x86 llvm-svn: 111041
2010-08-14 04:39:01 +08:00
// 256-bit load pattern fragments
Add comments near load pattern fragments indicating that all integer vector loads are promoted to v2i64 or v4i64 so that no one tries to reintroduce pattern fragments for other types. llvm-svn: 148771
2012-01-24 11:03:17 +08:00
// NOTE: all 256-bit integer vector loads are promoted to v4i64
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def loadv8f32 : PatFrag<(ops node:$ptr), (v8f32 (load node:$ptr))>;
def loadv4f64 : PatFrag<(ops node:$ptr), (v4f64 (load node:$ptr))>;
def loadv4i64 : PatFrag<(ops node:$ptr), (v4i64 (load node:$ptr))>;
AVX-512: Added VPERM* instructons and MOV* zmm-to-zmm instructions. Added a test for shuffles using VPERM. llvm-svn: 188147
2013-08-11 15:55:09 +08:00
// 512-bit load pattern fragments
def loadv16f32 : PatFrag<(ops node:$ptr), (v16f32 (load node:$ptr))>;
def loadv8f64 : PatFrag<(ops node:$ptr), (v8f64 (load node:$ptr))>;
[SKX] Enabling load/store instructions: encoding Instructions: VMOVAPD, VMOVAPS, VMOVDQA8, VMOVDQA16, VMOVDQA32,VMOVDQA64, VMOVDQU8, VMOVDQU16, VMOVDQU32,VMOVDQU64, VMOVUPD, VMOVUPS, Reviewed by Elena Demikhovsky <elena.demikhovsky@intel.com> llvm-svn: 214719
2014-08-04 22:35:15 +08:00
def loadv64i8 : PatFrag<(ops node:$ptr), (v64i8 (load node:$ptr))>;
def loadv32i16 : PatFrag<(ops node:$ptr), (v32i16 (load node:$ptr))>;
AVX-512: aligned / unaligned load and store for 512-bit integer vectors. llvm-svn: 193156
2013-10-22 17:19:28 +08:00
def loadv16i32 : PatFrag<(ops node:$ptr), (v16i32 (load node:$ptr))>;
AVX-512: Added VPERM* instructons and MOV* zmm-to-zmm instructions. Added a test for shuffles using VPERM. llvm-svn: 188147
2013-08-11 15:55:09 +08:00
def loadv8i64 : PatFrag<(ops node:$ptr), (v8i64 (load node:$ptr))>;
// 128-/256-/512-bit extload pattern fragments
Enhance PR11334 fix to support extload from v2f32/v4f32 - Fix an remaining issue of PR11674 as well llvm-svn: 163528
2012-09-11 02:33:51 +08:00
def extloadv2f32 : PatFrag<(ops node:$ptr), (v2f64 (extloadvf32 node:$ptr))>;
def extloadv4f32 : PatFrag<(ops node:$ptr), (v4f64 (extloadvf32 node:$ptr))>;
AVX-512: Added VPERM* instructons and MOV* zmm-to-zmm instructions. Added a test for shuffles using VPERM. llvm-svn: 188147
2013-08-11 15:55:09 +08:00
def extloadv8f32 : PatFrag<(ops node:$ptr), (v8f64 (extloadvf32 node:$ptr))>;
Enhance PR11334 fix to support extload from v2f32/v4f32 - Fix an remaining issue of PR11674 as well llvm-svn: 163528
2012-09-11 02:33:51 +08:00
prevent folding a scalar FP load into a packed logical FP instruction (PR22371) Change the memory operands in sse12_fp_packed_scalar_logical_alias from scalars to vectors. That's what the hardware packed logical FP instructions define: 128-bit memory operands. There are no scalar versions of these instructions...because this is x86. Generating the wrong code (folding a scalar load into a 128-bit load) is still possible using the peephole optimization pass and the load folding tables. We won't completely solve this bug until we either fix the lowering in fabs/fneg/fcopysign and any other places where scalar FP logic is created or fix the load folding in foldMemoryOperandImpl() to make sure it isn't changing the size of the load. Differential Revision: http://reviews.llvm.org/D7474 llvm-svn: 229531
2015-02-18 04:08:21 +08:00
// These are needed to match a scalar load that is used in a vector-only
// math instruction such as the FP logical ops: andps, andnps, orps, xorps.
// The memory operand is required to be a 128-bit load, so it must be converted
// from a vector to a scalar.
def loadf32_128 : PatFrag<(ops node:$ptr),
(f32 (vector_extract (loadv4f32 node:$ptr), (iPTR 0)))>;
def loadf64_128 : PatFrag<(ops node:$ptr),
(f64 (vector_extract (loadv2f64 node:$ptr), (iPTR 0)))>;
Add versions 256-bit versions of alignedstore and alignedload, to be more strict about the alignment checking. This was found by inspection and I don't have any testcases so far, although the llvm testsuite runs without any problem. llvm-svn: 139625
2011-09-14 03:33:03 +08:00
// Like 'store', but always requires 128-bit vector alignment.
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def alignedstore : PatFrag<(ops node:$val, node:$ptr),
(store node:$val, node:$ptr), [{
return cast<StoreSDNode>(N)->getAlignment() >= 16;
}]>;
Add versions 256-bit versions of alignedstore and alignedload, to be more strict about the alignment checking. This was found by inspection and I don't have any testcases so far, although the llvm testsuite runs without any problem. llvm-svn: 139625
2011-09-14 03:33:03 +08:00
// Like 'store', but always requires 256-bit vector alignment.
def alignedstore256 : PatFrag<(ops node:$val, node:$ptr),
(store node:$val, node:$ptr), [{
return cast<StoreSDNode>(N)->getAlignment() >= 32;
}]>;
AVX-512: Added VPERM* instructons and MOV* zmm-to-zmm instructions. Added a test for shuffles using VPERM. llvm-svn: 188147
2013-08-11 15:55:09 +08:00
// Like 'store', but always requires 512-bit vector alignment.
def alignedstore512 : PatFrag<(ops node:$val, node:$ptr),
(store node:$val, node:$ptr), [{
return cast<StoreSDNode>(N)->getAlignment() >= 64;
}]>;
Add versions 256-bit versions of alignedstore and alignedload, to be more strict about the alignment checking. This was found by inspection and I don't have any testcases so far, although the llvm testsuite runs without any problem. llvm-svn: 139625
2011-09-14 03:33:03 +08:00
// Like 'load', but always requires 128-bit vector alignment.
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def alignedload : PatFrag<(ops node:$ptr), (load node:$ptr), [{
return cast<LoadSDNode>(N)->getAlignment() >= 16;
}]>;
Fix a regression from r147481. Original commit message from r147481: DAGCombine for transforming 128->256 casts into a vmovaps, rather then a vxorps + vinsertf128 pair if the original vector came from a load. Fix: Unaligned loads need to generate a vmovups. rdar://10974078 llvm-svn: 152366
2012-03-09 10:00:48 +08:00
// Like 'X86vzload', but always requires 128-bit vector alignment.
def alignedX86vzload : PatFrag<(ops node:$ptr), (X86vzload node:$ptr), [{
return cast<MemSDNode>(N)->getAlignment() >= 16;
}]>;
Add versions 256-bit versions of alignedstore and alignedload, to be more strict about the alignment checking. This was found by inspection and I don't have any testcases so far, although the llvm testsuite runs without any problem. llvm-svn: 139625
2011-09-14 03:33:03 +08:00
// Like 'load', but always requires 256-bit vector alignment.
def alignedload256 : PatFrag<(ops node:$ptr), (load node:$ptr), [{
return cast<LoadSDNode>(N)->getAlignment() >= 32;
}]>;
AVX-512: Added VPERM* instructons and MOV* zmm-to-zmm instructions. Added a test for shuffles using VPERM. llvm-svn: 188147
2013-08-11 15:55:09 +08:00
// Like 'load', but always requires 512-bit vector alignment.
def alignedload512 : PatFrag<(ops node:$ptr), (load node:$ptr), [{
return cast<LoadSDNode>(N)->getAlignment() >= 64;
}]>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def alignedloadfsf32 : PatFrag<(ops node:$ptr),
(f32 (alignedload node:$ptr))>;
def alignedloadfsf64 : PatFrag<(ops node:$ptr),
(f64 (alignedload node:$ptr))>;
Add comments to some pattern fragments in x86 llvm-svn: 111041
2010-08-14 04:39:01 +08:00
// 128-bit aligned load pattern fragments
Add comments near load pattern fragments indicating that all integer vector loads are promoted to v2i64 or v4i64 so that no one tries to reintroduce pattern fragments for other types. llvm-svn: 148771
2012-01-24 11:03:17 +08:00
// NOTE: all 128-bit integer vector loads are promoted to v2i64
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def alignedloadv4f32 : PatFrag<(ops node:$ptr),
(v4f32 (alignedload node:$ptr))>;
def alignedloadv2f64 : PatFrag<(ops node:$ptr),
(v2f64 (alignedload node:$ptr))>;
def alignedloadv2i64 : PatFrag<(ops node:$ptr),
(v2i64 (alignedload node:$ptr))>;
Add comments to some pattern fragments in x86 llvm-svn: 111041
2010-08-14 04:39:01 +08:00
// 256-bit aligned load pattern fragments
Add comments near load pattern fragments indicating that all integer vector loads are promoted to v2i64 or v4i64 so that no one tries to reintroduce pattern fragments for other types. llvm-svn: 148771
2012-01-24 11:03:17 +08:00
// NOTE: all 256-bit integer vector loads are promoted to v4i64
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def alignedloadv8f32 : PatFrag<(ops node:$ptr),
Add versions 256-bit versions of alignedstore and alignedload, to be more strict about the alignment checking. This was found by inspection and I don't have any testcases so far, although the llvm testsuite runs without any problem. llvm-svn: 139625
2011-09-14 03:33:03 +08:00
(v8f32 (alignedload256 node:$ptr))>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def alignedloadv4f64 : PatFrag<(ops node:$ptr),
Add versions 256-bit versions of alignedstore and alignedload, to be more strict about the alignment checking. This was found by inspection and I don't have any testcases so far, although the llvm testsuite runs without any problem. llvm-svn: 139625
2011-09-14 03:33:03 +08:00
(v4f64 (alignedload256 node:$ptr))>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def alignedloadv4i64 : PatFrag<(ops node:$ptr),
Add versions 256-bit versions of alignedstore and alignedload, to be more strict about the alignment checking. This was found by inspection and I don't have any testcases so far, although the llvm testsuite runs without any problem. llvm-svn: 139625
2011-09-14 03:33:03 +08:00
(v4i64 (alignedload256 node:$ptr))>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
AVX-512: Added VPERM* instructons and MOV* zmm-to-zmm instructions. Added a test for shuffles using VPERM. llvm-svn: 188147
2013-08-11 15:55:09 +08:00
// 512-bit aligned load pattern fragments
def alignedloadv16f32 : PatFrag<(ops node:$ptr),
(v16f32 (alignedload512 node:$ptr))>;
AVX-512: Added VMOVD, VMOVQ, VMOVSS, VMOVSD instructions. llvm-svn: 188637
2013-08-18 21:08:57 +08:00
def alignedloadv16i32 : PatFrag<(ops node:$ptr),
(v16i32 (alignedload512 node:$ptr))>;
AVX-512: Added VPERM* instructons and MOV* zmm-to-zmm instructions. Added a test for shuffles using VPERM. llvm-svn: 188147
2013-08-11 15:55:09 +08:00
def alignedloadv8f64 : PatFrag<(ops node:$ptr),
(v8f64 (alignedload512 node:$ptr))>;
def alignedloadv8i64 : PatFrag<(ops node:$ptr),
(v8i64 (alignedload512 node:$ptr))>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
// Like 'load', but uses special alignment checks suitable for use in
// memory operands in most SSE instructions, which are required to
// be naturally aligned on some targets but not on others. If the subtarget
// allows unaligned accesses, match any load, though this may require
// setting a feature bit in the processor (on startup, for example).
// Opteron 10h and later implement such a feature.
def memop : PatFrag<(ops node:$ptr), (load node:$ptr), [{
Fix program crashes due to alignment exceptions generated for SSE memop instructions (PR22371). r224330 introduced a bug by misinterpreting the "FeatureVectorUAMem" bit. The commit log says that change did not affect anything, but that's not correct. That change allowed SSE instructions to have unaligned mem operands folded into math ops, and that's not allowed in the default specification for any SSE variant. The bug is exposed when compiling for an AVX-capable CPU that had this feature flag but without enabling AVX codegen. Another mistake in r224330 was not adding the feature flag to all AVX CPUs; the AMD chips were excluded. This is part of the fix for PR22371 ( http://llvm.org/bugs/show_bug.cgi?id=22371 ). This feature bit is SSE-specific, so I've renamed it to "FeatureSSEUnalignedMem". Changed the existing test case for the feature bit to reflect the new name and renamed the test file itself to better reflect the feature. Added runs to fold-vex.ll to check for the failing codegen. Note that the feature bit is not set by default on any CPU because it may require a configuration register setting to enable the enhanced unaligned behavior. llvm-svn: 227983
2015-02-04 01:13:04 +08:00
return Subtarget->hasSSEUnalignedMem()
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
|| cast<LoadSDNode>(N)->getAlignment() >= 16;
}]>;
def memopfsf32 : PatFrag<(ops node:$ptr), (f32 (memop node:$ptr))>;
def memopfsf64 : PatFrag<(ops node:$ptr), (f64 (memop node:$ptr))>;
Add comments to some pattern fragments in x86 llvm-svn: 111041
2010-08-14 04:39:01 +08:00
// 128-bit memop pattern fragments
Add comments near load pattern fragments indicating that all integer vector loads are promoted to v2i64 or v4i64 so that no one tries to reintroduce pattern fragments for other types. llvm-svn: 148771
2012-01-24 11:03:17 +08:00
// NOTE: all 128-bit integer vector loads are promoted to v2i64
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def memopv4f32 : PatFrag<(ops node:$ptr), (v4f32 (memop node:$ptr))>;
def memopv2f64 : PatFrag<(ops node:$ptr), (v2f64 (memop node:$ptr))>;
def memopv2i64 : PatFrag<(ops node:$ptr), (v2i64 (memop node:$ptr))>;
prevent folding a scalar FP load into a packed logical FP instruction (PR22371) Change the memory operands in sse12_fp_packed_scalar_logical_alias from scalars to vectors. That's what the hardware packed logical FP instructions define: 128-bit memory operands. There are no scalar versions of these instructions...because this is x86. Generating the wrong code (folding a scalar load into a 128-bit load) is still possible using the peephole optimization pass and the load folding tables. We won't completely solve this bug until we either fix the lowering in fabs/fneg/fcopysign and any other places where scalar FP logic is created or fix the load folding in foldMemoryOperandImpl() to make sure it isn't changing the size of the load. Differential Revision: http://reviews.llvm.org/D7474 llvm-svn: 229531
2015-02-18 04:08:21 +08:00
// These are needed to match a scalar memop that is used in a vector-only
// math instruction such as the FP logical ops: andps, andnps, orps, xorps.
// The memory operand is required to be a 128-bit load, so it must be converted
// from a vector to a scalar.
def memopfsf32_128 : PatFrag<(ops node:$ptr),
(f32 (vector_extract (memopv4f32 node:$ptr), (iPTR 0)))>;
def memopfsf64_128 : PatFrag<(ops node:$ptr),
(f64 (vector_extract (memopv2f64 node:$ptr), (iPTR 0)))>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
// SSSE3 uses MMX registers for some instructions. They aren't aligned on a
// 16-byte boundary.
// FIXME: 8 byte alignment for mmx reads is not required
def memop64 : PatFrag<(ops node:$ptr), (load node:$ptr), [{
return cast<LoadSDNode>(N)->getAlignment() >= 8;
}]>;
Massive rewrite of MMX: The x86_mmx type is used for MMX intrinsics, parameters and return values where these use MMX registers, and is also supported in load, store, and bitcast. Only the above operations generate MMX instructions, and optimizations do not operate on or produce MMX intrinsics. MMX-sized vectors <2 x i32> etc. are lowered to XMM or split into smaller pieces. Optimizations may occur on these forms and the result casted back to x86_mmx, provided the result feeds into a previous existing x86_mmx operation. The point of all this is prevent optimizations from introducing MMX operations, which is unsafe due to the EMMS problem. llvm-svn: 115243
2010-10-01 07:57:10 +08:00
def memopmmx : PatFrag<(ops node:$ptr), (x86mmx (memop64 node:$ptr))>;
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
// MOVNT Support
// Like 'store', but requires the non-temporal bit to be set
def nontemporalstore : PatFrag<(ops node:$val, node:$ptr),
(st node:$val, node:$ptr), [{
if (StoreSDNode *ST = dyn_cast<StoreSDNode>(N))
return ST->isNonTemporal();
return false;
}]>;
def alignednontemporalstore : PatFrag<(ops node:$val, node:$ptr),
Remove tabs. llvm-svn: 160477
2012-07-19 08:11:40 +08:00
(st node:$val, node:$ptr), [{
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
if (StoreSDNode *ST = dyn_cast<StoreSDNode>(N))
return ST->isNonTemporal() && !ST->isTruncatingStore() &&
ST->getAddressingMode() == ISD::UNINDEXED &&
ST->getAlignment() >= 16;
return false;
}]>;
def unalignednontemporalstore : PatFrag<(ops node:$val, node:$ptr),
Remove tabs. llvm-svn: 160477
2012-07-19 08:11:40 +08:00
(st node:$val, node:$ptr), [{
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
if (StoreSDNode *ST = dyn_cast<StoreSDNode>(N))
return ST->isNonTemporal() &&
ST->getAlignment() < 16;
return false;
}]>;
Masked gather and scatter - added DAGCombine visitors and AVX-512 instruction selection patterns. All other patches, including tests will follow. http://reviews.llvm.org/D7665 llvm-svn: 236211
2015-04-30 16:38:48 +08:00
def mgatherv8i32 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_gather node:$src1, node:$src2, node:$src3) , [{
if (MaskedGatherSDNode *Mgt = dyn_cast<MaskedGatherSDNode>(N))
return (Mgt->getIndex().getValueType() == MVT::v8i32 ||
Mgt->getBasePtr().getValueType() == MVT::v8i32);
return false;
}]>;
def mgatherv8i64 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_gather node:$src1, node:$src2, node:$src3) , [{
if (MaskedGatherSDNode *Mgt = dyn_cast<MaskedGatherSDNode>(N))
return (Mgt->getIndex().getValueType() == MVT::v8i64 ||
Mgt->getBasePtr().getValueType() == MVT::v8i64);
return false;
}]>;
def mgatherv16i32 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_gather node:$src1, node:$src2, node:$src3) , [{
if (MaskedGatherSDNode *Mgt = dyn_cast<MaskedGatherSDNode>(N))
return (Mgt->getIndex().getValueType() == MVT::v16i32 ||
Mgt->getBasePtr().getValueType() == MVT::v16i32);
return false;
}]>;
def mscatterv8i32 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_scatter node:$src1, node:$src2, node:$src3) , [{
if (MaskedScatterSDNode *Sc = dyn_cast<MaskedScatterSDNode>(N))
return (Sc->getIndex().getValueType() == MVT::v8i32 ||
Sc->getBasePtr().getValueType() == MVT::v8i32);
return false;
}]>;
def mscatterv8i64 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_scatter node:$src1, node:$src2, node:$src3) , [{
if (MaskedScatterSDNode *Sc = dyn_cast<MaskedScatterSDNode>(N))
return (Sc->getIndex().getValueType() == MVT::v8i64 ||
Sc->getBasePtr().getValueType() == MVT::v8i64);
return false;
}]>;
def mscatterv16i32 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_scatter node:$src1, node:$src2, node:$src3) , [{
if (MaskedScatterSDNode *Sc = dyn_cast<MaskedScatterSDNode>(N))
return (Sc->getIndex().getValueType() == MVT::v16i32 ||
Sc->getBasePtr().getValueType() == MVT::v16i32);
return false;
}]>;
Add comments to some pattern fragments in x86 llvm-svn: 111041
2010-08-14 04:39:01 +08:00
// 128-bit bitconvert pattern fragments
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def bc_v4f32 : PatFrag<(ops node:$in), (v4f32 (bitconvert node:$in))>;
def bc_v2f64 : PatFrag<(ops node:$in), (v2f64 (bitconvert node:$in))>;
def bc_v16i8 : PatFrag<(ops node:$in), (v16i8 (bitconvert node:$in))>;
def bc_v8i16 : PatFrag<(ops node:$in), (v8i16 (bitconvert node:$in))>;
def bc_v4i32 : PatFrag<(ops node:$in), (v4i32 (bitconvert node:$in))>;
def bc_v2i64 : PatFrag<(ops node:$in), (v2i64 (bitconvert node:$in))>;
Add comments to some pattern fragments in x86 llvm-svn: 111041
2010-08-14 04:39:01 +08:00
// 256-bit bitconvert pattern fragments
Add a bunch more X86 AVX2 instructions and their corresponding intrinsics. llvm-svn: 143529
2011-11-02 12:42:13 +08:00
def bc_v32i8 : PatFrag<(ops node:$in), (v32i8 (bitconvert node:$in))>;
def bc_v16i16 : PatFrag<(ops node:$in), (v16i16 (bitconvert node:$in))>;
Add more 256-bit forms for a bunch of regular AVX instructions Add 64-bit (GR64) versions of some instructions (which are not described in their SSE forms, but are described in AVX) llvm-svn: 109063
2010-07-22 07:53:50 +08:00
def bc_v8i32 : PatFrag<(ops node:$in), (v8i32 (bitconvert node:$in))>;
AVX Codegen support for 256-bit versions of vandps, vandpd, vorps, vorpd, vxorps, vxorpd llvm-svn: 135023
2011-07-13 09:15:33 +08:00
def bc_v4i64 : PatFrag<(ops node:$in), (v4i64 (bitconvert node:$in))>;
AVX-512: added intrinsic vcvtpd2ps (with rounding mode and without) llvm-svn: 198593
2014-01-06 16:45:54 +08:00
def bc_v8f32 : PatFrag<(ops node:$in), (v8f32 (bitconvert node:$in))>;
Add more 256-bit forms for a bunch of regular AVX instructions Add 64-bit (GR64) versions of some instructions (which are not described in their SSE forms, but are described in AVX) llvm-svn: 109063
2010-07-22 07:53:50 +08:00
Don't use v16i32 for load pattern matching. All 512-bit loads are cated to v8i64. llvm-svn: 188534
2013-08-16 14:07:34 +08:00
// 512-bit bitconvert pattern fragments
def bc_v16i32 : PatFrag<(ops node:$in), (v16i32 (bitconvert node:$in))>;
def bc_v8i64 : PatFrag<(ops node:$in), (v8i64 (bitconvert node:$in))>;
AVX-512: Added intrinsics for vcvt, vcvtt, vrndscale, vcmp Printing rounding control. Enncoding for EVEX_RC (rounding control). llvm-svn: 198277
2014-01-01 23:12:34 +08:00
def bc_v8f64 : PatFrag<(ops node:$in), (v8f64 (bitconvert node:$in))>;
def bc_v16f32 : PatFrag<(ops node:$in), (v16f32 (bitconvert node:$in))>;
Don't use v16i32 for load pattern matching. All 512-bit loads are cated to v8i64. llvm-svn: 188534
2013-08-16 14:07:34 +08:00
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
def vzmovl_v2i64 : PatFrag<(ops node:$src),
(bitconvert (v2i64 (X86vzmovl
(v2i64 (scalar_to_vector (loadi64 node:$src))))))>;
def vzmovl_v4i32 : PatFrag<(ops node:$src),
(bitconvert (v4i32 (X86vzmovl
(v4i32 (scalar_to_vector (loadi32 node:$src))))))>;
def vzload_v2i64 : PatFrag<(ops node:$src),
(bitconvert (v2i64 (X86vzload node:$src)))>;
def fp32imm0 : PatLeaf<(f32 fpimm), [{
return N->isExactlyValue(+0.0);
}]>;
AVX-512: Added intrinsics for vcvt, vcvtt, vrndscale, vcmp Printing rounding control. Enncoding for EVEX_RC (rounding control). llvm-svn: 198277
2014-01-01 23:12:34 +08:00
def I8Imm : SDNodeXForm<imm, [{
// Transformation function: get the low 8 bits.
Reapply r235977 "[DebugInfo] Add debug locations to constant SD nodes" [DebugInfo] Add debug locations to constant SD nodes This adds debug location to constant nodes of Selection DAG and updates all places that create constants to pass debug locations (see PR13269). Can't guarantee that all locations are correct, but in a lot of cases choice is obvious, so most of them should be. At least all tests pass. Tests for these changes do not cover everything, instead just check it for SDNodes, ARM and AArch64 where it's easy to get incorrect locations on constants. This is not complete fix as FastISel contains workaround for wrong debug locations, which drops locations from instructions on processing constants, but there isn't currently a way to use debug locations from constants there as llvm::Constant doesn't cache it (yet). Although this is a bit different issue, not directly related to these changes. Differential Revision: http://reviews.llvm.org/D9084 llvm-svn: 235989
2015-04-28 22:05:47 +08:00
return getI8Imm((uint8_t)N->getZExtValue(), SDLoc(N));
AVX-512: Added intrinsics for vcvt, vcvtt, vrndscale, vcmp Printing rounding control. Enncoding for EVEX_RC (rounding control). llvm-svn: 198277
2014-01-01 23:12:34 +08:00
}]>;
def FROUND_NO_EXC : ImmLeaf<i32, [{ return Imm == 8; }]>;
[AVX512] Add enum for the static rounding types No functional change. This will be used by the new FMA intrinsic lowering code. We can probably add NO_EXC here as well, I am just not too familiar with this part of AVX512 yet. We can add that later. llvm-svn: 215662
2014-08-15 01:13:26 +08:00
def FROUND_CURRENT : ImmLeaf<i32, [{
return Imm == X86::STATIC_ROUNDING::CUR_DIRECTION;
}]>;
AVX-512: Added intrinsics for vcvt, vcvtt, vrndscale, vcmp Printing rounding control. Enncoding for EVEX_RC (rounding control). llvm-svn: 198277
2014-01-01 23:12:34 +08:00
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
// BYTE_imm - Transform bit immediates into byte immediates.
def BYTE_imm : SDNodeXForm<imm, [{
// Transformation function: imm >> 3
Reapply r235977 "[DebugInfo] Add debug locations to constant SD nodes" [DebugInfo] Add debug locations to constant SD nodes This adds debug location to constant nodes of Selection DAG and updates all places that create constants to pass debug locations (see PR13269). Can't guarantee that all locations are correct, but in a lot of cases choice is obvious, so most of them should be. At least all tests pass. Tests for these changes do not cover everything, instead just check it for SDNodes, ARM and AArch64 where it's easy to get incorrect locations on constants. This is not complete fix as FastISel contains workaround for wrong debug locations, which drops locations from instructions on processing constants, but there isn't currently a way to use debug locations from constants there as llvm::Constant doesn't cache it (yet). Although this is a bit different issue, not directly related to these changes. Differential Revision: http://reviews.llvm.org/D9084 llvm-svn: 235989
2015-04-28 22:05:47 +08:00
return getI32Imm(N->getZExtValue() >> 3, SDLoc(N));
Move some SIMD fragment code into X86InstrFragmentsSIMD so that the utility classes can be used from multiple files. This will aid transitioning to a new refactored x86 SIMD specification. llvm-svn: 108213
2010-07-13 07:41:28 +08:00
}]>;
Added INSERT and EXTRACT intructions from AVX-512 ISA. All insertf*/extractf* functions replaced with insert/extract since we have insertf and inserti forms. Added lowering for INSERT_VECTOR_ELT / EXTRACT_VECTOR_ELT for 512-bit vectors. Added lowering for EXTRACT/INSERT subvector for 512-bit vectors. Added a test. llvm-svn: 187491
2013-07-31 19:35:14 +08:00
// EXTRACT_get_vextract128_imm xform function: convert extract_subvector index
// to VEXTRACTF128/VEXTRACTI128 imm.
def EXTRACT_get_vextract128_imm : SDNodeXForm<extract_subvector, [{
Reapply r235977 "[DebugInfo] Add debug locations to constant SD nodes" [DebugInfo] Add debug locations to constant SD nodes This adds debug location to constant nodes of Selection DAG and updates all places that create constants to pass debug locations (see PR13269). Can't guarantee that all locations are correct, but in a lot of cases choice is obvious, so most of them should be. At least all tests pass. Tests for these changes do not cover everything, instead just check it for SDNodes, ARM and AArch64 where it's easy to get incorrect locations on constants. This is not complete fix as FastISel contains workaround for wrong debug locations, which drops locations from instructions on processing constants, but there isn't currently a way to use debug locations from constants there as llvm::Constant doesn't cache it (yet). Although this is a bit different issue, not directly related to these changes. Differential Revision: http://reviews.llvm.org/D9084 llvm-svn: 235989
2015-04-28 22:05:47 +08:00
return getI8Imm(X86::getExtractVEXTRACT128Immediate(N), SDLoc(N));
[AVX] VEXTRACTF128 support. This commit includes patterns for matching EXTRACT_SUBVECTOR to VEXTRACTF128 along with support routines to examine and translate index values. VINSERTF128 comes next. With these two in place we can begin supporting more AVX operations as INSERT/EXTRACT can be used as a fallback when 256-bit support is not available. llvm-svn: 124797
2011-02-03 23:50:00 +08:00
}]>;
Added INSERT and EXTRACT intructions from AVX-512 ISA. All insertf*/extractf* functions replaced with insert/extract since we have insertf and inserti forms. Added lowering for INSERT_VECTOR_ELT / EXTRACT_VECTOR_ELT for 512-bit vectors. Added lowering for EXTRACT/INSERT subvector for 512-bit vectors. Added a test. llvm-svn: 187491
2013-07-31 19:35:14 +08:00
// INSERT_get_vinsert128_imm xform function: convert insert_subvector index to
// VINSERTF128/VINSERTI128 imm.
def INSERT_get_vinsert128_imm : SDNodeXForm<insert_subvector, [{
Reapply r235977 "[DebugInfo] Add debug locations to constant SD nodes" [DebugInfo] Add debug locations to constant SD nodes This adds debug location to constant nodes of Selection DAG and updates all places that create constants to pass debug locations (see PR13269). Can't guarantee that all locations are correct, but in a lot of cases choice is obvious, so most of them should be. At least all tests pass. Tests for these changes do not cover everything, instead just check it for SDNodes, ARM and AArch64 where it's easy to get incorrect locations on constants. This is not complete fix as FastISel contains workaround for wrong debug locations, which drops locations from instructions on processing constants, but there isn't currently a way to use debug locations from constants there as llvm::Constant doesn't cache it (yet). Although this is a bit different issue, not directly related to these changes. Differential Revision: http://reviews.llvm.org/D9084 llvm-svn: 235989
2015-04-28 22:05:47 +08:00
return getI8Imm(X86::getInsertVINSERT128Immediate(N), SDLoc(N));
[AVX] Support VSINSERTF128 with more patterns and appropriate infrastructure. This makes lowering 256-bit vectors to 128-bit vectors simple when 256-bit vector support is not available. llvm-svn: 124868
2011-02-05 00:08:29 +08:00
}]>;
Added INSERT and EXTRACT intructions from AVX-512 ISA. All insertf*/extractf* functions replaced with insert/extract since we have insertf and inserti forms. Added lowering for INSERT_VECTOR_ELT / EXTRACT_VECTOR_ELT for 512-bit vectors. Added lowering for EXTRACT/INSERT subvector for 512-bit vectors. Added a test. llvm-svn: 187491
2013-07-31 19:35:14 +08:00
// EXTRACT_get_vextract256_imm xform function: convert extract_subvector index
// to VEXTRACTF64x4 imm.
def EXTRACT_get_vextract256_imm : SDNodeXForm<extract_subvector, [{
Reapply r235977 "[DebugInfo] Add debug locations to constant SD nodes" [DebugInfo] Add debug locations to constant SD nodes This adds debug location to constant nodes of Selection DAG and updates all places that create constants to pass debug locations (see PR13269). Can't guarantee that all locations are correct, but in a lot of cases choice is obvious, so most of them should be. At least all tests pass. Tests for these changes do not cover everything, instead just check it for SDNodes, ARM and AArch64 where it's easy to get incorrect locations on constants. This is not complete fix as FastISel contains workaround for wrong debug locations, which drops locations from instructions on processing constants, but there isn't currently a way to use debug locations from constants there as llvm::Constant doesn't cache it (yet). Although this is a bit different issue, not directly related to these changes. Differential Revision: http://reviews.llvm.org/D9084 llvm-svn: 235989
2015-04-28 22:05:47 +08:00
return getI8Imm(X86::getExtractVEXTRACT256Immediate(N), SDLoc(N));
Added INSERT and EXTRACT intructions from AVX-512 ISA. All insertf*/extractf* functions replaced with insert/extract since we have insertf and inserti forms. Added lowering for INSERT_VECTOR_ELT / EXTRACT_VECTOR_ELT for 512-bit vectors. Added lowering for EXTRACT/INSERT subvector for 512-bit vectors. Added a test. llvm-svn: 187491
2013-07-31 19:35:14 +08:00
}]>;
// INSERT_get_vinsert256_imm xform function: convert insert_subvector index to
// VINSERTF64x4 imm.
def INSERT_get_vinsert256_imm : SDNodeXForm<insert_subvector, [{
Reapply r235977 "[DebugInfo] Add debug locations to constant SD nodes" [DebugInfo] Add debug locations to constant SD nodes This adds debug location to constant nodes of Selection DAG and updates all places that create constants to pass debug locations (see PR13269). Can't guarantee that all locations are correct, but in a lot of cases choice is obvious, so most of them should be. At least all tests pass. Tests for these changes do not cover everything, instead just check it for SDNodes, ARM and AArch64 where it's easy to get incorrect locations on constants. This is not complete fix as FastISel contains workaround for wrong debug locations, which drops locations from instructions on processing constants, but there isn't currently a way to use debug locations from constants there as llvm::Constant doesn't cache it (yet). Although this is a bit different issue, not directly related to these changes. Differential Revision: http://reviews.llvm.org/D9084 llvm-svn: 235989
2015-04-28 22:05:47 +08:00
return getI8Imm(X86::getInsertVINSERT256Immediate(N), SDLoc(N));
Added INSERT and EXTRACT intructions from AVX-512 ISA. All insertf*/extractf* functions replaced with insert/extract since we have insertf and inserti forms. Added lowering for INSERT_VECTOR_ELT / EXTRACT_VECTOR_ELT for 512-bit vectors. Added lowering for EXTRACT/INSERT subvector for 512-bit vectors. Added a test. llvm-svn: 187491
2013-07-31 19:35:14 +08:00
}]>;
def vextract128_extract : PatFrag<(ops node:$bigvec, node:$index),
(extract_subvector node:$bigvec,
node:$index), [{
return X86::isVEXTRACT128Index(N);
}], EXTRACT_get_vextract128_imm>;
def vinsert128_insert : PatFrag<(ops node:$bigvec, node:$smallvec,
node:$index),
(insert_subvector node:$bigvec, node:$smallvec,
node:$index), [{
return X86::isVINSERT128Index(N);
}], INSERT_get_vinsert128_imm>;
def vextract256_extract : PatFrag<(ops node:$bigvec, node:$index),
[AVX] VEXTRACTF128 support. This commit includes patterns for matching EXTRACT_SUBVECTOR to VEXTRACTF128 along with support routines to examine and translate index values. VINSERTF128 comes next. With these two in place we can begin supporting more AVX operations as INSERT/EXTRACT can be used as a fallback when 256-bit support is not available. llvm-svn: 124797
2011-02-03 23:50:00 +08:00
(extract_subvector node:$bigvec,
node:$index), [{
Added INSERT and EXTRACT intructions from AVX-512 ISA. All insertf*/extractf* functions replaced with insert/extract since we have insertf and inserti forms. Added lowering for INSERT_VECTOR_ELT / EXTRACT_VECTOR_ELT for 512-bit vectors. Added lowering for EXTRACT/INSERT subvector for 512-bit vectors. Added a test. llvm-svn: 187491
2013-07-31 19:35:14 +08:00
return X86::isVEXTRACT256Index(N);
}], EXTRACT_get_vextract256_imm>;
[AVX] Support VSINSERTF128 with more patterns and appropriate infrastructure. This makes lowering 256-bit vectors to 128-bit vectors simple when 256-bit vector support is not available. llvm-svn: 124868
2011-02-05 00:08:29 +08:00
Added INSERT and EXTRACT intructions from AVX-512 ISA. All insertf*/extractf* functions replaced with insert/extract since we have insertf and inserti forms. Added lowering for INSERT_VECTOR_ELT / EXTRACT_VECTOR_ELT for 512-bit vectors. Added lowering for EXTRACT/INSERT subvector for 512-bit vectors. Added a test. llvm-svn: 187491
2013-07-31 19:35:14 +08:00
def vinsert256_insert : PatFrag<(ops node:$bigvec, node:$smallvec,
[AVX] Support VSINSERTF128 with more patterns and appropriate infrastructure. This makes lowering 256-bit vectors to 128-bit vectors simple when 256-bit vector support is not available. llvm-svn: 124868
2011-02-05 00:08:29 +08:00
node:$index),
(insert_subvector node:$bigvec, node:$smallvec,
node:$index), [{
Added INSERT and EXTRACT intructions from AVX-512 ISA. All insertf*/extractf* functions replaced with insert/extract since we have insertf and inserti forms. Added lowering for INSERT_VECTOR_ELT / EXTRACT_VECTOR_ELT for 512-bit vectors. Added lowering for EXTRACT/INSERT subvector for 512-bit vectors. Added a test. llvm-svn: 187491
2013-07-31 19:35:14 +08:00
return X86::isVINSERT256Index(N);
}], INSERT_get_vinsert256_imm>;
- Handle special scalar_to_vector case: splats. Using a native 128-bit shuffle before inserting on a 256-bit vector. - Add AVX versions of movd/movq instructions - Introduce a few COPY patterns to match insert_subvector instructions. This turns a trivial insert_subvector instruction into a register copy, coalescing the xmm into a ymm and avoid emiting on more instruction. llvm-svn: 136002
2011-07-26 07:05:25 +08:00
AVX-512: Moved patterns for masked load/store under avx_store, avx_load classes. No functional changes. llvm-svn: 231069
2015-03-03 23:03:35 +08:00
def masked_load_aligned128 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_load node:$src1, node:$src2, node:$src3), [{
Reduce dyn_cast<> to isa<> or cast<> where possible. No functional change intended. llvm-svn: 234586
2015-04-10 19:24:51 +08:00
if (auto *Load = dyn_cast<MaskedLoadSDNode>(N))
return Load->getAlignment() >= 16;
AVX-512: Moved patterns for masked load/store under avx_store, avx_load classes. No functional changes. llvm-svn: 231069
2015-03-03 23:03:35 +08:00
return false;
}]>;
def masked_load_aligned256 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_load node:$src1, node:$src2, node:$src3), [{
Reduce dyn_cast<> to isa<> or cast<> where possible. No functional change intended. llvm-svn: 234586
2015-04-10 19:24:51 +08:00
if (auto *Load = dyn_cast<MaskedLoadSDNode>(N))
return Load->getAlignment() >= 32;
AVX-512: Moved patterns for masked load/store under avx_store, avx_load classes. No functional changes. llvm-svn: 231069
2015-03-03 23:03:35 +08:00
return false;
}]>;
def masked_load_aligned512 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_load node:$src1, node:$src2, node:$src3), [{
Reduce dyn_cast<> to isa<> or cast<> where possible. No functional change intended. llvm-svn: 234586
2015-04-10 19:24:51 +08:00
if (auto *Load = dyn_cast<MaskedLoadSDNode>(N))
return Load->getAlignment() >= 64;
AVX-512: Moved patterns for masked load/store under avx_store, avx_load classes. No functional changes. llvm-svn: 231069
2015-03-03 23:03:35 +08:00
return false;
}]>;
def masked_load_unaligned : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_load node:$src1, node:$src2, node:$src3), [{
Reduce dyn_cast<> to isa<> or cast<> where possible. No functional change intended. llvm-svn: 234586
2015-04-10 19:24:51 +08:00
return isa<MaskedLoadSDNode>(N);
AVX-512: Moved patterns for masked load/store under avx_store, avx_load classes. No functional changes. llvm-svn: 231069
2015-03-03 23:03:35 +08:00
}]>;
def masked_store_aligned128 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_store node:$src1, node:$src2, node:$src3), [{
Reduce dyn_cast<> to isa<> or cast<> where possible. No functional change intended. llvm-svn: 234586
2015-04-10 19:24:51 +08:00
if (auto *Store = dyn_cast<MaskedStoreSDNode>(N))
return Store->getAlignment() >= 16;
AVX-512: Moved patterns for masked load/store under avx_store, avx_load classes. No functional changes. llvm-svn: 231069
2015-03-03 23:03:35 +08:00
return false;
}]>;
def masked_store_aligned256 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_store node:$src1, node:$src2, node:$src3), [{
Reduce dyn_cast<> to isa<> or cast<> where possible. No functional change intended. llvm-svn: 234586
2015-04-10 19:24:51 +08:00
if (auto *Store = dyn_cast<MaskedStoreSDNode>(N))
return Store->getAlignment() >= 32;
AVX-512: Moved patterns for masked load/store under avx_store, avx_load classes. No functional changes. llvm-svn: 231069
2015-03-03 23:03:35 +08:00
return false;
}]>;
def masked_store_aligned512 : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_store node:$src1, node:$src2, node:$src3), [{
Reduce dyn_cast<> to isa<> or cast<> where possible. No functional change intended. llvm-svn: 234586
2015-04-10 19:24:51 +08:00
if (auto *Store = dyn_cast<MaskedStoreSDNode>(N))
return Store->getAlignment() >= 64;
AVX-512: Moved patterns for masked load/store under avx_store, avx_load classes. No functional changes. llvm-svn: 231069
2015-03-03 23:03:35 +08:00
return false;
}]>;
def masked_store_unaligned : PatFrag<(ops node:$src1, node:$src2, node:$src3),
(masked_store node:$src1, node:$src2, node:$src3), [{
Reduce dyn_cast<> to isa<> or cast<> where possible. No functional change intended. llvm-svn: 234586
2015-04-10 19:24:51 +08:00
return isa<MaskedStoreSDNode>(N);
AVX-512: Moved patterns for masked load/store under avx_store, avx_load classes. No functional changes. llvm-svn: 231069
2015-03-03 23:03:35 +08:00
}]>;