maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/test/CodeGen/X86/vector-shuffle-256-v4.ll

876 lines
32 KiB
LLVM
Raw Normal View History

[x86] Back out a bad choice about lowering v4i64 and pave the way for a more sane approach to AVX2 support. Fundamentally, there is no useful way to lower integer vectors in AVX. None. We always end up with a VINSERTF128 in the end, so we might as well eagerly switch to the floating point domain and do everything there. This cleans up lots of weird and unlikely to be correct differences between integer and floating point shuffles when we only have AVX1. The other nice consequence is that by doing things this way we will make it much easier to write the integer lowering routines as we won't need to duplicate the logic to check for AVX vs. AVX2 in each one -- if we actually try to lower a 256-bit vector as an integer vector, we have AVX2 and can rely on it. I think this will make the code much simpler and more comprehensible. Currently, I've disabled *all* support for AVX2 so that we always fall back to AVX. This keeps everything working rather than asserting. That will go away with the subsequent series of patches that provide a baseline AVX2 implementation. Please note, I'm going to implement AVX2 *without access to hardware*. That means I cannot correctness test this path. I will be relying on those with access to AVX2 hardware to do correctness testing and fix bugs here, but as a courtesy I'm trying to sketch out the framework for the new-style vector shuffle lowering in the context of the AVX2 ISA. llvm-svn: 218228
2014-09-22 08:32:15 +08:00
; RUN: llc < %s -mcpu=x86-64 -mattr=+avx -x86-experimental-vector-shuffle-lowering | FileCheck %s --check-prefix=ALL --check-prefix=AVX --check-prefix=AVX1
; RUN: llc < %s -mcpu=x86-64 -mattr=+avx2 -x86-experimental-vector-shuffle-lowering | FileCheck %s --check-prefix=ALL --check-prefix=AVX --check-prefix=AVX2
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
target triple = "x86_64-unknown-unknown"
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
define <4 x double> @shuffle_v4f64_0000(<4 x double> %a, <4 x double> %b) {
; AVX1-LABEL: shuffle_v4f64_0000:
; AVX1: # BB#0:
; AVX1-NEXT: vmovddup {{.*#+}} xmm0 = xmm0[0,0]
; AVX1-NEXT: vinsertf128 $1, %xmm0, %ymm0, %ymm0
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4f64_0000:
[x86] Add some vector-register broadcast operations to the 256-bit v4 tests which were missing them. llvm-svn: 218657
2014-09-30 10:32:36 +08:00
; AVX2: # BB#0:
[x86] Teach the new vector shuffle lowering about VBROADCAST and VPBROADCAST. This has the somewhat expected pervasive impact. I don't know why I forgot about this. Everything seems good with lots of significant improvements in the tests. llvm-svn: 218724
2014-10-01 08:41:21 +08:00
; AVX2-NEXT: vbroadcastsd %xmm0, %ymm0
[x86] Add some vector-register broadcast operations to the 256-bit v4 tests which were missing them. llvm-svn: 218657
2014-09-30 10:32:36 +08:00
; AVX2-NEXT: retq
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 0, i32 0, i32 0>
ret <4 x double> %shuffle
}
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
define <4 x double> @shuffle_v4f64_0001(<4 x double> %a, <4 x double> %b) {
; AVX1-LABEL: shuffle_v4f64_0001:
; AVX1: # BB#0:
; AVX1-NEXT: vmovddup {{.*#+}} xmm1 = xmm0[0,0]
; AVX1-NEXT: vinsertf128 $1, %xmm0, %ymm1, %ymm0
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4f64_0001:
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermpd {{.*#+}} ymm0 = ymm0[0,0,0,1]
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 0, i32 0, i32 1>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x double> @shuffle_v4f64_0020(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4f64_0020:
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; AVX1: # BB#0:
; AVX1-NEXT: vextractf128 $1, %ymm0, %xmm1
; AVX1-NEXT: vunpcklpd {{.*#+}} xmm1 = xmm1[0],xmm0[0]
; AVX1-NEXT: vmovddup {{.*#+}} xmm0 = xmm0[0,0]
; AVX1-NEXT: vinsertf128 $1, %xmm1, %ymm0, %ymm0
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4f64_0020:
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermpd {{.*#+}} ymm0 = ymm0[0,0,2,0]
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 0, i32 2, i32 0>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x double> @shuffle_v4f64_0300(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4f64_0300:
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX1: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-NEXT: vperm2f128 {{.*#+}} ymm1 = ymm0[2,3,0,1]
; AVX1-NEXT: vpermilpd {{.*#+}} ymm1 = ymm1[0,1,2,2]
[x86] Add a really preposterous number of patterns for matching all of the various ways in which blends can be used to do vector element insertion for lowering with the scalar math instruction forms that effectively re-blend with the high elements after performing the operation. This then allows me to bail on the element insertion lowering path when we have SSE4.1 and are going to be doing a normal blend, which in turn restores the last of the blends lost from the new vector shuffle lowering when I got it to prioritize insertion in other cases (for example when we don't *have* a blend instruction). Without the patterns, using blends here would have regressed sse-scalar-fp-arith.ll *completely* with the new vector shuffle lowering. For completeness, I've added RUN-lines with the new lowering here. This is somewhat superfluous as I'm about to flip the default, but hey, it shows that this actually significantly changed behavior. The patterns I've added are just ridiculously repetative. Suggestions on making them better very much welcome. In particular, handling the commuted form of the v2f64 patterns is somewhat obnoxious. llvm-svn: 219033
2014-10-04 06:43:17 +08:00
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0],ymm1[1,2,3]
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4f64_0300:
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermpd {{.*#+}} ymm0 = ymm0[0,3,0,0]
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 3, i32 0, i32 0>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x double> @shuffle_v4f64_1000(<4 x double> %a, <4 x double> %b) {
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; AVX1-LABEL: shuffle_v4f64_1000:
; AVX1: # BB#0:
; AVX1-NEXT: vpermilpd {{.*#+}} xmm1 = xmm0[1,0]
; AVX1-NEXT: vmovddup {{.*#+}} xmm0 = xmm0[0,0]
; AVX1-NEXT: vinsertf128 $1, %xmm0, %ymm1, %ymm0
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4f64_1000:
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermpd {{.*#+}} ymm0 = ymm0[1,0,0,0]
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 1, i32 0, i32 0, i32 0>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x double> @shuffle_v4f64_2200(<4 x double> %a, <4 x double> %b) {
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; AVX1-LABEL: shuffle_v4f64_2200:
; AVX1: # BB#0:
; AVX1-NEXT: vperm2f128 {{.*#+}} ymm0 = ymm0[2,3,0,1]
; AVX1-NEXT: vmovddup {{.*#+}} ymm0 = ymm0[0,0,2,2]
; AVX1-NEXT: retq
;
; AVX2-LABEL: shuffle_v4f64_2200:
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermpd {{.*#+}} ymm0 = ymm0[2,2,0,0]
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 2, i32 2, i32 0, i32 0>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x double> @shuffle_v4f64_3330(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4f64_3330:
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX1: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-NEXT: vperm2f128 {{.*#+}} ymm1 = ymm0[2,3,0,1]
; AVX1-NEXT: vpermilpd {{.*#+}} ymm1 = ymm1[1,1,2,2]
; AVX1-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[0,0,3,2]
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm1[0,1],ymm0[2],ymm1[3]
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4f64_3330:
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermpd {{.*#+}} ymm0 = ymm0[3,3,3,0]
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 3, i32 3, i32 3, i32 0>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x double> @shuffle_v4f64_3210(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4f64_3210:
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX1: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-NEXT: vperm2f128 {{.*#+}} ymm0 = ymm0[2,3,0,1]
; AVX1-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,3,2]
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4f64_3210:
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermpd {{.*#+}} ymm0 = ymm0[3,2,1,0]
[x86] Teach the new vector shuffle lowering to use AVX2 instructions for v4f64 and v8f32 shuffles when they are lane-crossing. We have fully general lane-crossing permutation functions in AVX2 that make this easy. Part of this also changes exactly when and how these vectors are split up when we don't have AVX2. This isn't always a win but it usually is a win, so on the balance I think its better. The primary regressions are all things that just need to be fixed anyways such as modeling when a blend can be completely accomplished via VINSERTF128, etc. Also, this highlights one of the few remaining big features: we do a really poor job of inserting elements into AVX registers efficiently. This completes almost all of the big tricks I have in mind for AVX2. The only things left that I plan to add: 1) element insertion smarts 2) palignr and other fairly specialized lowerings when they happen to apply llvm-svn: 218449
2014-09-25 19:03:55 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
define <4 x double> @shuffle_v4f64_0023(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_0023:
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[0,0,2,3]
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL-NEXT: retq
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 0, i32 2, i32 3>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
define <4 x double> @shuffle_v4f64_0022(<4 x double> %a, <4 x double> %b) {
; ALL-LABEL: shuffle_v4f64_0022:
; ALL: # BB#0:
; ALL-NEXT: vmovddup {{.*#+}} ymm0 = ymm0[0,0,2,2]
; ALL-NEXT: retq
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 0, i32 2, i32 2>
ret <4 x double> %shuffle
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
define <4 x double> @shuffle_v4f64_1032(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_1032:
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,3,2]
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL-NEXT: retq
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 1, i32 0, i32 3, i32 2>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
define <4 x double> @shuffle_v4f64_1133(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_1133:
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,1,3,3]
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL-NEXT: retq
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 1, i32 1, i32 3, i32 3>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
define <4 x double> @shuffle_v4f64_1023(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_1023:
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,2,3]
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL-NEXT: retq
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 1, i32 0, i32 2, i32 3>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
define <4 x double> @shuffle_v4f64_1022(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_1022:
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,2,2]
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL-NEXT: retq
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 1, i32 0, i32 2, i32 2>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
define <4 x double> @shuffle_v4f64_0423(<4 x double> %a, <4 x double> %b) {
; AVX1-LABEL: shuffle_v4f64_0423:
; AVX1: # BB#0:
; AVX1-NEXT: vmovddup {{.*#+}} ymm1 = ymm1[0,0,2,2]
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0],ymm1[1],ymm0[2,3]
; AVX1-NEXT: retq
;
[x86] Teach the new vector shuffle lowering about VBROADCAST and VPBROADCAST. This has the somewhat expected pervasive impact. I don't know why I forgot about this. Everything seems good with lots of significant improvements in the tests. llvm-svn: 218724
2014-10-01 08:41:21 +08:00
; AVX2-LABEL: shuffle_v4f64_0423:
; AVX2: # BB#0:
; AVX2-NEXT: vbroadcastsd %xmm1, %ymm1
; AVX2-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0],ymm1[1],ymm0[2,3]
; AVX2-NEXT: retq
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 4, i32 2, i32 3>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
define <4 x double> @shuffle_v4f64_0462(<4 x double> %a, <4 x double> %b) {
; ALL-LABEL: shuffle_v4f64_0462:
; ALL: # BB#0:
; ALL-NEXT: vmovddup {{.*#+}} ymm1 = ymm1[0,0,2,2]
; ALL-NEXT: vmovddup {{.*#+}} ymm0 = ymm0[0,0,2,2]
; ALL-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0],ymm1[1,2],ymm0[3]
; ALL-NEXT: retq
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 4, i32 6, i32 2>
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
define <4 x double> @shuffle_v4f64_0426(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_0426:
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vunpcklpd {{.*#+}} ymm0 = ymm0[0],ymm1[0],ymm0[2],ymm1[2]
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL-NEXT: retq
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 4, i32 2, i32 6>
ret <4 x double> %shuffle
[x86] Teach the new AVX v4f64 shuffle lowering to use UNPCK instructions where applicable for blending. llvm-svn: 215737
2014-08-16 01:42:00 +08:00
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Teach the new AVX v4f64 shuffle lowering to use UNPCK instructions where applicable for blending. llvm-svn: 215737
2014-08-16 01:42:00 +08:00
define <4 x double> @shuffle_v4f64_1537(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_1537:
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vunpckhpd {{.*#+}} ymm0 = ymm0[1],ymm1[1],ymm0[3],ymm1[3]
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL-NEXT: retq
[x86] Teach the new AVX v4f64 shuffle lowering to use UNPCK instructions where applicable for blending. llvm-svn: 215737
2014-08-16 01:42:00 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 1, i32 5, i32 3, i32 7>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Teach the new AVX v4f64 shuffle lowering to use UNPCK instructions where applicable for blending. llvm-svn: 215737
2014-08-16 01:42:00 +08:00
define <4 x double> @shuffle_v4f64_4062(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_4062:
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vunpcklpd {{.*#+}} ymm0 = ymm1[0],ymm0[0],ymm1[2],ymm0[2]
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL-NEXT: retq
[x86] Teach the new AVX v4f64 shuffle lowering to use UNPCK instructions where applicable for blending. llvm-svn: 215737
2014-08-16 01:42:00 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 4, i32 0, i32 6, i32 2>
ret <4 x double> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Teach the new AVX v4f64 shuffle lowering to use UNPCK instructions where applicable for blending. llvm-svn: 215737
2014-08-16 01:42:00 +08:00
define <4 x double> @shuffle_v4f64_5173(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_5173:
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vunpckhpd {{.*#+}} ymm0 = ymm1[1],ymm0[1],ymm1[3],ymm0[3]
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL-NEXT: retq
[x86] Teach the new AVX v4f64 shuffle lowering to use UNPCK instructions where applicable for blending. llvm-svn: 215737
2014-08-16 01:42:00 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 5, i32 1, i32 7, i32 3>
ret <4 x double> %shuffle
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
define <4 x double> @shuffle_v4f64_5163(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_5163:
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vshufpd {{.*#+}} ymm0 = ymm1[1],ymm0[1],ymm1[2],ymm0[3]
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
; ALL-NEXT: retq
[x86] Add the initial skeleton of type-based dispatch for AVX vectors in the new shuffle lowering and an implementation for v4 shuffles. This allows us to handle non-half-crossing shuffles directly for v4 shuffles, both integer and floating point. This currently misses places where we could perform the blend via UNPCK instructions, but otherwise generates equally good or better code for the test cases included to the existing vector shuffle lowering. There are a few cases that are entertainingly better. ;] llvm-svn: 215702
2014-08-15 19:01:40 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 5, i32 1, i32 6, i32 3>
ret <4 x double> %shuffle
}
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
[x86] Add some more comprehensive tests for v4f64 blending. llvm-svn: 218207
2014-09-21 19:12:19 +08:00
define <4 x double> @shuffle_v4f64_0527(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_0527:
[x86] Add some more comprehensive tests for v4f64 blending. llvm-svn: 218207
2014-09-21 19:12:19 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0],ymm1[1],ymm0[2],ymm1[3]
[x86] Add some more comprehensive tests for v4f64 blending. llvm-svn: 218207
2014-09-21 19:12:19 +08:00
; ALL-NEXT: retq
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 5, i32 2, i32 7>
ret <4 x double> %shuffle
}
define <4 x double> @shuffle_v4f64_4163(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_4163:
[x86] Add some more comprehensive tests for v4f64 blending. llvm-svn: 218207
2014-09-21 19:12:19 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vblendpd {{.*#+}} ymm0 = ymm1[0],ymm0[1],ymm1[2],ymm0[3]
[x86] Add some more comprehensive tests for v4f64 blending. llvm-svn: 218207
2014-09-21 19:12:19 +08:00
; ALL-NEXT: retq
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 4, i32 1, i32 6, i32 3>
ret <4 x double> %shuffle
}
define <4 x double> @shuffle_v4f64_0145(<4 x double> %a, <4 x double> %b) {
[x86] Teach the new vector shuffle lowering how to lower 128-bit shuffles using AVX and AVX2 instructions. This fixes PR21138, one of the few remaining regressions impacting benchmarks from the new vector shuffle lowering. You may note that it "regresses" many of the vperm2x128 test cases -- these were actually "improved" by the naive lowering that the new shuffle lowering previously did. This regression gave me fits. I had this patch ready-to-go about an hour after flipping the switch but wasn't sure how to have the best of both worlds here and thought the correct solution might be a completely different approach to lowering these vector shuffles. I'm now convinced this is the correct lowering and the missed optimizations shown in vperm2x128 are actually due to missing target-independent DAG combines. I've even written most of the needed DAG combine and will submit it shortly, but this part is ready and should help some real-world benchmarks out. llvm-svn: 219079
2014-10-05 19:41:36 +08:00
; ALL-LABEL: shuffle_v4f64_0145:
; ALL: # BB#0:
; ALL-NEXT: vinsertf128 $1, %xmm1, %ymm0, %ymm0
; ALL-NEXT: retq
[x86] Add some more comprehensive tests for v4f64 blending. llvm-svn: 218207
2014-09-21 19:12:19 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 1, i32 4, i32 5>
ret <4 x double> %shuffle
}
define <4 x double> @shuffle_v4f64_4501(<4 x double> %a, <4 x double> %b) {
[x86] Teach the new vector shuffle lowering how to lower 128-bit shuffles using AVX and AVX2 instructions. This fixes PR21138, one of the few remaining regressions impacting benchmarks from the new vector shuffle lowering. You may note that it "regresses" many of the vperm2x128 test cases -- these were actually "improved" by the naive lowering that the new shuffle lowering previously did. This regression gave me fits. I had this patch ready-to-go about an hour after flipping the switch but wasn't sure how to have the best of both worlds here and thought the correct solution might be a completely different approach to lowering these vector shuffles. I'm now convinced this is the correct lowering and the missed optimizations shown in vperm2x128 are actually due to missing target-independent DAG combines. I've even written most of the needed DAG combine and will submit it shortly, but this part is ready and should help some real-world benchmarks out. llvm-svn: 219079
2014-10-05 19:41:36 +08:00
; ALL-LABEL: shuffle_v4f64_4501:
; ALL: # BB#0:
; ALL-NEXT: vinsertf128 $1, %xmm0, %ymm1, %ymm0
; ALL-NEXT: retq
[x86] Add some more comprehensive tests for v4f64 blending. llvm-svn: 218207
2014-09-21 19:12:19 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 4, i32 5, i32 0, i32 1>
ret <4 x double> %shuffle
}
define <4 x double> @shuffle_v4f64_0167(<4 x double> %a, <4 x double> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: shuffle_v4f64_0167:
[x86] Add some more comprehensive tests for v4f64 blending. llvm-svn: 218207
2014-09-21 19:12:19 +08:00
; ALL: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0,1],ymm1[2,3]
[x86] Add some more comprehensive tests for v4f64 blending. llvm-svn: 218207
2014-09-21 19:12:19 +08:00
; ALL-NEXT: retq
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 1, i32 6, i32 7>
ret <4 x double> %shuffle
}
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
define <4 x double> @shuffle_v4f64_1054(<4 x double> %a, <4 x double> %b) {
[x86] Teach the x86 vector shuffle lowering to detect mergable 128-bit lanes. By special casing these we can often either reduce the total number of shuffles significantly or reduce the number of (high latency on Haswell) AVX2 shuffles that potentially cross 128-bit lanes. Even when these don't actually cross lanes, they have much higher latency to support that. Doing two of them and a blend is worse than doing a single insert across the 128-bit lanes to blend and then doing a single interleaved shuffle. While this seems like a narrow case, it kept cropping up on me and the difference is *huge* as you can see in many of the test cases. I first hit this trying to perfectly fix the interleaving shuffle patterns used by Halide for AVX2. llvm-svn: 222533
2014-11-21 21:56:05 +08:00
; ALL-LABEL: shuffle_v4f64_1054:
; ALL: # BB#0:
; ALL-NEXT: vinsertf128 $1, %xmm1, %ymm0, %ymm0
; ALL-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,3,2]
; ALL-NEXT: retq
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 1, i32 0, i32 5, i32 4>
ret <4 x double> %shuffle
}
define <4 x double> @shuffle_v4f64_3254(<4 x double> %a, <4 x double> %b) {
[x86] Teach the x86 vector shuffle lowering to detect mergable 128-bit lanes. By special casing these we can often either reduce the total number of shuffles significantly or reduce the number of (high latency on Haswell) AVX2 shuffles that potentially cross 128-bit lanes. Even when these don't actually cross lanes, they have much higher latency to support that. Doing two of them and a blend is worse than doing a single insert across the 128-bit lanes to blend and then doing a single interleaved shuffle. While this seems like a narrow case, it kept cropping up on me and the difference is *huge* as you can see in many of the test cases. I first hit this trying to perfectly fix the interleaving shuffle patterns used by Halide for AVX2. llvm-svn: 222533
2014-11-21 21:56:05 +08:00
; ALL-LABEL: shuffle_v4f64_3254:
; ALL: # BB#0:
; ALL-NEXT: vperm2f128 {{.*#+}} ymm0 = ymm0[2,3],ymm1[0,1]
; ALL-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,3,2]
; ALL-NEXT: retq
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 3, i32 2, i32 5, i32 4>
ret <4 x double> %shuffle
}
define <4 x double> @shuffle_v4f64_3276(<4 x double> %a, <4 x double> %b) {
[x86] Teach the x86 vector shuffle lowering to detect mergable 128-bit lanes. By special casing these we can often either reduce the total number of shuffles significantly or reduce the number of (high latency on Haswell) AVX2 shuffles that potentially cross 128-bit lanes. Even when these don't actually cross lanes, they have much higher latency to support that. Doing two of them and a blend is worse than doing a single insert across the 128-bit lanes to blend and then doing a single interleaved shuffle. While this seems like a narrow case, it kept cropping up on me and the difference is *huge* as you can see in many of the test cases. I first hit this trying to perfectly fix the interleaving shuffle patterns used by Halide for AVX2. llvm-svn: 222533
2014-11-21 21:56:05 +08:00
; ALL-LABEL: shuffle_v4f64_3276:
; ALL: # BB#0:
; ALL-NEXT: vperm2f128 {{.*#+}} ymm0 = ymm0[2,3],ymm1[2,3]
; ALL-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,3,2]
; ALL-NEXT: retq
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 3, i32 2, i32 7, i32 6>
ret <4 x double> %shuffle
}
define <4 x double> @shuffle_v4f64_1076(<4 x double> %a, <4 x double> %b) {
[x86] Make the previous logic significantly less conservative and get a bunch more improvements. Non-lane-crossing is fine, the key is that lane merging only makes sense for single-input shuffles. Not sure why I got so turned around here. The code all works, I was just using the wrong model for it. This only updates v4 and v8 lowering. The v16 and v32 lowering requires restructuring the entire check sequence. llvm-svn: 222537
2014-11-21 22:33:24 +08:00
; ALL-LABEL: shuffle_v4f64_1076:
; ALL: # BB#0:
; ALL-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0,1],ymm1[2,3]
; ALL-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,3,2]
; ALL-NEXT: retq
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 1, i32 0, i32 7, i32 6>
ret <4 x double> %shuffle
}
[x86] Add some tests for a common unpack pattern of vector shuffle that has a remarkably unique and efficient lowering. While we get this some of the time already, we miss a few cases and there wasn't a principled reason we got it. We should at least test this. v8 already has tests for this pattern. llvm-svn: 222607
2014-11-22 13:44:43 +08:00
define <4 x double> @shuffle_v4f64_0415(<4 x double> %a, <4 x double> %b) {
; AVX1-LABEL: shuffle_v4f64_0415:
; AVX1: # BB#0:
; AVX1-NEXT: vunpckhpd {{.*#+}} xmm2 = xmm0[1],xmm1[1]
; AVX1-NEXT: vunpcklpd {{.*#+}} xmm0 = xmm0[0],xmm1[0]
; AVX1-NEXT: vinsertf128 $1, %xmm2, %ymm0, %ymm0
; AVX1-NEXT: retq
;
; AVX2-LABEL: shuffle_v4f64_0415:
; AVX2: # BB#0:
; AVX2-NEXT: vpermpd {{.*#+}} ymm1 = ymm1[0,0,2,1]
; AVX2-NEXT: vpermpd {{.*#+}} ymm0 = ymm0[0,1,1,3]
; AVX2-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0],ymm1[1],ymm0[2],ymm1[3]
; AVX2-NEXT: retq
%shuffle = shufflevector <4 x double> %a, <4 x double> %b, <4 x i32> <i32 0, i32 4, i32 1, i32 5>
ret <4 x double> %shuffle
}
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
define <4 x i64> @shuffle_v4i64_0000(<4 x i64> %a, <4 x i64> %b) {
; AVX1-LABEL: shuffle_v4i64_0000:
; AVX1: # BB#0:
; AVX1-NEXT: vmovddup {{.*#+}} xmm0 = xmm0[0,0]
; AVX1-NEXT: vinsertf128 $1, %xmm0, %ymm0, %ymm0
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_0000:
[x86] Add some vector-register broadcast operations to the 256-bit v4 tests which were missing them. llvm-svn: 218657
2014-09-30 10:32:36 +08:00
; AVX2: # BB#0:
[x86] Teach the new vector shuffle lowering about VBROADCAST and VPBROADCAST. This has the somewhat expected pervasive impact. I don't know why I forgot about this. Everything seems good with lots of significant improvements in the tests. llvm-svn: 218724
2014-10-01 08:41:21 +08:00
; AVX2-NEXT: vbroadcastsd %xmm0, %ymm0
[x86] Add some vector-register broadcast operations to the 256-bit v4 tests which were missing them. llvm-svn: 218657
2014-09-30 10:32:36 +08:00
; AVX2-NEXT: retq
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 0, i32 0, i32 0, i32 0>
ret <4 x i64> %shuffle
}
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
define <4 x i64> @shuffle_v4i64_0001(<4 x i64> %a, <4 x i64> %b) {
; AVX1-LABEL: shuffle_v4i64_0001:
; AVX1: # BB#0:
; AVX1-NEXT: vmovddup {{.*#+}} xmm1 = xmm0[0,0]
; AVX1-NEXT: vinsertf128 $1, %xmm0, %ymm1, %ymm0
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_0001:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[0,0,0,1]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Move the AVX v4i64 test cases down to group them together. Increasingly I don't want to mix the integer and floating point tests, especially with AVX where they are handled quite differently. llvm-svn: 218233
2014-09-22 11:05:23 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 0, i32 0, i32 0, i32 1>
ret <4 x i64> %shuffle
}
define <4 x i64> @shuffle_v4i64_0020(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_0020:
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; AVX1: # BB#0:
; AVX1-NEXT: vextractf128 $1, %ymm0, %xmm1
; AVX1-NEXT: vunpcklpd {{.*#+}} xmm1 = xmm1[0],xmm0[0]
; AVX1-NEXT: vmovddup {{.*#+}} xmm0 = xmm0[0,0]
; AVX1-NEXT: vinsertf128 $1, %xmm1, %ymm0, %ymm0
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_0020:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[0,0,2,0]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Move the AVX v4i64 test cases down to group them together. Increasingly I don't want to mix the integer and floating point tests, especially with AVX where they are handled quite differently. llvm-svn: 218233
2014-09-22 11:05:23 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 0, i32 0, i32 2, i32 0>
ret <4 x i64> %shuffle
}
define <4 x i64> @shuffle_v4i64_0112(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_0112:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1: # BB#0:
; AVX1-NEXT: vextractf128 $1, %ymm0, %xmm1
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-NEXT: vshufpd {{.*#+}} xmm1 = xmm0[1],xmm1[0]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1-NEXT: vinsertf128 $1, %xmm1, %ymm0, %ymm0
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_0112:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[0,1,1,2]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Move the AVX v4i64 test cases down to group them together. Increasingly I don't want to mix the integer and floating point tests, especially with AVX where they are handled quite differently. llvm-svn: 218233
2014-09-22 11:05:23 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 0, i32 1, i32 1, i32 2>
ret <4 x i64> %shuffle
}
define <4 x i64> @shuffle_v4i64_0300(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_0300:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-NEXT: vperm2f128 {{.*#+}} ymm1 = ymm0[2,3,0,1]
; AVX1-NEXT: vpermilpd {{.*#+}} ymm1 = ymm1[0,1,2,2]
[x86] Add a really preposterous number of patterns for matching all of the various ways in which blends can be used to do vector element insertion for lowering with the scalar math instruction forms that effectively re-blend with the high elements after performing the operation. This then allows me to bail on the element insertion lowering path when we have SSE4.1 and are going to be doing a normal blend, which in turn restores the last of the blends lost from the new vector shuffle lowering when I got it to prioritize insertion in other cases (for example when we don't *have* a blend instruction). Without the patterns, using blends here would have regressed sse-scalar-fp-arith.ll *completely* with the new vector shuffle lowering. For completeness, I've added RUN-lines with the new lowering here. This is somewhat superfluous as I'm about to flip the default, but hey, it shows that this actually significantly changed behavior. The patterns I've added are just ridiculously repetative. Suggestions on making them better very much welcome. In particular, handling the commuted form of the v2f64 patterns is somewhat obnoxious. llvm-svn: 219033
2014-10-04 06:43:17 +08:00
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0],ymm1[1,2,3]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_0300:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[0,3,0,0]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Move the AVX v4i64 test cases down to group them together. Increasingly I don't want to mix the integer and floating point tests, especially with AVX where they are handled quite differently. llvm-svn: 218233
2014-09-22 11:05:23 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 0, i32 3, i32 0, i32 0>
ret <4 x i64> %shuffle
}
define <4 x i64> @shuffle_v4i64_1000(<4 x i64> %a, <4 x i64> %b) {
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; AVX1-LABEL: shuffle_v4i64_1000:
; AVX1: # BB#0:
; AVX1-NEXT: vpermilpd {{.*#+}} xmm1 = xmm0[1,0]
; AVX1-NEXT: vmovddup {{.*#+}} xmm0 = xmm0[0,0]
; AVX1-NEXT: vinsertf128 $1, %xmm0, %ymm1, %ymm0
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_1000:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[1,0,0,0]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Move the AVX v4i64 test cases down to group them together. Increasingly I don't want to mix the integer and floating point tests, especially with AVX where they are handled quite differently. llvm-svn: 218233
2014-09-22 11:05:23 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 1, i32 0, i32 0, i32 0>
ret <4 x i64> %shuffle
}
define <4 x i64> @shuffle_v4i64_2200(<4 x i64> %a, <4 x i64> %b) {
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; AVX1-LABEL: shuffle_v4i64_2200:
; AVX1: # BB#0:
; AVX1-NEXT: vperm2f128 {{.*#+}} ymm0 = ymm0[2,3,0,1]
; AVX1-NEXT: vmovddup {{.*#+}} ymm0 = ymm0[0,0,2,2]
; AVX1-NEXT: retq
;
; AVX2-LABEL: shuffle_v4i64_2200:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[2,2,0,0]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Move the AVX v4i64 test cases down to group them together. Increasingly I don't want to mix the integer and floating point tests, especially with AVX where they are handled quite differently. llvm-svn: 218233
2014-09-22 11:05:23 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 2, i32 2, i32 0, i32 0>
ret <4 x i64> %shuffle
}
define <4 x i64> @shuffle_v4i64_3330(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_3330:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-NEXT: vperm2f128 {{.*#+}} ymm1 = ymm0[2,3,0,1]
; AVX1-NEXT: vpermilpd {{.*#+}} ymm1 = ymm1[1,1,2,2]
; AVX1-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[0,0,3,2]
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm1[0,1],ymm0[2],ymm1[3]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_3330:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[3,3,3,0]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Move the AVX v4i64 test cases down to group them together. Increasingly I don't want to mix the integer and floating point tests, especially with AVX where they are handled quite differently. llvm-svn: 218233
2014-09-22 11:05:23 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 3, i32 3, i32 3, i32 0>
ret <4 x i64> %shuffle
}
define <4 x i64> @shuffle_v4i64_3210(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_3210:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-NEXT: vperm2f128 {{.*#+}} ymm0 = ymm0[2,3,0,1]
; AVX1-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,3,2]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_3210:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[3,2,1,0]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Move the AVX v4i64 test cases down to group them together. Increasingly I don't want to mix the integer and floating point tests, especially with AVX where they are handled quite differently. llvm-svn: 218233
2014-09-22 11:05:23 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
ret <4 x i64> %shuffle
}
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
define <4 x i64> @shuffle_v4i64_0124(<4 x i64> %a, <4 x i64> %b) {
; AVX1-LABEL: shuffle_v4i64_0124:
; AVX1: # BB#0:
; AVX1-NEXT: vmovddup {{.*#+}} xmm1 = xmm1[0,0]
; AVX1-NEXT: vinsertf128 $1, %xmm1, %ymm0, %ymm1
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0,1,2],ymm1[3]
; AVX1-NEXT: retq
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_0124:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Teach the new vector shuffle lowering about VBROADCAST and VPBROADCAST. This has the somewhat expected pervasive impact. I don't know why I forgot about this. Everything seems good with lots of significant improvements in the tests. llvm-svn: 218724
2014-10-01 08:41:21 +08:00
; AVX2-NEXT: vpbroadcastq %xmm1, %ymm1
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm0[0,1,2,3,4,5],ymm1[6,7]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 0, i32 1, i32 2, i32 4>
ret <4 x i64> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x i64> @shuffle_v4i64_0142(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_0142:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1: # BB#0:
[x86] Teach the new vector shuffle lowering how to lower 128-bit shuffles using AVX and AVX2 instructions. This fixes PR21138, one of the few remaining regressions impacting benchmarks from the new vector shuffle lowering. You may note that it "regresses" many of the vperm2x128 test cases -- these were actually "improved" by the naive lowering that the new shuffle lowering previously did. This regression gave me fits. I had this patch ready-to-go about an hour after flipping the switch but wasn't sure how to have the best of both worlds here and thought the correct solution might be a completely different approach to lowering these vector shuffles. I'm now convinced this is the correct lowering and the missed optimizations shown in vperm2x128 are actually due to missing target-independent DAG combines. I've even written most of the needed DAG combine and will submit it shortly, but this part is ready and should help some real-world benchmarks out. llvm-svn: 219079
2014-10-05 19:41:36 +08:00
; AVX1-NEXT: vinsertf128 $1, %xmm1, %ymm1, %ymm1
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[0,1,2,2]
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0,1],ymm1[2],ymm0[3]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_0142:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Teach the new vector shuffle lowering how to lower 128-bit shuffles using AVX and AVX2 instructions. This fixes PR21138, one of the few remaining regressions impacting benchmarks from the new vector shuffle lowering. You may note that it "regresses" many of the vperm2x128 test cases -- these were actually "improved" by the naive lowering that the new shuffle lowering previously did. This regression gave me fits. I had this patch ready-to-go about an hour after flipping the switch but wasn't sure how to have the best of both worlds here and thought the correct solution might be a completely different approach to lowering these vector shuffles. I'm now convinced this is the correct lowering and the missed optimizations shown in vperm2x128 are actually due to missing target-independent DAG combines. I've even written most of the needed DAG combine and will submit it shortly, but this part is ready and should help some real-world benchmarks out. llvm-svn: 219079
2014-10-05 19:41:36 +08:00
; AVX2-NEXT: vinserti128 $1, %xmm1, %ymm1, %ymm1
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[0,1,2,2]
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm0[0,1,2,3],ymm1[4,5],ymm0[6,7]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 0, i32 1, i32 4, i32 2>
ret <4 x i64> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x i64> @shuffle_v4i64_0412(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_0412:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1: # BB#0:
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; AVX1-NEXT: vextractf128 $1, %ymm0, %xmm2
; AVX1-NEXT: vshufpd {{.*#+}} xmm2 = xmm0[1],xmm2[0]
; AVX1-NEXT: vinsertf128 $1, %xmm2, %ymm0, %ymm0
; AVX1-NEXT: vmovddup {{.*#+}} ymm1 = ymm1[0,0,2,2]
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0],ymm1[1],ymm0[2,3]
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_0412:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[0,1,1,2]
[x86] Teach the new vector shuffle lowering about VBROADCAST and VPBROADCAST. This has the somewhat expected pervasive impact. I don't know why I forgot about this. Everything seems good with lots of significant improvements in the tests. llvm-svn: 218724
2014-10-01 08:41:21 +08:00
; AVX2-NEXT: vpbroadcastq %xmm1, %ymm1
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm0[0,1],ymm1[2,3],ymm0[4,5,6,7]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 0, i32 4, i32 1, i32 2>
ret <4 x i64> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x i64> @shuffle_v4i64_4012(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_4012:
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; AVX1: # BB#0:
; AVX1-NEXT: vextractf128 $1, %ymm0, %xmm2
; AVX1-NEXT: vshufpd {{.*#+}} xmm2 = xmm0[1],xmm2[0]
; AVX1-NEXT: vmovddup {{.*#+}} xmm0 = xmm0[0,0]
; AVX1-NEXT: vinsertf128 $1, %xmm2, %ymm0, %ymm0
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm1[0],ymm0[1,2,3]
; AVX1-NEXT: retq
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_4012:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[0,0,1,2]
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm1[0,1],ymm0[2,3,4,5,6,7]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 4, i32 0, i32 1, i32 2>
ret <4 x i64> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x i64> @shuffle_v4i64_0145(<4 x i64> %a, <4 x i64> %b) {
[x86] Teach the new vector shuffle lowering how to lower 128-bit shuffles using AVX and AVX2 instructions. This fixes PR21138, one of the few remaining regressions impacting benchmarks from the new vector shuffle lowering. You may note that it "regresses" many of the vperm2x128 test cases -- these were actually "improved" by the naive lowering that the new shuffle lowering previously did. This regression gave me fits. I had this patch ready-to-go about an hour after flipping the switch but wasn't sure how to have the best of both worlds here and thought the correct solution might be a completely different approach to lowering these vector shuffles. I'm now convinced this is the correct lowering and the missed optimizations shown in vperm2x128 are actually due to missing target-independent DAG combines. I've even written most of the needed DAG combine and will submit it shortly, but this part is ready and should help some real-world benchmarks out. llvm-svn: 219079
2014-10-05 19:41:36 +08:00
; ALL-LABEL: shuffle_v4i64_0145:
; ALL: # BB#0:
; ALL-NEXT: vinsertf128 $1, %xmm1, %ymm0, %ymm0
; ALL-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 0, i32 1, i32 4, i32 5>
ret <4 x i64> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x i64> @shuffle_v4i64_0451(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_0451:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1: # BB#0:
[x86] Teach the vector shuffle lowering to make a more nuanced decision between splitting a vector into 128-bit lanes and recombining them vs. decomposing things into single-input shuffles and a final blend. This handles a large number of cases in AVX1 where the cross-lane shuffles would be much more expensive to represent even though we end up with a fast blend at the root. Instead, we can do a better job of shuffling in a single lane and then inserting it into the other lanes. This fixes the remaining bits of Halide's regression captured in PR21281 for AVX1. However, the bug persists in AVX2 because I've made this change reasonably conservative. The cases where it makes sense in AVX2 to split into 128-bit lanes are much more rare because we can often do full permutations across all elements of the 256-bit vector. However, the particular test case in PR21281 is an example of one of the rare cases where it is *always* better to work in a single 128-bit lane. I'm going to try to teach the logic to detect and form the good code even in AVX2 next, but it will need to use a separate heuristic. Finally, there is one pesky regression here where we previously would craftily use vpermilps in AVX1 to shuffle both high and low halves at the same time. We no longer pull that off, and not for any really good reason. Ultimately, I think this is just another missing nuance to the selection heuristic that I'll try to add in afterward, but this change already seems strictly worth doing considering the magnitude of the improvements in common matrix math shuffle patterns. As always, please let me know if this causes a surprising regression for you. llvm-svn: 221861
2014-11-13 12:06:10 +08:00
; AVX1-NEXT: vunpckhpd {{.*#+}} xmm2 = xmm1[1],xmm0[1]
; AVX1-NEXT: vunpcklpd {{.*#+}} xmm0 = xmm0[0],xmm1[0]
; AVX1-NEXT: vinsertf128 $1, %xmm2, %ymm0, %ymm0
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_0451:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Teach the new vector shuffle lowering how to lower 128-bit shuffles using AVX and AVX2 instructions. This fixes PR21138, one of the few remaining regressions impacting benchmarks from the new vector shuffle lowering. You may note that it "regresses" many of the vperm2x128 test cases -- these were actually "improved" by the naive lowering that the new shuffle lowering previously did. This regression gave me fits. I had this patch ready-to-go about an hour after flipping the switch but wasn't sure how to have the best of both worlds here and thought the correct solution might be a completely different approach to lowering these vector shuffles. I'm now convinced this is the correct lowering and the missed optimizations shown in vperm2x128 are actually due to missing target-independent DAG combines. I've even written most of the needed DAG combine and will submit it shortly, but this part is ready and should help some real-world benchmarks out. llvm-svn: 219079
2014-10-05 19:41:36 +08:00
; AVX2-NEXT: vinserti128 $1, %xmm0, %ymm0, %ymm0
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm1 = ymm1[0,0,1,3]
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm0[0,1],ymm1[2,3,4,5],ymm0[6,7]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 0, i32 4, i32 5, i32 1>
ret <4 x i64> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x i64> @shuffle_v4i64_4501(<4 x i64> %a, <4 x i64> %b) {
[x86] Teach the new vector shuffle lowering how to lower 128-bit shuffles using AVX and AVX2 instructions. This fixes PR21138, one of the few remaining regressions impacting benchmarks from the new vector shuffle lowering. You may note that it "regresses" many of the vperm2x128 test cases -- these were actually "improved" by the naive lowering that the new shuffle lowering previously did. This regression gave me fits. I had this patch ready-to-go about an hour after flipping the switch but wasn't sure how to have the best of both worlds here and thought the correct solution might be a completely different approach to lowering these vector shuffles. I'm now convinced this is the correct lowering and the missed optimizations shown in vperm2x128 are actually due to missing target-independent DAG combines. I've even written most of the needed DAG combine and will submit it shortly, but this part is ready and should help some real-world benchmarks out. llvm-svn: 219079
2014-10-05 19:41:36 +08:00
; ALL-LABEL: shuffle_v4i64_4501:
; ALL: # BB#0:
; ALL-NEXT: vinsertf128 $1, %xmm0, %ymm1, %ymm0
; ALL-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 4, i32 5, i32 0, i32 1>
ret <4 x i64> %shuffle
}
[x86] Re-generate a bunch of the v4f64 test cases with my new script. This expands the integer cases to cover the fact that AVX2 moves their lane-crossing shuffles into the integer domain. It also adds proper support for AVX2 run lines and the "ALL" group when it doesn't matter. llvm-svn: 218206
2014-09-21 19:07:41 +08:00
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
define <4 x i64> @shuffle_v4i64_4015(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_4015:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1: # BB#0:
[x86] Teach the vector shuffle lowering to make a more nuanced decision between splitting a vector into 128-bit lanes and recombining them vs. decomposing things into single-input shuffles and a final blend. This handles a large number of cases in AVX1 where the cross-lane shuffles would be much more expensive to represent even though we end up with a fast blend at the root. Instead, we can do a better job of shuffling in a single lane and then inserting it into the other lanes. This fixes the remaining bits of Halide's regression captured in PR21281 for AVX1. However, the bug persists in AVX2 because I've made this change reasonably conservative. The cases where it makes sense in AVX2 to split into 128-bit lanes are much more rare because we can often do full permutations across all elements of the 256-bit vector. However, the particular test case in PR21281 is an example of one of the rare cases where it is *always* better to work in a single 128-bit lane. I'm going to try to teach the logic to detect and form the good code even in AVX2 next, but it will need to use a separate heuristic. Finally, there is one pesky regression here where we previously would craftily use vpermilps in AVX1 to shuffle both high and low halves at the same time. We no longer pull that off, and not for any really good reason. Ultimately, I think this is just another missing nuance to the selection heuristic that I'll try to add in afterward, but this change already seems strictly worth doing considering the magnitude of the improvements in common matrix math shuffle patterns. As always, please let me know if this causes a surprising regression for you. llvm-svn: 221861
2014-11-13 12:06:10 +08:00
; AVX1-NEXT: vunpckhpd {{.*#+}} xmm2 = xmm0[1],xmm1[1]
; AVX1-NEXT: vunpcklpd {{.*#+}} xmm0 = xmm1[0],xmm0[0]
; AVX1-NEXT: vinsertf128 $1, %xmm2, %ymm0, %ymm0
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_4015:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[x86] Teach the new vector shuffle lowering how to lower 128-bit shuffles using AVX and AVX2 instructions. This fixes PR21138, one of the few remaining regressions impacting benchmarks from the new vector shuffle lowering. You may note that it "regresses" many of the vperm2x128 test cases -- these were actually "improved" by the naive lowering that the new shuffle lowering previously did. This regression gave me fits. I had this patch ready-to-go about an hour after flipping the switch but wasn't sure how to have the best of both worlds here and thought the correct solution might be a completely different approach to lowering these vector shuffles. I'm now convinced this is the correct lowering and the missed optimizations shown in vperm2x128 are actually due to missing target-independent DAG combines. I've even written most of the needed DAG combine and will submit it shortly, but this part is ready and should help some real-world benchmarks out. llvm-svn: 219079
2014-10-05 19:41:36 +08:00
; AVX2-NEXT: vinserti128 $1, %xmm1, %ymm1, %ymm1
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[0,0,1,3]
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm1[0,1],ymm0[2,3,4,5],ymm1[6,7]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Begin stubbing out the AVX support in the new vector shuffle lowering scheme. Currently, this just directly bails to the fallback path of splitting the 256-bit vector into two 128-bit vectors, operating there, and then joining the results back together. While the results are far from perfect, they are *shockingly* good for what we're doing here. I'll be layering the rest of the functionality on top of this piece by piece and updating tests as I go. Note that 256-bit vectors in this mode are still somewhat WIP. While I think the code paths that I'm adding here are clean and good-to-go, there are still a lot of 128-bit assumptions that I'll need to stomp out as I march through the functional spread here. llvm-svn: 215637
2014-08-14 20:13:59 +08:00
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 4, i32 0, i32 1, i32 5>
ret <4 x i64> %shuffle
}
[x86] Start fixing a really subtle and terrible form of miscompile in these DAG combines. The DAG auto-CSE thing is truly terrible. Due to it, when RAUW-ing a node with its operand, you can cause its uses to CSE to itself, which then causes their uses to become your uses which causes them to be picked up by the RAUW. For nodes that are determined to be "no-ops", this is "fine". But if the RAUW is one of several steps to enact a transformation, this causes the DAG to really silently eat an discard nodes that you would never expect. It took days for me to actually pinpoint a test case triggering this and a really frustrating amount of time to even comprehend the bug because I never even thought about the ability of RAUW to iteratively consume nodes due to CSE-ing them into itself. To fix this, we have to build up a brand-new chain of operations any time we are combining across (potentially) intervening nodes. But once the logic is added to do this, another issue surfaces: CombineTo eagerly deletes the one node combined, *but no others*. This is... really frustrating. If deleting it makes its operands become dead, those operand nodes often won't go onto the worklist in the order you would want -- they're already on it and not near the top. That means things higher on the worklist will get combined prior to these dead nodes being GCed out of the worklist, and if the chain is long, the immediate users won't be enough to re-detect where the root of the chain is that became single-use again after deleting the dead nodes. The better way to do this is to never immediately delete nodes, and instead to just enqueue them so we can recursively delete them. The combined-from node is typically not on the worklist anyways by virtue of having been popped off.... But that in turn breaks other tests that *require* CombineTo to delete unused nodes. :: sigh :: Fortunately, there is a better way. This whole routine should have been returning the replacement rather than using CombineTo which is quite hacky. Switch to that, and all the pieces fall together. I suspect the same kind of miscompile is possible in the half-shuffle folding code, and potentially the recursive folding code. I'll be switching those over to a pattern more like this one for safety's sake even though I don't immediately have any test cases for them. Note that the only way I got a test case for this instance was with *heavily* DAG combined 256-bit shuffle sequences generated by my fuzzer. ;] llvm-svn: 216319
2014-08-23 18:25:15 +08:00
[x86] Fix a large collection of bugs that crept in as I fleshed out the AVX support. New test cases included. Note that none of the existing test cases covered these buggy code paths. =/ Also, it is clear from this that SHUFPS and SHUFPD are the most bug prone shuffle instructions in x86. =[ These were all detected by fuzz-testing. (I <3 fuzz testing.) llvm-svn: 218522
2014-09-27 01:11:02 +08:00
define <4 x i64> @shuffle_v4i64_2u35(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_2u35:
[x86] Fix a large collection of bugs that crept in as I fleshed out the AVX support. New test cases included. Note that none of the existing test cases covered these buggy code paths. =/ Also, it is clear from this that SHUFPS and SHUFPD are the most bug prone shuffle instructions in x86. =[ These were all detected by fuzz-testing. (I <3 fuzz testing.) llvm-svn: 218522
2014-09-27 01:11:02 +08:00
; AVX1: # BB#0:
; AVX1-NEXT: vextractf128 $1, %ymm0, %xmm0
[x86] Teach the vector shuffle lowering to make a more nuanced decision between splitting a vector into 128-bit lanes and recombining them vs. decomposing things into single-input shuffles and a final blend. This handles a large number of cases in AVX1 where the cross-lane shuffles would be much more expensive to represent even though we end up with a fast blend at the root. Instead, we can do a better job of shuffling in a single lane and then inserting it into the other lanes. This fixes the remaining bits of Halide's regression captured in PR21281 for AVX1. However, the bug persists in AVX2 because I've made this change reasonably conservative. The cases where it makes sense in AVX2 to split into 128-bit lanes are much more rare because we can often do full permutations across all elements of the 256-bit vector. However, the particular test case in PR21281 is an example of one of the rare cases where it is *always* better to work in a single 128-bit lane. I'm going to try to teach the logic to detect and form the good code even in AVX2 next, but it will need to use a separate heuristic. Finally, there is one pesky regression here where we previously would craftily use vpermilps in AVX1 to shuffle both high and low halves at the same time. We no longer pull that off, and not for any really good reason. Ultimately, I think this is just another missing nuance to the selection heuristic that I'll try to add in afterward, but this change already seems strictly worth doing considering the magnitude of the improvements in common matrix math shuffle patterns. As always, please let me know if this causes a surprising regression for you. llvm-svn: 221861
2014-11-13 12:06:10 +08:00
; AVX1-NEXT: vunpckhpd {{.*#+}} xmm1 = xmm0[1],xmm1[1]
; AVX1-NEXT: vinsertf128 $1, %xmm1, %ymm0, %ymm0
[x86] Fix a large collection of bugs that crept in as I fleshed out the AVX support. New test cases included. Note that none of the existing test cases covered these buggy code paths. =/ Also, it is clear from this that SHUFPS and SHUFPD are the most bug prone shuffle instructions in x86. =[ These were all detected by fuzz-testing. (I <3 fuzz testing.) llvm-svn: 218522
2014-09-27 01:11:02 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_2u35:
[x86] Fix a large collection of bugs that crept in as I fleshed out the AVX support. New test cases included. Note that none of the existing test cases covered these buggy code paths. =/ Also, it is clear from this that SHUFPS and SHUFPD are the most bug prone shuffle instructions in x86. =[ These were all detected by fuzz-testing. (I <3 fuzz testing.) llvm-svn: 218522
2014-09-27 01:11:02 +08:00
; AVX2: # BB#0:
[x86] Teach the new vector shuffle lowering how to lower 128-bit shuffles using AVX and AVX2 instructions. This fixes PR21138, one of the few remaining regressions impacting benchmarks from the new vector shuffle lowering. You may note that it "regresses" many of the vperm2x128 test cases -- these were actually "improved" by the naive lowering that the new shuffle lowering previously did. This regression gave me fits. I had this patch ready-to-go about an hour after flipping the switch but wasn't sure how to have the best of both worlds here and thought the correct solution might be a completely different approach to lowering these vector shuffles. I'm now convinced this is the correct lowering and the missed optimizations shown in vperm2x128 are actually due to missing target-independent DAG combines. I've even written most of the needed DAG combine and will submit it shortly, but this part is ready and should help some real-world benchmarks out. llvm-svn: 219079
2014-10-05 19:41:36 +08:00
; AVX2-NEXT: vinserti128 $1, %xmm1, %ymm1, %ymm1
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[2,1,3,3]
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm0[0,1,2,3,4,5],ymm1[6,7]
[x86] Fix a large collection of bugs that crept in as I fleshed out the AVX support. New test cases included. Note that none of the existing test cases covered these buggy code paths. =/ Also, it is clear from this that SHUFPS and SHUFPD are the most bug prone shuffle instructions in x86. =[ These were all detected by fuzz-testing. (I <3 fuzz testing.) llvm-svn: 218522
2014-09-27 01:11:02 +08:00
; AVX2-NEXT: retq
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 2, i32 undef, i32 3, i32 5>
ret <4 x i64> %shuffle
}
define <4 x i64> @shuffle_v4i64_1251(<4 x i64> %a, <4 x i64> %b) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: shuffle_v4i64_1251:
[x86] Fix a large collection of bugs that crept in as I fleshed out the AVX support. New test cases included. Note that none of the existing test cases covered these buggy code paths. =/ Also, it is clear from this that SHUFPS and SHUFPD are the most bug prone shuffle instructions in x86. =[ These were all detected by fuzz-testing. (I <3 fuzz testing.) llvm-svn: 218522
2014-09-27 01:11:02 +08:00
; AVX1: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-NEXT: vperm2f128 {{.*#+}} ymm2 = ymm0[2,3,0,1]
; AVX1-NEXT: vshufpd {{.*#+}} ymm0 = ymm0[1],ymm2[0],ymm0[2],ymm2[3]
; AVX1-NEXT: vpermilpd {{.*#+}} xmm1 = xmm1[1,0]
[x86] Fix a large collection of bugs that crept in as I fleshed out the AVX support. New test cases included. Note that none of the existing test cases covered these buggy code paths. =/ Also, it is clear from this that SHUFPS and SHUFPD are the most bug prone shuffle instructions in x86. =[ These were all detected by fuzz-testing. (I <3 fuzz testing.) llvm-svn: 218522
2014-09-27 01:11:02 +08:00
; AVX1-NEXT: vinsertf128 $1, %xmm1, %ymm0, %ymm1
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0,1],ymm1[2],ymm0[3]
[x86] Fix a large collection of bugs that crept in as I fleshed out the AVX support. New test cases included. Note that none of the existing test cases covered these buggy code paths. =/ Also, it is clear from this that SHUFPS and SHUFPD are the most bug prone shuffle instructions in x86. =[ These were all detected by fuzz-testing. (I <3 fuzz testing.) llvm-svn: 218522
2014-09-27 01:11:02 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: shuffle_v4i64_1251:
[x86] Fix a large collection of bugs that crept in as I fleshed out the AVX support. New test cases included. Note that none of the existing test cases covered these buggy code paths. =/ Also, it is clear from this that SHUFPS and SHUFPD are the most bug prone shuffle instructions in x86. =[ These were all detected by fuzz-testing. (I <3 fuzz testing.) llvm-svn: 218522
2014-09-27 01:11:02 +08:00
; AVX2: # BB#0:
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpermq {{.*#+}} ymm1 = ymm1[0,1,1,3]
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[1,2,2,1]
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm0[0,1,2,3],ymm1[4,5],ymm0[6,7]
[x86] Fix a large collection of bugs that crept in as I fleshed out the AVX support. New test cases included. Note that none of the existing test cases covered these buggy code paths. =/ Also, it is clear from this that SHUFPS and SHUFPD are the most bug prone shuffle instructions in x86. =[ These were all detected by fuzz-testing. (I <3 fuzz testing.) llvm-svn: 218522
2014-09-27 01:11:02 +08:00
; AVX2-NEXT: retq
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 1, i32 2, i32 5, i32 1>
ret <4 x i64> %shuffle
}
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
define <4 x i64> @shuffle_v4i64_1054(<4 x i64> %a, <4 x i64> %b) {
; AVX1-LABEL: shuffle_v4i64_1054:
; AVX1: # BB#0:
; AVX1-NEXT: vinsertf128 $1, %xmm1, %ymm0, %ymm0
[x86] Teach the x86 vector shuffle lowering to detect mergable 128-bit lanes. By special casing these we can often either reduce the total number of shuffles significantly or reduce the number of (high latency on Haswell) AVX2 shuffles that potentially cross 128-bit lanes. Even when these don't actually cross lanes, they have much higher latency to support that. Doing two of them and a blend is worse than doing a single insert across the 128-bit lanes to blend and then doing a single interleaved shuffle. While this seems like a narrow case, it kept cropping up on me and the difference is *huge* as you can see in many of the test cases. I first hit this trying to perfectly fix the interleaving shuffle patterns used by Halide for AVX2. llvm-svn: 222533
2014-11-21 21:56:05 +08:00
; AVX1-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,3,2]
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
; AVX1-NEXT: retq
;
; AVX2-LABEL: shuffle_v4i64_1054:
; AVX2: # BB#0:
[x86] Teach the x86 vector shuffle lowering to detect mergable 128-bit lanes. By special casing these we can often either reduce the total number of shuffles significantly or reduce the number of (high latency on Haswell) AVX2 shuffles that potentially cross 128-bit lanes. Even when these don't actually cross lanes, they have much higher latency to support that. Doing two of them and a blend is worse than doing a single insert across the 128-bit lanes to blend and then doing a single interleaved shuffle. While this seems like a narrow case, it kept cropping up on me and the difference is *huge* as you can see in many of the test cases. I first hit this trying to perfectly fix the interleaving shuffle patterns used by Halide for AVX2. llvm-svn: 222533
2014-11-21 21:56:05 +08:00
; AVX2-NEXT: vinserti128 $1, %xmm1, %ymm0, %ymm0
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
; AVX2-NEXT: vpshufd {{.*#+}} ymm0 = ymm0[2,3,0,1,6,7,4,5]
; AVX2-NEXT: retq
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 1, i32 0, i32 5, i32 4>
ret <4 x i64> %shuffle
}
define <4 x i64> @shuffle_v4i64_3254(<4 x i64> %a, <4 x i64> %b) {
; AVX1-LABEL: shuffle_v4i64_3254:
; AVX1: # BB#0:
[x86] Teach the x86 vector shuffle lowering to detect mergable 128-bit lanes. By special casing these we can often either reduce the total number of shuffles significantly or reduce the number of (high latency on Haswell) AVX2 shuffles that potentially cross 128-bit lanes. Even when these don't actually cross lanes, they have much higher latency to support that. Doing two of them and a blend is worse than doing a single insert across the 128-bit lanes to blend and then doing a single interleaved shuffle. While this seems like a narrow case, it kept cropping up on me and the difference is *huge* as you can see in many of the test cases. I first hit this trying to perfectly fix the interleaving shuffle patterns used by Halide for AVX2. llvm-svn: 222533
2014-11-21 21:56:05 +08:00
; AVX1-NEXT: vperm2f128 {{.*#+}} ymm0 = ymm0[2,3],ymm1[0,1]
; AVX1-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,3,2]
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
; AVX1-NEXT: retq
;
; AVX2-LABEL: shuffle_v4i64_3254:
; AVX2: # BB#0:
[x86] Teach the x86 vector shuffle lowering to detect mergable 128-bit lanes. By special casing these we can often either reduce the total number of shuffles significantly or reduce the number of (high latency on Haswell) AVX2 shuffles that potentially cross 128-bit lanes. Even when these don't actually cross lanes, they have much higher latency to support that. Doing two of them and a blend is worse than doing a single insert across the 128-bit lanes to blend and then doing a single interleaved shuffle. While this seems like a narrow case, it kept cropping up on me and the difference is *huge* as you can see in many of the test cases. I first hit this trying to perfectly fix the interleaving shuffle patterns used by Halide for AVX2. llvm-svn: 222533
2014-11-21 21:56:05 +08:00
; AVX2-NEXT: vperm2i128 {{.*#+}} ymm0 = ymm0[2,3],ymm1[0,1]
; AVX2-NEXT: vpshufd {{.*#+}} ymm0 = ymm0[2,3,0,1,6,7,4,5]
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
; AVX2-NEXT: retq
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 3, i32 2, i32 5, i32 4>
ret <4 x i64> %shuffle
}
define <4 x i64> @shuffle_v4i64_3276(<4 x i64> %a, <4 x i64> %b) {
; AVX1-LABEL: shuffle_v4i64_3276:
; AVX1: # BB#0:
[x86] Teach the x86 vector shuffle lowering to detect mergable 128-bit lanes. By special casing these we can often either reduce the total number of shuffles significantly or reduce the number of (high latency on Haswell) AVX2 shuffles that potentially cross 128-bit lanes. Even when these don't actually cross lanes, they have much higher latency to support that. Doing two of them and a blend is worse than doing a single insert across the 128-bit lanes to blend and then doing a single interleaved shuffle. While this seems like a narrow case, it kept cropping up on me and the difference is *huge* as you can see in many of the test cases. I first hit this trying to perfectly fix the interleaving shuffle patterns used by Halide for AVX2. llvm-svn: 222533
2014-11-21 21:56:05 +08:00
; AVX1-NEXT: vperm2f128 {{.*#+}} ymm0 = ymm0[2,3],ymm1[2,3]
; AVX1-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,3,2]
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
; AVX1-NEXT: retq
;
; AVX2-LABEL: shuffle_v4i64_3276:
; AVX2: # BB#0:
[x86] Teach the x86 vector shuffle lowering to detect mergable 128-bit lanes. By special casing these we can often either reduce the total number of shuffles significantly or reduce the number of (high latency on Haswell) AVX2 shuffles that potentially cross 128-bit lanes. Even when these don't actually cross lanes, they have much higher latency to support that. Doing two of them and a blend is worse than doing a single insert across the 128-bit lanes to blend and then doing a single interleaved shuffle. While this seems like a narrow case, it kept cropping up on me and the difference is *huge* as you can see in many of the test cases. I first hit this trying to perfectly fix the interleaving shuffle patterns used by Halide for AVX2. llvm-svn: 222533
2014-11-21 21:56:05 +08:00
; AVX2-NEXT: vperm2i128 {{.*#+}} ymm0 = ymm0[2,3],ymm1[2,3]
; AVX2-NEXT: vpshufd {{.*#+}} ymm0 = ymm0[2,3,0,1,6,7,4,5]
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
; AVX2-NEXT: retq
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 3, i32 2, i32 7, i32 6>
ret <4 x i64> %shuffle
}
define <4 x i64> @shuffle_v4i64_1076(<4 x i64> %a, <4 x i64> %b) {
; AVX1-LABEL: shuffle_v4i64_1076:
; AVX1: # BB#0:
[x86] Make the previous logic significantly less conservative and get a bunch more improvements. Non-lane-crossing is fine, the key is that lane merging only makes sense for single-input shuffles. Not sure why I got so turned around here. The code all works, I was just using the wrong model for it. This only updates v4 and v8 lowering. The v16 and v32 lowering requires restructuring the entire check sequence. llvm-svn: 222537
2014-11-21 22:33:24 +08:00
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0,1],ymm1[2,3]
; AVX1-NEXT: vpermilpd {{.*#+}} ymm0 = ymm0[1,0,3,2]
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
; AVX1-NEXT: retq
;
; AVX2-LABEL: shuffle_v4i64_1076:
; AVX2: # BB#0:
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm0[0,1,2,3],ymm1[4,5,6,7]
[x86] Make the previous logic significantly less conservative and get a bunch more improvements. Non-lane-crossing is fine, the key is that lane merging only makes sense for single-input shuffles. Not sure why I got so turned around here. The code all works, I was just using the wrong model for it. This only updates v4 and v8 lowering. The v16 and v32 lowering requires restructuring the entire check sequence. llvm-svn: 222537
2014-11-21 22:33:24 +08:00
; AVX2-NEXT: vpshufd {{.*#+}} ymm0 = ymm0[2,3,0,1,6,7,4,5]
[x86] Add a bunch of test cases to 256-bit shuffles that exercise merging 128-bit subvectors and also shuffling all the elements of those subvectors. Currently we generate pretty bad code for many of these, but I'm testing a patch that should dramatically improve this in addition to making the shuffle lowering robust to other changes. llvm-svn: 222525
2014-11-21 20:17:50 +08:00
; AVX2-NEXT: retq
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 1, i32 0, i32 7, i32 6>
ret <4 x i64> %shuffle
}
[x86] Add some tests for a common unpack pattern of vector shuffle that has a remarkably unique and efficient lowering. While we get this some of the time already, we miss a few cases and there wasn't a principled reason we got it. We should at least test this. v8 already has tests for this pattern. llvm-svn: 222607
2014-11-22 13:44:43 +08:00
define <4 x i64> @shuffle_v4i64_0415(<4 x i64> %a, <4 x i64> %b) {
; AVX1-LABEL: shuffle_v4i64_0415:
; AVX1: # BB#0:
; AVX1-NEXT: vunpckhpd {{.*#+}} xmm2 = xmm0[1],xmm1[1]
; AVX1-NEXT: vunpcklpd {{.*#+}} xmm0 = xmm0[0],xmm1[0]
; AVX1-NEXT: vinsertf128 $1, %xmm2, %ymm0, %ymm0
; AVX1-NEXT: retq
;
; AVX2-LABEL: shuffle_v4i64_0415:
; AVX2: # BB#0:
; AVX2-NEXT: vpermq {{.*#+}} ymm1 = ymm1[0,0,2,1]
; AVX2-NEXT: vpermq {{.*#+}} ymm0 = ymm0[0,1,1,3]
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm0[0,1],ymm1[2,3],ymm0[4,5],ymm1[6,7]
; AVX2-NEXT: retq
%shuffle = shufflevector <4 x i64> %a, <4 x i64> %b, <4 x i32> <i32 0, i32 4, i32 1, i32 5>
ret <4 x i64> %shuffle
}
[x86] Start fixing a really subtle and terrible form of miscompile in these DAG combines. The DAG auto-CSE thing is truly terrible. Due to it, when RAUW-ing a node with its operand, you can cause its uses to CSE to itself, which then causes their uses to become your uses which causes them to be picked up by the RAUW. For nodes that are determined to be "no-ops", this is "fine". But if the RAUW is one of several steps to enact a transformation, this causes the DAG to really silently eat an discard nodes that you would never expect. It took days for me to actually pinpoint a test case triggering this and a really frustrating amount of time to even comprehend the bug because I never even thought about the ability of RAUW to iteratively consume nodes due to CSE-ing them into itself. To fix this, we have to build up a brand-new chain of operations any time we are combining across (potentially) intervening nodes. But once the logic is added to do this, another issue surfaces: CombineTo eagerly deletes the one node combined, *but no others*. This is... really frustrating. If deleting it makes its operands become dead, those operand nodes often won't go onto the worklist in the order you would want -- they're already on it and not near the top. That means things higher on the worklist will get combined prior to these dead nodes being GCed out of the worklist, and if the chain is long, the immediate users won't be enough to re-detect where the root of the chain is that became single-use again after deleting the dead nodes. The better way to do this is to never immediately delete nodes, and instead to just enqueue them so we can recursively delete them. The combined-from node is typically not on the worklist anyways by virtue of having been popped off.... But that in turn breaks other tests that *require* CombineTo to delete unused nodes. :: sigh :: Fortunately, there is a better way. This whole routine should have been returning the replacement rather than using CombineTo which is quite hacky. Switch to that, and all the pieces fall together. I suspect the same kind of miscompile is possible in the half-shuffle folding code, and potentially the recursive folding code. I'll be switching those over to a pattern more like this one for safety's sake even though I don't immediately have any test cases for them. Note that the only way I got a test case for this instance was with *heavily* DAG combined 256-bit shuffle sequences generated by my fuzzer. ;] llvm-svn: 216319
2014-08-23 18:25:15 +08:00
define <4 x i64> @stress_test1(<4 x i64> %a, <4 x i64> %b) {
[x86] Mechanically update a bunch of tests' check lines using the latest version of the script. Changes include: - Using the VEX prefix - Skipping more detail when we have useful shuffle comments to match - Matching more shuffle comments that have been added to the printer (yay!) - Matching the destination registers of some AVX instructions - Stripping trailing whitespace that crept in - Fixing indentation issues Nothing interesting going on here. I'm just trying really hard to ensure these changes don't show up in the diffs with actual changes to the backend. llvm-svn: 228132
2015-02-04 18:46:53 +08:00
; ALL-LABEL: stress_test1:
; ALL: retq
[x86] Start fixing a really subtle and terrible form of miscompile in these DAG combines. The DAG auto-CSE thing is truly terrible. Due to it, when RAUW-ing a node with its operand, you can cause its uses to CSE to itself, which then causes their uses to become your uses which causes them to be picked up by the RAUW. For nodes that are determined to be "no-ops", this is "fine". But if the RAUW is one of several steps to enact a transformation, this causes the DAG to really silently eat an discard nodes that you would never expect. It took days for me to actually pinpoint a test case triggering this and a really frustrating amount of time to even comprehend the bug because I never even thought about the ability of RAUW to iteratively consume nodes due to CSE-ing them into itself. To fix this, we have to build up a brand-new chain of operations any time we are combining across (potentially) intervening nodes. But once the logic is added to do this, another issue surfaces: CombineTo eagerly deletes the one node combined, *but no others*. This is... really frustrating. If deleting it makes its operands become dead, those operand nodes often won't go onto the worklist in the order you would want -- they're already on it and not near the top. That means things higher on the worklist will get combined prior to these dead nodes being GCed out of the worklist, and if the chain is long, the immediate users won't be enough to re-detect where the root of the chain is that became single-use again after deleting the dead nodes. The better way to do this is to never immediately delete nodes, and instead to just enqueue them so we can recursively delete them. The combined-from node is typically not on the worklist anyways by virtue of having been popped off.... But that in turn breaks other tests that *require* CombineTo to delete unused nodes. :: sigh :: Fortunately, there is a better way. This whole routine should have been returning the replacement rather than using CombineTo which is quite hacky. Switch to that, and all the pieces fall together. I suspect the same kind of miscompile is possible in the half-shuffle folding code, and potentially the recursive folding code. I'll be switching those over to a pattern more like this one for safety's sake even though I don't immediately have any test cases for them. Note that the only way I got a test case for this instance was with *heavily* DAG combined 256-bit shuffle sequences generated by my fuzzer. ;] llvm-svn: 216319
2014-08-23 18:25:15 +08:00
%c = shufflevector <4 x i64> %b, <4 x i64> undef, <4 x i32> <i32 3, i32 1, i32 1, i32 0>
%d = shufflevector <4 x i64> %c, <4 x i64> undef, <4 x i32> <i32 3, i32 undef, i32 2, i32 undef>
%e = shufflevector <4 x i64> %b, <4 x i64> undef, <4 x i32> <i32 3, i32 3, i32 1, i32 undef>
%f = shufflevector <4 x i64> %d, <4 x i64> %e, <4 x i32> <i32 5, i32 1, i32 1, i32 0>
ret <4 x i64> %f
}
[x86] Teach the new vector shuffle lowering the basics about insertion of a single element into a zero vector for v4f64 and v4i64 in AVX. Ironically, there is less to see here because xor+blend is so crazy fast that we can't really beat that to zero the high 128-bit lane. llvm-svn: 218214
2014-09-21 20:49:46 +08:00
define <4 x i64> @insert_reg_and_zero_v4i64(i64 %a) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: insert_reg_and_zero_v4i64:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1: # BB#0:
; AVX1-NEXT: vmovq %rdi, %xmm0
[x86] Adjust the patterns for lowering X86vzmovl nodes which don't perform a load to use blendps rather than movss when it is available. For non-loads, blendps is *much* faster. It can execute on two ports in Sandy Bridge and Ivy Bridge, and *three* ports on Haswell. This fixes one of the "regressions" from aggressively taking the "insertion" path in the new vector shuffle lowering. This does highlight one problem with blendps -- it isn't commuted as heavily as it should be. That's future work though. llvm-svn: 219022
2014-10-04 05:38:49 +08:00
; AVX1-NEXT: vxorpd %ymm1, %ymm1, %ymm1
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0],ymm1[1,2,3]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: insert_reg_and_zero_v4i64:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
; AVX2-NEXT: vmovq %rdi, %xmm0
; AVX2-NEXT: vpxor %ymm1, %ymm1, %ymm1
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm0[0,1],ymm1[2,3,4,5,6,7]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Teach the new vector shuffle lowering the basics about insertion of a single element into a zero vector for v4f64 and v4i64 in AVX. Ironically, there is less to see here because xor+blend is so crazy fast that we can't really beat that to zero the high 128-bit lane. llvm-svn: 218214
2014-09-21 20:49:46 +08:00
%v = insertelement <4 x i64> undef, i64 %a, i64 0
%shuffle = shufflevector <4 x i64> %v, <4 x i64> zeroinitializer, <4 x i32> <i32 0, i32 5, i32 6, i32 7>
ret <4 x i64> %shuffle
}
define <4 x i64> @insert_mem_and_zero_v4i64(i64* %ptr) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX1-LABEL: insert_mem_and_zero_v4i64:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1: # BB#0:
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; AVX1-NEXT: vmovq {{.*#+}} xmm0 = mem[0],zero
[x86] Adjust the patterns for lowering X86vzmovl nodes which don't perform a load to use blendps rather than movss when it is available. For non-loads, blendps is *much* faster. It can execute on two ports in Sandy Bridge and Ivy Bridge, and *three* ports on Haswell. This fixes one of the "regressions" from aggressively taking the "insertion" path in the new vector shuffle lowering. This does highlight one problem with blendps -- it isn't commuted as heavily as it should be. That's future work though. llvm-svn: 219022
2014-10-04 05:38:49 +08:00
; AVX1-NEXT: vxorpd %ymm1, %ymm1, %ymm1
; AVX1-NEXT: vblendpd {{.*#+}} ymm0 = ymm0[0],ymm1[1,2,3]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX1-NEXT: retq
;
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-LABEL: insert_mem_and_zero_v4i64:
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2: # BB#0:
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; AVX2-NEXT: vmovq {{.*#+}} xmm0 = mem[0],zero
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: vpxor %ymm1, %ymm1, %ymm1
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; AVX2-NEXT: vpblendd {{.*#+}} ymm0 = ymm0[0,1],ymm1[2,3,4,5,6,7]
[x86] Teach the new vector shuffle lowering to lower v4i64 vector shuffles using the AVX2 instructions. This is the first step of cutting in real AVX2 support. Note that I have spotted at least one bug in the test cases already, but I suspect it was already present and just is getting surfaced. Will investigate next. llvm-svn: 218338
2014-09-24 06:39:02 +08:00
; AVX2-NEXT: retq
[x86] Teach the new vector shuffle lowering the basics about insertion of a single element into a zero vector for v4f64 and v4i64 in AVX. Ironically, there is less to see here because xor+blend is so crazy fast that we can't really beat that to zero the high 128-bit lane. llvm-svn: 218214
2014-09-21 20:49:46 +08:00
%a = load i64* %ptr
%v = insertelement <4 x i64> undef, i64 %a, i64 0
%shuffle = shufflevector <4 x i64> %v, <4 x i64> zeroinitializer, <4 x i32> <i32 0, i32 5, i32 6, i32 7>
ret <4 x i64> %shuffle
}
define <4 x double> @insert_reg_and_zero_v4f64(double %a) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: insert_reg_and_zero_v4f64:
[x86] Teach the new vector shuffle lowering the basics about insertion of a single element into a zero vector for v4f64 and v4i64 in AVX. Ironically, there is less to see here because xor+blend is so crazy fast that we can't really beat that to zero the high 128-bit lane. llvm-svn: 218214
2014-09-21 20:49:46 +08:00
; ALL: # BB#0:
[x86] Give movss and movsd execution domains in the x86 backend. This associates movss and movsd with the packed single and packed double execution domains (resp.). While this is largely cosmetic, as we now don't have weird ping-pong-ing between single and double precision, it is also useful because it avoids the domain fixing algorithm from seeing domain breaks that don't actually exist. It will also be much more important if we have an execution domain default other than packed single, as that would cause us to mix movss and movsd with integer vector code on a regular basis, a very bad mixture. llvm-svn: 228135
2015-02-04 18:58:53 +08:00
; ALL-NEXT: vxorpd %xmm1, %xmm1, %xmm1
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; ALL-NEXT: vmovsd {{.*#+}} xmm0 = xmm0[0],xmm1[1]
[x86] Teach the new vector shuffle lowering the basics about insertion of a single element into a zero vector for v4f64 and v4i64 in AVX. Ironically, there is less to see here because xor+blend is so crazy fast that we can't really beat that to zero the high 128-bit lane. llvm-svn: 218214
2014-09-21 20:49:46 +08:00
; ALL-NEXT: retq
%v = insertelement <4 x double> undef, double %a, i32 0
%shuffle = shufflevector <4 x double> %v, <4 x double> zeroinitializer, <4 x i32> <i32 0, i32 5, i32 6, i32 7>
ret <4 x double> %shuffle
}
define <4 x double> @insert_mem_and_zero_v4f64(double* %ptr) {
[x86] Update the exact FileCheck syntax of the 256-bit and 512-bit shuffle tests to match that used in the script I posted and now used consistently in 128-bit tests. Nothing interesting changing here, just using the label name as the FileCheck label and a slightly more general comment marker consumption strategy. llvm-svn: 218709
2014-10-01 06:04:45 +08:00
; ALL-LABEL: insert_mem_and_zero_v4f64:
[x86] Teach the new vector shuffle lowering the basics about insertion of a single element into a zero vector for v4f64 and v4i64 in AVX. Ironically, there is less to see here because xor+blend is so crazy fast that we can't really beat that to zero the high 128-bit lane. llvm-svn: 218214
2014-09-21 20:49:46 +08:00
; ALL: # BB#0:
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
; ALL-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
[x86] Teach the new vector shuffle lowering the basics about insertion of a single element into a zero vector for v4f64 and v4i64 in AVX. Ironically, there is less to see here because xor+blend is so crazy fast that we can't really beat that to zero the high 128-bit lane. llvm-svn: 218214
2014-09-21 20:49:46 +08:00
; ALL-NEXT: retq
%a = load double* %ptr
%v = insertelement <4 x double> undef, double %a, i32 0
%shuffle = shufflevector <4 x double> %v, <4 x double> zeroinitializer, <4 x i32> <i32 0, i32 5, i32 6, i32 7>
ret <4 x double> %shuffle
}
[x86] Teach the new vector shuffle lowering about VBROADCAST and VPBROADCAST. This has the somewhat expected pervasive impact. I don't know why I forgot about this. Everything seems good with lots of significant improvements in the tests. llvm-svn: 218724
2014-10-01 08:41:21 +08:00
define <4 x double> @splat_mem_v4f64(double* %ptr) {
; ALL-LABEL: splat_mem_v4f64:
; ALL: # BB#0:
; ALL-NEXT: vbroadcastsd (%rdi), %ymm0
; ALL-NEXT: retq
%a = load double* %ptr
%v = insertelement <4 x double> undef, double %a, i32 0
%shuffle = shufflevector <4 x double> %v, <4 x double> undef, <4 x i32> <i32 0, i32 0, i32 0, i32 0>
ret <4 x double> %shuffle
}
define <4 x i64> @splat_mem_v4i64(i64* %ptr) {
; AVX1-LABEL: splat_mem_v4i64:
; AVX1: # BB#0:
[x86] Mechanically update a bunch of tests' check lines using the latest version of the script. Changes include: - Using the VEX prefix - Skipping more detail when we have useful shuffle comments to match - Matching more shuffle comments that have been added to the printer (yay!) - Matching the destination registers of some AVX instructions - Stripping trailing whitespace that crept in - Fixing indentation issues Nothing interesting going on here. I'm just trying really hard to ensure these changes don't show up in the diffs with actual changes to the backend. llvm-svn: 228132
2015-02-04 18:46:53 +08:00
; AVX1-NEXT: vmovddup {{.*#+}} xmm0 = mem[0,0]
[x86] Teach the new vector shuffle lowering about VBROADCAST and VPBROADCAST. This has the somewhat expected pervasive impact. I don't know why I forgot about this. Everything seems good with lots of significant improvements in the tests. llvm-svn: 218724
2014-10-01 08:41:21 +08:00
; AVX1-NEXT: vinsertf128 $1, %xmm0, %ymm0, %ymm0
; AVX1-NEXT: retq
;
; AVX2-LABEL: splat_mem_v4i64:
; AVX2: # BB#0:
; AVX2-NEXT: vbroadcastsd (%rdi), %ymm0
; AVX2-NEXT: retq
%a = load i64* %ptr
%v = insertelement <4 x i64> undef, i64 %a, i64 0
%shuffle = shufflevector <4 x i64> %v, <4 x i64> undef, <4 x i32> <i32 0, i32 0, i32 0, i32 0>
ret <4 x i64> %shuffle
}
Fix a broadcast related regression on the vector shuffle lowering. Summary: Test by Robert Lougher! Reviewers: chandlerc Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D5745 llvm-svn: 219617
2014-10-14 00:16:16 +08:00
define <4 x double> @splat_mem_v4f64_2(double* %p) {
; ALL-LABEL: splat_mem_v4f64_2:
; ALL: # BB#0:
; ALL-NEXT: vbroadcastsd (%rdi), %ymm0
; ALL-NEXT: retq
%1 = load double* %p
%2 = insertelement <2 x double> undef, double %1, i32 0
%3 = shufflevector <2 x double> %2, <2 x double> undef, <4 x i32> zeroinitializer
ret <4 x double> %3
}
[X86][SSE] Shuffle mask decode support for zero extend, scalar float/double moves and integer load instructions This patch adds shuffle mask decodes for integer zero extends (pmovzx** and movq xmm,xmm) and scalar float/double loads/moves (movss/movsd). Also adds shuffle mask decodes for integer loads (movd/movq). Differential Revision: http://reviews.llvm.org/D7228 llvm-svn: 227688
2015-01-31 22:09:36 +08:00
define <4 x double> @splat_v4f64(<2 x double> %r) {
; AVX1-LABEL: splat_v4f64:
; AVX1: # BB#0:
; AVX1-NEXT: vmovddup {{.*#+}} xmm0 = xmm0[0,0]
; AVX1-NEXT: vinsertf128 $1, %xmm0, %ymm0, %ymm0
; AVX1-NEXT: retq
;
Fix a broadcast related regression on the vector shuffle lowering. Summary: Test by Robert Lougher! Reviewers: chandlerc Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D5745 llvm-svn: 219617
2014-10-14 00:16:16 +08:00
; AVX2-LABEL: splat_v4f64:
; AVX2: # BB#0:
; AVX2-NEXT: vbroadcastsd %xmm0, %ymm0
; AVX2-NEXT: retq
%1 = shufflevector <2 x double> %r, <2 x double> undef, <4 x i32> zeroinitializer
ret <4 x double> %1
}