maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/test/CodeGen/Thumb2/mve-pred-loadstore.ll

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

627 lines
21 KiB
LLVM
Raw Normal View History

[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
[ARM] Remove unused check labels. NFC
2020-11-12 16:37:46 +08:00
; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=CHECK-LE
; RUN: llc -mtriple=thumbebv8.1m.main-none-none-eabi -mattr=+mve -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=CHECK-BE
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
define arm_aapcs_vfpcc <4 x i32> @load_v4i1(<4 x i1> *%src, <4 x i32> %a) {
; CHECK-LE-LABEL: load_v4i1:
; CHECK-LE: @ %bb.0: @ %entry
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-LE-NEXT: ldrb r0, [r0]
; CHECK-LE-NEXT: vmov.i8 q1, #0x0
; CHECK-LE-NEXT: vmov.i8 q2, #0xff
; CHECK-LE-NEXT: vmsr p0, r0
; CHECK-LE-NEXT: vpsel q1, q2, q1
Revert "[ARM] Match dual lane vmovs from insert_vector_elt" This one needed more testing.
2020-12-18 21:33:40 +08:00
; CHECK-LE-NEXT: vmov.u8 r0, q1[2]
[ARM] Match dual lane vmovs from insert_vector_elt MVE has a dual lane vector move instruction, capable of moving two general purpose registers into lanes of a vector register. They look like one of: vmov q0[2], q0[0], r2, r0 vmov q0[3], q0[1], r3, r1 They only accept these lane indices though (and only insert into an i32), either moving lanes 1 and 3, or 0 and 2. This patch adds some tablegen patterns for them, selecting from vector inserts elements. Because the insert_elements are know to be canonicalized to ascending order there are several patterns that we need to select. These lane indices are: 3 2 1 0 -> vmovqrr 31; vmovqrr 20 3 2 1 -> vmovqrr 31; vmov 2 3 1 -> vmovqrr 31 2 1 0 -> vmovqrr 20; vmov 1 2 0 -> vmovqrr 20 With the top one being the most common. All other potential patterns of lane indices will be matched by a combination of these and the individual vmov pattern already present. This does mean that we are selecting several machine instructions at once due to the need to re-arrange the inserts, but in this case there is nothing else that will attempt to match an insert_vector_elt node. This is a recommit of 6cc3d80a84884a79967fffa4596c14001b8ba8a3 after fixing the backward instruction definitions.
2020-12-19 00:13:08 +08:00
; CHECK-LE-NEXT: vmov.u8 r1, q1[0]
; CHECK-LE-NEXT: vmov q2[2], q2[0], r1, r0
Revert "[ARM] Match dual lane vmovs from insert_vector_elt" This one needed more testing.
2020-12-18 21:33:40 +08:00
; CHECK-LE-NEXT: vmov.u8 r0, q1[3]
[ARM] Match dual lane vmovs from insert_vector_elt MVE has a dual lane vector move instruction, capable of moving two general purpose registers into lanes of a vector register. They look like one of: vmov q0[2], q0[0], r2, r0 vmov q0[3], q0[1], r3, r1 They only accept these lane indices though (and only insert into an i32), either moving lanes 1 and 3, or 0 and 2. This patch adds some tablegen patterns for them, selecting from vector inserts elements. Because the insert_elements are know to be canonicalized to ascending order there are several patterns that we need to select. These lane indices are: 3 2 1 0 -> vmovqrr 31; vmovqrr 20 3 2 1 -> vmovqrr 31; vmov 2 3 1 -> vmovqrr 31 2 1 0 -> vmovqrr 20; vmov 1 2 0 -> vmovqrr 20 With the top one being the most common. All other potential patterns of lane indices will be matched by a combination of these and the individual vmov pattern already present. This does mean that we are selecting several machine instructions at once due to the need to re-arrange the inserts, but in this case there is nothing else that will attempt to match an insert_vector_elt node. This is a recommit of 6cc3d80a84884a79967fffa4596c14001b8ba8a3 after fixing the backward instruction definitions.
2020-12-19 00:13:08 +08:00
; CHECK-LE-NEXT: vmov.u8 r1, q1[1]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
[ARM] Match dual lane vmovs from insert_vector_elt MVE has a dual lane vector move instruction, capable of moving two general purpose registers into lanes of a vector register. They look like one of: vmov q0[2], q0[0], r2, r0 vmov q0[3], q0[1], r3, r1 They only accept these lane indices though (and only insert into an i32), either moving lanes 1 and 3, or 0 and 2. This patch adds some tablegen patterns for them, selecting from vector inserts elements. Because the insert_elements are know to be canonicalized to ascending order there are several patterns that we need to select. These lane indices are: 3 2 1 0 -> vmovqrr 31; vmovqrr 20 3 2 1 -> vmovqrr 31; vmov 2 3 1 -> vmovqrr 31 2 1 0 -> vmovqrr 20; vmov 1 2 0 -> vmovqrr 20 With the top one being the most common. All other potential patterns of lane indices will be matched by a combination of these and the individual vmov pattern already present. This does mean that we are selecting several machine instructions at once due to the need to re-arrange the inserts, but in this case there is nothing else that will attempt to match an insert_vector_elt node. This is a recommit of 6cc3d80a84884a79967fffa4596c14001b8ba8a3 after fixing the backward instruction definitions.
2020-12-19 00:13:08 +08:00
; CHECK-LE-NEXT: vmov q2[3], q2[1], r1, r0
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-LE-NEXT: vcmp.i32 ne, q2, zr
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_v4i1:
; CHECK-BE: @ %bb.0: @ %entry
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: ldrb r0, [r0]
; CHECK-BE-NEXT: vmov.i8 q1, #0x0
; CHECK-BE-NEXT: vmov.i8 q2, #0xff
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: rbit r0, r0
; CHECK-BE-NEXT: lsrs r0, r0, #28
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: vmsr p0, r0
; CHECK-BE-NEXT: vpsel q1, q2, q1
Revert "[ARM] Match dual lane vmovs from insert_vector_elt" This one needed more testing.
2020-12-18 21:33:40 +08:00
; CHECK-BE-NEXT: vmov.u8 r0, q1[2]
[ARM] Match dual lane vmovs from insert_vector_elt MVE has a dual lane vector move instruction, capable of moving two general purpose registers into lanes of a vector register. They look like one of: vmov q0[2], q0[0], r2, r0 vmov q0[3], q0[1], r3, r1 They only accept these lane indices though (and only insert into an i32), either moving lanes 1 and 3, or 0 and 2. This patch adds some tablegen patterns for them, selecting from vector inserts elements. Because the insert_elements are know to be canonicalized to ascending order there are several patterns that we need to select. These lane indices are: 3 2 1 0 -> vmovqrr 31; vmovqrr 20 3 2 1 -> vmovqrr 31; vmov 2 3 1 -> vmovqrr 31 2 1 0 -> vmovqrr 20; vmov 1 2 0 -> vmovqrr 20 With the top one being the most common. All other potential patterns of lane indices will be matched by a combination of these and the individual vmov pattern already present. This does mean that we are selecting several machine instructions at once due to the need to re-arrange the inserts, but in this case there is nothing else that will attempt to match an insert_vector_elt node. This is a recommit of 6cc3d80a84884a79967fffa4596c14001b8ba8a3 after fixing the backward instruction definitions.
2020-12-19 00:13:08 +08:00
; CHECK-BE-NEXT: vmov.u8 r1, q1[0]
; CHECK-BE-NEXT: vmov q2[2], q2[0], r1, r0
Revert "[ARM] Match dual lane vmovs from insert_vector_elt" This one needed more testing.
2020-12-18 21:33:40 +08:00
; CHECK-BE-NEXT: vmov.u8 r0, q1[3]
[ARM] Match dual lane vmovs from insert_vector_elt MVE has a dual lane vector move instruction, capable of moving two general purpose registers into lanes of a vector register. They look like one of: vmov q0[2], q0[0], r2, r0 vmov q0[3], q0[1], r3, r1 They only accept these lane indices though (and only insert into an i32), either moving lanes 1 and 3, or 0 and 2. This patch adds some tablegen patterns for them, selecting from vector inserts elements. Because the insert_elements are know to be canonicalized to ascending order there are several patterns that we need to select. These lane indices are: 3 2 1 0 -> vmovqrr 31; vmovqrr 20 3 2 1 -> vmovqrr 31; vmov 2 3 1 -> vmovqrr 31 2 1 0 -> vmovqrr 20; vmov 1 2 0 -> vmovqrr 20 With the top one being the most common. All other potential patterns of lane indices will be matched by a combination of these and the individual vmov pattern already present. This does mean that we are selecting several machine instructions at once due to the need to re-arrange the inserts, but in this case there is nothing else that will attempt to match an insert_vector_elt node. This is a recommit of 6cc3d80a84884a79967fffa4596c14001b8ba8a3 after fixing the backward instruction definitions.
2020-12-19 00:13:08 +08:00
; CHECK-BE-NEXT: vmov.u8 r1, q1[1]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: vrev64.32 q1, q0
[ARM] Match dual lane vmovs from insert_vector_elt MVE has a dual lane vector move instruction, capable of moving two general purpose registers into lanes of a vector register. They look like one of: vmov q0[2], q0[0], r2, r0 vmov q0[3], q0[1], r3, r1 They only accept these lane indices though (and only insert into an i32), either moving lanes 1 and 3, or 0 and 2. This patch adds some tablegen patterns for them, selecting from vector inserts elements. Because the insert_elements are know to be canonicalized to ascending order there are several patterns that we need to select. These lane indices are: 3 2 1 0 -> vmovqrr 31; vmovqrr 20 3 2 1 -> vmovqrr 31; vmov 2 3 1 -> vmovqrr 31 2 1 0 -> vmovqrr 20; vmov 1 2 0 -> vmovqrr 20 With the top one being the most common. All other potential patterns of lane indices will be matched by a combination of these and the individual vmov pattern already present. This does mean that we are selecting several machine instructions at once due to the need to re-arrange the inserts, but in this case there is nothing else that will attempt to match an insert_vector_elt node. This is a recommit of 6cc3d80a84884a79967fffa4596c14001b8ba8a3 after fixing the backward instruction definitions.
2020-12-19 00:13:08 +08:00
; CHECK-BE-NEXT: vmov q2[3], q2[1], r1, r0
Revert "[ARM] Match dual lane vmovs from insert_vector_elt" This one needed more testing.
2020-12-18 21:33:40 +08:00
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
[ARM] Match dual lane vmovs from insert_vector_elt MVE has a dual lane vector move instruction, capable of moving two general purpose registers into lanes of a vector register. They look like one of: vmov q0[2], q0[0], r2, r0 vmov q0[3], q0[1], r3, r1 They only accept these lane indices though (and only insert into an i32), either moving lanes 1 and 3, or 0 and 2. This patch adds some tablegen patterns for them, selecting from vector inserts elements. Because the insert_elements are know to be canonicalized to ascending order there are several patterns that we need to select. These lane indices are: 3 2 1 0 -> vmovqrr 31; vmovqrr 20 3 2 1 -> vmovqrr 31; vmov 2 3 1 -> vmovqrr 31 2 1 0 -> vmovqrr 20; vmov 1 2 0 -> vmovqrr 20 With the top one being the most common. All other potential patterns of lane indices will be matched by a combination of these and the individual vmov pattern already present. This does mean that we are selecting several machine instructions at once due to the need to re-arrange the inserts, but in this case there is nothing else that will attempt to match an insert_vector_elt node. This is a recommit of 6cc3d80a84884a79967fffa4596c14001b8ba8a3 after fixing the backward instruction definitions.
2020-12-19 00:13:08 +08:00
; CHECK-BE-NEXT: vcmp.i32 ne, q2, zr
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.32 q0, q1
; CHECK-BE-NEXT: bx lr
entry:
%c = load <4 x i1>, <4 x i1>* %src
%s = select <4 x i1> %c, <4 x i32> %a, <4 x i32> zeroinitializer
ret <4 x i32> %s
}
define arm_aapcs_vfpcc <8 x i16> @load_v8i1(<8 x i1> *%src, <8 x i16> %a) {
; CHECK-LE-LABEL: load_v8i1:
; CHECK-LE: @ %bb.0: @ %entry
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-LE-NEXT: ldrb r0, [r0]
; CHECK-LE-NEXT: vmov.i8 q1, #0x0
; CHECK-LE-NEXT: vmov.i8 q2, #0xff
; CHECK-LE-NEXT: vmsr p0, r0
; CHECK-LE-NEXT: vpsel q2, q2, q1
; CHECK-LE-NEXT: vmov.u8 r0, q2[0]
; CHECK-LE-NEXT: vmov.16 q1[0], r0
; CHECK-LE-NEXT: vmov.u8 r0, q2[1]
; CHECK-LE-NEXT: vmov.16 q1[1], r0
; CHECK-LE-NEXT: vmov.u8 r0, q2[2]
; CHECK-LE-NEXT: vmov.16 q1[2], r0
; CHECK-LE-NEXT: vmov.u8 r0, q2[3]
; CHECK-LE-NEXT: vmov.16 q1[3], r0
; CHECK-LE-NEXT: vmov.u8 r0, q2[4]
; CHECK-LE-NEXT: vmov.16 q1[4], r0
; CHECK-LE-NEXT: vmov.u8 r0, q2[5]
; CHECK-LE-NEXT: vmov.16 q1[5], r0
; CHECK-LE-NEXT: vmov.u8 r0, q2[6]
; CHECK-LE-NEXT: vmov.16 q1[6], r0
; CHECK-LE-NEXT: vmov.u8 r0, q2[7]
; CHECK-LE-NEXT: vmov.16 q1[7], r0
; CHECK-LE-NEXT: vcmp.i16 ne, q1, zr
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_v8i1:
; CHECK-BE: @ %bb.0: @ %entry
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: ldrb r0, [r0]
; CHECK-BE-NEXT: vmov.i8 q1, #0x0
; CHECK-BE-NEXT: vmov.i8 q2, #0xff
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: rbit r0, r0
; CHECK-BE-NEXT: lsrs r0, r0, #24
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: vmsr p0, r0
; CHECK-BE-NEXT: vpsel q2, q2, q1
; CHECK-BE-NEXT: vmov.u8 r0, q2[0]
; CHECK-BE-NEXT: vmov.16 q1[0], r0
; CHECK-BE-NEXT: vmov.u8 r0, q2[1]
; CHECK-BE-NEXT: vmov.16 q1[1], r0
; CHECK-BE-NEXT: vmov.u8 r0, q2[2]
; CHECK-BE-NEXT: vmov.16 q1[2], r0
; CHECK-BE-NEXT: vmov.u8 r0, q2[3]
; CHECK-BE-NEXT: vmov.16 q1[3], r0
; CHECK-BE-NEXT: vmov.u8 r0, q2[4]
; CHECK-BE-NEXT: vmov.16 q1[4], r0
; CHECK-BE-NEXT: vmov.u8 r0, q2[5]
; CHECK-BE-NEXT: vmov.16 q1[5], r0
; CHECK-BE-NEXT: vmov.u8 r0, q2[6]
; CHECK-BE-NEXT: vmov.16 q1[6], r0
; CHECK-BE-NEXT: vmov.u8 r0, q2[7]
; CHECK-BE-NEXT: vmov.16 q1[7], r0
; CHECK-BE-NEXT: vcmp.i16 ne, q1, zr
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: vrev64.16 q1, q0
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
; CHECK-BE-NEXT: vrev32.16 q0, q0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.16 q0, q1
; CHECK-BE-NEXT: bx lr
entry:
%c = load <8 x i1>, <8 x i1>* %src
%s = select <8 x i1> %c, <8 x i16> %a, <8 x i16> zeroinitializer
ret <8 x i16> %s
}
define arm_aapcs_vfpcc <16 x i8> @load_v16i1(<16 x i1> *%src, <16 x i8> %a) {
; CHECK-LE-LABEL: load_v16i1:
; CHECK-LE: @ %bb.0: @ %entry
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-LE-NEXT: ldrh r0, [r0]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-LE-NEXT: vmsr p0, r0
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_v16i1:
; CHECK-BE: @ %bb.0: @ %entry
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: ldrh r0, [r0]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: vrev64.8 q1, q0
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: rbit r0, r0
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: vrev32.8 q0, q0
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: lsrs r0, r0, #16
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: vmsr p0, r0
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.8 q0, q1
; CHECK-BE-NEXT: bx lr
entry:
%c = load <16 x i1>, <16 x i1>* %src
%s = select <16 x i1> %c, <16 x i8> %a, <16 x i8> zeroinitializer
ret <16 x i8> %s
}
define arm_aapcs_vfpcc <2 x i64> @load_v2i1(<2 x i1> *%src, <2 x i64> %a) {
; CHECK-LE-LABEL: load_v2i1:
; CHECK-LE: @ %bb.0: @ %entry
; CHECK-LE-NEXT: ldrb r0, [r0]
[ARM] Make MVE v2i1 predicates legal MVE can treat v16i1, v8i1, v4i1 and v2i1 as different views onto the same 16bit VPR.P0 register, with v2i1 holding two 8 bit values for the two halves. This was never treated as a legal type in llvm in the past as there are not many 64bit instructions and no 64bit compares. There are a few instructions that could use it though, notably a VSELECT (as it can handle any size using the underlying v16i8 VPSEL), AND/OR/XOR for similar reasons, some gathers/scatter and long multiplies and VCTP64 instructions. This patch goes through and makes v2i1 a legal type, handling all the cases that fall out of that. It also makes VSELECT legal for v2i64 as a side benefit. A lot of the codegen changes as a result - usually in way that is a little better or a little worse, but still expensive. Costs can change a little too in the process, again in a way that expensive things remain expensive. A lot of the tests that changed are mainly to ensure correctness - the code can hopefully be improved in the future where it comes up in practice. The intrinsics currently remain using the v4i1 they previously did to emulate a v2i1. This will be changed in a followup patch but this one was already large enough. Differential Revision: https://reviews.llvm.org/D114449
2021-12-03 22:05:41 +08:00
; CHECK-LE-NEXT: vmov.i8 q1, #0x0
; CHECK-LE-NEXT: vmov.i8 q2, #0xff
; CHECK-LE-NEXT: vmsr p0, r0
; CHECK-LE-NEXT: vpsel q1, q2, q1
; CHECK-LE-NEXT: vmov.u8 r0, q1[1]
; CHECK-LE-NEXT: vmov.u8 r1, q1[0]
; CHECK-LE-NEXT: vmov q1[2], q1[0], r1, r0
; CHECK-LE-NEXT: vmov q1[3], q1[1], r1, r0
; CHECK-LE-NEXT: vcmp.i32 ne, q1, zr
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
; CHECK-LE-NEXT: vpsel q0, q0, q1
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_v2i1:
; CHECK-BE: @ %bb.0: @ %entry
; CHECK-BE-NEXT: ldrb r0, [r0]
[ARM] Make MVE v2i1 predicates legal MVE can treat v16i1, v8i1, v4i1 and v2i1 as different views onto the same 16bit VPR.P0 register, with v2i1 holding two 8 bit values for the two halves. This was never treated as a legal type in llvm in the past as there are not many 64bit instructions and no 64bit compares. There are a few instructions that could use it though, notably a VSELECT (as it can handle any size using the underlying v16i8 VPSEL), AND/OR/XOR for similar reasons, some gathers/scatter and long multiplies and VCTP64 instructions. This patch goes through and makes v2i1 a legal type, handling all the cases that fall out of that. It also makes VSELECT legal for v2i64 as a side benefit. A lot of the codegen changes as a result - usually in way that is a little better or a little worse, but still expensive. Costs can change a little too in the process, again in a way that expensive things remain expensive. A lot of the tests that changed are mainly to ensure correctness - the code can hopefully be improved in the future where it comes up in practice. The intrinsics currently remain using the v4i1 they previously did to emulate a v2i1. This will be changed in a followup patch but this one was already large enough. Differential Revision: https://reviews.llvm.org/D114449
2021-12-03 22:05:41 +08:00
; CHECK-BE-NEXT: vmov.i8 q1, #0x0
; CHECK-BE-NEXT: vmov.i8 q2, #0xff
; CHECK-BE-NEXT: rbit r0, r0
; CHECK-BE-NEXT: lsrs r0, r0, #30
; CHECK-BE-NEXT: vmsr p0, r0
; CHECK-BE-NEXT: vpsel q1, q2, q1
; CHECK-BE-NEXT: vmov.u8 r0, q1[1]
; CHECK-BE-NEXT: vmov.u8 r1, q1[0]
; CHECK-BE-NEXT: vmov q1[2], q1[0], r1, r0
; CHECK-BE-NEXT: vmov q1[3], q1[1], r1, r0
; CHECK-BE-NEXT: vcmp.i32 ne, q1, zr
; CHECK-BE-NEXT: vmov.i32 q1, #0x0
; CHECK-BE-NEXT: vpsel q0, q0, q1
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: bx lr
entry:
%c = load <2 x i1>, <2 x i1>* %src
%s = select <2 x i1> %c, <2 x i64> %a, <2 x i64> zeroinitializer
ret <2 x i64> %s
}
define arm_aapcs_vfpcc void @store_v4i1(<4 x i1> *%dst, <4 x i32> %a) {
; CHECK-LE-LABEL: store_v4i1:
; CHECK-LE: @ %bb.0: @ %entry
; CHECK-LE-NEXT: vcmp.i32 eq, q0, zr
[ARM] A predicate cast of a predicate cast is a predicate cast The adds some very basic folding of PREDICATE_CASTS, removing cases when they are chained together. These would already be removed eventually, as these are lowered to copies. This just allows it to happen earlier, which can help other simplifications. Differential Revision: https://reviews.llvm.org/D67591 llvm-svn: 372012
2019-09-17 01:29:07 +08:00
; CHECK-LE-NEXT: movs r3, #0
; CHECK-LE-NEXT: vmrs r1, p0
; CHECK-LE-NEXT: and r2, r1, #1
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-LE-NEXT: rsbs r2, r2, #0
[ARM] A predicate cast of a predicate cast is a predicate cast The adds some very basic folding of PREDICATE_CASTS, removing cases when they are chained together. These would already be removed eventually, as these are lowered to copies. This just allows it to happen earlier, which can help other simplifications. Differential Revision: https://reviews.llvm.org/D67591 llvm-svn: 372012
2019-09-17 01:29:07 +08:00
; CHECK-LE-NEXT: bfi r3, r2, #0, #1
; CHECK-LE-NEXT: ubfx r2, r1, #4, #1
; CHECK-LE-NEXT: rsbs r2, r2, #0
; CHECK-LE-NEXT: bfi r3, r2, #1, #1
; CHECK-LE-NEXT: ubfx r2, r1, #8, #1
; CHECK-LE-NEXT: ubfx r1, r1, #12, #1
; CHECK-LE-NEXT: rsbs r2, r2, #0
; CHECK-LE-NEXT: bfi r3, r2, #2, #1
; CHECK-LE-NEXT: rsbs r1, r1, #0
; CHECK-LE-NEXT: bfi r3, r1, #3, #1
; CHECK-LE-NEXT: strb r3, [r0]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: store_v4i1:
; CHECK-BE: @ %bb.0: @ %entry
; CHECK-BE-NEXT: vrev64.32 q1, q0
[ARM] A predicate cast of a predicate cast is a predicate cast The adds some very basic folding of PREDICATE_CASTS, removing cases when they are chained together. These would already be removed eventually, as these are lowered to copies. This just allows it to happen earlier, which can help other simplifications. Differential Revision: https://reviews.llvm.org/D67591 llvm-svn: 372012
2019-09-17 01:29:07 +08:00
; CHECK-BE-NEXT: movs r3, #0
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: vcmp.i32 eq, q1, zr
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: vmrs r1, p0
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: ubfx r2, r1, #12, #1
[ARM] A predicate cast of a predicate cast is a predicate cast The adds some very basic folding of PREDICATE_CASTS, removing cases when they are chained together. These would already be removed eventually, as these are lowered to copies. This just allows it to happen earlier, which can help other simplifications. Differential Revision: https://reviews.llvm.org/D67591 llvm-svn: 372012
2019-09-17 01:29:07 +08:00
; CHECK-BE-NEXT: rsbs r2, r2, #0
; CHECK-BE-NEXT: bfi r3, r2, #0, #1
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: ubfx r2, r1, #8, #1
[ARM] A predicate cast of a predicate cast is a predicate cast The adds some very basic folding of PREDICATE_CASTS, removing cases when they are chained together. These would already be removed eventually, as these are lowered to copies. This just allows it to happen earlier, which can help other simplifications. Differential Revision: https://reviews.llvm.org/D67591 llvm-svn: 372012
2019-09-17 01:29:07 +08:00
; CHECK-BE-NEXT: rsbs r2, r2, #0
; CHECK-BE-NEXT: bfi r3, r2, #1, #1
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: ubfx r2, r1, #4, #1
; CHECK-BE-NEXT: and r1, r1, #1
[ARM] A predicate cast of a predicate cast is a predicate cast The adds some very basic folding of PREDICATE_CASTS, removing cases when they are chained together. These would already be removed eventually, as these are lowered to copies. This just allows it to happen earlier, which can help other simplifications. Differential Revision: https://reviews.llvm.org/D67591 llvm-svn: 372012
2019-09-17 01:29:07 +08:00
; CHECK-BE-NEXT: rsbs r2, r2, #0
; CHECK-BE-NEXT: bfi r3, r2, #2, #1
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: rsbs r1, r1, #0
[ARM] A predicate cast of a predicate cast is a predicate cast The adds some very basic folding of PREDICATE_CASTS, removing cases when they are chained together. These would already be removed eventually, as these are lowered to copies. This just allows it to happen earlier, which can help other simplifications. Differential Revision: https://reviews.llvm.org/D67591 llvm-svn: 372012
2019-09-17 01:29:07 +08:00
; CHECK-BE-NEXT: bfi r3, r1, #3, #1
; CHECK-BE-NEXT: strb r3, [r0]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: bx lr
entry:
%c = icmp eq <4 x i32> %a, zeroinitializer
store <4 x i1> %c, <4 x i1>* %dst
ret void
}
define arm_aapcs_vfpcc void @store_v8i1(<8 x i1> *%dst, <8 x i16> %a) {
; CHECK-LE-LABEL: store_v8i1:
; CHECK-LE: @ %bb.0: @ %entry
; CHECK-LE-NEXT: vcmp.i16 eq, q0, zr
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-LE-NEXT: vmrs r2, p0
[ARM] A predicate cast of a predicate cast is a predicate cast The adds some very basic folding of PREDICATE_CASTS, removing cases when they are chained together. These would already be removed eventually, as these are lowered to copies. This just allows it to happen earlier, which can help other simplifications. Differential Revision: https://reviews.llvm.org/D67591 llvm-svn: 372012
2019-09-17 01:29:07 +08:00
; CHECK-LE-NEXT: and r1, r2, #1
; CHECK-LE-NEXT: rsbs r3, r1, #0
; CHECK-LE-NEXT: movs r1, #0
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-LE-NEXT: bfi r1, r3, #0, #1
; CHECK-LE-NEXT: ubfx r3, r2, #2, #1
; CHECK-LE-NEXT: rsbs r3, r3, #0
; CHECK-LE-NEXT: bfi r1, r3, #1, #1
; CHECK-LE-NEXT: ubfx r3, r2, #4, #1
; CHECK-LE-NEXT: rsbs r3, r3, #0
; CHECK-LE-NEXT: bfi r1, r3, #2, #1
; CHECK-LE-NEXT: ubfx r3, r2, #6, #1
; CHECK-LE-NEXT: rsbs r3, r3, #0
; CHECK-LE-NEXT: bfi r1, r3, #3, #1
; CHECK-LE-NEXT: ubfx r3, r2, #8, #1
; CHECK-LE-NEXT: rsbs r3, r3, #0
; CHECK-LE-NEXT: bfi r1, r3, #4, #1
; CHECK-LE-NEXT: ubfx r3, r2, #10, #1
; CHECK-LE-NEXT: rsbs r3, r3, #0
; CHECK-LE-NEXT: bfi r1, r3, #5, #1
; CHECK-LE-NEXT: ubfx r3, r2, #12, #1
; CHECK-LE-NEXT: ubfx r2, r2, #14, #1
; CHECK-LE-NEXT: rsbs r3, r3, #0
; CHECK-LE-NEXT: bfi r1, r3, #6, #1
; CHECK-LE-NEXT: rsbs r2, r2, #0
; CHECK-LE-NEXT: bfi r1, r2, #7, #1
; CHECK-LE-NEXT: strb r1, [r0]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: store_v8i1:
; CHECK-BE: @ %bb.0: @ %entry
; CHECK-BE-NEXT: vrev64.16 q1, q0
; CHECK-BE-NEXT: vcmp.i16 eq, q1, zr
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: vmrs r2, p0
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: ubfx r1, r2, #14, #1
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: rsbs r3, r1, #0
; CHECK-BE-NEXT: movs r1, #0
; CHECK-BE-NEXT: bfi r1, r3, #0, #1
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: ubfx r3, r2, #12, #1
[ARM] A predicate cast of a predicate cast is a predicate cast The adds some very basic folding of PREDICATE_CASTS, removing cases when they are chained together. These would already be removed eventually, as these are lowered to copies. This just allows it to happen earlier, which can help other simplifications. Differential Revision: https://reviews.llvm.org/D67591 llvm-svn: 372012
2019-09-17 01:29:07 +08:00
; CHECK-BE-NEXT: rsbs r3, r3, #0
; CHECK-BE-NEXT: bfi r1, r3, #1, #1
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: ubfx r3, r2, #10, #1
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: rsbs r3, r3, #0
; CHECK-BE-NEXT: bfi r1, r3, #2, #1
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: ubfx r3, r2, #8, #1
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: rsbs r3, r3, #0
; CHECK-BE-NEXT: bfi r1, r3, #3, #1
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: ubfx r3, r2, #6, #1
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: rsbs r3, r3, #0
; CHECK-BE-NEXT: bfi r1, r3, #4, #1
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: ubfx r3, r2, #4, #1
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: rsbs r3, r3, #0
; CHECK-BE-NEXT: bfi r1, r3, #5, #1
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: ubfx r3, r2, #2, #1
; CHECK-BE-NEXT: and r2, r2, #1
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: rsbs r3, r3, #0
; CHECK-BE-NEXT: bfi r1, r3, #6, #1
; CHECK-BE-NEXT: rsbs r2, r2, #0
; CHECK-BE-NEXT: bfi r1, r2, #7, #1
; CHECK-BE-NEXT: strb r1, [r0]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: bx lr
entry:
%c = icmp eq <8 x i16> %a, zeroinitializer
store <8 x i1> %c, <8 x i1>* %dst
ret void
}
define arm_aapcs_vfpcc void @store_v16i1(<16 x i1> *%dst, <16 x i8> %a) {
; CHECK-LE-LABEL: store_v16i1:
; CHECK-LE: @ %bb.0: @ %entry
; CHECK-LE-NEXT: vcmp.i8 eq, q0, zr
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-LE-NEXT: vmrs r1, p0
; CHECK-LE-NEXT: strh r1, [r0]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: store_v16i1:
; CHECK-BE: @ %bb.0: @ %entry
; CHECK-BE-NEXT: vrev64.8 q1, q0
; CHECK-BE-NEXT: vcmp.i8 eq, q1, zr
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: vmrs r1, p0
[ARM] Make a BE predicate bitcast consistent with the rest of llvm We were storing predicate registers, such as a <8 x i1>, in the opposite order to how the rest of llvm expects. This actually turns out to be correct for the one place that usually uses it - the ScalarizeMaskedMemIntrin pass, but only because the pass was incorrect itself. This fixes the order so that bits are stored in the opposite order and bitcasts work as expected. This allows the Scalarization pass to be fixed, as in https://reviews.llvm.org/D94765. Differential Revision: https://reviews.llvm.org/D94867
2021-02-08 19:37:17 +08:00
; CHECK-BE-NEXT: rbit r1, r1
; CHECK-BE-NEXT: lsrs r1, r1, #16
[ARM] Fix loads and stores for predicate vectors These predicate vectors can usually be loaded and stored with a single instruction, a VSTR_P0. However this instruction will store the entire P0 predicate, 16 bits, zeroextended to 32bits. Each lane of the the v4i1/v8i1/v16i1 representing 4/2/1 bits. As far as I understand, when llvm says "store this v4i1", it really does need to store 4 bits (or 8, that being the size of a byte, with this bottom 4 as the interesting bits). For example a bitcast from a v8i1 to a i8 is defined as a store followed by a load, which is how the code is expanded. So this instead lowers the v4i1/v8i1 load/store through some shuffles to get the bits into the correct positions. This, as you might imagine, is not as efficient as a single instruction. But I believe it is needed for correctness. v16i1 equally should not load/store 32bits, only storing the 16bits of data. Stack loads/stores are still using the VSTR_P0 (as can be seen by the test not changing). This is fine as they are self-consistent, it is only "externally observable loads/stores" (from our point of view) that need to be corrected. Differential revision: https://reviews.llvm.org/D67085 llvm-svn: 371419
2019-09-10 00:35:49 +08:00
; CHECK-BE-NEXT: strh r1, [r0]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: bx lr
entry:
%c = icmp eq <16 x i8> %a, zeroinitializer
store <16 x i1> %c, <16 x i1>* %dst
ret void
}
define arm_aapcs_vfpcc void @store_v2i1(<2 x i1> *%dst, <2 x i64> %a) {
; CHECK-LE-LABEL: store_v2i1:
; CHECK-LE: @ %bb.0: @ %entry
[ARM] Create VMOVRRD from adjacent vector extracts This adds a combine for extract(x, n); extract(x, n+1) -> VMOVRRD(extract x, n/2). This allows two vector lanes to be moved at the same time in a single instruction, and thanks to the other VMOVRRD folds we have added recently can help reduce the amount of executed instructions. Floating point types are very similar, but will include a bitcast to an integer type. This also adds a shouldRewriteCopySrc, to prevent copy propagation from DPR to SPR, which can break as not all DPR regs can be extracted from directly. Otherwise the machine verifier is unhappy. Differential Revision: https://reviews.llvm.org/D100244
2021-04-20 22:15:43 +08:00
; CHECK-LE-NEXT: vmov r1, r2, d0
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: orrs r1, r2
[ARM] Make MVE v2i1 predicates legal MVE can treat v16i1, v8i1, v4i1 and v2i1 as different views onto the same 16bit VPR.P0 register, with v2i1 holding two 8 bit values for the two halves. This was never treated as a legal type in llvm in the past as there are not many 64bit instructions and no 64bit compares. There are a few instructions that could use it though, notably a VSELECT (as it can handle any size using the underlying v16i8 VPSEL), AND/OR/XOR for similar reasons, some gathers/scatter and long multiplies and VCTP64 instructions. This patch goes through and makes v2i1 a legal type, handling all the cases that fall out of that. It also makes VSELECT legal for v2i64 as a side benefit. A lot of the codegen changes as a result - usually in way that is a little better or a little worse, but still expensive. Costs can change a little too in the process, again in a way that expensive things remain expensive. A lot of the tests that changed are mainly to ensure correctness - the code can hopefully be improved in the future where it comes up in practice. The intrinsics currently remain using the v4i1 they previously did to emulate a v2i1. This will be changed in a followup patch but this one was already large enough. Differential Revision: https://reviews.llvm.org/D114449
2021-12-03 22:05:41 +08:00
; CHECK-LE-NEXT: mov.w r2, #0
[ARM] Peek through And 1 in IsCMPZCSINC We can be in situations where And 1 zext nodes will not have been yet, preventing us from detecting removable cmpz/csinc patterns. This peeks through those nodes allowing us to simplify more code. Differential Revision: https://reviews.llvm.org/D115176
2021-12-08 23:40:23 +08:00
; CHECK-LE-NEXT: csetm r1, eq
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: bfi r2, r1, #0, #1
[ARM] Make MVE v2i1 predicates legal MVE can treat v16i1, v8i1, v4i1 and v2i1 as different views onto the same 16bit VPR.P0 register, with v2i1 holding two 8 bit values for the two halves. This was never treated as a legal type in llvm in the past as there are not many 64bit instructions and no 64bit compares. There are a few instructions that could use it though, notably a VSELECT (as it can handle any size using the underlying v16i8 VPSEL), AND/OR/XOR for similar reasons, some gathers/scatter and long multiplies and VCTP64 instructions. This patch goes through and makes v2i1 a legal type, handling all the cases that fall out of that. It also makes VSELECT legal for v2i64 as a side benefit. A lot of the codegen changes as a result - usually in way that is a little better or a little worse, but still expensive. Costs can change a little too in the process, again in a way that expensive things remain expensive. A lot of the tests that changed are mainly to ensure correctness - the code can hopefully be improved in the future where it comes up in practice. The intrinsics currently remain using the v4i1 they previously did to emulate a v2i1. This will be changed in a followup patch but this one was already large enough. Differential Revision: https://reviews.llvm.org/D114449
2021-12-03 22:05:41 +08:00
; CHECK-LE-NEXT: vmov r1, r3, d1
; CHECK-LE-NEXT: orrs r1, r3
[ARM] Peek through And 1 in IsCMPZCSINC We can be in situations where And 1 zext nodes will not have been yet, preventing us from detecting removable cmpz/csinc patterns. This peeks through those nodes allowing us to simplify more code. Differential Revision: https://reviews.llvm.org/D115176
2021-12-08 23:40:23 +08:00
; CHECK-LE-NEXT: csetm r1, eq
[ARM] Make MVE v2i1 predicates legal MVE can treat v16i1, v8i1, v4i1 and v2i1 as different views onto the same 16bit VPR.P0 register, with v2i1 holding two 8 bit values for the two halves. This was never treated as a legal type in llvm in the past as there are not many 64bit instructions and no 64bit compares. There are a few instructions that could use it though, notably a VSELECT (as it can handle any size using the underlying v16i8 VPSEL), AND/OR/XOR for similar reasons, some gathers/scatter and long multiplies and VCTP64 instructions. This patch goes through and makes v2i1 a legal type, handling all the cases that fall out of that. It also makes VSELECT legal for v2i64 as a side benefit. A lot of the codegen changes as a result - usually in way that is a little better or a little worse, but still expensive. Costs can change a little too in the process, again in a way that expensive things remain expensive. A lot of the tests that changed are mainly to ensure correctness - the code can hopefully be improved in the future where it comes up in practice. The intrinsics currently remain using the v4i1 they previously did to emulate a v2i1. This will be changed in a followup patch but this one was already large enough. Differential Revision: https://reviews.llvm.org/D114449
2021-12-03 22:05:41 +08:00
; CHECK-LE-NEXT: bfi r2, r1, #1, #1
; CHECK-LE-NEXT: strb r2, [r0]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: store_v2i1:
; CHECK-BE: @ %bb.0: @ %entry
; CHECK-BE-NEXT: vrev64.32 q1, q0
[ARM] Create VMOVRRD from adjacent vector extracts This adds a combine for extract(x, n); extract(x, n+1) -> VMOVRRD(extract x, n/2). This allows two vector lanes to be moved at the same time in a single instruction, and thanks to the other VMOVRRD folds we have added recently can help reduce the amount of executed instructions. Floating point types are very similar, but will include a bitcast to an integer type. This also adds a shouldRewriteCopySrc, to prevent copy propagation from DPR to SPR, which can break as not all DPR regs can be extracted from directly. Otherwise the machine verifier is unhappy. Differential Revision: https://reviews.llvm.org/D100244
2021-04-20 22:15:43 +08:00
; CHECK-BE-NEXT: vmov r1, r2, d3
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: orrs r1, r2
[ARM] Make MVE v2i1 predicates legal MVE can treat v16i1, v8i1, v4i1 and v2i1 as different views onto the same 16bit VPR.P0 register, with v2i1 holding two 8 bit values for the two halves. This was never treated as a legal type in llvm in the past as there are not many 64bit instructions and no 64bit compares. There are a few instructions that could use it though, notably a VSELECT (as it can handle any size using the underlying v16i8 VPSEL), AND/OR/XOR for similar reasons, some gathers/scatter and long multiplies and VCTP64 instructions. This patch goes through and makes v2i1 a legal type, handling all the cases that fall out of that. It also makes VSELECT legal for v2i64 as a side benefit. A lot of the codegen changes as a result - usually in way that is a little better or a little worse, but still expensive. Costs can change a little too in the process, again in a way that expensive things remain expensive. A lot of the tests that changed are mainly to ensure correctness - the code can hopefully be improved in the future where it comes up in practice. The intrinsics currently remain using the v4i1 they previously did to emulate a v2i1. This will be changed in a followup patch but this one was already large enough. Differential Revision: https://reviews.llvm.org/D114449
2021-12-03 22:05:41 +08:00
; CHECK-BE-NEXT: mov.w r2, #0
[ARM] Peek through And 1 in IsCMPZCSINC We can be in situations where And 1 zext nodes will not have been yet, preventing us from detecting removable cmpz/csinc patterns. This peeks through those nodes allowing us to simplify more code. Differential Revision: https://reviews.llvm.org/D115176
2021-12-08 23:40:23 +08:00
; CHECK-BE-NEXT: csetm r1, eq
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: bfi r2, r1, #0, #1
[ARM] Make MVE v2i1 predicates legal MVE can treat v16i1, v8i1, v4i1 and v2i1 as different views onto the same 16bit VPR.P0 register, with v2i1 holding two 8 bit values for the two halves. This was never treated as a legal type in llvm in the past as there are not many 64bit instructions and no 64bit compares. There are a few instructions that could use it though, notably a VSELECT (as it can handle any size using the underlying v16i8 VPSEL), AND/OR/XOR for similar reasons, some gathers/scatter and long multiplies and VCTP64 instructions. This patch goes through and makes v2i1 a legal type, handling all the cases that fall out of that. It also makes VSELECT legal for v2i64 as a side benefit. A lot of the codegen changes as a result - usually in way that is a little better or a little worse, but still expensive. Costs can change a little too in the process, again in a way that expensive things remain expensive. A lot of the tests that changed are mainly to ensure correctness - the code can hopefully be improved in the future where it comes up in practice. The intrinsics currently remain using the v4i1 they previously did to emulate a v2i1. This will be changed in a followup patch but this one was already large enough. Differential Revision: https://reviews.llvm.org/D114449
2021-12-03 22:05:41 +08:00
; CHECK-BE-NEXT: vmov r1, r3, d2
; CHECK-BE-NEXT: orrs r1, r3
[ARM] Peek through And 1 in IsCMPZCSINC We can be in situations where And 1 zext nodes will not have been yet, preventing us from detecting removable cmpz/csinc patterns. This peeks through those nodes allowing us to simplify more code. Differential Revision: https://reviews.llvm.org/D115176
2021-12-08 23:40:23 +08:00
; CHECK-BE-NEXT: csetm r1, eq
[ARM] Make MVE v2i1 predicates legal MVE can treat v16i1, v8i1, v4i1 and v2i1 as different views onto the same 16bit VPR.P0 register, with v2i1 holding two 8 bit values for the two halves. This was never treated as a legal type in llvm in the past as there are not many 64bit instructions and no 64bit compares. There are a few instructions that could use it though, notably a VSELECT (as it can handle any size using the underlying v16i8 VPSEL), AND/OR/XOR for similar reasons, some gathers/scatter and long multiplies and VCTP64 instructions. This patch goes through and makes v2i1 a legal type, handling all the cases that fall out of that. It also makes VSELECT legal for v2i64 as a side benefit. A lot of the codegen changes as a result - usually in way that is a little better or a little worse, but still expensive. Costs can change a little too in the process, again in a way that expensive things remain expensive. A lot of the tests that changed are mainly to ensure correctness - the code can hopefully be improved in the future where it comes up in practice. The intrinsics currently remain using the v4i1 they previously did to emulate a v2i1. This will be changed in a followup patch but this one was already large enough. Differential Revision: https://reviews.llvm.org/D114449
2021-12-03 22:05:41 +08:00
; CHECK-BE-NEXT: bfi r2, r1, #1, #1
; CHECK-BE-NEXT: strb r2, [r0]
[ARM] MVE predicate store patterns Stack loads and stores were already working, but direct stores were not. This adds the patterns for them, same as predicate loads. Differential Revision: https://reviews.llvm.org/D66213 llvm-svn: 368988
2019-08-15 18:41:42 +08:00
; CHECK-BE-NEXT: bx lr
entry:
%c = icmp eq <2 x i64> %a, zeroinitializer
store <2 x i1> %c, <2 x i1>* %dst
ret void
}
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
define arm_aapcs_vfpcc <4 x i32> @load_predcastzext(i16* %i, <4 x i32> %a) {
; CHECK-LE-LABEL: load_predcastzext:
; CHECK-LE: @ %bb.0:
; CHECK-LE-NEXT: ldrh r0, [r0]
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
; CHECK-LE-NEXT: vmsr p0, r0
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_predcastzext:
; CHECK-BE: @ %bb.0:
; CHECK-BE-NEXT: ldrh r0, [r0]
; CHECK-BE-NEXT: vrev64.32 q1, q0
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
; CHECK-BE-NEXT: vmsr p0, r0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.32 q0, q1
; CHECK-BE-NEXT: bx lr
%l = load i16, i16* %i, align 4
%lz = zext i16 %l to i32
%c = tail call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %lz)
%s = select <4 x i1> %c, <4 x i32> %a, <4 x i32> zeroinitializer
ret <4 x i32> %s
}
define arm_aapcs_vfpcc <4 x i32> @load_bc4(i32* %i, <4 x i32> %a) {
; CHECK-LE-LABEL: load_bc4:
; CHECK-LE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_bc4:
; CHECK-BE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vrev64.32 q1, q0
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.32 q0, q1
; CHECK-BE-NEXT: bx lr
%l = load i32, i32* %i, align 4
%c = tail call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %l)
%s = select <4 x i1> %c, <4 x i32> %a, <4 x i32> zeroinitializer
ret <4 x i32> %s
}
define arm_aapcs_vfpcc <8 x i16> @load_predcast8(i32* %i, <8 x i16> %a) {
; CHECK-LE-LABEL: load_predcast8:
; CHECK-LE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_predcast8:
; CHECK-BE: @ %bb.0:
; CHECK-BE-NEXT: vrev64.16 q1, q0
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vrev32.16 q0, q0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.16 q0, q1
; CHECK-BE-NEXT: bx lr
%l = load i32, i32* %i, align 4
%c = tail call <8 x i1> @llvm.arm.mve.pred.i2v.v8i1(i32 %l)
%s = select <8 x i1> %c, <8 x i16> %a, <8 x i16> zeroinitializer
ret <8 x i16> %s
}
define arm_aapcs_vfpcc <16 x i8> @load_predcast16(i32* %i, <16 x i8> %a) {
; CHECK-LE-LABEL: load_predcast16:
; CHECK-LE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_predcast16:
; CHECK-BE: @ %bb.0:
; CHECK-BE-NEXT: vrev64.8 q1, q0
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vrev32.8 q0, q0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.8 q0, q1
; CHECK-BE-NEXT: bx lr
%l = load i32, i32* %i, align 4
%c = tail call <16 x i1> @llvm.arm.mve.pred.i2v.v16i1(i32 %l)
%s = select <16 x i1> %c, <16 x i8> %a, <16 x i8> zeroinitializer
ret <16 x i8> %s
}
define arm_aapcs_vfpcc <4 x i32> @load_bc4_align2(i32* %i, <4 x i32> %a) {
; CHECK-LE-LABEL: load_bc4_align2:
; CHECK-LE: @ %bb.0:
; CHECK-LE-NEXT: ldr r0, [r0]
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
; CHECK-LE-NEXT: vmsr p0, r0
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_bc4_align2:
; CHECK-BE: @ %bb.0:
; CHECK-BE-NEXT: ldr r0, [r0]
; CHECK-BE-NEXT: vrev64.32 q1, q0
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
; CHECK-BE-NEXT: vmsr p0, r0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.32 q0, q1
; CHECK-BE-NEXT: bx lr
%l = load i32, i32* %i, align 2
%c = tail call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %l)
%s = select <4 x i1> %c, <4 x i32> %a, <4 x i32> zeroinitializer
ret <4 x i32> %s
}
define arm_aapcs_vfpcc <4 x i32> @load_bc4_offset(i16* %i, <4 x i32> %a) {
; CHECK-LE-LABEL: load_bc4_offset:
; CHECK-LE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: adds r0, #6
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_bc4_offset:
; CHECK-BE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: adds r0, #6
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vrev64.32 q1, q0
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.32 q0, q1
; CHECK-BE-NEXT: bx lr
%g = getelementptr inbounds i16, i16* %i, i32 3
%gb = bitcast i16* %g to i32*
%l = load i32, i32* %gb, align 4
%c = tail call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %l)
%s = select <4 x i1> %c, <4 x i32> %a, <4 x i32> zeroinitializer
ret <4 x i32> %s
}
define arm_aapcs_vfpcc <4 x i32> @load_bc4_range4(i32* %i, <4 x i32> %a) {
; CHECK-LE-LABEL: load_bc4_range4:
; CHECK-LE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: vldr p0, [r0, #4]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_bc4_range4:
; CHECK-BE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: vldr p0, [r0, #4]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vrev64.32 q1, q0
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.32 q0, q1
; CHECK-BE-NEXT: bx lr
%g = getelementptr inbounds i32, i32* %i, i32 1
%l = load i32, i32* %g, align 4
%c = tail call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %l)
%s = select <4 x i1> %c, <4 x i32> %a, <4 x i32> zeroinitializer
ret <4 x i32> %s
}
define arm_aapcs_vfpcc <4 x i32> @load_bc4_range(i32* %i, <4 x i32> %a) {
; CHECK-LE-LABEL: load_bc4_range:
; CHECK-LE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: vldr p0, [r0, #508]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_bc4_range:
; CHECK-BE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: vldr p0, [r0, #508]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vrev64.32 q1, q0
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.32 q0, q1
; CHECK-BE-NEXT: bx lr
%g = getelementptr inbounds i32, i32* %i, i32 127
%l = load i32, i32* %g, align 4
%c = tail call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %l)
%s = select <4 x i1> %c, <4 x i32> %a, <4 x i32> zeroinitializer
ret <4 x i32> %s
}
define arm_aapcs_vfpcc <4 x i32> @load_bc4_range2(i32* %i, <4 x i32> %a) {
; CHECK-LE-LABEL: load_bc4_range2:
; CHECK-LE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: vldr p0, [r0, #-508]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_bc4_range2:
; CHECK-BE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: vldr p0, [r0, #-508]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vrev64.32 q1, q0
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.32 q0, q1
; CHECK-BE-NEXT: bx lr
%g = getelementptr inbounds i32, i32* %i, i32 -127
%l = load i32, i32* %g, align 4
%c = tail call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %l)
%s = select <4 x i1> %c, <4 x i32> %a, <4 x i32> zeroinitializer
ret <4 x i32> %s
}
define arm_aapcs_vfpcc <4 x i32> @load_bc4_range3(i32* %i, <4 x i32> %a) {
; CHECK-LE-LABEL: load_bc4_range3:
; CHECK-LE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: add.w r0, r0, #512
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_bc4_range3:
; CHECK-BE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: add.w r0, r0, #512
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vrev64.32 q1, q0
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.32 q0, q1
; CHECK-BE-NEXT: bx lr
%g = getelementptr inbounds i32, i32* %i, i32 128
%l = load i32, i32* %g, align 4
%c = tail call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %l)
%s = select <4 x i1> %c, <4 x i32> %a, <4 x i32> zeroinitializer
ret <4 x i32> %s
}
define arm_aapcs_vfpcc <4 x i32> @load_bc4_range5(i32* %i, <4 x i32> %a) {
; CHECK-LE-LABEL: load_bc4_range5:
; CHECK-LE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: sub.w r0, r0, #512
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vmov.i32 q1, #0x0
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-LE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-LE-NEXT: vpsel q0, q0, q1
; CHECK-LE-NEXT: bx lr
;
; CHECK-BE-LABEL: load_bc4_range5:
; CHECK-BE: @ %bb.0:
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: sub.w r0, r0, #512
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vrev64.32 q1, q0
[ARM] Fold predicate_cast(load) into vldr p0 This adds a simple tablegen pattern for folding predicate_cast(load) into vldr p0, providing the alignment and offset are correct. Differential Revision: https://reviews.llvm.org/D86702
2020-09-04 18:29:59 +08:00
; CHECK-BE-NEXT: vldr p0, [r0]
[ARM] Extra predicate load tests. NFC
2020-09-04 00:52:37 +08:00
; CHECK-BE-NEXT: vmov.i32 q0, #0x0
; CHECK-BE-NEXT: vpsel q1, q1, q0
; CHECK-BE-NEXT: vrev64.32 q0, q1
; CHECK-BE-NEXT: bx lr
%g = getelementptr inbounds i32, i32* %i, i32 -128
%l = load i32, i32* %g, align 4
%c = tail call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %l)
%s = select <4 x i1> %c, <4 x i32> %a, <4 x i32> zeroinitializer
ret <4 x i32> %s
}
declare <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32)
declare <8 x i1> @llvm.arm.mve.pred.i2v.v8i1(i32)
declare <16 x i1> @llvm.arm.mve.pred.i2v.v16i1(i32)