Commit Graph

3621 Commits

Author SHA1 Message Date
Jay Foad 7816ad918f [AMDGPU] Restrict v_cndmask_b32 abs/neg modifiers to f32
Summary:
D64497 allowed abs/neg source modifiers on v_cndmask_b32 but it doesn't
make any sense to apply them to f16 operands; they would interpret the
bits of the value as an f32, giving nonsensical results. This patch
restricts them to f32 operands.

Reviewers: arsenm, hakzsam

Subscribers: kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, hiraditya, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64636

llvm-svn: 365904
2019-07-12 15:02:59 +00:00
Fangrui Song b251cc0d91 Delete dead stores
llvm-svn: 365903
2019-07-12 14:58:15 +00:00
Michael Liao 16d3c1ac03 [AMDGPU] Skip calculating callee saved registers for entry function.
Reviewers: arsenm

Subscribers: kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, hiraditya, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64596

llvm-svn: 365846
2019-07-11 23:53:30 +00:00
Matt Arsenault e5fb434d92 AMDGPU: s_waitcnt field should be treated as unsigned
Also make it an ImmLeaf, so it should work with global isel as well,
which was part of the point of moving it in the first place.

llvm-svn: 365842
2019-07-11 23:42:57 +00:00
Stanislav Mekhanoshin 28550c8680 [AMDGPU] Fixed asan error with agpr spilling
Instruction was used after it was erased.

llvm-svn: 365837
2019-07-11 22:30:11 +00:00
Stanislav Mekhanoshin 937ff6e701 [AMDGPU] gfx908 agpr spilling
Differential Revision: https://reviews.llvm.org/D64594

llvm-svn: 365833
2019-07-11 21:54:13 +00:00
Stanislav Mekhanoshin 7d2019bb96 [AMDGPU] gfx908 hazard recognizer
Differential Revision: https://reviews.llvm.org/D64593

llvm-svn: 365829
2019-07-11 21:30:34 +00:00
Stanislav Mekhanoshin b83e283e65 [AMDGPU] gfx908 scheduling
Differential Revision: https://reviews.llvm.org/D64590

llvm-svn: 365826
2019-07-11 21:25:00 +00:00
Stanislav Mekhanoshin e67cc380a8 [AMDGPU] gfx908 mfma support
Differential Revision: https://reviews.llvm.org/D64584

llvm-svn: 365824
2019-07-11 21:19:33 +00:00
Matt Arsenault b725d27350 AMDGPU/GlobalISel: Move kernel argument handling to separate function
llvm-svn: 365782
2019-07-11 14:18:25 +00:00
Jay Foad c1b7db9eda Remove some redundant code from r290372 and improve a comment.
llvm-svn: 365741
2019-07-11 08:49:52 +00:00
Stanislav Mekhanoshin e93279fd1b [AMDGPU] gfx908 atomic fadd and atomic pk_fadd
Differential Revision: https://reviews.llvm.org/D64435

llvm-svn: 365717
2019-07-11 00:10:17 +00:00
Stanislav Mekhanoshin c0ae1be066 [AMDGPU] gfx908 dot instruction support
Differential Revision: https://reviews.llvm.org/D64431

llvm-svn: 365715
2019-07-11 00:00:27 +00:00
Matt Arsenault 6ce1b4fec5 GlobalISel: Legalization for G_FMINNUM/G_FMAXNUM
llvm-svn: 365658
2019-07-10 16:31:19 +00:00
Matt Arsenault 58426a3707 AMDGPU: Serialize mode from MachineFunctionInfo
llvm-svn: 365653
2019-07-10 16:09:26 +00:00
Jay Foad bba37e89a5 [AMDGPU] Allow abs/neg source modifiers on v_cndmask_b32
Summary:
D59191 added support for these modifiers in the assembler and
disassembler. This patch just teaches instruction selection that it can
use them.

Reviewers: arsenm, tstellar

Subscribers: kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, hiraditya, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64497

llvm-svn: 365640
2019-07-10 14:53:47 +00:00
Tom Stellard d0ba79fe7b AMDGPU/GlobalISel: Add support for wide loads >= 256-bits
Summary:
This adds support for the most commonly used wide load types:
<8xi32>, <16xi32>, <4xi64>, and <8xi64>

Reviewers: arsenm

Reviewed By: arsenm

Subscribers: hiraditya, kzhuravl, jvesely, wdng, nhaehnle, yaxunl, rovka, kristof.beyls, dstuttard, tpr, t-tye, volkan, Petar.Avramovic, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D57399

llvm-svn: 365586
2019-07-10 00:22:41 +00:00
Matt Arsenault b1843e130a GlobalISel: Implement lower for G_FCOPYSIGN
In SelectionDAG AMDGPU treated these as legal, but this was mostly
because the bitcasts required for FP types were painful. Theoretically
the bitpattern should eventually match to bfi, so don't bother trying
to get the patterns to import.

llvm-svn: 365583
2019-07-09 23:34:29 +00:00
Matt Arsenault 3f1a34546c AMDGPU/GlobalISel: Fix legality for G_BUILD_VECTOR
llvm-svn: 365575
2019-07-09 22:48:04 +00:00
Stanislav Mekhanoshin 1e9eae95af [AMDGPU] gfx908 v_pk_fmac_f16 support
Differential Revision: https://reviews.llvm.org/D64433

llvm-svn: 365573
2019-07-09 22:42:24 +00:00
Stanislav Mekhanoshin 50d7f46460 [AMDGPU] gfx908 mAI instructions, MC part
Differential Revision: https://reviews.llvm.org/D64446

llvm-svn: 365563
2019-07-09 21:43:09 +00:00
Craig Topper 84a1f07363 [X86][AMDGPU][DAGCombiner] Move call to allowsMemoryAccess into isLoadBitCastBeneficial/isStoreBitCastBeneficial to allow X86 to bypass it
Basically the problem is that X86 doesn't set the Fast flag from
allowsMemoryAccess on certain CPUs due to slow unaligned memory
subtarget features. This prevents bitcasts from being folded into
loads and stores. But all vector loads and stores of the same width
are the same cost on X86.

This patch merges the allowsMemoryAccess call into isLoadBitCastBeneficial to allow X86 to skip it.

Differential Revision: https://reviews.llvm.org/D64295

llvm-svn: 365549
2019-07-09 19:55:28 +00:00
Stanislav Mekhanoshin 9e77d0c6df [AMDGPU] gfx908 register file changes
Differential Revision: https://reviews.llvm.org/D64438

llvm-svn: 365546
2019-07-09 19:41:51 +00:00
Stanislav Mekhanoshin 22b2c3d651 [AMDGPU] gfx908 target
Differential Revision: https://reviews.llvm.org/D64429

llvm-svn: 365525
2019-07-09 18:10:06 +00:00
Christudasan Devadasan b2d24bd540 [AMDGPU] Created a sub-register class for the return address operand in the return instruction.
Function return instruction lowering, currently uses the fixed register pair s[30:31] for holding
the return address. It can be any SGPR pair other than the CSRs. Created an SGPR pair sub-register class
exclusive of the CSRs, and used this regclass while lowering the return instruction.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D63924

llvm-svn: 365512
2019-07-09 16:48:42 +00:00
Matt Arsenault 4dd5755d01 AMDGPU/GlobalISel: Legalize more concat_vectors
llvm-svn: 365488
2019-07-09 14:17:31 +00:00
Matt Arsenault 6bdb92d833 AMDGPU/GlobalISel: Improve regbankselect for icmp s16
Account for 64-bit scalar eq/ne when available.

llvm-svn: 365487
2019-07-09 14:13:09 +00:00
Matt Arsenault 8b8eee5904 AMDGPU/GlobalISel: Make s16 G_ICMP legal
llvm-svn: 365486
2019-07-09 14:10:43 +00:00
Matt Arsenault e6d10f97dd AMDGPU/GlobalISel: Select G_SUB
llvm-svn: 365484
2019-07-09 14:05:11 +00:00
Matt Arsenault 872f38be7e AMDGPU/GlobalISel: Select G_UNMERGE_VALUES
llvm-svn: 365483
2019-07-09 14:02:26 +00:00
Matt Arsenault 9b7ffc4e55 AMDGPU/GlobalISel: Select G_MERGE_VALUES
llvm-svn: 365482
2019-07-09 14:02:20 +00:00
Stanislav Mekhanoshin c776dc0b60 [AMDGPU] Added td definitions for HW regs
Infrastructure work for future commit. NFC.

Differential Revision: https://reviews.llvm.org/D64370

llvm-svn: 365432
2019-07-09 03:20:33 +00:00
Stanislav Mekhanoshin 818d748a45 [AMDGPU] Always use s_memtime for readcyclecounter
Differential Revision: https://reviews.llvm.org/D64369

llvm-svn: 365431
2019-07-09 03:10:18 +00:00
Matt Arsenault 9e7cbc0e7d AMDGPU: Split extload/zextload local load patterns
This will help removing the custom load predicates, allowing the
global isel emitter to handle them.

llvm-svn: 365398
2019-07-08 22:08:23 +00:00
Bill Wendling c8933c4070 Add parentheses to silence warning.
llvm-svn: 365394
2019-07-08 22:00:33 +00:00
Matt Arsenault 8561844321 AMDGPU: Fix unused variable in release build
llvm-svn: 365378
2019-07-08 19:47:42 +00:00
Matt Arsenault acc9e1e4c2 AMDGPU: Fix stray typing
llvm-svn: 365373
2019-07-08 19:05:19 +00:00
Matt Arsenault 71dfb7ec5c AMDGPU: Make s34 the FP register
Make the FP register callee saved.

This is tricky because now the FP needs to be spilled in the prolog
relative to the incoming SP register, rather than the frame register
used throughout the rest of the function. I don't like how this
bypassess the standard mechanism for CSR spills just to get the
correct insert point. I may look for a better solution, since all CSR
VGPRs may also need to have all lanes activated. Another option might
be to make getFrameIndexReference change the base register if the
frame index is a CSR, and then try to figure out the right insertion
point in emitProlog.

If there is a free VGPR lane available for SGPR spilling, try to use
it for the FP. If that would require intrtoducing a new VGPR spill,
try to use a free call clobbered SGPR. Only fallback to introducing a
new VGPR spill as a last resort.

This also doesn't attempt to handle SGPR spilling with scalar stores.

llvm-svn: 365372
2019-07-08 19:03:38 +00:00
Matt Arsenault 5e643036cb AMDGPU: Move DEBUG_TYPE definition below includes
llvm-svn: 365369
2019-07-08 18:48:39 +00:00
Matt Arsenault 224d8cd987 AMDGPU: Remove mubuf specific PatFrags
These are identical to the *_global PatFrag, and will only create more
work to get the GlobalISel importer to handle them.

llvm-svn: 365350
2019-07-08 16:53:53 +00:00
Matt Arsenault 430b0497e7 AMDGPU: Move waitcnt intrinsic to instruction definition pattern
llvm-svn: 365349
2019-07-08 16:53:48 +00:00
Dmitry Preobrazhensky 2eff0318c6 [AMDGPU][MC] Corrected parsing of FLAT offset modifier
Summary of changes:

- simplified handling of FLAT offset: offset_s13 and offset_u12 have been replaced with flat_offset;
- provided information about error position for pre-gfx9 targets;
- improved errors handling.

Reviewers: artem.tamazov, arsenm, rampitec

Differential Revision: https://reviews.llvm.org/D64244

llvm-svn: 365321
2019-07-08 14:27:37 +00:00
Jay Foad 38902350ef [AMDGPU] Use a named predicate instead of a magic number.
Reviewers: arsenm

Reviewed By: arsenm

Subscribers: arsenm, kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, hiraditya, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64201

llvm-svn: 365294
2019-07-08 07:04:58 +00:00
Matt Arsenault 5e9610a3f5 AMDGPU: Fix assert in clang test
llvm-svn: 365245
2019-07-05 21:09:53 +00:00
Matt Arsenault e7e23e3e91 AMDGPU: Make AMDGPUPerfHintAnalysis an SCC pass
Add a string attribute instead of directly setting
MachineFunctionInfo. This avoids trying to get the analysis in the
MachineFunctionInfo in a way that doesn't work with the new pass
manager.

This will also avoid re-visiting the call graph for every single
function.

llvm-svn: 365241
2019-07-05 20:26:13 +00:00
Michael Liao 8d6ea2d48c [CodeGen] Enhance `MachineInstrSpan` to allow the end of MBB to be used.
Summary:
- Explicitly specify the parent MBB to allow the end iterator to be
  used.

Reviewers: aprantl, MatzeB, craig.topper, qcolombet

Subscribers: arsenm, jvesely, nhaehnle, hiraditya, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64261

llvm-svn: 365240
2019-07-05 20:23:59 +00:00
Christudasan Devadasan 652ad423bb [NFC] A test commit to check the access permission. Removed a blank line.
llvm-svn: 365223
2019-07-05 17:07:42 +00:00
Yaxun Liu a62413526d [AMDGPU] Added a new metadata for multi grid sync implicit argument
Patch by Christudasan Devadasan.

Differential Revision: https://reviews.llvm.org/D63886

llvm-svn: 365217
2019-07-05 16:05:17 +00:00
Jay Foad 7e0c10b55f [AMDGPU] DPP combiner: recognize identities for more opcodes
Summary:
This allows the DPP combiner to kick in more often. For example the
exclusive scan generated by the atomic optimizer for a divergent atomic
add used to look like this:

        v_mov_b32_e32 v3, v1
        v_mov_b32_e32 v5, v1
        v_mov_b32_e32 v6, v1
        v_mov_b32_dpp v3, v2  wave_shr:1 row_mask:0xf bank_mask:0xf
        s_nop 1
        v_add_u32_dpp v4, v3, v3  row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:0
        v_mov_b32_dpp v5, v3  row_shr:2 row_mask:0xf bank_mask:0xf
        v_mov_b32_dpp v6, v3  row_shr:3 row_mask:0xf bank_mask:0xf
        v_add3_u32 v3, v4, v5, v6
        v_mov_b32_e32 v4, v1
        s_nop 1
        v_mov_b32_dpp v4, v3  row_shr:4 row_mask:0xf bank_mask:0xe
        v_add_u32_e32 v3, v3, v4
        v_mov_b32_e32 v4, v1
        s_nop 1
        v_mov_b32_dpp v4, v3  row_shr:8 row_mask:0xf bank_mask:0xc
        v_add_u32_e32 v3, v3, v4
        v_mov_b32_e32 v4, v1
        s_nop 1
        v_mov_b32_dpp v4, v3  row_bcast:15 row_mask:0xa bank_mask:0xf
        v_add_u32_e32 v3, v3, v4
        s_nop 1
        v_mov_b32_dpp v1, v3  row_bcast:31 row_mask:0xc bank_mask:0xf
        v_add_u32_e32 v1, v3, v1
        v_add_u32_e32 v1, v2, v1
        v_readlane_b32 s0, v1, 63

But now most of the dpp movs are combined into adds:

        v_mov_b32_e32 v3, v1
        v_mov_b32_e32 v5, v1
        s_nop 0
        v_mov_b32_dpp v3, v2  wave_shr:1 row_mask:0xf bank_mask:0xf
        s_nop 1
        v_add_u32_dpp v4, v3, v3  row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:0
        v_mov_b32_dpp v5, v3  row_shr:2 row_mask:0xf bank_mask:0xf
        v_mov_b32_dpp v1, v3  row_shr:3 row_mask:0xf bank_mask:0xf
        v_add3_u32 v1, v4, v5, v1
        s_nop 1
        v_add_u32_dpp v1, v1, v1  row_shr:4 row_mask:0xf bank_mask:0xe
        s_nop 1
        v_add_u32_dpp v1, v1, v1  row_shr:8 row_mask:0xf bank_mask:0xc
        s_nop 1
        v_add_u32_dpp v1, v1, v1  row_bcast:15 row_mask:0xa bank_mask:0xf
        s_nop 1
        v_add_u32_dpp v1, v1, v1  row_bcast:31 row_mask:0xc bank_mask:0xf
        v_add_u32_e32 v1, v2, v1
        v_readlane_b32 s0, v1, 63

Reviewers: arsenm, vpykhtin

Subscribers: kzhuravl, nemanjai, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, hiraditya, kbarton, MaskRay, jfb, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64207

llvm-svn: 365211
2019-07-05 14:52:48 +00:00
Tim Renouf 5816889c74 [AMDGPU] Custom lower INSERT_SUBVECTOR v3, v4, v5, v8
Summary:
Since the changes to introduce vec3 and vec5, INSERT_VECTOR for these
sizes has been marked "expand", which made LegalizeDAG lower it to loads
and stores via a stack slot. The code got optimized a bit later, but the
now-unused stack slot was never deleted.

This commit avoids that problem by custom lowering INSERT_SUBVECTOR into
an EXTRACT_VECTOR_ELT and INSERT_VECTOR_ELT for each element in the
subvector to insert.

V2: Addressed review comments re test.

Differential Revision: https://reviews.llvm.org/D63160

Change-Id: I9e3c13e36f68cfa3431bb9814851cc1f673274e1
llvm-svn: 365148
2019-07-04 17:38:24 +00:00