[AMDGPU] Check MI bundles for hazards
Summary: GCNHazardRecognizer fails to identify hazards that are in and around bundles. This patch allows the hazard recognizer to consider bundled instructions in both scheduler and hazard recognizer mode. We ignore “bundledness” for the purpose of detecting hazards and examine the instructions individually.
Reviewers: arsenm, msearles, rampitec
Reviewed By: rampitec
Subscribers: kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61564
llvm-svn: 360199
2019-05-08 06:12:15 +08:00
|
|
|
# RUN: llc -march=amdgcn -mcpu=gfx902 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,XNACK,GCX9 %s
|
|
|
|
# RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,NOXNACK,GFX9 %s
|
2019-06-13 02:44:11 +08:00
|
|
|
# RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-WavefrontSize32,+WavefrontSize64 -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,NOXNACK,GFX10 %s
|
[AMDGPU] Check MI bundles for hazards
Summary: GCNHazardRecognizer fails to identify hazards that are in and around bundles. This patch allows the hazard recognizer to consider bundled instructions in both scheduler and hazard recognizer mode. We ignore “bundledness” for the purpose of detecting hazards and examine the instructions individually.
Reviewers: arsenm, msearles, rampitec
Reviewed By: rampitec
Subscribers: kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61564
llvm-svn: 360199
2019-05-08 06:12:15 +08:00
|
|
|
|
|
|
|
# GCN-LABEL: name: break_smem_clause_simple_load_smrd8_ptr_hidden_bundle
|
[AMDGPU] Enable serializing of argument info.
Summary:
- Support serialization of all arguments in machine function info. This
enables fabricating MIR tests depending on argument info.
Reviewers: arsenm, rampitec
Subscribers: kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D64096
llvm-svn: 364995
2019-07-03 10:00:21 +08:00
|
|
|
# GCN: bb.0:
|
[AMDGPU] Check MI bundles for hazards
Summary: GCNHazardRecognizer fails to identify hazards that are in and around bundles. This patch allows the hazard recognizer to consider bundled instructions in both scheduler and hazard recognizer mode. We ignore “bundledness” for the purpose of detecting hazards and examine the instructions individually.
Reviewers: arsenm, msearles, rampitec
Reviewed By: rampitec
Subscribers: kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61564
llvm-svn: 360199
2019-05-08 06:12:15 +08:00
|
|
|
# GCN: }
|
|
|
|
# XNACK-NEXT: S_NOP
|
|
|
|
# NOXNACK-NOT: S_NOP
|
|
|
|
# GCN: S_LOAD_DWORDX2_IMM
|
|
|
|
---
|
|
|
|
name: break_smem_clause_simple_load_smrd8_ptr_hidden_bundle
|
|
|
|
body: |
|
|
|
|
bb.0:
|
|
|
|
BUNDLE implicit-def $sgpr6_sgpr7 {
|
|
|
|
$sgpr10_sgpr11 = S_LOAD_DWORDX2_IMM $sgpr12_sgpr13, 0, 0, 0
|
|
|
|
}
|
|
|
|
$sgpr14_sgpr15 = S_LOAD_DWORDX2_IMM $sgpr10_sgpr11, 0, 0, 0
|
|
|
|
S_ENDPGM 0
|
|
|
|
...
|
|
|
|
|
|
|
|
# GFX9-LABEL: name: hazard_precedes_bundle
|
|
|
|
# GFX9: S_MOV_B32
|
|
|
|
# GFX9-NEXT: S_NOP
|
|
|
|
# GFX9: BUNDLE
|
|
|
|
# GFX9-NEXT: S_NOP
|
|
|
|
---
|
|
|
|
name: hazard_precedes_bundle
|
|
|
|
body: |
|
|
|
|
bb.0:
|
|
|
|
$m0 = S_MOV_B32 $sgpr7
|
|
|
|
S_SENDMSG 3, implicit $exec, implicit $m0
|
|
|
|
$m0 = S_MOV_B32 $sgpr8
|
|
|
|
BUNDLE implicit-def $vgpr0 {
|
|
|
|
$vgpr0 = V_INTERP_P1_F32 killed $vgpr4, 0, 0, implicit $m0, implicit $exec
|
|
|
|
}
|
|
|
|
S_ENDPGM 0
|
|
|
|
...
|
|
|
|
|
|
|
|
# GCN-LABEL: name: vmem_vcc_hazard_ignore_bundle_instr
|
|
|
|
# GCN: S_LOAD_DWORDX2_IMM
|
|
|
|
# GCN-NEXT: }
|
|
|
|
# GCN-NEXT: S_NOP
|
|
|
|
# GCN-NEXT: S_NOP
|
|
|
|
# GCN-NEXT: S_NOP
|
|
|
|
# GCN-NEXT: S_NOP
|
|
|
|
# GCN: BUFFER_LOAD_DWORD_OFFEN
|
|
|
|
---
|
|
|
|
name: vmem_vcc_hazard_ignore_bundle_instr
|
|
|
|
body: |
|
|
|
|
bb.0:
|
|
|
|
$sgpr0_sgpr1_sgpr2_sgpr3 = IMPLICIT_DEF
|
|
|
|
$vgpr0 = IMPLICIT_DEF
|
|
|
|
BUNDLE implicit-def $vgpr1, implicit $vgpr0, implicit $vgpr0, implicit-def $vcc, implicit $vcc, implicit $exec {
|
|
|
|
$vgpr1 = V_ADDC_U32_e32 $vgpr0, $vgpr0, implicit-def $vcc, implicit $vcc, implicit $exec
|
|
|
|
}
|
|
|
|
BUNDLE implicit-def $sgpr0_sgpr1, implicit $sgpr10_sgpr11 {
|
|
|
|
$sgpr0_sgpr1 = S_LOAD_DWORDX2_IMM $sgpr10_sgpr11, 0, 0, 0
|
|
|
|
}
|
[AMDGPU] Extend buffer intrinsics with swizzling
Summary:
Extend cachepolicy operand in the new VMEM buffer intrinsics
to supply information whether the buffer data is swizzled.
Also, propagate this information to MIR.
Intrinsics updated:
int_amdgcn_raw_buffer_load
int_amdgcn_raw_buffer_load_format
int_amdgcn_raw_buffer_store
int_amdgcn_raw_buffer_store_format
int_amdgcn_raw_tbuffer_load
int_amdgcn_raw_tbuffer_store
int_amdgcn_struct_buffer_load
int_amdgcn_struct_buffer_load_format
int_amdgcn_struct_buffer_store
int_amdgcn_struct_buffer_store_format
int_amdgcn_struct_tbuffer_load
int_amdgcn_struct_tbuffer_store
Furthermore, disable merging of VMEM buffer instructions
in SI Load/Store optimizer, if the "swizzled" bit on the instruction
is on.
The default value of the bit is 0, meaning that data in buffer
is linear and buffer instructions can be merged.
There is no difference in the generated code with this commit.
However, in the future it will be expected that front-ends
use buffer intrinsics with correct "swizzled" bit set.
Reviewers: arsenm, nhaehnle, tpr
Reviewed By: nhaehnle
Subscribers: arsenm, kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, arphaman, jfb, Petar.Avramovic, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D68200
llvm-svn: 373491
2019-10-03 01:22:36 +08:00
|
|
|
$vgpr1 = BUFFER_LOAD_DWORD_OFFEN $vgpr0, $sgpr0_sgpr1_sgpr2_sgpr3, $vcc_lo, 0, 0, 0, 0, 0, 0, implicit $exec
|
[AMDGPU] Check MI bundles for hazards
Summary: GCNHazardRecognizer fails to identify hazards that are in and around bundles. This patch allows the hazard recognizer to consider bundled instructions in both scheduler and hazard recognizer mode. We ignore “bundledness” for the purpose of detecting hazards and examine the instructions individually.
Reviewers: arsenm, msearles, rampitec
Reviewed By: rampitec
Subscribers: kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61564
llvm-svn: 360199
2019-05-08 06:12:15 +08:00
|
|
|
S_ENDPGM 0
|
|
|
|
...
|
|
|
|
|
|
|
|
# GCN-LABEL: name: vmem_vcc_min_of_two_after_bundle
|
|
|
|
# GCN: bb.2:
|
|
|
|
# GCN-NEXT: S_NOP
|
|
|
|
# GCN-NEXT: S_NOP
|
|
|
|
# GCN-NEXT: S_NOP
|
|
|
|
# GCN-NEXT: S_NOP
|
|
|
|
# GCN-NEXT: S_NOP
|
|
|
|
# GCN-NEXT: BUFFER_LOAD_DWORD_OFFEN
|
|
|
|
---
|
|
|
|
name: vmem_vcc_min_of_two_after_bundle
|
|
|
|
body: |
|
|
|
|
bb.0:
|
|
|
|
successors: %bb.2
|
|
|
|
|
|
|
|
BUNDLE implicit-def $vgpr1, implicit $vgpr0 {
|
|
|
|
$sgpr0_sgpr1_sgpr2_sgpr3 = IMPLICIT_DEF
|
|
|
|
$vgpr0 = IMPLICIT_DEF
|
|
|
|
$vgpr1 = V_ADDC_U32_e32 $vgpr0, $vgpr0, implicit-def $vcc, implicit $vcc, implicit $exec
|
|
|
|
}
|
|
|
|
S_NOP 0
|
|
|
|
S_BRANCH %bb.2
|
|
|
|
|
|
|
|
bb.1:
|
|
|
|
successors: %bb.2
|
|
|
|
|
|
|
|
BUNDLE implicit-def $vgpr1, implicit $vgpr0 {
|
|
|
|
$vgpr1 = V_ADDC_U32_e32 $vgpr0, $vgpr0, implicit-def $vcc, implicit $vcc, implicit $exec
|
|
|
|
}
|
|
|
|
|
|
|
|
bb.2:
|
[AMDGPU] Extend buffer intrinsics with swizzling
Summary:
Extend cachepolicy operand in the new VMEM buffer intrinsics
to supply information whether the buffer data is swizzled.
Also, propagate this information to MIR.
Intrinsics updated:
int_amdgcn_raw_buffer_load
int_amdgcn_raw_buffer_load_format
int_amdgcn_raw_buffer_store
int_amdgcn_raw_buffer_store_format
int_amdgcn_raw_tbuffer_load
int_amdgcn_raw_tbuffer_store
int_amdgcn_struct_buffer_load
int_amdgcn_struct_buffer_load_format
int_amdgcn_struct_buffer_store
int_amdgcn_struct_buffer_store_format
int_amdgcn_struct_tbuffer_load
int_amdgcn_struct_tbuffer_store
Furthermore, disable merging of VMEM buffer instructions
in SI Load/Store optimizer, if the "swizzled" bit on the instruction
is on.
The default value of the bit is 0, meaning that data in buffer
is linear and buffer instructions can be merged.
There is no difference in the generated code with this commit.
However, in the future it will be expected that front-ends
use buffer intrinsics with correct "swizzled" bit set.
Reviewers: arsenm, nhaehnle, tpr
Reviewed By: nhaehnle
Subscribers: arsenm, kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, arphaman, jfb, Petar.Avramovic, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D68200
llvm-svn: 373491
2019-10-03 01:22:36 +08:00
|
|
|
$vgpr1 = BUFFER_LOAD_DWORD_OFFEN $vgpr0, $sgpr0_sgpr1_sgpr2_sgpr3, $vcc_lo, 0, 0, 0, 0, 0, 0, implicit $exec
|
[AMDGPU] Check MI bundles for hazards
Summary: GCNHazardRecognizer fails to identify hazards that are in and around bundles. This patch allows the hazard recognizer to consider bundled instructions in both scheduler and hazard recognizer mode. We ignore “bundledness” for the purpose of detecting hazards and examine the instructions individually.
Reviewers: arsenm, msearles, rampitec
Reviewed By: rampitec
Subscribers: kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61564
llvm-svn: 360199
2019-05-08 06:12:15 +08:00
|
|
|
...
|