This reverts commit ba07f300c6.
A build-vector sequence is made of pairs: rotate+insert. When constructing
a single vector, this results in a chain of 2*N instructions. The rotate
operation is a permute operation, but the insert uses a multiplication
resource: insert and rotate can execute in the same cycle, but obviously
they cannot operate on the same vector. The original halving idea is still
beneficial since it does allow for insert/rotate overlap, and for hiding
insert's latency.
Use integer vector scalar move instruction when move 0 to avoid add a integer-float move instruction.
Differential Revision: https://reviews.llvm.org/D116365
The patterns of the immediate comparison instruction is rewrite here, and put similar code to a class.
Do not change any function of the original code, making the code more concise.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D116215
For vectors with repeating values, old codegen would rotate and insert
every duplicate element. This patch replaces that behavior with a splat
of the most common element, vinsert/vror only occur when needed.
They are the same as for the other HVX vectors, but types need to be
listed explicitly. Also, add a detailed codegen testcase.
Co-authored-by: Abhikrant Sharma <quic_abhikran@quicinc.com>
After consuming all vector registers, the scalable vector values will be
passed indirectly. The pointer values will be saved in general
registers. If all general registers are used up, we will report an error to
notify users the compiler does not support passing scalable vector
values through the stack. In this patch, we remove the restriction. After
all general registers are used up, we use the stack to save the
pointers which point to the indirect passed scalable vector values.
Differential Revision: https://reviews.llvm.org/D116310
A 'CMOV 1, 0, CC, %cpsr, Cmp' is the same as a 'CSINC 0, 0, CC, Cmp',
and can be treated the same in IsCMPZCSINC added in D114013. This allows
us to remove the unnecessary CMOV in the same way that we could remove a
CSINC.
Differential Revision: https://reviews.llvm.org/D115188
Basic zmm reg-reg moves (with predication) are more port limited than xmm/ymm moves, so we need to add a separate class for them.
We still appear to be missing move-elimination patterns for most of the intel models, which looks to be one of the main diffs for basic codegen analysis between llvm-mca and uops.info
Load/stores are a bit messier and might be better handled as overrides.
X86 allows inlining functions if the callee target features are a
subset of the caller target features. This ensures that we don't
inline something into a caller that does not support it.
However, this does not account for possible call ABI mismatches as
a result of inlining. If a call passing a vector argument was
originally in a -avx function, calling another -avx function, the
vector is passed in xmm. If we now inline it into a +avx function,
then it will be passed in ymm, even though the callee expects it in xmm.
Fix this by scanning over all calls in the function and checking
whether ABI incompatibility is possible. Calls that only pass scalar
types are excluded, as I believe those always use the same ABI
independent of target features.
Fixes https://github.com/llvm/llvm-project/issues/52660.
Differential Revision: https://reviews.llvm.org/D116036
The implicit defines may come from a partial define in an instruction.
It does not mean the defining instruction and the COPY instruction have
the same vl and vtype. When the source comes from the implicit defines,
do not convert the whole register copies to vmv.v.v.
Differential Revision: https://reviews.llvm.org/D115866
In 'trunc' i16/32/64 to i1 pattern the 'and $src, 1' node supply operand to 'setcc'.
The latter is selected to S_CMP_EQ/V_CMP_EQ dependent on the divergence. In case the 'and' is scalar
and 'setcc' is divergent, we need VGPR to SGPR copy to adjust input operand for V_CMP_EQ.
This patch changes the S_AND_B32 to V_AND_B32_e64 in the 'trunc to i1' divergent patterns.
Reviewed By: rampitec
Differential Revision: https://reviews.llvm.org/D116241
Converts RSHRN/RSHRN2 to RADDHN/RADDHN2 when the shift amount is half
the width of the vector element. The latter has twice the throughput
and half the latency on Arm out-of-order cores. Setting up the zero
register adds no latency.
Differential Revision: https://reviews.llvm.org/D116166
This reverts commit a954558e87.
Thanks Yuanfang's help. I think I found the root cause of the buildbot
fail.
The failed test has both Memory and Immediate X86Operand. All data of
different operand kinds share the same memory space by a union
definition. So it has chance we get the wrong result if we don't check
the operand kind.
It's probably it happen to be the correct value in my local environment
so that I can't reproduce the fail.
Differential Revision: https://reviews.llvm.org/D116090
The 'r' constraint uses the GPR class. There is generic support
for bitcasting and extending/truncating non-integer VTs to the
required integer VT. This doesn't work for scalable vectors and
instead crashes.
To prevent this, explicitly reject vectors. Fixed vectors might
work without crashing, but it doesn't seem worthwhile to allow.
While there remove an unnecessary level of indentation in the
"vr" and "vm" constraint handling.
Differential Revision: https://reviews.llvm.org/D115810
For fixed and scalable vectors, each intrinsic x is lowered to vmx.mm,
dropping the mask, which is safe to do as masked-off elements are
undef anyway.
Differential Revision: https://reviews.llvm.org/D115339
Removing dead frame indices for VGPR to AGPR spills is incorrect
when the frame index is shared by multiple objects, which may
occur due to stack slot coloring. The problem is that subsequent
code that processes the other object will assert because the stack
frame index is marked dead.
Removing dead frame indices is needed prior to stack slot
coloring, which is what happens with SGPR to VGPR spills. These
spills are lowered prior to stack slot coloring, but the VGPR
to AGPR spills are processed afterwards during the Prolog/Epilog
Inserter pass. This patch marks the VGPR to AGPR spill slot as
dead if the slot is not used by another object.
Differential Revision: https://reviews.llvm.org/D115996
-0.0 requires a constant pool. +0.0 can be made with vmv.v.x x0.
Not doing this in getNeutralElement for fear of changing other targets.
Differential Revision: https://reviews.llvm.org/D115978
This adds support for strict conversions between fp types and between
integer and fp.
NOTE: RISCV has static rounding mode instructions, but the constrainted
intrinsic metadata is not used to select static rounding modes. Dynamic
rounding mode is always used.
Differential Revision: https://reviews.llvm.org/D115997
Converts concat_vectors(Vd, trunc(smin(smax Vm, -2^n), 2^n-1) to
sqxtn2(Vd, Vm). Deliberately not handling v2i64 ~> v2i32 as the
min/max nodes are not legal (same thing we did for the SQXTN
patterns in https://reviews.llvm.org/D103263).
Differential Revision: https://reviews.llvm.org/D116105
Fix issue in TargetLowering::expandROT where we only attempt to flip a rotation if the other direction has better support - this matches TargetLowering::expandFunnelShift
This allows us to enable ISD::ROTR lowering on SSE targets, which particularly simplifies/improves codegen for splat amount and AVX2 per-element shifts.
The loop vectorizer can interleave scalar loops even if it doesn't
vectorize them. I don't believe we intended to enable this when
we enabled interleaving for vector instructions.
Disable interleaving for VF=1 like X86 and AMDGPU already do. Test
lifted from AMDGPU.
Differential Revision: https://reviews.llvm.org/D115975
Artifact combiner is not able to access individual elements after using
LCMTy style merge/unmerge, extract and insert to change vector number of
elements (pad with undef or split to sub-vector instructions).
Use unmerge to individual elements instead and then merge elements into
requested types.
Change argument lowering for vectors and moreElementsVector to use
buildPadVectorWithUndefElements and buildDeleteTrailingVectorElements.
FewerElementsVector had a few helpers that had different behavior,
introduce new helper for most of the opcodes.
FewerElementsVector helper is more flexible since it can create leftover
instruction smaller then requested type (useful in case target wants to
avoid pad with undef and use fewer registers). If target does not want
leftover of different type it should call more elements first.
Some helpers were performing more elements first to have split without
leftover. Opcodes that used this helper use clampMaxNumElementsStrict
(does more elements first) in LegalizerInfo to avoid test changes.
Fixes failures caused by failing to combine artifacts created during
more/fewer elements vector.
Differential Revision: https://reviews.llvm.org/D114198
D115225 tried to roll back the effects on symbols of MS inline asm
introduced by D113096. But the combination of the conditions cannot
match all the changes. As a result, there are still fails after the
patch.
This patch fixes the problem by checking the exact conditions for MS
global variables, i.e., variable (by FrontendSize != 0) + non rip/eip
(by DefaultBaseReg == 0), so that we can fully roll back for D113096.
Reviewed By: skan
Differential Revision: https://reviews.llvm.org/D116090
Enable transform (X & Y) == Y ---> (~X & Y) == 0 and (X & Y) != Y ---> (~X & Y) != 0 when have Zbb extension to use more andn instruction.
Differential Revision: https://reviews.llvm.org/D115922
This change enables divergence-driven instruction selection for the build_vector DAG nodes.
It also enables packed i16 instructions for GFX9.
Reviewed By: rampitec
Differential Revision: https://reviews.llvm.org/D116187
Currently the return address ABI registers s[30:31], which fall in the call
clobbered register range, are added as a live-in on the function entry to
preserve its value when we have calls so that it gets saved and restored
around the calls.
But the DWARF unwind information (CFI) needs to track where the return address
resides in a frame and the above approach makes it difficult to track the
return address when the CFI information is emitted during the frame lowering,
due to the involvment of understanding the control flow.
This patch moves the return address ABI registers s[30:31] into callee saved
registers range and stops adding live-in for return address registers, so that
the CFI machinery will know where the return address resides when CSR
save/restore happen during the frame lowering.
And doing the above poses an issue that now the return instruction uses undefined
register `sgpr30_sgpr31`. This is resolved by hiding the return address register
use by the return instruction through the `SI_RETURN` pseudo instruction, which
doesn't take any input operands, until the `SI_RETURN` pseudo gets lowered to the
`S_SETPC_B64_return` during the `expandPostRAPseudo()`.
As an added benefit, this patch simplifies overall return instruction handling.
Note: The AMDGPU CFI changes are there only in the downstream code and another
version of this patch will be posted for review for the downstream code.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D114652
The areFunctionArgsABICompatible() hook currently accepts a list of
pointer arguments, though what we're actually interested in is the
ABI compatibility after these pointer arguments have been converted
into value arguments.
This means that a) the current API is incompatible with opaque
pointers (because it requires inspection of pointee types) and
b) it can only be used in the specific context of ArgPromotion.
I would like to reuse the API when inspecting calls during inlining.
This patch converts it into an areTypesABICompatible() hook, which
accepts a list of types. This makes the method more generally usable,
and compatible with opaque pointers from an API perspective (the
actual usage in ArgPromotion/Attributor is still incompatible,
I'll follow up on that in separate patches).
Differential Revision: https://reviews.llvm.org/D116031
The current code makes the assumption that equality
comparison can be performed with a word comparison
instruction. While this is true if the entire 64-bit
results are used, it does not generally work. It is
possible that the low order words and high order
words produce different results and a user of only
one will get the wrong result.
This patch adds an and of the result words so that
each word has the result of the comparison of the
entire doubleword that contains it.
Differential revision: https://reviews.llvm.org/D115678
Commit 150681f increases
cost of producing MMA types (vector pair and quad).
However, it increases the cost for getUserCost() which is
used in unrolling. As a result, loops that contain these
types already (from the user code) cannot be unrolled
(even with the user's unroll pragma). This was an unintended
sideeffect. Reverting that portion of the commit to allow
unrolling such loops.
Differential revision: https://reviews.llvm.org/D115424
Converts concat_vectors((trunc (lshr)), (trunc (lshr))) to UZP2
when the shift amount is half the width of the vector element.
Differential Revision: https://reviews.llvm.org/D116021
Move shift-by-constant handling and move it into its only user (VSHIFT intrinsics lowering).
This is some prep-work for getTargetVShiftNode to no longer take a scalar shift amount - we're introducing temporary ISD::EXTRACT_VECTOR_ELT nodes via SelectionDAG::getSplatValue to accommodate this which can cause various issues, including unnecessary scalarization and xmm->gpr->xmm transfers, and causes problems for 32-bit codegen if we fail to remove an (illegal) i64 scalar extracted from a (legal) vXi64 vector.
Pull out the "rotl(x,y) --> (unpack(x,x) << zext(splat(y % bw))) >> bw" special case from vXi8 lowering so we can reuse it for vXi32 types as well.
There's still some regressions with vXi16 to handle before this becomes entirely general.
It also allows us to remove the now unnecessary hack for handling amount-modulo before splatting.
Attempt to lower a shuffle as a permute instruction(zip/uzp/trn) for fixed length SVE.
Reviewed By: paulwalker-arm
Differential Revision: https://reviews.llvm.org/D113376
Depends on D115940 for the `Binary_rv_vr_vv` pattern class op isel
fragment used for divisions.
Reviewed By: kaz7
Differential Revision: https://reviews.llvm.org/D116035
Matteo Croce reported a bpf backend fatal error in
https://github.com/llvm/llvm-project/issues/52779
A simplified case looks like:
$ cat bug.c
extern int do_smth(int);
int test() {
return __builtin_btf_type_id(*(typeof(do_smth) *)do_smth, 1);
}
$ clang -target bpf -O2 -g -c bug.c
fatal error: error in backend: Empty type name for BTF_TYPE_ID_REMOTE reloc
...
The reason for the fatal error is that the relocation is against
a DISubroutineType like type 13 below:
!10 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
!11 = !{}
!12 = !DILocation(line: 3, column: 10, scope: !7)
!13 = !DISubroutineType(types: !14)
!14 = !{!10, !10}
The DISubroutineType doesn't have a name and there is no way for
downstream bpfloader/kernel to do proper relocation for it.
But we can improve error message to be more specific for this case.
The patch improved the error message to be:
fatal error: error in backend: SubroutineType not supported for BTF_TYPE_ID_REMOTE reloc
Differential Revision: https://reviews.llvm.org/D116063
Summary: When disassembling, symbolize a branch target operand
to print a label instead of a real address.
Reviewed By: shchenz
Differential Revision: https://reviews.llvm.org/D114492
These actions should only be used for adjusting the register types
(and the memory type as needed to satisfy the register
type). Unaligned accesses should be split as a type of lowering.
This has the effect of improving the code in many cases since now we
produce zextloads instead of separate loads with ands. The load/store
legality rules still seem far more complicated than necessary though.
ctlz/cttz get lowered to the set of target opcodes
This change enables the ISel to select SALU or VALU form according to the SDNode divergence.
CTLZ - S_FLBIT_I32_B32 if uniform and V_FFBH_U32_e64 if divergent
CTTZ - S_FF1_I32_B32 if uniform and V_FFBL_B32_e64 if divergent
Also @llvm.amdgcn.sffbh.i32 gets lowered to S_FLBIT_I32 if uniform and V_FFBH_I32_e64 if divergent
NOTE: 64bit versions S_FF1_I32_B64 and S_FLBIT_I32_B64 are not currently supported by the DAG ISel.
ctlz/cttz with i64 input are split into two 32bit instructions. Nevertheless, they already have the patterns
and were equipped with the divergence predicates to make sure they will be selected correctly when enabled.
Reviewed By: rampitec
Differential Revision: https://reviews.llvm.org/D116044
The availability of SVE should be sufficient to enable scalable
auto-vectorization.
This patch adds a new TTI interface to query the target what style of
vectorization it wants when scalable vectors are available. For other
targets than AArch64, this currently defaults to 'FixedWidthOnly'.
Differential Revision: https://reviews.llvm.org/D115651
This mnemonic has been supported by GAS for years and
it was added to the PowerPC ISA as of ISA 3.1. We will
support the mnemonic to be compatible with GAS.
This patch enables divergence predicates for min/max nodes.
It makes ISD::MIN/MAX selected to S_MIN_I(U)32/S_MAX_I(U)32 or V_MIN_I(U)32_e64/V_MAX_I(U)32_e64
Reviewed By: rampitec
Differential Revision: https://reviews.llvm.org/D115954
The "not" is defined as XOR $src -1.
We need to transform this pattern to either S_NOT_B32 or V_NOT_B32_e32
dependent on the "xor" node divergence.
Reviewed By: rampitec, foad
Differential Revision: https://reviews.llvm.org/D115884
This makes use of the code in D114013 to fold away unnecessary
CMPZ/CSINC starting from a CMOV, in a similar way to how we fold away
CSINV/CSINC/etc
Differential Revision: https://reviews.llvm.org/D115185
The IceLake scheduler model is still mainly a copy of the SkylakeServer model.
This patch adjusts the fp shuffle classes to account for most instructions now working on Port 1 as well as Port 5.
This is based off Agner + uops.info as well as the PR48110 report.
Differential Revision: https://reviews.llvm.org/D115752
The race condition happened when two pass managers ran on two different modules but modified/read the global variables.
To address this, I considered using singletons and freestanding functions to allow getting/setting `HardwareLimits` and `RegisterEncoding`, or making it local to the pass. I chose the latter and made it a member of `WaitcntsBrackets`, to minimizes the amount of global state.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D115896
Both these preference helper functions have initial support with
this change. The loop unrolling preferences are set with initial
settings to control thresholds, size and attributes of loops to
unroll with some tuning done. The peeling preferences may need
some tuning as well as the initial support looks much like what
other architectures utilize.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D113798
This patch partially resolves an issue for VLS code generation
where a mask is generated from a smaller width integer comparison
than the instruction using the mask requires.
Instead of sign extending a p register by converting it to a z
register, extending that, and converting back, we instead just
do an unpack of the p register.
A separate issue causes the code generation to still be poor when
the mask generation would fit in a neon register, as we then use
a neon comparison operation and have to convert that to a p register.
This will be resolved in a separate patch.
Reviewed By: peterwaller-arm
Differential Revision: https://reviews.llvm.org/D111221
This extends the custom lowering for truncating stores on
fixed length vectors in SVE to support masked truncating stores.
It also adds a DAG combine for truncates followed by masked
stores.
Reviewed By: peterwaller-arm, paulwalker-arm
Differential Revision: https://reviews.llvm.org/D108115
D111441 included trunc isel patterns for sve_int_pred_pattern_a
but no accompanying tests. This patch adds the missing tests and
also simplifies the isel patterns that use sve_cnt_shl_imm.
Differential Revision: https://reviews.llvm.org/D115512
The LOD bias operand is of type 'half' when the A16-bit is ON' for MIMG instructions.
'bias' is only 16-bit but occupies 32-bits with upper 16-bits containing junk.
The patch fixes both the paths(ISelDAG and GlobalISel) for proper encoding of LOD bias operand.
Differential Revision: https://reviews.llvm.org/D111754
Some MVE instructions have qr variants that take a Q and R register,
splatting the R register for each lane. This is usually handled fine for
standard splats as we sink the splat into the loop and combine the
resulting dup into the qr instruction. It does not work for constant
splats though, as we generate a vmovimm or constant pool load instead.
This intercepts that, generating a vdup of the constant instead where we
can turn the result into a qr instruction variant.
Differential Revision: https://reviews.llvm.org/D115242
This supports bitcode compilation using `clang -fwasm-exceptions`.
---
The current situation:
Currently the backend requires two options for Wasm EH:
`-wasm-enable-eh` and `-exception-model=wasm`. Wasm SjLj requires two
options as well: `-wasm-enable-sjlj` and `-exception-model=wasm`. When
using Wasm EH via Emscripten, you only need to pass `-fwasm-exceptions`,
and these options will be added within the clang driver. This
description will focus on the case of Wasm EH going forward, but Wasm
SjLj's case is similar.
When you pass `-fwasm-exceptions` to emcc and clang driver, the clang
driver adds these options to the command line that calls the clang
frontend (`clang -cc1`): `-mllvm -wasm-enable-eh` and
`-exception-model=wasm`. `-wasm-enable-eh` is prefixed with `-mllvm`, so
it is passed as is to the backend. But `-exception-model` is parsed and
processed within the clang frontend and stored in `LangOptions` class.
This info is later transferred to `TargetOptions` class, and then
eventually passed to `MCAsmInfo` class. All LLVM code queries this
`MCAsmInfo` to get the exception model.
---
Problem:
The problem is the whole `LangOptions` processing is bypassed when
compiling bitcode, so the information transfer of `LangOptions` ->
`TargetOptions` -> `MCAsmInfo` does not happen. They are all set to
`ExceptionHandling::None`, which is the default value.
---
What other targets do, and why we can't do the same:
Other targets support bitcode compilation by the clang driver, but they
can do that by using different triples. For example, X86 target supports
multiple triples, each of which has its own subclass of `MCAsmInfo`, so
it can hardcode the appropriate exception model within those subclasses'
constructors. But we don't have separate triples for each exception
mode: none, emscripten, and wasm.
---
What this CL does:
If we can figure out whether `-wasm-enable-eh` is passed to the backend,
we can programatically set the exception model from the backend, rather
than requiring it to be passed.
So we check `WasmEnableEH` and `WasmEnableSjLj` variables, which are
`cl::opt` for `-wasm-enable-eh` and `-wasm-enable-sjlj`, in
`WebAssemblyMCAsmInfo` constructor, and if either of them is set, we set
`MCAsmInfo.ExceptionType` to Wasm. `TargetOptions` cannot be updated
there, so we make sure they are the same later.
Fixes https://github.com/emscripten-core/emscripten/issues/15712.
Reviewed By: dschuff
Differential Revision: https://reviews.llvm.org/D115893
Our Zfhmin support is only MC layer, but these are CodeGen layer
interfaces. If f16 isn't a Legal type for CodeGen with Zfhmin, then
these interfaces should keep their non-Zfh behavior.
Reviewed By: luismarques
Differential Revision: https://reviews.llvm.org/D115822
This patch adds support for prologue and epilogue generation for the z/OS target under the XPLINK64 ABI for functions with a stack size of less than 1048576 bytes (huge stack frames).
Reviewed By: uweigand
Differential Revision: https://reviews.llvm.org/D114457
When hitting an else clause the type Stack should be reset to as it was at the start of the if, without taking into account the Type inserted into the Stack during the then branch of the if.
Reviewed By: aardappel
Differential Revision: https://reviews.llvm.org/D115748
When have Zbs extension, we could use bexti to fold (and (not (srl X, C)), 1) to (xor (bexti X, C), 1).
Differential Revision: https://reviews.llvm.org/D115629
According to the v-spec, the source and destination VR of vmv<nr>r.v should be aligned for the VR group size.
Differential Revision: https://reviews.llvm.org/D115720
This is an optimization, but also fixes a compile failure when no free
VGPRs are available. The problem still exists for gfx908 where a
scratch register is still required. This also still exists for the
SGPR to AGPR case.
This was a workaround for not supporting indirect calls when
instcombine didn't eliminate constant expression casts of the callee
at -O0. Indirect calls are supposed to work now, so drop the hack.
These are deprecated and should be replaced with getAlign().
Some of these asserts don't do anything because Load/Store/AllocaInst never have a 0 align value.
When references to the symbol `swift_async_extendedFramePointerFlags`
are emitted they have to be weak.
References to the symbol `swift_async_extendedFramePointerFlags` get
emitted only by frame lowering code. Therefore, the backend needs to track
references to the symbol and mark them weak.
Differential Revision: https://reviews.llvm.org/D115672
Instead of bailing and using the default expansion, we can more efficiently use the shl(unpack(x,x),unpack(amt,zero)) pattern for vXi8 rotl, as we'll then use vXi16 fast PMULLW (or PSLLVW).
This required some minor changes to improve constant folding during unpack shuffle creation and convertShiftLeftToScale to support constants that have already been lowered to constant pools.
Attempt to lower a shuffle as a permute instruction(rev/revb/revh/revw) for fixed length SVE.
Reviewed By: paulwalker-arm
Differential Revision: https://reviews.llvm.org/D114960
Teach convertToThreeAddress to use the V_FMA_F16_gfx9 pseudo (i.e. the
standard instruction in GFX9 onwards) instead of V_FMA_F16 (the legacy
pseudo for GFX8 compatibility, which is no longer supported in GFX10).
This follows the example of macToMad in SIFoldOperands.
Differential Revision: https://reviews.llvm.org/D115731
Pseudos like V_MAD_U16 and V_FMA_F16 map down to what GFX9 calls
v_mad_legacy_u16 and v_fma_legacy_f16, which are documented to have the
same zeroing behaviour as on GFX8.
Differential Revision: https://reviews.llvm.org/D115729
We currently rely on generic promotion to vXi16/vXi32 types for rotation lowering on various AVX512 targets.
We can more efficiently perform this by making use of the shl(unpack(x,x),amt) style pattern that we already use for vXi8 rotation by splat amounts, either by widening to a larger vector type or unpacking lo/hi halves of the subvectors so we can access whatever vXi16/vXi32 per-element shifts are supported.
This uncovered an issue in the supportedVectorShiftWithImm/supportedVectorVarShift legality checkers which was using hasAVX512() instead of useAVX512Regs() to detect support for 512-bit vector shifts.
NOTE: I'm actually hoping to eventually reuse this code for shl(unpack(y,x),amt) funnel shift lowering (vXi8 and wider), but initially I just want to ensure we have efficient ISD::ROTL lowering for all targets.
Differential Revision: https://reviews.llvm.org/D115180
Approximately revert D103431.
LDS variables are allocated at kernel launch and deallocated at kernel exit.
The address is therefore kernel execution dependent. Global variables are
initialized by values written to .data, which can't be done for a LDS variable
as there is no kernel running, or by a global constructor. Initializing the
global to the address of some LDS allocated by a global constructor is possible
but indistinguishable from undef.
Assigning the address of a LDS variable to a global should be a sema error. It
isn't for openmp, haven't checked other languages. Failing that it could be set
to undef, perhaps in this pass.
Reviewed By: rampitec
Differential Revision: https://reviews.llvm.org/D115413
The IceLake scheduler model is still mainly a copy of the SkylakeServer model.
This patch adjusts the integer shuffle classes to account for most instructions now working on Port 1 as well as Port 5.
This is based off Agner + uops.info as well as the PR48110 report.
Differential Revision: https://reviews.llvm.org/D115547
Test that STRICT_FMINNUM/FMAXNUM are lowered to libcalls for f32/f64.
The RISC-V instructions don't match the behavior of fmin/fmax libcalls
with respect to SNaN.
Promoting FMINNUM/FMAXNUM for f16 needs more work outside of the
RISC-V backend.
Reviewed By: asb, arcbbb
Differential Revision: https://reviews.llvm.org/D115680
This patch revises the warning fix done in
a93b1792f1. Specifically, it rolls the
MRI.getType call into the assert, thereby avoiding the named variable.
In order to support constrained FP intrinsics we need to model FRM
dependency. Whether or not a instruction uses FRM is based on a 3
bit field in the instruction. Because of this we can't add
'Uses = [FRM]' to the tablegen descriptions.
This patch examines the immediate after isel and adds an implicit
use of FRM. This idea came from Roger Ferrer Ibanez.
Other ideas:
We could be overly conservative and just pretend all instructions with
frm field read the FRM register. Or we could have pseudoinstructions
for CodeGen with rounding mode.
Reviewed By: asb, frasercrmck, arcbbb
Differential Revision: https://reviews.llvm.org/D115555
We already do this for splat nodes that carry a VL, but not for
splats that use VLMAX.
Reviewed By: frasercrmck
Differential Revision: https://reviews.llvm.org/D115483
When possible, optimize TRUNCATE to generate Wasm SIMD narrow
instructions (i16x8.narrow_i32x4_u, i8x16.narrow_i16x8_u), rather than generate
lots of extract_lane and replace_lane.
Closes#50350.
If either operand has an element with allbits set, then we don't need the equivalent element from the other operand, as allbits are guaranteed to be set.
As reported from a failing firefox build, we can sometimes get frame
indices with negative offsets from a t2LDRi8. This adds support for
them, to prevent the crash.
Converts concat_vectors(V64 (trunc V128), V64 (trunc V128)), which
would otherwise be lowered as xtn followed by xtn2, to uzp1.
Differential Revision: https://reviews.llvm.org/D115435
Fix a couple of things that were causing stack protection to not work
correctly in functions that have scalable vectors on the stack:
* Use TypeSize when determining if accesses to a variable are
considered out-of-bounds so that the behaviour is correct for
scalable vectors.
* When stack protection is enabled move the stack protector location
to the top of the SVE locals, so that any overflow in them (or the
other locals which are below that) will be detected.
Fixes: https://github.com/llvm/llvm-project/issues/51137
Differential Revision: https://reviews.llvm.org/D111631
This patch updates expandCALL_RVMARKER to wrap the call, marker and
objc runtime call in an instruction bundle. This ensures later passes,
like machine block placement, cannot break them up.
On AArch64, the instruction sequence is already wrapped in a bundle.
Keeping the whole instruction sequence together is highly desirable for
performance and outweighs potential other benefits from breaking the
sequence up.
Reviewed By: ahatanak
Differential Revision: https://reviews.llvm.org/D115230
Instead of having unary instruction include a 'let' in their class
body, add rs2val as a template parameter. Then we can use a let
in FPUnaryOp_r and FPUnaryOp_r_frm. This reduces the overall
verbosity of the FP files.
Reviewed By: achieveartificialintelligence
Differential Revision: https://reviews.llvm.org/D115537
This patch is one of the TODO of commit, 283879793d
We build the GenericTable for these opcodes, and also extend class RISCVOpcode, to store the names of opcodes. Then we call the parseInsnDirectiveOpcode to parse the opcode filed in .insn directive. We only allow users to write the recognized opcode names, or just write the immediate values in the 7 bits range.
Documentation: https://sourceware.org/binutils/docs-2.37/as/RISC_002dV_002dFormats.html
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D115224
This patch adds support for prologue and epilogue generation for
the z/OS target under the XPLINK64 ABI for functions with a stack
size of less than 1048576 bytes (huge stack frames).
Reviewed by: uweigand, Kai
Differential Revision: https://reviews.llvm.org/D114457
MachineOutliner may outline a "patchable-function-entry" function whose body has
a TargetOpcode::PATCHABLE_FUNCTION_ENTER MachineInstr. This is incorrect because
the special code sequence must stay unchanged to be used at run-time.
Avoid outlining PATCHABLE_FUNCTION_ENTER. While here, avoid outlining FENTRY_CALL too
(which doesn't reproduce currently) to allow phase ordering flexibility.
Fixes#52635
Reviewed By: paquette
Differential Revision: https://reviews.llvm.org/D115614
This patch implements a fix to recognize global symbols that represent
WebAssembly appropriately and generate the necessary .tabletype
directives.
Reviewed By: sbc100
Differential Revision: https://reviews.llvm.org/D115511
The reduction instructions only reads the first element. The
execution time for a splat may take longer with a larger VL.
We should use the smallest VL we can.
Reviewed By: frasercrmck, HsiangKai
Differential Revision: https://reviews.llvm.org/D115536
This change moves optimized callbacks from each .o file to compiler-rt.
Reviewed By: vitalybuka, morehouse
Differential Revision: https://reviews.llvm.org/D115396
Fixed ABI arguments are compute specific and should not be added to
graphics shaders or functions, so do not try to add them.
Differential Revision: https://reviews.llvm.org/D115344
1. Fixed costs inconsistency for llvm.fma.vXf16 instinsiscs.
2. Added tests for llvm.sadd.sat, llvm.ssub.sat, llvm.uadd.sat, llvm.usub.sat
intrisics since they have special processing in cost model.
3. Minor intrisics' costs tests updat and refinement.
Reviewed By: rampitec
Differential Revision: https://reviews.llvm.org/D115385
-(Za + Zm * Zn) != (-Za + Zm * (-Zn))
when the FMA produces a zero output (e.g. all zero inputs can produce -0
output)
Add a PatFrag to check presence of nsz on the fneg, add tests which
ensure the combine does not fire in the absense of nsz.
See https://reviews.llvm.org/D90901 for a similar discussion on X86.
Differential Revision: https://reviews.llvm.org/D109525
Fix overrides to use both ports. Update the uops counts + port usage based off the most recent llvm-exegesis captures (PR36895) and what Intel AoM / Agner reports as well.
Currently the superalign option only increases the alignment of
variables that are moved into the module.lds block. Change that to all LDS
variables. Also only increase the alignment once, instead of once per function.
Reviewed By: rampitec
Differential Revision: https://reviews.llvm.org/D115488
This is a very old copy+paste typo - none of these cvt ops have an immediate operand.
Noticed while trying to merge MMX instructions into some existing SSE instruction scheduler instregex patterns.
This is a very old copy+paste typo - none of these binops have an immediate operand.
Noticed while trying to merge MMX instructions into some existing SSE instruction scheduler instregex patterns.
At the moment these are identical to WriteShuffle256 (which we were using), but it should be WriteVPMOV256 to match the AVX2 instruction, plus it will help us remove some unnecessary overrides by tweaking the WriteVPMOV256 class.
Also, as D115547 shows, we still need to split off 128-bit extensions/truncations sched classes to remove some other overrides.
This change moves optimized callbacks from each .o file to compiler-rt.
Reviewed By: vitalybuka, morehouse
Differential Revision: https://reviews.llvm.org/D115396
By adding the register class and funct as template parameters we
can share the classes with all 3 extensions.
I've used "let SchedRW =" to avoid repeating scheduler classes on
multiple lines where we previously inherited from the Sched class.
A subsequent patch will add mayRaiseFPException and FRM dependencies.
Reducing the number of classes means less repeating for those changes.
This of course conflicts with the Zfinx patch D93298.
Reviewed By: achieveartificialintelligence
Differential Revision: https://reviews.llvm.org/D115469
We only used this to mark it as a reserved register. But that's not
important if we don't do anything else with it.
I think if we were ever to do anything with it, we would need to
model it as a super register of FRM and FFLAGS. But it might be
easier to reference both FRM and FFLAGS in implicit defs/uses
for anything we were to do with "fcsr".
Reviewed By: sepavloff
Differential Revision: https://reviews.llvm.org/D115455
When this pass was originally implemented, the fix pass was enabled
using a llvm command-line flag. This works fine, except in the case of
LTO, where the flag is not passed into the linker plugin in order to
enable the function pass in the LTO backend.
Now LTO exists, the expectation now is to use target features rather
than command-line arguments to control code generation, as this ensures
that different command-line arguments in different files are correctly
represented, and target-features always get to the LTO plugin as they
are encoded into LLVM IR.
The fall-out of this change is that the fix pass has to always be added
to the backend pass pipeline, so now it makes no changes if the function
does not have the right target feature to enable it. This should make a
minimal difference to compile time.
One advantage is it's now much easier to enable when compiling for a
Cortex-A53, as CPUs imply their own individual sets of target-features,
in a more fine-grained way. I haven't done this yet, but it is an
option, if the fix should be enabled in more places.
Existing tests of the user interface are unaffected, the changes are to
reflect that the argument is now turned into a target feature.
Reviewed By: tmatheson
Differential Revision: https://reviews.llvm.org/D114703
Adds x-ray support for hexagon to llvm codegen, clang driver,
compiler-rt libs.
Differential Revision: https://reviews.llvm.org/D113638
Reapplying this after 543a9ad7c4,
which fixes the leak introduced there.
While enabling vector superclasses with D109301,
the AV spills are converted into VGPR spills by
introducing appropriate copies. The whole thing
ended up adding two instructions per spill (a copy
+ vgpr spill pseudo) and caused an incorrect
liverange update during inline spiller.
This patch adds the pseudo instructions for all
AV spills from 32b to 1024b and handles them in
the way all other spills are lowered.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D115439
Originially there are two places that does parsing - `parseArchString` and
`parseFeatures`, each with its code on dependency check and implication.
This patch extracts common parts of the two as functions of `RISCVISAInfo`
and let them 2 use it.
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D112359
This results in a very minor improvement in most cases, generating
stores of xzr instead of moving zero to a vector register.
Differential Revision: https://reviews.llvm.org/D115479
D113096 solved the "undefined reference to xxx" issue by adding
constraint *m for the global var. But it has strong side effect due to
the symbol in the assembly being replaced with constraint variable.
This leads to some lowering fails. https://godbolt.org/z/h3nWoerPe
This patch fix the problem by use the constraint *m as place holder
rather than real constraint. It has negligible effect for the existing
code generation.
Reviewed By: skan
Differential Revision: https://reviews.llvm.org/D115225
There are two signatures of setSpecialOperandAttr in TargetInstrInfo.
One of them is only called from PPCInstrInfo which has an override
of it.
Remove it from TargetInstrInfo and make it a non-virtual method in
PPCInstrInfo.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D115404
D113805 improved handling of i32 divu/remu on RV64. The basic idea
from that can be extended to (mul (and X, C2), C1) where C2 is any
mask constant.
We can replace the and with an SLLI by shifting by the number of
leading zeros in C2 if we also shift C1 left by XLen - lzcnt(C1)
bits. This will give the full product XLen additional trailing zeros,
putting the result in the output of MULHU. If we can't use ANDI,
ZEXT.H, or ZEXT.W, this will avoid materializing C2 in a register.
The downside is it make take 1 additional instruction to create C1.
But since that's not on the critical path, it can hopefully be
interleaved with other operations.
The previous tablegen pattern is replaced by custom isel code.
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D115310
This patch adds on an overhead cost for gathers and scatters, which
is a rough estimate based on performance investigations I have
performed on SVE hardware for various micro-benchmarks.
Differential Revision: https://reviews.llvm.org/D115143
Krzysztof Parzyszek