Summary:
This patch enables the register allocator to spill/fill lists of 2, 3
and 4 SVE vectors registers to/from the stack. This is implemented with
pseudo instructions that get expanded to individual LDR_ZXI/STR_ZXI
instructions in AArch64ExpandPseudoInsts.
Patch by Sander de Smalen.
Reviewed By: efriedma
Differential Revision: https://reviews.llvm.org/D75988
This fixes a bug in `tryOptArithImmedIntegerCompare`.
It is unsafe to update the predicate on a MachineOperand when optimizing a
G_ICMP, because it may be used in more than one place.
For example, when we are optimizing G_SELECT, we allow compares which are used
in more than one G_SELECT. If we modify the G_ICMP, then we'll break one of
the G_SELECTs.
Since the compare is being produced to either
1) Select a G_ICMP
2) Fold a G_ICMP into an instruction when profitable
there's no reason to actually modify it. The change is local to the specific
compare.
Instead, pass a `CmpInst::Predicate` to `tryOptArithImmedIntegerCompare` which
can be modified by reference.
Differential Revision: https://reviews.llvm.org/D80585
Confusingly, these were unrelated and had different semantics. The
G_PTR_MASK instruction predates the llvm.ptrmask intrinsic, but has a
different format. G_PTR_MASK only allows clearing the low bits of a
pointer, and only a constant number of bits. The ptrmask intrinsic
allows an arbitrary mask. Replace G_PTR_MASK to match the intrinsic.
Only selects the cases that look like the old instruction. More work
is needed to select the general case. Also new legalization code is
still needed to deal with the case where the incoming mask size does
not match the pointer size, which has a specified behavior in the
langref.
Add the remaining cast instruction opcodes to the base implementation
of getUserCost and directly return the result. This allows
getInstructionThroughput to return getUserCost for the casts. This
has required changes to PPC and SystemZ because they implement
getUserCost and/or getCastInstrCost with adjustments for vector
operations. Adjusts have also been made in the remaining backends
that implement the method so that they still produce a cost of zero
or one for cost kinds other than throughput.
Differential Revision: https://reviews.llvm.org/D79848
Although writing to wzr/xzr is correct since we don't care about the result
of the sub, only the flags, doing so causes tail merge blocks to fail.
Writing to an unused virtual register instead allows the optimization to fire,
improving performance significantly on 256.bzip2.
Differential Revision: https://reviews.llvm.org/D80460
(This patch is by Jessica, I'm just committing it on her behalf because I need
a post-legalizer combiner for something else).
This supersedes D77250, which did equivalent work in the selector. This can be
done pre-legalization or post-legalization. Post-legalization is more likely to
hit, since G_IMPLICIT_DEFs tend to appear during legalization. There's no reason
to not do it pre-legalization though-- if it can be caught earlier, great.
(I also think that it might be worth reimplementing D78769 using a
target-specific post-legalization combine too after thinking about it for a
while.)
Differential Revision: https://reviews.llvm.org/D78852
Summary:
This patch fixes a problem when pmull2 instruction is not
generated for vmull_high_p64 intrinsic.
ISel has a pattern for int_aarch64_neon_pmull64 intrinsic to generate
PMULL2 instruction. That pattern assumes that extraction operations
are located in the same basic block. We need to sink them
if they are not. Handle operands of int_aarch64_neon_pmull64
into AArch64TargetLowering::shouldSinkOperands.
Reviewed by: efriedma
Differential Revision: https://reviews.llvm.org/D80320
The set of patterns for unpredicated load/store was incomplete: it only
included non-extending stores. Fill out the remaining patterns for
extending stores, and add the corresponding support to frame offset
lowering.
Differential Revision: https://reviews.llvm.org/D80349
The offsets were wrong. The result is now the same as what the compiler
would generate for a function that spills lr normally.
Differential Revision: https://reviews.llvm.org/D80238
If we don't know anything about the alignment of a pointer, Align(1) is
still correct: all pointers are at least 1-byte aligned.
Included in this patch is a bugfix for an issue discovered during this
cleanup: pointers with "dereferenceable" attributes/metadata were
assumed to be aligned according to the type of the pointer. This
wasn't intentional, as far as I can tell, so Loads.cpp was fixed to
stop making this assumption. Frontends may need to be updated. I
updated clang's handling of C++ references, and added a release note for
this.
Differential Revision: https://reviews.llvm.org/D80072
With the new SVE stack layout, we now need to provide a Darwin variant
for all the calling conventions based on the main AAPCS CSR save order.
This also changes APCS_SwiftError to have a Darwin and a non-Darwin
version, assuming it could be used on other platforms these days, and
restricts the AArch64_CXX_TLS calling convention to Darwin.
Differential Revision: https://reviews.llvm.org/D73805
These should legalize like undefs and select into copies.
The ll test is copied from the x86 test, minus the half fp case because
we don't currently support that.
Summary:
This patch adds IR intrinsics for the mnemonics USDOT and SUDOT of the
8.6 extension of Armv8-a.
Reviewers: sdesmalen, efriedma, david-arm
Subscribers: tschuett, kristof.beyls, hiraditya, rkruppe, psnobl, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D79876
Along the lines of D77454 and D79968. Unlike loads and stores, the
default alignment is getPrefTypeAlign, to match the existing handling in
various places, including SelectionDAG and InstCombine.
Differential Revision: https://reviews.llvm.org/D80044
This is the first checkin to support Marvell ThunderX3T110.
Initial definition of the micro-ops of the instructions in ThunderX3T110
is included.
Differential Revision: https://reviews.llvm.org/D78129
Currently the AsmPrinter cannot emit some floating point constant
expressions in global initializers. Avoid generating them.
Reviewers: dmgreen, t.p.northover, arsenm, efriedma, Gerolf
Reviewed By: efriedma
Differential Revision: https://reviews.llvm.org/D79865
Summary:
The optimization has been refactored to fix certain bugs and
limitations. The condition for lowering to S[LR]I has been changed
to reflect the manual pseudocode description of SLI and SRI operation.
The optimization can now handle more cases of operand type and order.
Subscribers: kristof.beyls, hiraditya, danielkiss, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D79233
This covers forms involving "CPY (immediate, zeroing)".
This doesn't handle the case where the operands are reversed, and the
condition is freely invertible. Not sure how to handle that. Maybe a
DAGCombine.
Differential Revision: https://reviews.llvm.org/D79598
This fixes a couple verifier failures on this bot:
http://green.lab.llvm.org/green/job/test-suite-verify-machineinstrs-aarch64-globalisel-O0-g/
The failures show up in eeprof-1.c and pr17377.c in the GCC C Torture Suite.
Specifically:
*** Bad machine code: MBB has allocatable live-in, but isn't entry or landing-pad. ***
- function: foo
- basic block: %bb.3 if.end (0x7fac7106dfc8)
- p. register: $lr
and
*** Bad machine code: Using an undefined physical register ***
- function: f
- basic block: %bb.1 entry (0x7f8941092588)
- instruction: %18:gpr64 = COPY $lr
- operand 1: $lr
Unlike SDAG, we were setting LR as a live in to the block containing the
returnaddress.
Also, this ensures that we don't add LR as a livein to the entry block twice.
In MachineBasicBlock.h there's a comment saying
"Note that it is an error to add the same register to the same set more than
once unless the intention is to call sortUniqueLiveIns after all registers are
added."
so it's probably good to avoid adding LR twice.
Surprisingly the verifier doesn't complain about that. Maybe it should.
Differential Revision: https://reviews.llvm.org/D79657
This patch stores the alignment for ConstantPoolSDNode as an
Align and updates the getConstantPool interface to take a MaybeAlign.
Removing getAlignment() will be done as a follow up.
Differential Revision: https://reviews.llvm.org/D79436
Now using patterns, since there's a single-instruction lowering. (We
could convert to VSELECT and pattern-match that, but there doesn't seem
to be much point.)
I think this might be the first instruction to use nested multiclasses
this way? It seems like a good way to reduce duplication between
different integer widths. Let me know if it seems like an improvement.
Also, while I'm here, fix the return type of SETCC so we don't try to
merge a sign-extend with a SETCC.
Differential Revision: https://reviews.llvm.org/D79193
Since G_ICMP can be selected to a SUBS, we can fold shifts into such compares.
E.g.
```
cmp w1, w0, lsl #3
cmp w1, w0, lsr #3
cmp w1, w0, asr #3
```
This is done the same way as for adds and subtracts, using
`selectShiftedRegister`.
This gives some minor code size savings on CTMark.
https://reviews.llvm.org/D79365
Make the kind of cost explicit throughout the cost model which,
apart from making the cost clear, will allow the generic parts to
calculate better costs. It will also allow some backends to
approximate and correlate the different costs if they wish. Another
benefit is that it will also help simplify the cost model around
immediate and intrinsic costs, where we currently have multiple APIs.
RFC thread:
http://lists.llvm.org/pipermail/llvm-dev/2020-April/141263.html
Differential Revision: https://reviews.llvm.org/D79002
Summary:
This patch adds AArch64ISD nodes for [S|U]MIN_PRED
and [S|U]MAX_PRED, and lowers both SVE intrinsics and
IR operations for min and max to these nodes.
There are two forms of these instructions for SVE: a predicated
form and an immediate (unpredicated) form. The patterns
which existed for the latter have been updated to match a
predicated node with an immediate and map this
to the immediate instruction.
Reviewers: sdesmalen, efriedma, dancgr, rengolin
Reviewed By: efriedma
Subscribers: huihuiz, tschuett, kristof.beyls, hiraditya, rkruppe, psnobl, cfe-commits, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D79087
Summary:
This patch implements custom floating-point reduction ISD nodes that
have vector results, which are used to lower the following intrinsics:
* llvm.aarch64.sve.fadda
* llvm.aarch64.sve.faddv
* llvm.aarch64.sve.fmaxv
* llvm.aarch64.sve.fmaxnmv
* llvm.aarch64.sve.fminv
* llvm.aarch64.sve.fminnmv
SVE reduction instructions keep their result within a vector register,
with all other bits set to zero.
Changes in this patch were implemented by Paul Walker and Sander de
Smalen.
Reviewers: sdesmalen, efriedma, rengolin
Reviewed By: efriedma
Differential Revision: https://reviews.llvm.org/D78723
Summary:
This patch tries to ensure that we do something sensible when
generating code for the ISD::INSERT_VECTOR_ELT DAG node when operating
on scalable vectors. Previously we always returned 'undef' when
inserting an element into an out-of-bounds lane index, whereas now
we only do this for fixed length vectors. For scalable vectors it
is assumed that the backend will do the right thing in the same way
that we have to deal with variable lane indices.
In this patch I have permitted a few basic combinations for scalable
vector types where it makes sense, but in general avoided most cases
for now as they currently require the use of BUILD_VECTOR nodes.
This patch includes tests for all scalable vector types when inserting
into lane 0, but I've only included one or two vector types for other
cases such as variable lane inserts.
Differential Revision: https://reviews.llvm.org/D78992
Summary:
AArch64's system register ERXTS_EL1 is present in the backend as a
component of the Arm Reliability, Availability and Serviceability (RAS)
extension. However, it has been removed from the specification before
its final release.
This patch removes the register.
Reviewers: SjoerdMeijer, DavidSpickett
Reviewed By: DavidSpickett
Subscribers: DavidSpickett, kristof.beyls, hiraditya, danielkiss, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D79007
Generalize the 16-bit FPR to 32-bit GPR logic to work for all cases where
destination size is bigger than source size.
Also fixed CheckCopy() always returning true instead of the result of
isValidCopy().
Differential Revision: https://reviews.llvm.org/D77530
Patch by tambre (Raul Tambre)
Similar to code in `getAArch64Cmp` in AArch64ISelLowering.
When we get a compare against a constant, sometimes, that constant isn't valid
for selecting an immediate form.
However, sometimes, you can get a valid constant by adding 1 or subtracting 1,
and updating the condition code.
This implements the following transformations when valid:
- x slt c => x sle c - 1
- x sge c => x sgt c - 1
- x ult c => x ule c - 1
- x uge c => x ugt c - 1
- x sle c => x slt c + 1
- x sgt c => s sge c + 1
- x ule c => x ult c + 1
- x ugt c => s uge c + 1
Valid meaning the constant doesn't wrap around when we fudge it, and the result
gives us a compare which can be selected into an immediate form.
This also moves `getImmedFromMO` higher up in the file so we can use it.
Differential Revision: https://reviews.llvm.org/D78769
Add support for reserving LR in:
* the driver through `-ffixed-x30`
* cc1 through `-target-feature +reserve-x30`
* the backend through `-mattr=+reserve-x30`
* a subtarget feature `reserve-x30`
the same way we're doing for the other registers.
Cullen Rhodes