Commit Graph

526 Commits

Author SHA1 Message Date
Matt Arsenault 692bd5ec2f R600: Handle fnearbyint
The difference from rint isn't really relevant here,
so treat them as equivalent. OpenCL doesn't have nearbyint,
so this is sort of pointless other than for completeness.

llvm-svn: 211229
2014-06-18 22:03:45 +00:00
Marek Olsak 51b8e7b2e7 R600/SI: add gather4 and getlod intrinsics (v3)
This contains all the previous patches + getlod support on top of it.
It doesn't use SDNodes anymore, so it's quite small.
It also adds v16i8 to SReg_128, which is used for the sampler descriptor.

Reviewed-by: Tom Stellard
llvm-svn: 211228
2014-06-18 22:00:29 +00:00
Jan Vesely 85f0dbce5c R600: Expand vector fceil
Move fp64 fceil tests to fceil64.ll

v2: rebase

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
llvm-svn: 211194
2014-06-18 17:57:29 +00:00
Matt Arsenault 43160e7af2 R600/SI: Add intrinsics for brev instructions
llvm-svn: 211187
2014-06-18 17:13:57 +00:00
Matt Arsenault dbc9aae1fb R600/SI: Prettier operand printing for 64-bit ops.
Copy what is done for 32-bit already so the order is about the same.

llvm-svn: 211186
2014-06-18 17:13:51 +00:00
Matt Arsenault 4601093267 R600: Implement f64 ftrunc, ffloor and fceil.
CI has instructions for these, so this fixes them for older hardware.

llvm-svn: 211183
2014-06-18 17:05:30 +00:00
Matt Arsenault e8208ec95b R600: Custom lower f64 frint for pre-CI
llvm-svn: 211182
2014-06-18 17:05:26 +00:00
Jan Vesely ecf5133a2b R600: Implement 64bit SRA
v2: Use capitalized variable name

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
llvm-svn: 211159
2014-06-18 12:27:17 +00:00
Jan Vesely 900ff2e74b R600: Implement 64bit SRL
v2: use C++ style comment

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
llvm-svn: 211158
2014-06-18 12:27:15 +00:00
Jan Vesely 25f362766e R600: Implement 64bit SHL
v2: Use c++ style comment

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
llvm-svn: 211157
2014-06-18 12:27:13 +00:00
Matt Arsenault 295b86e81d R600/SI: Match cttz_zero_undef
llvm-svn: 211116
2014-06-17 17:36:27 +00:00
Matt Arsenault 8579601050 R600/SI: Match ctlz_zero_undef
llvm-svn: 211115
2014-06-17 17:36:24 +00:00
Tom Stellard 880a80ad07 R600: Use LDS and vectors for private memory
llvm-svn: 211110
2014-06-17 16:53:14 +00:00
Tom Stellard aad4659470 SelectionDAG: Expand i64 = FP_TO_SINT i32
llvm-svn: 211108
2014-06-17 16:53:07 +00:00
Matt Arsenault f302c941d8 R600: Add a rotr testcase I forgot to add
llvm-svn: 211002
2014-06-15 21:09:00 +00:00
Matt Arsenault 717c1d0319 R600: Remove a few more things from AMDILISelLowering
Try to keep all the setOperationActions for integer ops
together.

llvm-svn: 211001
2014-06-15 21:08:58 +00:00
Matt Arsenault b5dff9ab50 R600: Fix assert on vector sdiv
llvm-svn: 211000
2014-06-15 21:08:54 +00:00
Matt Arsenault cf9a9a148e R600: Report that integer division is expensive.
Divides by weird constants now emit much better code.

llvm-svn: 210995
2014-06-15 19:48:16 +00:00
NAKAMURA Takumi e876f5b61e Don't expect tests always crashing. Add "REQUIRES:asserts".
llvm-svn: 210983
2014-06-15 01:01:11 +00:00
Matt Arsenault 5eb038a9f2 R600: Add failing testcases.
These are reduced from assert in the
OpenCV CvtColor8u.BGR5652GRAY test.

llvm-svn: 210969
2014-06-14 04:26:09 +00:00
Matt Arsenault e682a19a1c R600: Fix asserts related to constant initializers
This would assert if a constant address space was extern
and therefore didn't have an initializer. If the initializer
was undef, it would hit the unreachable unhandled initializer case.

An extern global should never really occur since we don't have
machine linking, but bugpoint likes to remove initializers.

llvm-svn: 210967
2014-06-14 04:26:05 +00:00
Tim Northover 420a216817 IR: add "cmpxchg weak" variant to support permitted failure.
This commit adds a weak variant of the cmpxchg operation, as described
in C++11. A cmpxchg instruction with this modifier is permitted to
fail to store, even if the comparison indicated it should.

As a result, cmpxchg instructions must return a flag indicating
success in addition to their original iN value loaded. Thus, for
uniformity *all* cmpxchg instructions now return "{ iN, i1 }". The
second flag is 1 when the store succeeded.

At the DAG level, a new ATOMIC_CMP_SWAP_WITH_SUCCESS node has been
added as the natural representation for the new cmpxchg instructions.
It is a strong cmpxchg.

By default this gets Expanded to the existing ATOMIC_CMP_SWAP during
Legalization, so existing backends should see no change in behaviour.
If they wish to deal with the enhanced node instead, they can call
setOperationAction on it. Beware: as a node with 2 results, it cannot
be selected from TableGen.

Currently, no use is made of the extra information provided in this
patch. Test updates are almost entirely adapting the input IR to the
new scheme.

Summary for out of tree users:
------------------------------

+ Legacy Bitcode files are upgraded during read.
+ Legacy assembly IR files will be invalid.
+ Front-ends must adapt to different type for "cmpxchg".
+ Backends should be unaffected by default.

llvm-svn: 210903
2014-06-13 14:24:07 +00:00
Matt Arsenault 825fb0b094 R600/SI: Fix selection error on i64 rotl / rotr.
Evergreen is still broken due to missing shl_parts.

llvm-svn: 210885
2014-06-13 04:00:30 +00:00
Matt Arsenault 5d47d4ac7e R600: Mostly remove remaining AMDIL intrinsics.
Delete all unused ones, and add new AMDGPU named intrinsics for
the ones that are. Handle the old AMDIL names for comptability (although
remove their GCCBuiltin names) and add tests since there weren't any
for these before.

llvm-svn: 210827
2014-06-12 21:15:44 +00:00
Tom Stellard 7783b0adf4 Revert "SelectionDAG: Enable (and (setcc x), (setcc y)) -> (setcc (and x, y)) for vectors"
This reverts commit r210540, adds a testcase for the regression it
caused, and marks the R600 test it was supposed to fix as XFAIL.

llvm-svn: 210792
2014-06-12 16:04:47 +00:00
Matt Arsenault 2c81994f92 R600/SI: Use a register set to -1 for data0 on ds_inc*/ds_dec*
There is not such thing as a 0-data ds instruction, and the data
operand needs to be a vgpr set to something meaningful.

llvm-svn: 210756
2014-06-12 08:21:54 +00:00
Matt Arsenault 2acc7a4570 R600/SI: Fix bitcast between v2i32 and f64
This is the same problem fixed in r210664 for more types.

The test passes without this fix. For some reason
I'm only hitting this when creating selects lowered
to v2i32 selects.

llvm-svn: 210692
2014-06-11 19:31:13 +00:00
Matt Arsenault caa0ec2851 R600/SI: Add common 64-bit LDS atomics
llvm-svn: 210680
2014-06-11 18:08:54 +00:00
Matt Arsenault c793e1d9dc R600/SI: Add 32-bit LDS atomic cmpxchg
llvm-svn: 210678
2014-06-11 18:08:48 +00:00
Matt Arsenault 9e874541ac R600/SI: Use LDS atomic inc / dec
llvm-svn: 210677
2014-06-11 18:08:45 +00:00
Matt Arsenault 0e69e8128c R600/SI: Add other LDS atomic operations
llvm-svn: 210676
2014-06-11 18:08:42 +00:00
Matt Arsenault 7ddcd83d49 R600/SI: Fix backwards names for local atomic instructions.
The manual lists them as *_RTN_U32, not *_U32_RTN, which is more
consistent with how every other sized instruction is named.

llvm-svn: 210674
2014-06-11 18:08:37 +00:00
Matt Arsenault 725741004c R600/SI: Refactor local atomics.
Use patterns that will also match the immediate offset to
match the normal read / writes.

llvm-svn: 210673
2014-06-11 18:08:34 +00:00
Matt Arsenault 364a6747aa R600/SI: Use v_cvt_f32_ubyte* instructions
This eliminates extra extract instructions when loading an i8 vector to
a float vector.

llvm-svn: 210666
2014-06-11 17:50:44 +00:00
Matt Arsenault 064c206d23 R600/SI: Fix selection failure on scalar_to_vector
There seem to be only 2 places that produce these,
and it's kind of tricky to hit them.

Also fixes failure to bitcast between i64 and v2f32,
although this for some reason wasn't actually broken in the
simple bitcast testcase, but did in the scalar_to_vector one.

llvm-svn: 210664
2014-06-11 17:40:32 +00:00
Matt Arsenault 6042506b5c R600: Use BCNT_INT for evergreen
llvm-svn: 210569
2014-06-10 19:18:28 +00:00
Matt Arsenault 8333e4378e R600/SI: Implement i64 ctpop
llvm-svn: 210568
2014-06-10 19:18:24 +00:00
Matt Arsenault b5b5110b5c R600/SI: Use bcnt instruction for ctpop
llvm-svn: 210567
2014-06-10 19:18:21 +00:00
Matt Arsenault 6e43965fbc R600: Handle fcopysign
llvm-svn: 210564
2014-06-10 19:00:20 +00:00
Matt Arsenault b2cbf799d1 R600/SI: Handle sign_extend and zero_extend to i64 with patterns.
llvm-svn: 210563
2014-06-10 18:54:59 +00:00
Tom Stellard 3ca1bfc728 SelectionDAG: Expand SELECT_CC to SELECT + SETCC
This consolidates code from the Hexagon, R600, and XCore targets.

No functionality change intended.

llvm-svn: 210539
2014-06-10 16:01:22 +00:00
Alp Toker d3d017cf00 Reduce verbiage of lit.local.cfg files
We can just split targets_to_build in one place and make it immutable.

llvm-svn: 210496
2014-06-09 22:42:55 +00:00
Matt Arsenault 689f325099 R600/SI: Keep 64-bit not on SALU
llvm-svn: 210476
2014-06-09 16:36:31 +00:00
Matt Arsenault 13ccc8f1bc R600: Fix selection failure for vector bswap
llvm-svn: 210475
2014-06-09 16:20:25 +00:00
Matt Arsenault 284ae0899b R600: Add more and testcases
llvm-svn: 210453
2014-06-09 08:36:53 +00:00
Rafael Espindola 42a4c9f9e0 Allow aliases to be unnamed_addr.
Alias with unnamed_addr were in a strange state. It is stored in GlobalValue,
the language reference talks about "unnamed_addr aliases" but the verifier
was rejecting them.

It seems natural to allow unnamed_addr in aliases:

* It is a property of how it is accessed, not of the data itself.
* It is perfectly possible to write code that depends on the address
of an alias.

This patch then makes unname_addr legal for aliases. One side effect is that
the syntax changes for a corner case: In globals, unnamed_addr is now printed
before the address space.

llvm-svn: 210302
2014-06-06 01:20:28 +00:00
Matt Arsenault 3e332a4862 R600: Fix test. Using wrong check prefix.
llvm-svn: 210244
2014-06-05 08:00:36 +00:00
Matt Arsenault 151304691c R600/SI: Match rsq instructions
llvm-svn: 210226
2014-06-05 00:15:55 +00:00
Matt Arsenault aeca2fa9f7 R600/SI: Fix [s|u]int_to_fp for i1
llvm-svn: 209971
2014-05-31 06:47:42 +00:00
Matt Arsenault 05e96f4444 R600: Try to convert BFE back to standard bit ops when possible.
This allows existing DAG combines to work on them, and then
we can re-match to BFE if necessary during instruction selection.

llvm-svn: 209462
2014-05-22 18:09:12 +00:00
Matt Arsenault 5565f65e13 R600: Add dag combine for BFE
llvm-svn: 209461
2014-05-22 18:09:07 +00:00
Matt Arsenault bf8694d36d R600: Implement ComputeNumSignBitsForTargetNode for BFE
llvm-svn: 209460
2014-05-22 18:09:03 +00:00
Matt Arsenault 493c5f1bc4 R600: Expand mul24 for GPUs without it
llvm-svn: 209458
2014-05-22 18:00:24 +00:00
Matt Arsenault f15a05623e R600: Expand mad24 for GPUs without it
llvm-svn: 209457
2014-05-22 18:00:20 +00:00
Matt Arsenault eb260206c3 R600: Add intrinsics for mad24
llvm-svn: 209456
2014-05-22 18:00:15 +00:00
Matt Arsenault c3a73c3087 R600/SI: Match fp_to_uint / uint_to_fp for f64
llvm-svn: 209388
2014-05-22 03:20:30 +00:00
Matt Arsenault 6a57fd8b47 R600: Partially fix constant initializers for structs and vectors.
This should extend the current workaround to work with structs
that only contain legal, scalar types.

llvm-svn: 209331
2014-05-21 22:42:42 +00:00
Matt Arsenault 310f761a42 R600: Add failing testcases for constant initializers.
Constant initializers involving illegal types hit an assertion.

Patch by: Jan Vesely <jan.vesely@rutgers.edu>

llvm-svn: 209330
2014-05-21 22:42:38 +00:00
Tom Stellard f719ee9e76 R600/SI: Promote f32 SELECT to i32
llvm-svn: 209024
2014-05-16 20:56:41 +00:00
Tom Stellard 73b98ed8cf R600/SI: Only use SALU instructions for 64-bit add in a block of CF depth 0
llvm-svn: 208886
2014-05-15 14:41:54 +00:00
Tom Stellard 365a2b49f2 R600/SI: Use VALU instructions for i1 ops
llvm-svn: 208885
2014-05-15 14:41:50 +00:00
Jay Foad a0653a3e6c Rename ComputeMaskedBits to computeKnownBits. "Masked" has been
inappropriate since it lost its Mask parameter in r154011.

llvm-svn: 208811
2014-05-14 21:14:37 +00:00
Matt Arsenault 4b0402e317 R600/SI: Try to fix BFE operands when moving to VALU
This was broken by r208479

llvm-svn: 208740
2014-05-13 23:45:50 +00:00
Matt Arsenault 62b1737081 R600: Add mul24 intrinsics
llvm-svn: 208604
2014-05-12 17:49:57 +00:00
Matt Arsenault 2adca6090f Make SimplifyDemandedBits understand BUILD_PAIR
llvm-svn: 208598
2014-05-12 17:14:48 +00:00
Vincent Lejeune 29c0c210fc R600/SI: Fold fabs/fneg into src input modifier
llvm-svn: 208480
2014-05-10 19:18:39 +00:00
Vincent Lejeune 94af31fbe8 R600/SI: Prettier display of input modifiers
llvm-svn: 208479
2014-05-10 19:18:33 +00:00
Tom Stellard 4c00b52e1a R600/SI: Teach SIInstrInfo::moveToVALU() how to move S_LOAD_*_IMM instructions
llvm-svn: 208432
2014-05-09 16:42:22 +00:00
Tom Stellard d6cb8e8efd R600/SI: Fix SMRD pattern for offsets > 32 bits
We were dropping the high bits of 64-bit immediate offsets.

llvm-svn: 208431
2014-05-09 16:42:21 +00:00
Tom Stellard a2acad785a R600: Expand i64 SELECT_CC
llvm-svn: 208430
2014-05-09 16:42:19 +00:00
Tom Stellard afa8b532b1 R600: Move MIN/MAX matching from LowerOperation() to PerformDAGCombine()
llvm-svn: 208429
2014-05-09 16:42:16 +00:00
Tom Stellard 45b3dcd35b R600: Expand i64 ISD:SUB
llvm-svn: 208005
2014-05-05 21:47:15 +00:00
Tom Stellard 3dbf1f8df0 R600: Expand vector sin and cos.
v2: move code to AMDGPUISelLowering.cpp
    squash with tests (both EG and SI)

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
llvm-svn: 207845
2014-05-02 15:41:47 +00:00
Tom Stellard 605e116e8e R600: Expand TruncStore i64 -> {i16,i8}
llvm-svn: 207844
2014-05-02 15:41:46 +00:00
Matt Arsenault 06028dd7be R600/SI: Fix verifier error with pseudo store instructions.
Use i32 instead of specifying SReg_32. When this is
the pseudo INDIRECT_BASE_ADDR, this would give a bogus
verifier error.

llvm-svn: 207770
2014-05-01 16:37:52 +00:00
Tom Stellard 1bd80725b3 R600/SI: Use VALU instructions for copying i1 values
We can't use SALU instructions for this since they ignore the EXEC mask
and are always executed.

This fixes several OpenCV tests.

llvm-svn: 207661
2014-04-30 15:31:33 +00:00
Tom Stellard 0c354f25c9 R600/SI: Teach moveToVALU how to handle some SMRD instructions
llvm-svn: 207660
2014-04-30 15:31:29 +00:00
Tom Stellard 919bb6b83f R600/SI: Custom lower SI_IF and SI_ELSE to avoid machine verifier errors
SI_IF and SI_ELSE are terminators which also produce a value.  For
these instructions ISel always inserts a COPY to move their value
to another basic block.  This COPY ends up between SI_(IF|ELSE)
and the S_BRANCH* instruction at the end of the block.

This breaks MachineBasicBlock::getFirstTerminator() and also the
machine verifier which assumes that terminators are grouped together at
the end of blocks.

To solve this we coalesce the copy away right after ISel to make sure
there are no instructions in between terminators at the end of blocks.

llvm-svn: 207591
2014-04-29 23:12:53 +00:00
Tom Stellard 58ac7440e6 R600/SI: Only select SALU instructions in the entry or exit block
SALU instructions ignore control flow, so it is not always safe to use
them within branches.  This is a partial solution to this problem
until we can come up with something better.

llvm-svn: 207590
2014-04-29 23:12:48 +00:00
Tom Stellard 676f571999 R600: optimize the UDIVREM 64 algorithm
This is a squash of several optimization commits:
 - calculate DIV_Lo and DIV_Hi separately
 - use BFE_U32 if we are operating on 32bit values
 - use precomputed constants instead of shifting in UDVIREM
 - skip the first 32 iterations of udivrem

v2: Check whether BFE is supported before using it

Patch by: Jan Vesely

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
Reviewed-by: Tom Stellard <thomas.stellard@amd.com>
llvm-svn: 207589
2014-04-29 23:12:46 +00:00
Matt Arsenault 4c6ab696e2 R600: Add a test that used to be broken that I forgot to add
llvm-svn: 207017
2014-04-23 19:45:05 +00:00
Matt Arsenault 16353871c3 R600: Emit error instead of unreachable on function call
llvm-svn: 206904
2014-04-22 16:42:00 +00:00
Matt Arsenault 5dbd5db518 R600: Make sign_extend_inreg legal.
Don't know why I didn't just do this in the first place.

llvm-svn: 206862
2014-04-22 03:49:30 +00:00
Matt Arsenault 78b8670aac R600/SI: Try to use scalar BFE.
Use scalar BFE with constant shift and offset when possible.
This is complicated by the fact that the scalar version packs
the two operands of the vector version into one.

llvm-svn: 206558
2014-04-18 05:19:26 +00:00
Matt Arsenault 27cc958dff R600/SI: Match sign_extend_inreg to s_sext_i32_i8 and s_sext_i32_i16
llvm-svn: 206547
2014-04-18 01:53:18 +00:00
Tom Stellard 1aa6cb4d88 R600/SI: Use SReg_64 instead of VSrc_64 when selecting BUILD_PAIR
llvm-svn: 206541
2014-04-18 00:36:21 +00:00
Tom Stellard 868fd92e54 R600/SI: Stop using i128 as the resource descriptor type
Having i128 as a legal type complicates the legalization phase.  v4i32
is already a legal type, so we will use that instead.

This fixes several piglit tests.

llvm-svn: 206500
2014-04-17 21:00:11 +00:00
Matt Arsenault a90d22fad5 R600/SI: f64 frint is legal on CI
llvm-svn: 206475
2014-04-17 17:06:37 +00:00
Matt Arsenault 51df0c1965 R600/SI: Fix zext from i1 to i64
llvm-svn: 206437
2014-04-17 02:03:08 +00:00
Matt Arsenault 4ef2588b65 R600: Extend r600 sign_extend_inreg tests for EG
Patch by: Jan Vesely <jan.vesely@rutgers.edu>

llvm-svn: 206349
2014-04-16 01:41:34 +00:00
Matt Arsenault 4d7d38333b R600/SI: Print more immediates in hex format
Print in decimal for inline immediates, and hex otherwise. Use hex
always for offsets in addressing offsets.

This approximately matches what the shader compiler does.

llvm-svn: 206335
2014-04-15 22:32:49 +00:00
Matt Arsenault 470acd81a8 R600/SI: Fix loads of i1
llvm-svn: 206330
2014-04-15 22:28:39 +00:00
Tom Stellard a1a5d9aa2e SelectionDAG: Use helper function to improve legalization of ISD::MUL
The TargetLowering::expandMUL() helper contains lowering code extracted
from the DAGTypeLegalizer and allows the SelectionDAGLegalizer to expand more
ISD::MUL patterns without having to use a library call.

llvm-svn: 206037
2014-04-11 16:12:01 +00:00
Matt Arsenault 2c33562cd6 R600/SI: Match not instruction.
llvm-svn: 205837
2014-04-09 07:16:16 +00:00
Tom Stellard 204e61bbdf R600/SI: Handle INSERT_SUBREG in SIFixSGPRCopies
llvm-svn: 205732
2014-04-07 19:45:45 +00:00
Tom Stellard 50122a5890 R600: Match 24-bit arithmetic patterns in a Target DAGCombine
Moving these patterns from TableGen files to PerformDAGCombine()
should allow us to generate better code by eliminating unnecessary
shifts and extensions earlier.

This also fixes a bug where the MAD pattern was calling
SimplifyDemandedBits with a 24-bit mask on the first operand
even when the full pattern wasn't being matched.  This occasionally
resulted in some instructions being incorrectly deleted from the
program.

v2:
  - Fix bug with 64-bit mul

llvm-svn: 205731
2014-04-07 19:45:41 +00:00
Tom Stellard a0150cb6a9 R600: Correct opcode for BFE_INT
Acording to AMD documentation, the correct opcode for
BFE_INT is 0x5, not 0x4

Fixes Arithm/Absdiff.Mat/3 OpenCV test

Patch by: Bruno Jiménez

llvm-svn: 205562
2014-04-03 20:19:29 +00:00
Tom Stellard 7ed0b5235a R600/SI: Lower 64-bit immediates using REG_SEQUENCE
llvm-svn: 205561
2014-04-03 20:19:27 +00:00
Tom Stellard 36a031870b TargetLibraryInfo: Disable memcpy and memset on R600
There are no implementations of these for R600.

llvm-svn: 205455
2014-04-02 19:53:29 +00:00
Matt Arsenault 553751b9bc Fix missing RUN line in test
llvm-svn: 205341
2014-04-01 18:34:13 +00:00