maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/test/CodeGen/X86/atomic-unordered.ll

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

2754 lines
85 KiB
LLVM
Raw Normal View History

[Test] Add codegen tests for unordered and monotonic integer operations llvm-svn: 353266
2019-02-06 11:19:04 +08:00
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
[SelectionDAG] Enable lowering unordered atomics loads w/LoadSDNode (and stores w/StoreSDNode) by default Enable the new SelectionDAG representation for unordered loads and stores introduced in r371441 by default. As a reminder, the new lowering changes the representation of an unordered atomic load from an AtomicSDNode - which is essentially a black box which gets passed through without combines messing with it - to a LoadSDNode w/a atomic marker on the MMO. The later parallels the way we handle volatiles, and I've audited the code to ensure that every location which checks one checks the other. This has been fairly heavily fuzzed, and I examined diffs in a reasonable large corpus of assembly by hand, so I'm reasonable sure this is correct for the common case. Late in the review for this, it was discovered that I hadn't correctly handled cases which could be legalized into CAS operations. This points out that there's a strong bias in the IR of the frontend I'm working with towards only legal atomics. If there are problems with this patch, the most likely area will be legalization. Differential Revision: https://reviews.llvm.org/D69219
2019-10-30 02:07:36 +08:00
; RUN: llc -O0 < %s -mtriple=x86_64-linux-generic -verify-machineinstrs -mcpu=skylake -x86-experimental-unordered-atomic-isel=0 | FileCheck --check-prefixes=CHECK,CHECK-O0,CHECK-O0-CUR %s
; RUN: llc -O3 < %s -mtriple=x86_64-linux-generic -verify-machineinstrs -mcpu=skylake -x86-experimental-unordered-atomic-isel=0 | FileCheck --check-prefixes=CHECK,CHECK-O3,CHECK-O3-CUR %s
; RUN: llc -O0 < %s -mtriple=x86_64-linux-generic -verify-machineinstrs -mcpu=skylake -x86-experimental-unordered-atomic-isel=1 | FileCheck --check-prefixes=CHECK,CHECK-O0,CHECK-O0-EX %s
; RUN: llc -O3 < %s -mtriple=x86_64-linux-generic -verify-machineinstrs -mcpu=skylake -x86-experimental-unordered-atomic-isel=1 | FileCheck --check-prefixes=CHECK,CHECK-O3,CHECK-O3-EX %s
[Test] Add codegen tests for unordered and monotonic integer operations llvm-svn: 353266
2019-02-06 11:19:04 +08:00
define i8 @load_i8(i8* %ptr) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: load_i8:
; CHECK: # %bb.0:
; CHECK-NEXT: movb (%rdi), %al
; CHECK-NEXT: retq
[Test] Add codegen tests for unordered and monotonic integer operations llvm-svn: 353266
2019-02-06 11:19:04 +08:00
%v = load atomic i8, i8* %ptr unordered, align 1
ret i8 %v
}
define void @store_i8(i8* %ptr, i8 %v) {
; CHECK-O0-LABEL: store_i8:
; CHECK-O0: # %bb.0:
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: # kill: def $sil killed $sil killed $esi
; CHECK-O0-NEXT: movb %sil, (%rdi)
[Test] Add codegen tests for unordered and monotonic integer operations llvm-svn: 353266
2019-02-06 11:19:04 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: store_i8:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movb %sil, (%rdi)
; CHECK-O3-NEXT: retq
store atomic i8 %v, i8* %ptr unordered, align 1
ret void
}
define i16 @load_i16(i16* %ptr) {
; CHECK-O0-LABEL: load_i16:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movw (%rdi), %ax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_i16:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movzwl (%rdi), %eax
; CHECK-O3-NEXT: retq
%v = load atomic i16, i16* %ptr unordered, align 2
ret i16 %v
}
define void @store_i16(i16* %ptr, i16 %v) {
; CHECK-O0-LABEL: store_i16:
; CHECK-O0: # %bb.0:
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: # kill: def $si killed $si killed $esi
; CHECK-O0-NEXT: movw %si, (%rdi)
[Test] Add codegen tests for unordered and monotonic integer operations llvm-svn: 353266
2019-02-06 11:19:04 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: store_i16:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movw %si, (%rdi)
; CHECK-O3-NEXT: retq
store atomic i16 %v, i16* %ptr unordered, align 2
ret void
}
define i32 @load_i32(i32* %ptr) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: load_i32:
; CHECK: # %bb.0:
; CHECK-NEXT: movl (%rdi), %eax
; CHECK-NEXT: retq
[Test] Add codegen tests for unordered and monotonic integer operations llvm-svn: 353266
2019-02-06 11:19:04 +08:00
%v = load atomic i32, i32* %ptr unordered, align 4
ret i32 %v
}
define void @store_i32(i32* %ptr, i32 %v) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: store_i32:
; CHECK: # %bb.0:
; CHECK-NEXT: movl %esi, (%rdi)
; CHECK-NEXT: retq
[Test] Add codegen tests for unordered and monotonic integer operations llvm-svn: 353266
2019-02-06 11:19:04 +08:00
store atomic i32 %v, i32* %ptr unordered, align 4
ret void
}
define i64 @load_i64(i64* %ptr) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: load_i64:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: retq
[Test] Add codegen tests for unordered and monotonic integer operations llvm-svn: 353266
2019-02-06 11:19:04 +08:00
%v = load atomic i64, i64* %ptr unordered, align 8
ret i64 %v
}
define void @store_i64(i64* %ptr, i64 %v) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: store_i64:
; CHECK: # %bb.0:
; CHECK-NEXT: movq %rsi, (%rdi)
; CHECK-NEXT: retq
[Test] Add codegen tests for unordered and monotonic integer operations llvm-svn: 353266
2019-02-06 11:19:04 +08:00
store atomic i64 %v, i64* %ptr unordered, align 8
ret void
}
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
;; The tests in the rest of this file are intended to show transforms which we
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
;; either *can't* do for legality, or don't currently implement. The later
;; are noted carefully where relevant.
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
;; Start w/some clearly illegal ones.
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
; Must use a full width op, not a byte op
define void @narrow_writeback_or(i64* %ptr) {
; CHECK-O0-LABEL: narrow_writeback_or:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: orq $7, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: narrow_writeback_or:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: orq $7, (%rdi)
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %ptr unordered, align 8
%v.new = or i64 %v, 7
store atomic i64 %v.new, i64* %ptr unordered, align 8
ret void
}
; Must use a full width op, not a byte op
define void @narrow_writeback_and(i64* %ptr) {
; CHECK-O0-LABEL: narrow_writeback_and:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O0-NEXT: andl $-256, %eax
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
; CHECK-O0-NEXT: movl %eax, %ecx
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: movq %rcx, (%rdi)
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: narrow_writeback_and:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movl $4294967040, %eax # imm = 0xFFFFFF00
; CHECK-O3-NEXT: andq %rax, (%rdi)
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %ptr unordered, align 8
%v.new = and i64 %v, 4294967040 ;; 0xFFFF_FF00
store atomic i64 %v.new, i64* %ptr unordered, align 8
ret void
}
; Must use a full width op, not a byte op
define void @narrow_writeback_xor(i64* %ptr) {
; CHECK-O0-LABEL: narrow_writeback_xor:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorq $7, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: narrow_writeback_xor:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: xorq $7, (%rdi)
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %ptr unordered, align 8
%v.new = xor i64 %v, 7
store atomic i64 %v.new, i64* %ptr unordered, align 8
ret void
}
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
;; Next batch of tests are exercising cases where store widening would
;; improve codegeneration. Note that widening is only legal if the
;; resulting type would be atomic. Each tests has a well aligned, and
;; unaligned variant to ensure we get correct codegen here.
;; Note: It's not a legality issue, but there's a gotcha here to be aware
;; of. Once we widen a pair of atomic stores, we loose the information
;; that the original atomicity requirement was half the width. Given that,
;; we can't then split the load again. This challenges our usual iterative
;; approach to incremental improvement.
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
; Legal if wider type is also atomic (TODO)
define void @widen_store(i32* %p0, i32 %v1, i32 %v2) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: widen_store:
; CHECK: # %bb.0:
; CHECK-NEXT: movl %esi, (%rdi)
; CHECK-NEXT: movl %edx, 4(%rdi)
; CHECK-NEXT: retq
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
%p1 = getelementptr i32, i32* %p0, i64 1
store atomic i32 %v1, i32* %p0 unordered, align 8
store atomic i32 %v2, i32* %p1 unordered, align 4
ret void
}
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
; This one is *NOT* legal to widen. With weaker alignment,
; the wider type might cross a cache line and violate the
; atomicity requirement.
define void @widen_store_unaligned(i32* %p0, i32 %v1, i32 %v2) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: widen_store_unaligned:
; CHECK: # %bb.0:
; CHECK-NEXT: movl %esi, (%rdi)
; CHECK-NEXT: movl %edx, 4(%rdi)
; CHECK-NEXT: retq
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
%p1 = getelementptr i32, i32* %p0, i64 1
store atomic i32 %v1, i32* %p0 unordered, align 4
store atomic i32 %v2, i32* %p1 unordered, align 4
ret void
}
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
; Legal if wider type is also atomic (TODO)
define void @widen_broadcast(i32* %p0, i32 %v) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: widen_broadcast:
; CHECK: # %bb.0:
; CHECK-NEXT: movl %esi, (%rdi)
; CHECK-NEXT: movl %esi, 4(%rdi)
; CHECK-NEXT: retq
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
%p1 = getelementptr i32, i32* %p0, i64 1
store atomic i32 %v, i32* %p0 unordered, align 8
store atomic i32 %v, i32* %p1 unordered, align 4
ret void
}
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
; Not legal to widen due to alignment restriction
define void @widen_broadcast_unaligned(i32* %p0, i32 %v) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: widen_broadcast_unaligned:
; CHECK: # %bb.0:
; CHECK-NEXT: movl %esi, (%rdi)
; CHECK-NEXT: movl %esi, 4(%rdi)
; CHECK-NEXT: retq
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
%p1 = getelementptr i32, i32* %p0, i64 1
store atomic i32 %v, i32* %p0 unordered, align 4
store atomic i32 %v, i32* %p1 unordered, align 4
ret void
}
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
define i128 @load_i128(i128* %ptr) {
; CHECK-O0-LABEL: load_i128:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: pushq %rbx
; CHECK-O0-NEXT: .cfi_def_cfa_offset 16
; CHECK-O0-NEXT: .cfi_offset %rbx, -16
; CHECK-O0-NEXT: xorl %eax, %eax
; CHECK-O0-NEXT: movl %eax, %ecx
; CHECK-O0-NEXT: movq %rcx, %rax
; CHECK-O0-NEXT: movq %rcx, %rdx
; CHECK-O0-NEXT: movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: movq {{[-0-9]+}}(%r{{[sb]}}p), %rbx # 8-byte Reload
; CHECK-O0-NEXT: lock cmpxchg16b (%rdi)
; CHECK-O0-NEXT: popq %rbx
; CHECK-O0-NEXT: .cfi_def_cfa_offset 8
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_i128:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: pushq %rbx
; CHECK-O3-NEXT: .cfi_def_cfa_offset 16
; CHECK-O3-NEXT: .cfi_offset %rbx, -16
; CHECK-O3-NEXT: xorl %eax, %eax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: xorl %ecx, %ecx
; CHECK-O3-NEXT: xorl %ebx, %ebx
; CHECK-O3-NEXT: lock cmpxchg16b (%rdi)
; CHECK-O3-NEXT: popq %rbx
; CHECK-O3-NEXT: .cfi_def_cfa_offset 8
; CHECK-O3-NEXT: retq
%v = load atomic i128, i128* %ptr unordered, align 16
ret i128 %v
}
define void @store_i128(i128* %ptr, i128 %v) {
; CHECK-O0-LABEL: store_i128:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: pushq %rbx
; CHECK-O0-NEXT: .cfi_def_cfa_offset 16
; CHECK-O0-NEXT: .cfi_offset %rbx, -16
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: movq 8(%rdi), %rcx
[FastISel] Skip creating unnecessary vregs for arguments This behavior was added in r130928 for both FastISel and SD, and then disabled in r131156 for FastISel. This re-enables it for FastISel with the corresponding fix. This is triggered only when FastISel can't lower the arguments and falls back to SelectionDAG for it. FastISel contains a map of "register fixups" where at the end of the selection phase it replaces all uses of a register with another register that FastISel sometimes pre-assigned. Code at the end of SelectionDAGISel::runOnMachineFunction is doing the replacement at the very end of the function, while other pieces that come in before that look through the MachineFunction and assume everything is done. In this case, the real issue is that the code emitting COPY instructions for the liveins (physreg to vreg) (EmitLiveInCopies) is checking if the vreg assigned to the physreg is used, and if it's not, it will skip the COPY. If a register wasn't replaced with its assigned fixup yet, the copy will be skipped and we'll end up with uses of undefined registers. This fix moves the replacement of registers before the emission of copies for the live-ins. The initial motivation for this fix is to enable tail calls for swiftself functions, which were blocked because we couldn't prove that the swiftself argument (which is callee-save) comes from a function argument (live-in), because there was an extra copy (vreg to vreg). A few tests are affected by this: * llvm/test/CodeGen/AArch64/swifterror.ll: we used to spill x21 (callee-save) but never reload it because it's attached to the return. We now don't even spill it anymore. * llvm/test/CodeGen/*/swiftself.ll: we tail-call now. * llvm/test/CodeGen/AMDGPU/mubuf-legalize-operands.ll: I believe this test was not really testing the right thing, but it worked because the same registers were re-used. * llvm/test/CodeGen/ARM/cmpxchg-O0.ll: regalloc changes * llvm/test/CodeGen/ARM/swifterror.ll: get rid of a copy * llvm/test/CodeGen/Mips/*: get rid of spills and copies * llvm/test/CodeGen/SystemZ/swift-return.ll: smaller stack * llvm/test/CodeGen/X86/atomic-unordered.ll: smaller stack * llvm/test/CodeGen/X86/swifterror.ll: same as AArch64 * llvm/test/DebugInfo/X86/dbg-declare-arg.ll: stack size changed Differential Revision: https://reviews.llvm.org/D62361 llvm-svn: 362963
2019-06-11 00:53:37 +08:00
; CHECK-O0-NEXT: movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O0-NEXT: movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: jmp .LBB16_1
; CHECK-O0-NEXT: .LBB16_1: # %atomicrmw.start
; CHECK-O0-NEXT: # =>This Inner Loop Header: Depth=1
; CHECK-O0-NEXT: movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O0-NEXT: movq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Reload
; CHECK-O0-NEXT: movq {{[-0-9]+}}(%r{{[sb]}}p), %rbx # 8-byte Reload
; CHECK-O0-NEXT: movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
; CHECK-O0-NEXT: lock cmpxchg16b (%rsi)
; CHECK-O0-NEXT: movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: jne .LBB16_1
; CHECK-O0-NEXT: jmp .LBB16_2
; CHECK-O0-NEXT: .LBB16_2: # %atomicrmw.end
; CHECK-O0-NEXT: popq %rbx
; CHECK-O0-NEXT: .cfi_def_cfa_offset 8
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: store_i128:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: pushq %rbx
; CHECK-O3-NEXT: .cfi_def_cfa_offset 16
; CHECK-O3-NEXT: .cfi_offset %rbx, -16
; CHECK-O3-NEXT: movq %rdx, %rcx
; CHECK-O3-NEXT: movq %rsi, %rbx
; CHECK-O3-NEXT: movq (%rdi), %rax
; CHECK-O3-NEXT: movq 8(%rdi), %rdx
; CHECK-O3-NEXT: .p2align 4, 0x90
; CHECK-O3-NEXT: .LBB16_1: # %atomicrmw.start
; CHECK-O3-NEXT: # =>This Inner Loop Header: Depth=1
; CHECK-O3-NEXT: lock cmpxchg16b (%rdi)
; CHECK-O3-NEXT: jne .LBB16_1
; CHECK-O3-NEXT: # %bb.2: # %atomicrmw.end
; CHECK-O3-NEXT: popq %rbx
; CHECK-O3-NEXT: .cfi_def_cfa_offset 8
; CHECK-O3-NEXT: retq
store atomic i128 %v, i128* %ptr unordered, align 16
ret void
}
define i256 @load_i256(i256* %ptr) {
; CHECK-O0-LABEL: load_i256:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: subq $56, %rsp
; CHECK-O0-NEXT: .cfi_def_cfa_offset 64
; CHECK-O0-NEXT: movq %rdi, %rax
; CHECK-O0-NEXT: movl $32, %ecx
; CHECK-O0-NEXT: leaq {{[0-9]+}}(%rsp), %rdx
; CHECK-O0-NEXT: xorl %r8d, %r8d
; CHECK-O0-NEXT: movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: movq %rcx, %rdi
; CHECK-O0-NEXT: movl %r8d, %ecx
; CHECK-O0-NEXT: movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: callq __atomic_load
; CHECK-O0-NEXT: movq {{[0-9]+}}(%rsp), %rax
; CHECK-O0-NEXT: movq {{[0-9]+}}(%rsp), %rdx
; CHECK-O0-NEXT: movq {{[0-9]+}}(%rsp), %rsi
; CHECK-O0-NEXT: movq {{[0-9]+}}(%rsp), %rdi
; CHECK-O0-NEXT: movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
; CHECK-O0-NEXT: movq %rdi, 24(%r9)
; CHECK-O0-NEXT: movq %rsi, 16(%r9)
; CHECK-O0-NEXT: movq %rdx, 8(%r9)
; CHECK-O0-NEXT: movq %rax, (%r9)
; CHECK-O0-NEXT: movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
; CHECK-O0-NEXT: addq $56, %rsp
; CHECK-O0-NEXT: .cfi_def_cfa_offset 8
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_i256:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: pushq %rbx
; CHECK-O3-NEXT: .cfi_def_cfa_offset 16
; CHECK-O3-NEXT: subq $32, %rsp
; CHECK-O3-NEXT: .cfi_def_cfa_offset 48
; CHECK-O3-NEXT: .cfi_offset %rbx, -16
; CHECK-O3-NEXT: movq %rdi, %rbx
; CHECK-O3-NEXT: movq %rsp, %rdx
; CHECK-O3-NEXT: movl $32, %edi
; CHECK-O3-NEXT: xorl %ecx, %ecx
; CHECK-O3-NEXT: callq __atomic_load
; CHECK-O3-NEXT: vmovups (%rsp), %ymm0
; CHECK-O3-NEXT: vmovups %ymm0, (%rbx)
; CHECK-O3-NEXT: movq %rbx, %rax
; CHECK-O3-NEXT: addq $32, %rsp
; CHECK-O3-NEXT: .cfi_def_cfa_offset 16
; CHECK-O3-NEXT: popq %rbx
; CHECK-O3-NEXT: .cfi_def_cfa_offset 8
; CHECK-O3-NEXT: vzeroupper
; CHECK-O3-NEXT: retq
%v = load atomic i256, i256* %ptr unordered, align 16
ret i256 %v
}
define void @store_i256(i256* %ptr, i256 %v) {
; CHECK-O0-LABEL: store_i256:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: subq $40, %rsp
; CHECK-O0-NEXT: .cfi_def_cfa_offset 48
; CHECK-O0-NEXT: xorl %eax, %eax
; CHECK-O0-NEXT: leaq {{[0-9]+}}(%rsp), %r9
; CHECK-O0-NEXT: movq %rsi, {{[0-9]+}}(%rsp)
; CHECK-O0-NEXT: movq %rdx, {{[0-9]+}}(%rsp)
; CHECK-O0-NEXT: movq %rcx, {{[0-9]+}}(%rsp)
; CHECK-O0-NEXT: movq %r8, {{[0-9]+}}(%rsp)
; CHECK-O0-NEXT: movl $32, %ecx
; CHECK-O0-NEXT: movq %rdi, (%rsp) # 8-byte Spill
; CHECK-O0-NEXT: movq %rcx, %rdi
; CHECK-O0-NEXT: movq (%rsp), %rsi # 8-byte Reload
; CHECK-O0-NEXT: movq %r9, %rdx
; CHECK-O0-NEXT: movl %eax, %ecx
; CHECK-O0-NEXT: callq __atomic_store
; CHECK-O0-NEXT: addq $40, %rsp
; CHECK-O0-NEXT: .cfi_def_cfa_offset 8
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: store_i256:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: subq $40, %rsp
; CHECK-O3-NEXT: .cfi_def_cfa_offset 48
; CHECK-O3-NEXT: movq %rdi, %rax
; CHECK-O3-NEXT: movq %r8, {{[0-9]+}}(%rsp)
; CHECK-O3-NEXT: movq %rcx, {{[0-9]+}}(%rsp)
; CHECK-O3-NEXT: movq %rdx, {{[0-9]+}}(%rsp)
; CHECK-O3-NEXT: movq %rsi, {{[0-9]+}}(%rsp)
; CHECK-O3-NEXT: leaq {{[0-9]+}}(%rsp), %rdx
; CHECK-O3-NEXT: movl $32, %edi
; CHECK-O3-NEXT: movq %rax, %rsi
; CHECK-O3-NEXT: xorl %ecx, %ecx
; CHECK-O3-NEXT: callq __atomic_store
; CHECK-O3-NEXT: addq $40, %rsp
; CHECK-O3-NEXT: .cfi_def_cfa_offset 8
; CHECK-O3-NEXT: retq
store atomic i256 %v, i256* %ptr unordered, align 16
ret void
}
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
; Legal if wider type is also atomic (TODO)
define void @vec_store(i32* %p0, <2 x i32> %vec) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-CUR-LABEL: vec_store:
; CHECK-O0-CUR: # %bb.0:
; CHECK-O0-CUR-NEXT: vmovd %xmm0, %eax
; CHECK-O0-CUR-NEXT: vpextrd $1, %xmm0, %ecx
; CHECK-O0-CUR-NEXT: movl %eax, (%rdi)
; CHECK-O0-CUR-NEXT: movl %ecx, 4(%rdi)
; CHECK-O0-CUR-NEXT: retq
;
; CHECK-O3-CUR-LABEL: vec_store:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: vmovd %xmm0, %eax
; CHECK-O3-CUR-NEXT: vpextrd $1, %xmm0, %ecx
; CHECK-O3-CUR-NEXT: movl %eax, (%rdi)
; CHECK-O3-CUR-NEXT: movl %ecx, 4(%rdi)
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O0-EX-LABEL: vec_store:
; CHECK-O0-EX: # %bb.0:
; CHECK-O0-EX-NEXT: vmovd %xmm0, (%rdi)
; CHECK-O0-EX-NEXT: vpextrd $1, %xmm0, 4(%rdi)
; CHECK-O0-EX-NEXT: retq
;
; CHECK-O3-EX-LABEL: vec_store:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: vmovss %xmm0, (%rdi)
; CHECK-O3-EX-NEXT: vextractps $1, %xmm0, 4(%rdi)
; CHECK-O3-EX-NEXT: retq
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
%v1 = extractelement <2 x i32> %vec, i32 0
%v2 = extractelement <2 x i32> %vec, i32 1
%p1 = getelementptr i32, i32* %p0, i64 1
store atomic i32 %v1, i32* %p0 unordered, align 8
store atomic i32 %v2, i32* %p1 unordered, align 4
ret void
}
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
; Not legal to widen due to alignment restriction
define void @vec_store_unaligned(i32* %p0, <2 x i32> %vec) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-CUR-LABEL: vec_store_unaligned:
; CHECK-O0-CUR: # %bb.0:
; CHECK-O0-CUR-NEXT: vmovd %xmm0, %eax
; CHECK-O0-CUR-NEXT: vpextrd $1, %xmm0, %ecx
; CHECK-O0-CUR-NEXT: movl %eax, (%rdi)
; CHECK-O0-CUR-NEXT: movl %ecx, 4(%rdi)
; CHECK-O0-CUR-NEXT: retq
;
; CHECK-O3-CUR-LABEL: vec_store_unaligned:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: vmovd %xmm0, %eax
; CHECK-O3-CUR-NEXT: vpextrd $1, %xmm0, %ecx
; CHECK-O3-CUR-NEXT: movl %eax, (%rdi)
; CHECK-O3-CUR-NEXT: movl %ecx, 4(%rdi)
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O0-EX-LABEL: vec_store_unaligned:
; CHECK-O0-EX: # %bb.0:
; CHECK-O0-EX-NEXT: vmovd %xmm0, (%rdi)
; CHECK-O0-EX-NEXT: vpextrd $1, %xmm0, 4(%rdi)
; CHECK-O0-EX-NEXT: retq
;
; CHECK-O3-EX-LABEL: vec_store_unaligned:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: vmovss %xmm0, (%rdi)
; CHECK-O3-EX-NEXT: vextractps $1, %xmm0, 4(%rdi)
; CHECK-O3-EX-NEXT: retq
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
%v1 = extractelement <2 x i32> %vec, i32 0
%v2 = extractelement <2 x i32> %vec, i32 1
%p1 = getelementptr i32, i32* %p0, i64 1
store atomic i32 %v1, i32* %p0 unordered, align 4
store atomic i32 %v2, i32* %p1 unordered, align 4
ret void
}
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
; Legal if wider type is also atomic (TODO)
; Also, can avoid register move from xmm to eax (TODO)
define void @widen_broadcast2(i32* %p0, <2 x i32> %vec) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-CUR-LABEL: widen_broadcast2:
; CHECK-O0-CUR: # %bb.0:
; CHECK-O0-CUR-NEXT: vmovd %xmm0, %eax
; CHECK-O0-CUR-NEXT: movl %eax, (%rdi)
; CHECK-O0-CUR-NEXT: movl %eax, 4(%rdi)
; CHECK-O0-CUR-NEXT: retq
;
; CHECK-O3-CUR-LABEL: widen_broadcast2:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: vmovd %xmm0, %eax
; CHECK-O3-CUR-NEXT: movl %eax, (%rdi)
; CHECK-O3-CUR-NEXT: movl %eax, 4(%rdi)
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O0-EX-LABEL: widen_broadcast2:
; CHECK-O0-EX: # %bb.0:
; CHECK-O0-EX-NEXT: vmovd %xmm0, (%rdi)
; CHECK-O0-EX-NEXT: vmovd %xmm0, 4(%rdi)
; CHECK-O0-EX-NEXT: retq
;
; CHECK-O3-EX-LABEL: widen_broadcast2:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: vmovss %xmm0, (%rdi)
; CHECK-O3-EX-NEXT: vmovss %xmm0, 4(%rdi)
; CHECK-O3-EX-NEXT: retq
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
%v1 = extractelement <2 x i32> %vec, i32 0
%p1 = getelementptr i32, i32* %p0, i64 1
store atomic i32 %v1, i32* %p0 unordered, align 8
store atomic i32 %v1, i32* %p1 unordered, align 4
ret void
}
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
; Not legal to widen due to alignment restriction
define void @widen_broadcast2_unaligned(i32* %p0, <2 x i32> %vec) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-CUR-LABEL: widen_broadcast2_unaligned:
; CHECK-O0-CUR: # %bb.0:
; CHECK-O0-CUR-NEXT: vmovd %xmm0, %eax
; CHECK-O0-CUR-NEXT: movl %eax, (%rdi)
; CHECK-O0-CUR-NEXT: movl %eax, 4(%rdi)
; CHECK-O0-CUR-NEXT: retq
;
; CHECK-O3-CUR-LABEL: widen_broadcast2_unaligned:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: vmovd %xmm0, %eax
; CHECK-O3-CUR-NEXT: movl %eax, (%rdi)
; CHECK-O3-CUR-NEXT: movl %eax, 4(%rdi)
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O0-EX-LABEL: widen_broadcast2_unaligned:
; CHECK-O0-EX: # %bb.0:
; CHECK-O0-EX-NEXT: vmovd %xmm0, (%rdi)
; CHECK-O0-EX-NEXT: vmovd %xmm0, 4(%rdi)
; CHECK-O0-EX-NEXT: retq
;
; CHECK-O3-EX-LABEL: widen_broadcast2_unaligned:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: vmovss %xmm0, (%rdi)
; CHECK-O3-EX-NEXT: vmovss %xmm0, 4(%rdi)
; CHECK-O3-EX-NEXT: retq
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
%v1 = extractelement <2 x i32> %vec, i32 0
%p1 = getelementptr i32, i32* %p0, i64 1
store atomic i32 %v1, i32* %p0 unordered, align 4
store atomic i32 %v1, i32* %p1 unordered, align 4
ret void
}
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
; Legal if wider type is also atomic (TODO)
define void @widen_zero_init(i32* %p0, i32 %v1, i32 %v2) {
[SDAG] Update generic code to conservatively check for isAtomic in addition to isVolatile This is the first sweep of generic code to add isAtomic bailouts where appropriate. The intention here is to have the switch from AtomicSDNode to LoadSDNode/StoreSDNode be close to NFC; that is, I'm not looking to allow additional optimizations at this time. That will come later. See D66309 for context. Differential Revision: https://reviews.llvm.org/D66318 llvm-svn: 371786
2019-09-13 06:49:17 +08:00
; CHECK-LABEL: widen_zero_init:
; CHECK: # %bb.0:
; CHECK-NEXT: movl $0, (%rdi)
; CHECK-NEXT: movl $0, 4(%rdi)
; CHECK-NEXT: retq
[Tests] First batch of cornercase tests for unordered atomics Mixture of things we legally can't do, and things we're missing. Once D57601 is in, the later will serve as a punch list. llvm-svn: 353959
2019-02-14 02:00:58 +08:00
%p1 = getelementptr i32, i32* %p0, i64 1
store atomic i32 0, i32* %p0 unordered, align 8
store atomic i32 0, i32* %p1 unordered, align 4
ret void
}
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
; Not legal to widen due to alignment restriction
define void @widen_zero_init_unaligned(i32* %p0, i32 %v1, i32 %v2) {
[SDAG] Update generic code to conservatively check for isAtomic in addition to isVolatile This is the first sweep of generic code to add isAtomic bailouts where appropriate. The intention here is to have the switch from AtomicSDNode to LoadSDNode/StoreSDNode be close to NFC; that is, I'm not looking to allow additional optimizations at this time. That will come later. See D66309 for context. Differential Revision: https://reviews.llvm.org/D66318 llvm-svn: 371786
2019-09-13 06:49:17 +08:00
; CHECK-LABEL: widen_zero_init_unaligned:
; CHECK: # %bb.0:
; CHECK-NEXT: movl $0, (%rdi)
; CHECK-NEXT: movl $0, 4(%rdi)
; CHECK-NEXT: retq
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
%p1 = getelementptr i32, i32* %p0, i64 1
store atomic i32 0, i32* %p0 unordered, align 4
store atomic i32 0, i32* %p1 unordered, align 4
ret void
}
;; The next batch of tests are stressing load folding. Folding is legal
;; on x86, so these are simply checking optimization quality.
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; Legal, as expected
define i64 @load_fold_add1(i64* %p) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: load_fold_add1:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: addq $15, %rax
; CHECK-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p unordered, align 8
%ret = add i64 %v, 15
ret i64 %ret
}
define i64 @load_fold_add2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_add2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: addq (%rdi), %rsi
; CHECK-O0-NEXT: movq %rsi, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_add2:
; CHECK-O3: # %bb.0:
Allow code motion (and thus folding) for atomic (but unordered) memory operands Building on the work done in D57601, now that we can distinguish between atomic and volatile memory accesses, go ahead and allow code motion of unordered atomics. As seen in the diffs, this allows much better folding of memory operations into using instructions. (Mostly done by the PeepholeOpt pass.) Note: I have not reviewed all callers of hasOrderedMemoryRef since one of them - isSafeToMove - is very widely used. I'm relying on the documented semantics of each method to judge correctness. Differential Revision: https://reviews.llvm.org/D59345 llvm-svn: 356170
2019-03-15 01:20:59 +08:00
; CHECK-O3-NEXT: movq %rsi, %rax
; CHECK-O3-NEXT: addq (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = add i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_add3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_add3:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: addq (%rsi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-CUR-LABEL: load_fold_add3:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rsi), %rax
; CHECK-O3-CUR-NEXT: addq (%rdi), %rax
; CHECK-O3-CUR-NEXT: retq
Introduce infrastructure for an incremental port of SelectionDAG atomic load/store handling This is the first patch in a large sequence. The eventual goal is to have unordered atomic loads and stores - and possibly ordered atomics as well - handled through the normal ISEL codepaths for loads and stores. Today, there handled w/instances of AtomicSDNodes. The result of which is that all transforms need to be duplicated to work for unordered atomics. The benefit of the current design is that it's harder to introduce a silent miscompile by adding an transform which forgets about atomicity. See the thread on llvm-dev titled "FYI: proposed changes to atomic load/store in SelectionDAG" for further context. Note that this patch is NFC unless the experimental flag is set. The basic strategy I plan on taking is: introduce infrastructure and a flag for testing (this patch) Audit uses of isVolatile, and apply isAtomic conservatively* piecemeal conservative* update generic code and x86 backedge code in individual reviews w/tests for cases which didn't check volatile, but can be found with inspection flip the flag at the end (with minimal diffs) Work through todo list identified in (2) and (3) exposing performance ops (*) The "conservative" bit here is aimed at minimizing the number of diffs involved in (4). Ideally, there'd be none. In practice, getting it down to something reviewable by a human is the actual goal. Note that there are (currently) no paths which produce LoadSDNode or StoreSDNode with atomic MMOs, so we don't need to worry about preserving any behaviour there. We've taken a very similar strategy twice before with success - once at IR level, and once at the MI level (post ISEL). Differential Revision: https://reviews.llvm.org/D66309 llvm-svn: 371441
2019-09-10 03:23:22 +08:00
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-LABEL: load_fold_add3:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq (%rdi), %rax
; CHECK-O3-EX-NEXT: addq (%rsi), %rax
; CHECK-O3-EX-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = add i64 %v, %v2
ret i64 %ret
}
; Legal, as expected
define i64 @load_fold_sub1(i64* %p) {
; CHECK-O0-LABEL: load_fold_sub1:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: subq $15, %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_sub1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
; CHECK-O3-NEXT: addq $-15, %rax
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = sub i64 %v, 15
ret i64 %ret
}
define i64 @load_fold_sub2(i64* %p, i64 %v2) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: load_fold_sub2:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: subq %rsi, %rax
; CHECK-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p unordered, align 8
%ret = sub i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_sub3(i64* %p1, i64* %p2) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: load_fold_sub3:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: subq (%rsi), %rax
; CHECK-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = sub i64 %v, %v2
ret i64 %ret
}
; Legal, as expected
define i64 @load_fold_mul1(i64* %p) {
; CHECK-O0-LABEL: load_fold_mul1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: imulq $15, (%rdi), %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_mul1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
; CHECK-O3-NEXT: leaq (%rax,%rax,4), %rax
; CHECK-O3-NEXT: leaq (%rax,%rax,2), %rax
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = mul i64 %v, 15
ret i64 %ret
}
define i64 @load_fold_mul2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_mul2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: imulq (%rdi), %rsi
; CHECK-O0-NEXT: movq %rsi, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_mul2:
; CHECK-O3: # %bb.0:
Allow code motion (and thus folding) for atomic (but unordered) memory operands Building on the work done in D57601, now that we can distinguish between atomic and volatile memory accesses, go ahead and allow code motion of unordered atomics. As seen in the diffs, this allows much better folding of memory operations into using instructions. (Mostly done by the PeepholeOpt pass.) Note: I have not reviewed all callers of hasOrderedMemoryRef since one of them - isSafeToMove - is very widely used. I'm relying on the documented semantics of each method to judge correctness. Differential Revision: https://reviews.llvm.org/D59345 llvm-svn: 356170
2019-03-15 01:20:59 +08:00
; CHECK-O3-NEXT: movq %rsi, %rax
; CHECK-O3-NEXT: imulq (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = mul i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_mul3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_mul3:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: imulq (%rsi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-CUR-LABEL: load_fold_mul3:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rsi), %rax
; CHECK-O3-CUR-NEXT: imulq (%rdi), %rax
; CHECK-O3-CUR-NEXT: retq
Introduce infrastructure for an incremental port of SelectionDAG atomic load/store handling This is the first patch in a large sequence. The eventual goal is to have unordered atomic loads and stores - and possibly ordered atomics as well - handled through the normal ISEL codepaths for loads and stores. Today, there handled w/instances of AtomicSDNodes. The result of which is that all transforms need to be duplicated to work for unordered atomics. The benefit of the current design is that it's harder to introduce a silent miscompile by adding an transform which forgets about atomicity. See the thread on llvm-dev titled "FYI: proposed changes to atomic load/store in SelectionDAG" for further context. Note that this patch is NFC unless the experimental flag is set. The basic strategy I plan on taking is: introduce infrastructure and a flag for testing (this patch) Audit uses of isVolatile, and apply isAtomic conservatively* piecemeal conservative* update generic code and x86 backedge code in individual reviews w/tests for cases which didn't check volatile, but can be found with inspection flip the flag at the end (with minimal diffs) Work through todo list identified in (2) and (3) exposing performance ops (*) The "conservative" bit here is aimed at minimizing the number of diffs involved in (4). Ideally, there'd be none. In practice, getting it down to something reviewable by a human is the actual goal. Note that there are (currently) no paths which produce LoadSDNode or StoreSDNode with atomic MMOs, so we don't need to worry about preserving any behaviour there. We've taken a very similar strategy twice before with success - once at IR level, and once at the MI level (post ISEL). Differential Revision: https://reviews.llvm.org/D66309 llvm-svn: 371441
2019-09-10 03:23:22 +08:00
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-LABEL: load_fold_mul3:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq (%rdi), %rax
; CHECK-O3-EX-NEXT: imulq (%rsi), %rax
; CHECK-O3-EX-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = mul i64 %v, %v2
ret i64 %ret
}
; Legal to fold (TODO)
define i64 @load_fold_sdiv1(i64* %p) {
; CHECK-O0-LABEL: load_fold_sdiv1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: cqto
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movl $15, %ecx
; CHECK-O0-NEXT: idivq %rcx
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_sdiv1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rcx
; CHECK-O3-NEXT: movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
; CHECK-O3-NEXT: movq %rcx, %rax
; CHECK-O3-NEXT: imulq %rdx
; CHECK-O3-NEXT: addq %rcx, %rdx
; CHECK-O3-NEXT: movq %rdx, %rax
; CHECK-O3-NEXT: shrq $63, %rax
; CHECK-O3-NEXT: sarq $3, %rdx
[X86FixupLEAs] Turn optIncDec into a generic two address LEA optimizer. Support LEA64_32r properly. INC/DEC is really a special case of a more generic issue. We should also turn leas into add reg/reg or add reg/imm regardless of the slow lea flags. This also supports LEA64_32 which has 64 bit input registers and 32 bit output registers. So we need to convert the 64 bit inputs to their 32 bit equivalents to check if they are equal to base reg. One thing to note, the original code preserved the kill flags by adding operands to the new instruction instead of using addReg. But I think tied operands aren't supposed to have the kill flag set. I dropped the kill flags, but I could probably try to preserve it in the add reg/reg case if we think its important. Not sure which operand its supposed to go on for the LEA64_32r instruction due to the super reg implicit uses. Though I'm also not sure those are needed since they were probably just created by an INSERT_SUBREG from a 32-bit input. Differential Revision: https://reviews.llvm.org/D61472 llvm-svn: 361691
2019-05-25 14:17:47 +08:00
; CHECK-O3-NEXT: addq %rdx, %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = sdiv i64 %v, 15
ret i64 %ret
}
; Legal to fold (TODO)
define i64 @load_fold_sdiv2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_sdiv2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: cqto
; CHECK-O0-NEXT: idivq %rsi
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_sdiv2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq %rax, %rcx
; CHECK-O3-NEXT: orq %rsi, %rcx
; CHECK-O3-NEXT: shrq $32, %rcx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB35_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: cqto
; CHECK-O3-NEXT: idivq %rsi
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB35_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divl %esi
; CHECK-O3-NEXT: # kill: def $eax killed $eax def $rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = sdiv i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_sdiv3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_sdiv3:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: cqto
; CHECK-O0-NEXT: idivq (%rsi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_sdiv3:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq (%rsi), %rcx
; CHECK-O3-NEXT: movq %rax, %rdx
; CHECK-O3-NEXT: orq %rcx, %rdx
; CHECK-O3-NEXT: shrq $32, %rdx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB36_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: cqto
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: idivq %rcx
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB36_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divl %ecx
; CHECK-O3-NEXT: # kill: def $eax killed $eax def $rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = sdiv i64 %v, %v2
ret i64 %ret
}
; Legal to fold (TODO)
define i64 @load_fold_udiv1(i64* %p) {
; CHECK-O0-LABEL: load_fold_udiv1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorl %ecx, %ecx
; CHECK-O0-NEXT: movl %ecx, %edx
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movl $15, %esi
; CHECK-O0-NEXT: divq %rsi
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-CUR-LABEL: load_fold_udiv1:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rdi), %rax
; CHECK-O3-CUR-NEXT: movabsq $-8608480567731124087, %rcx # imm = 0x8888888888888889
; CHECK-O3-CUR-NEXT: mulq %rcx
; CHECK-O3-CUR-NEXT: movq %rdx, %rax
; CHECK-O3-CUR-NEXT: shrq $3, %rax
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O3-EX-LABEL: load_fold_udiv1:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movabsq $-8608480567731124087, %rax # imm = 0x8888888888888889
; CHECK-O3-EX-NEXT: mulq (%rdi)
; CHECK-O3-EX-NEXT: movq %rdx, %rax
; CHECK-O3-EX-NEXT: shrq $3, %rax
; CHECK-O3-EX-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p unordered, align 8
%ret = udiv i64 %v, 15
ret i64 %ret
}
define i64 @load_fold_udiv2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_udiv2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorl %ecx, %ecx
; CHECK-O0-NEXT: movl %ecx, %edx
; CHECK-O0-NEXT: divq %rsi
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_udiv2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq %rax, %rcx
; CHECK-O3-NEXT: orq %rsi, %rcx
; CHECK-O3-NEXT: shrq $32, %rcx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB38_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divq %rsi
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB38_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divl %esi
; CHECK-O3-NEXT: # kill: def $eax killed $eax def $rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = udiv i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_udiv3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_udiv3:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorl %ecx, %ecx
; CHECK-O0-NEXT: movl %ecx, %edx
; CHECK-O0-NEXT: divq (%rsi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_udiv3:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq (%rsi), %rcx
; CHECK-O3-NEXT: movq %rax, %rdx
; CHECK-O3-NEXT: orq %rcx, %rdx
; CHECK-O3-NEXT: shrq $32, %rdx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB39_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divq %rcx
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB39_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: xorl %edx, %edx
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: divl %ecx
; CHECK-O3-NEXT: # kill: def $eax killed $eax def $rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = udiv i64 %v, %v2
ret i64 %ret
}
; Legal to fold (TODO)
define i64 @load_fold_srem1(i64* %p) {
; CHECK-O0-LABEL: load_fold_srem1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: cqto
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movl $15, %ecx
; CHECK-O0-NEXT: idivq %rcx
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: movq %rdx, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_srem1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rcx
; CHECK-O3-NEXT: movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
; CHECK-O3-NEXT: movq %rcx, %rax
; CHECK-O3-NEXT: imulq %rdx
; CHECK-O3-NEXT: addq %rcx, %rdx
; CHECK-O3-NEXT: movq %rdx, %rax
; CHECK-O3-NEXT: shrq $63, %rax
; CHECK-O3-NEXT: sarq $3, %rdx
; CHECK-O3-NEXT: addq %rax, %rdx
; CHECK-O3-NEXT: leaq (%rdx,%rdx,4), %rax
; CHECK-O3-NEXT: leaq (%rax,%rax,2), %rax
; CHECK-O3-NEXT: subq %rax, %rcx
; CHECK-O3-NEXT: movq %rcx, %rax
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = srem i64 %v, 15
ret i64 %ret
}
; Legal, as expected
define i64 @load_fold_srem2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_srem2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: cqto
; CHECK-O0-NEXT: idivq %rsi
; CHECK-O0-NEXT: movq %rdx, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_srem2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq %rax, %rcx
; CHECK-O3-NEXT: orq %rsi, %rcx
; CHECK-O3-NEXT: shrq $32, %rcx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB41_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: cqto
; CHECK-O3-NEXT: idivq %rsi
; CHECK-O3-NEXT: movq %rdx, %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB41_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divl %esi
; CHECK-O3-NEXT: movl %edx, %eax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = srem i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_srem3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_srem3:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: cqto
; CHECK-O0-NEXT: idivq (%rsi)
; CHECK-O0-NEXT: movq %rdx, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_srem3:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq (%rsi), %rcx
; CHECK-O3-NEXT: movq %rax, %rdx
; CHECK-O3-NEXT: orq %rcx, %rdx
; CHECK-O3-NEXT: shrq $32, %rdx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB42_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: cqto
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: idivq %rcx
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: movq %rdx, %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB42_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divl %ecx
; CHECK-O3-NEXT: movl %edx, %eax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = srem i64 %v, %v2
ret i64 %ret
}
; Legal to fold (TODO)
define i64 @load_fold_urem1(i64* %p) {
; CHECK-O0-LABEL: load_fold_urem1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorl %ecx, %ecx
; CHECK-O0-NEXT: movl %ecx, %edx
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movl $15, %esi
; CHECK-O0-NEXT: divq %rsi
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: movq %rdx, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_urem1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rcx
; CHECK-O3-NEXT: movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
; CHECK-O3-NEXT: movq %rcx, %rax
; CHECK-O3-NEXT: mulq %rdx
; CHECK-O3-NEXT: shrq $3, %rdx
; CHECK-O3-NEXT: leaq (%rdx,%rdx,4), %rax
; CHECK-O3-NEXT: leaq (%rax,%rax,2), %rax
; CHECK-O3-NEXT: subq %rax, %rcx
; CHECK-O3-NEXT: movq %rcx, %rax
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = urem i64 %v, 15
ret i64 %ret
}
; Legal, as expected
define i64 @load_fold_urem2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_urem2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorl %ecx, %ecx
; CHECK-O0-NEXT: movl %ecx, %edx
; CHECK-O0-NEXT: divq %rsi
; CHECK-O0-NEXT: movq %rdx, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_urem2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq %rax, %rcx
; CHECK-O3-NEXT: orq %rsi, %rcx
; CHECK-O3-NEXT: shrq $32, %rcx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB44_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divq %rsi
; CHECK-O3-NEXT: movq %rdx, %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB44_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divl %esi
; CHECK-O3-NEXT: movl %edx, %eax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = urem i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_urem3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_urem3:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorl %ecx, %ecx
; CHECK-O0-NEXT: movl %ecx, %edx
; CHECK-O0-NEXT: divq (%rsi)
; CHECK-O0-NEXT: movq %rdx, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_urem3:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq (%rsi), %rcx
; CHECK-O3-NEXT: movq %rax, %rdx
; CHECK-O3-NEXT: orq %rcx, %rdx
; CHECK-O3-NEXT: shrq $32, %rdx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB45_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: xorl %edx, %edx
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: divq %rcx
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: movq %rdx, %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB45_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divl %ecx
; CHECK-O3-NEXT: movl %edx, %eax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = urem i64 %v, %v2
ret i64 %ret
}
; Legal, as expected
define i64 @load_fold_shl1(i64* %p) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: load_fold_shl1:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: shlq $15, %rax
; CHECK-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p unordered, align 8
%ret = shl i64 %v, 15
ret i64 %ret
}
define i64 @load_fold_shl2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_shl2:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: movq %rsi, %rcx
; CHECK-O0-NEXT: # kill: def $cl killed $rcx
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: shlq %cl, %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_shl2:
; CHECK-O3: # %bb.0:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: shlxq %rsi, (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = shl i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_shl3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_shl3:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: movq (%rsi), %rcx
; CHECK-O0-NEXT: # kill: def $cl killed $rcx
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: shlq %cl, %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_shl3:
; CHECK-O3: # %bb.0:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq (%rsi), %rax
; CHECK-O3-NEXT: shlxq %rax, (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = shl i64 %v, %v2
ret i64 %ret
}
; Legal, as expected
define i64 @load_fold_lshr1(i64* %p) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: load_fold_lshr1:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: shrq $15, %rax
; CHECK-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p unordered, align 8
%ret = lshr i64 %v, 15
ret i64 %ret
}
define i64 @load_fold_lshr2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_lshr2:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: movq %rsi, %rcx
; CHECK-O0-NEXT: # kill: def $cl killed $rcx
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: shrq %cl, %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_lshr2:
; CHECK-O3: # %bb.0:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: shrxq %rsi, (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = lshr i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_lshr3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_lshr3:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: movq (%rsi), %rcx
; CHECK-O0-NEXT: # kill: def $cl killed $rcx
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: shrq %cl, %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_lshr3:
; CHECK-O3: # %bb.0:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq (%rsi), %rax
; CHECK-O3-NEXT: shrxq %rax, (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = lshr i64 %v, %v2
ret i64 %ret
}
; Legal, as expected
define i64 @load_fold_ashr1(i64* %p) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: load_fold_ashr1:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: sarq $15, %rax
; CHECK-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p unordered, align 8
%ret = ashr i64 %v, 15
ret i64 %ret
}
define i64 @load_fold_ashr2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_ashr2:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: movq %rsi, %rcx
; CHECK-O0-NEXT: # kill: def $cl killed $rcx
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: sarq %cl, %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_ashr2:
; CHECK-O3: # %bb.0:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: sarxq %rsi, (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = ashr i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_ashr3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_ashr3:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: movq (%rsi), %rcx
; CHECK-O0-NEXT: # kill: def $cl killed $rcx
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: sarq %cl, %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_ashr3:
; CHECK-O3: # %bb.0:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq (%rsi), %rax
; CHECK-O3-NEXT: sarxq %rax, (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = ashr i64 %v, %v2
ret i64 %ret
}
; Legal, as expected
define i64 @load_fold_and1(i64* %p) {
; CHECK-O0-LABEL: load_fold_and1:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: andq $15, %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_and1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
; CHECK-O3-NEXT: andl $15, %eax
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = and i64 %v, 15
ret i64 %ret
}
define i64 @load_fold_and2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_and2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: andq (%rdi), %rsi
; CHECK-O0-NEXT: movq %rsi, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_and2:
; CHECK-O3: # %bb.0:
Allow code motion (and thus folding) for atomic (but unordered) memory operands Building on the work done in D57601, now that we can distinguish between atomic and volatile memory accesses, go ahead and allow code motion of unordered atomics. As seen in the diffs, this allows much better folding of memory operations into using instructions. (Mostly done by the PeepholeOpt pass.) Note: I have not reviewed all callers of hasOrderedMemoryRef since one of them - isSafeToMove - is very widely used. I'm relying on the documented semantics of each method to judge correctness. Differential Revision: https://reviews.llvm.org/D59345 llvm-svn: 356170
2019-03-15 01:20:59 +08:00
; CHECK-O3-NEXT: movq %rsi, %rax
; CHECK-O3-NEXT: andq (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = and i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_and3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_and3:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: andq (%rsi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-CUR-LABEL: load_fold_and3:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rsi), %rax
; CHECK-O3-CUR-NEXT: andq (%rdi), %rax
; CHECK-O3-CUR-NEXT: retq
Introduce infrastructure for an incremental port of SelectionDAG atomic load/store handling This is the first patch in a large sequence. The eventual goal is to have unordered atomic loads and stores - and possibly ordered atomics as well - handled through the normal ISEL codepaths for loads and stores. Today, there handled w/instances of AtomicSDNodes. The result of which is that all transforms need to be duplicated to work for unordered atomics. The benefit of the current design is that it's harder to introduce a silent miscompile by adding an transform which forgets about atomicity. See the thread on llvm-dev titled "FYI: proposed changes to atomic load/store in SelectionDAG" for further context. Note that this patch is NFC unless the experimental flag is set. The basic strategy I plan on taking is: introduce infrastructure and a flag for testing (this patch) Audit uses of isVolatile, and apply isAtomic conservatively* piecemeal conservative* update generic code and x86 backedge code in individual reviews w/tests for cases which didn't check volatile, but can be found with inspection flip the flag at the end (with minimal diffs) Work through todo list identified in (2) and (3) exposing performance ops (*) The "conservative" bit here is aimed at minimizing the number of diffs involved in (4). Ideally, there'd be none. In practice, getting it down to something reviewable by a human is the actual goal. Note that there are (currently) no paths which produce LoadSDNode or StoreSDNode with atomic MMOs, so we don't need to worry about preserving any behaviour there. We've taken a very similar strategy twice before with success - once at IR level, and once at the MI level (post ISEL). Differential Revision: https://reviews.llvm.org/D66309 llvm-svn: 371441
2019-09-10 03:23:22 +08:00
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-LABEL: load_fold_and3:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq (%rdi), %rax
; CHECK-O3-EX-NEXT: andq (%rsi), %rax
; CHECK-O3-EX-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = and i64 %v, %v2
ret i64 %ret
}
; Legal, as expected
define i64 @load_fold_or1(i64* %p) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: load_fold_or1:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: orq $15, %rax
; CHECK-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p unordered, align 8
%ret = or i64 %v, 15
ret i64 %ret
}
define i64 @load_fold_or2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_or2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: orq (%rdi), %rsi
; CHECK-O0-NEXT: movq %rsi, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_or2:
; CHECK-O3: # %bb.0:
Allow code motion (and thus folding) for atomic (but unordered) memory operands Building on the work done in D57601, now that we can distinguish between atomic and volatile memory accesses, go ahead and allow code motion of unordered atomics. As seen in the diffs, this allows much better folding of memory operations into using instructions. (Mostly done by the PeepholeOpt pass.) Note: I have not reviewed all callers of hasOrderedMemoryRef since one of them - isSafeToMove - is very widely used. I'm relying on the documented semantics of each method to judge correctness. Differential Revision: https://reviews.llvm.org/D59345 llvm-svn: 356170
2019-03-15 01:20:59 +08:00
; CHECK-O3-NEXT: movq %rsi, %rax
; CHECK-O3-NEXT: orq (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = or i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_or3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_or3:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: orq (%rsi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-CUR-LABEL: load_fold_or3:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rsi), %rax
; CHECK-O3-CUR-NEXT: orq (%rdi), %rax
; CHECK-O3-CUR-NEXT: retq
Introduce infrastructure for an incremental port of SelectionDAG atomic load/store handling This is the first patch in a large sequence. The eventual goal is to have unordered atomic loads and stores - and possibly ordered atomics as well - handled through the normal ISEL codepaths for loads and stores. Today, there handled w/instances of AtomicSDNodes. The result of which is that all transforms need to be duplicated to work for unordered atomics. The benefit of the current design is that it's harder to introduce a silent miscompile by adding an transform which forgets about atomicity. See the thread on llvm-dev titled "FYI: proposed changes to atomic load/store in SelectionDAG" for further context. Note that this patch is NFC unless the experimental flag is set. The basic strategy I plan on taking is: introduce infrastructure and a flag for testing (this patch) Audit uses of isVolatile, and apply isAtomic conservatively* piecemeal conservative* update generic code and x86 backedge code in individual reviews w/tests for cases which didn't check volatile, but can be found with inspection flip the flag at the end (with minimal diffs) Work through todo list identified in (2) and (3) exposing performance ops (*) The "conservative" bit here is aimed at minimizing the number of diffs involved in (4). Ideally, there'd be none. In practice, getting it down to something reviewable by a human is the actual goal. Note that there are (currently) no paths which produce LoadSDNode or StoreSDNode with atomic MMOs, so we don't need to worry about preserving any behaviour there. We've taken a very similar strategy twice before with success - once at IR level, and once at the MI level (post ISEL). Differential Revision: https://reviews.llvm.org/D66309 llvm-svn: 371441
2019-09-10 03:23:22 +08:00
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-LABEL: load_fold_or3:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq (%rdi), %rax
; CHECK-O3-EX-NEXT: orq (%rsi), %rax
; CHECK-O3-EX-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = or i64 %v, %v2
ret i64 %ret
}
; Legal, as expected
define i64 @load_fold_xor1(i64* %p) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: load_fold_xor1:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: xorq $15, %rax
; CHECK-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p unordered, align 8
%ret = xor i64 %v, 15
ret i64 %ret
}
define i64 @load_fold_xor2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_xor2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: xorq (%rdi), %rsi
; CHECK-O0-NEXT: movq %rsi, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_xor2:
; CHECK-O3: # %bb.0:
Allow code motion (and thus folding) for atomic (but unordered) memory operands Building on the work done in D57601, now that we can distinguish between atomic and volatile memory accesses, go ahead and allow code motion of unordered atomics. As seen in the diffs, this allows much better folding of memory operations into using instructions. (Mostly done by the PeepholeOpt pass.) Note: I have not reviewed all callers of hasOrderedMemoryRef since one of them - isSafeToMove - is very widely used. I'm relying on the documented semantics of each method to judge correctness. Differential Revision: https://reviews.llvm.org/D59345 llvm-svn: 356170
2019-03-15 01:20:59 +08:00
; CHECK-O3-NEXT: movq %rsi, %rax
; CHECK-O3-NEXT: xorq (%rdi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = xor i64 %v, %v2
ret i64 %ret
}
define i64 @load_fold_xor3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_xor3:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorq (%rsi), %rax
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-CUR-LABEL: load_fold_xor3:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rsi), %rax
; CHECK-O3-CUR-NEXT: xorq (%rdi), %rax
; CHECK-O3-CUR-NEXT: retq
Introduce infrastructure for an incremental port of SelectionDAG atomic load/store handling This is the first patch in a large sequence. The eventual goal is to have unordered atomic loads and stores - and possibly ordered atomics as well - handled through the normal ISEL codepaths for loads and stores. Today, there handled w/instances of AtomicSDNodes. The result of which is that all transforms need to be duplicated to work for unordered atomics. The benefit of the current design is that it's harder to introduce a silent miscompile by adding an transform which forgets about atomicity. See the thread on llvm-dev titled "FYI: proposed changes to atomic load/store in SelectionDAG" for further context. Note that this patch is NFC unless the experimental flag is set. The basic strategy I plan on taking is: introduce infrastructure and a flag for testing (this patch) Audit uses of isVolatile, and apply isAtomic conservatively* piecemeal conservative* update generic code and x86 backedge code in individual reviews w/tests for cases which didn't check volatile, but can be found with inspection flip the flag at the end (with minimal diffs) Work through todo list identified in (2) and (3) exposing performance ops (*) The "conservative" bit here is aimed at minimizing the number of diffs involved in (4). Ideally, there'd be none. In practice, getting it down to something reviewable by a human is the actual goal. Note that there are (currently) no paths which produce LoadSDNode or StoreSDNode with atomic MMOs, so we don't need to worry about preserving any behaviour there. We've taken a very similar strategy twice before with success - once at IR level, and once at the MI level (post ISEL). Differential Revision: https://reviews.llvm.org/D66309 llvm-svn: 371441
2019-09-10 03:23:22 +08:00
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-LABEL: load_fold_xor3:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq (%rdi), %rax
; CHECK-O3-EX-NEXT: xorq (%rsi), %rax
; CHECK-O3-EX-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = xor i64 %v, %v2
ret i64 %ret
}
define i1 @load_fold_icmp1(i64* %p) {
; CHECK-O0-LABEL: load_fold_icmp1:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: subq $15, %rax
; CHECK-O0-NEXT: sete %cl
; CHECK-O0-NEXT: movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: movb %cl, %al
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_icmp1:
; CHECK-O3: # %bb.0:
Allow code motion (and thus folding) for atomic (but unordered) memory operands Building on the work done in D57601, now that we can distinguish between atomic and volatile memory accesses, go ahead and allow code motion of unordered atomics. As seen in the diffs, this allows much better folding of memory operations into using instructions. (Mostly done by the PeepholeOpt pass.) Note: I have not reviewed all callers of hasOrderedMemoryRef since one of them - isSafeToMove - is very widely used. I'm relying on the documented semantics of each method to judge correctness. Differential Revision: https://reviews.llvm.org/D59345 llvm-svn: 356170
2019-03-15 01:20:59 +08:00
; CHECK-O3-NEXT: cmpq $15, (%rdi)
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: sete %al
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = icmp eq i64 %v, 15
ret i1 %ret
}
define i1 @load_fold_icmp2(i64* %p, i64 %v2) {
; CHECK-O0-LABEL: load_fold_icmp2:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: subq %rsi, %rax
; CHECK-O0-NEXT: sete %cl
; CHECK-O0-NEXT: movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: movb %cl, %al
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_fold_icmp2:
; CHECK-O3: # %bb.0:
Allow code motion (and thus folding) for atomic (but unordered) memory operands Building on the work done in D57601, now that we can distinguish between atomic and volatile memory accesses, go ahead and allow code motion of unordered atomics. As seen in the diffs, this allows much better folding of memory operations into using instructions. (Mostly done by the PeepholeOpt pass.) Note: I have not reviewed all callers of hasOrderedMemoryRef since one of them - isSafeToMove - is very widely used. I'm relying on the documented semantics of each method to judge correctness. Differential Revision: https://reviews.llvm.org/D59345 llvm-svn: 356170
2019-03-15 01:20:59 +08:00
; CHECK-O3-NEXT: cmpq %rsi, (%rdi)
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O3-NEXT: sete %al
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%ret = icmp eq i64 %v, %v2
ret i1 %ret
}
define i1 @load_fold_icmp3(i64* %p1, i64* %p2) {
; CHECK-O0-LABEL: load_fold_icmp3:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: movq (%rsi), %rcx
; CHECK-O0-NEXT: subq %rcx, %rax
; CHECK-O0-NEXT: sete %dl
; CHECK-O0-NEXT: movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: movb %dl, %al
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
; CHECK-O0-NEXT: retq
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-CUR-LABEL: load_fold_icmp3:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rsi), %rax
; CHECK-O3-CUR-NEXT: cmpq %rax, (%rdi)
; CHECK-O3-CUR-NEXT: sete %al
; CHECK-O3-CUR-NEXT: retq
Introduce infrastructure for an incremental port of SelectionDAG atomic load/store handling This is the first patch in a large sequence. The eventual goal is to have unordered atomic loads and stores - and possibly ordered atomics as well - handled through the normal ISEL codepaths for loads and stores. Today, there handled w/instances of AtomicSDNodes. The result of which is that all transforms need to be duplicated to work for unordered atomics. The benefit of the current design is that it's harder to introduce a silent miscompile by adding an transform which forgets about atomicity. See the thread on llvm-dev titled "FYI: proposed changes to atomic load/store in SelectionDAG" for further context. Note that this patch is NFC unless the experimental flag is set. The basic strategy I plan on taking is: introduce infrastructure and a flag for testing (this patch) Audit uses of isVolatile, and apply isAtomic conservatively* piecemeal conservative* update generic code and x86 backedge code in individual reviews w/tests for cases which didn't check volatile, but can be found with inspection flip the flag at the end (with minimal diffs) Work through todo list identified in (2) and (3) exposing performance ops (*) The "conservative" bit here is aimed at minimizing the number of diffs involved in (4). Ideally, there'd be none. In practice, getting it down to something reviewable by a human is the actual goal. Note that there are (currently) no paths which produce LoadSDNode or StoreSDNode with atomic MMOs, so we don't need to worry about preserving any behaviour there. We've taken a very similar strategy twice before with success - once at IR level, and once at the MI level (post ISEL). Differential Revision: https://reviews.llvm.org/D66309 llvm-svn: 371441
2019-09-10 03:23:22 +08:00
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-LABEL: load_fold_icmp3:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq (%rdi), %rax
; CHECK-O3-EX-NEXT: cmpq (%rsi), %rax
; CHECK-O3-EX-NEXT: sete %al
; CHECK-O3-EX-NEXT: retq
[Tests] Add a bunch of tests for load folding w/unordered atomics llvm-svn: 353964
2019-02-14 02:26:01 +08:00
%v = load atomic i64, i64* %p1 unordered, align 8
%v2 = load atomic i64, i64* %p2 unordered, align 8
%ret = icmp eq i64 %v, %v2
ret i1 %ret
}
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
;; The next batch of tests check for read-modify-write patterns
;; Legally, it's okay to use a memory operand here as long as the operand
;; is well aligned (i.e. doesn't cross a cache line boundary). We are
;; required not to narrow the store though!
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; Legal, as expected
define void @rmw_fold_add1(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_add1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: addq $15, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_add1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: addq $15, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = add i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_add2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_add2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: addq %rsi, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_add2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: addq %rsi, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = add i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_sub1(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_sub1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: addq $-15, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_sub1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: addq $-15, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = sub i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_sub2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_sub2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: subq %rsi, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_sub2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: subq %rsi, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = sub i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_mul1(i64* %p, i64 %v) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: rmw_fold_mul1:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: leaq (%rax,%rax,4), %rax
; CHECK-NEXT: leaq (%rax,%rax,2), %rax
; CHECK-NEXT: movq %rax, (%rdi)
; CHECK-NEXT: retq
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
%prev = load atomic i64, i64* %p unordered, align 8
%val = mul i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal to fold (TODO)
define void @rmw_fold_mul2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_mul2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: imulq %rsi, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_mul2:
; CHECK-O3: # %bb.0:
Allow code motion (and thus folding) for atomic (but unordered) memory operands Building on the work done in D57601, now that we can distinguish between atomic and volatile memory accesses, go ahead and allow code motion of unordered atomics. As seen in the diffs, this allows much better folding of memory operations into using instructions. (Mostly done by the PeepholeOpt pass.) Note: I have not reviewed all callers of hasOrderedMemoryRef since one of them - isSafeToMove - is very widely used. I'm relying on the documented semantics of each method to judge correctness. Differential Revision: https://reviews.llvm.org/D59345 llvm-svn: 356170
2019-03-15 01:20:59 +08:00
; CHECK-O3-NEXT: imulq (%rdi), %rsi
; CHECK-O3-NEXT: movq %rsi, (%rdi)
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = mul i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_sdiv1(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_sdiv1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: movabsq $-8608480567731124087, %rcx # imm = 0x8888888888888889
; CHECK-O0-NEXT: movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: imulq %rcx
; CHECK-O0-NEXT: movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
; CHECK-O0-NEXT: addq %rax, %rdx
; CHECK-O0-NEXT: movq %rdx, %rcx
; CHECK-O0-NEXT: shrq $63, %rcx
; CHECK-O0-NEXT: sarq $3, %rdx
; CHECK-O0-NEXT: addq %rcx, %rdx
; CHECK-O0-NEXT: movq %rdx, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_sdiv1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rcx
; CHECK-O3-NEXT: movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
; CHECK-O3-NEXT: movq %rcx, %rax
; CHECK-O3-NEXT: imulq %rdx
; CHECK-O3-NEXT: addq %rcx, %rdx
; CHECK-O3-NEXT: movq %rdx, %rax
; CHECK-O3-NEXT: shrq $63, %rax
; CHECK-O3-NEXT: sarq $3, %rdx
; CHECK-O3-NEXT: addq %rax, %rdx
; CHECK-O3-NEXT: movq %rdx, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = sdiv i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_sdiv2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_sdiv2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: cqto
; CHECK-O0-NEXT: idivq %rsi
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_sdiv2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq %rax, %rcx
; CHECK-O3-NEXT: orq %rsi, %rcx
; CHECK-O3-NEXT: shrq $32, %rcx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB74_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O3-NEXT: cqto
; CHECK-O3-NEXT: idivq %rsi
; CHECK-O3-NEXT: movq %rax, (%rdi)
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB74_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divl %esi
; CHECK-O3-NEXT: # kill: def $eax killed $eax def $rax
; CHECK-O3-NEXT: movq %rax, (%rdi)
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = sdiv i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_udiv1(i64* %p, i64 %v) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-LABEL: rmw_fold_udiv1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: movabsq $-8608480567731124087, %rcx # imm = 0x8888888888888889
; CHECK-O0-NEXT: mulq %rcx
; CHECK-O0-NEXT: shrq $3, %rdx
; CHECK-O0-NEXT: movq %rdx, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-CUR-LABEL: rmw_fold_udiv1:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rdi), %rax
; CHECK-O3-CUR-NEXT: movabsq $-8608480567731124087, %rcx # imm = 0x8888888888888889
; CHECK-O3-CUR-NEXT: mulq %rcx
; CHECK-O3-CUR-NEXT: shrq $3, %rdx
; CHECK-O3-CUR-NEXT: movq %rdx, (%rdi)
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O3-EX-LABEL: rmw_fold_udiv1:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movabsq $-8608480567731124087, %rax # imm = 0x8888888888888889
; CHECK-O3-EX-NEXT: mulq (%rdi)
; CHECK-O3-EX-NEXT: shrq $3, %rdx
; CHECK-O3-EX-NEXT: movq %rdx, (%rdi)
; CHECK-O3-EX-NEXT: retq
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
%prev = load atomic i64, i64* %p unordered, align 8
%val = udiv i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_udiv2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_udiv2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorl %ecx, %ecx
; CHECK-O0-NEXT: movl %ecx, %edx
; CHECK-O0-NEXT: divq %rsi
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_udiv2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq %rax, %rcx
; CHECK-O3-NEXT: orq %rsi, %rcx
; CHECK-O3-NEXT: shrq $32, %rcx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB76_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divq %rsi
; CHECK-O3-NEXT: movq %rax, (%rdi)
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB76_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divl %esi
; CHECK-O3-NEXT: # kill: def $eax killed $eax def $rax
; CHECK-O3-NEXT: movq %rax, (%rdi)
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = udiv i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_srem1(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_srem1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: movabsq $-8608480567731124087, %rcx # imm = 0x8888888888888889
; CHECK-O0-NEXT: movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: imulq %rcx
; CHECK-O0-NEXT: movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
; CHECK-O0-NEXT: addq %rax, %rdx
; CHECK-O0-NEXT: movq %rdx, %rcx
; CHECK-O0-NEXT: shrq $63, %rcx
; CHECK-O0-NEXT: sarq $3, %rdx
; CHECK-O0-NEXT: addq %rcx, %rdx
; CHECK-O0-NEXT: leaq (%rdx,%rdx,4), %rcx
; CHECK-O0-NEXT: leaq (%rcx,%rcx,2), %rcx
; CHECK-O0-NEXT: subq %rcx, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_srem1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rcx
; CHECK-O3-NEXT: movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
; CHECK-O3-NEXT: movq %rcx, %rax
; CHECK-O3-NEXT: imulq %rdx
; CHECK-O3-NEXT: addq %rcx, %rdx
; CHECK-O3-NEXT: movq %rdx, %rax
; CHECK-O3-NEXT: shrq $63, %rax
; CHECK-O3-NEXT: sarq $3, %rdx
; CHECK-O3-NEXT: addq %rax, %rdx
; CHECK-O3-NEXT: leaq (%rdx,%rdx,4), %rax
; CHECK-O3-NEXT: leaq (%rax,%rax,2), %rax
; CHECK-O3-NEXT: subq %rax, %rcx
; CHECK-O3-NEXT: movq %rcx, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = srem i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_srem2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_srem2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: cqto
; CHECK-O0-NEXT: idivq %rsi
; CHECK-O0-NEXT: movq %rdx, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_srem2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq %rax, %rcx
; CHECK-O3-NEXT: orq %rsi, %rcx
; CHECK-O3-NEXT: shrq $32, %rcx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB78_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O3-NEXT: cqto
; CHECK-O3-NEXT: idivq %rsi
; CHECK-O3-NEXT: movq %rdx, (%rdi)
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB78_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divl %esi
; CHECK-O3-NEXT: # kill: def $edx killed $edx def $rdx
; CHECK-O3-NEXT: movq %rdx, (%rdi)
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = srem i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_urem1(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_urem1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: movabsq $-8608480567731124087, %rcx # imm = 0x8888888888888889
; CHECK-O0-NEXT: movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
; CHECK-O0-NEXT: mulq %rcx
; CHECK-O0-NEXT: shrq $3, %rdx
; CHECK-O0-NEXT: leaq (%rdx,%rdx,4), %rax
; CHECK-O0-NEXT: leaq (%rax,%rax,2), %rax
; CHECK-O0-NEXT: movq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Reload
; CHECK-O0-NEXT: subq %rax, %rcx
; CHECK-O0-NEXT: movq %rcx, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_urem1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rcx
; CHECK-O3-NEXT: movabsq $-8608480567731124087, %rdx # imm = 0x8888888888888889
; CHECK-O3-NEXT: movq %rcx, %rax
; CHECK-O3-NEXT: mulq %rdx
; CHECK-O3-NEXT: shrq $3, %rdx
; CHECK-O3-NEXT: leaq (%rdx,%rdx,4), %rax
; CHECK-O3-NEXT: leaq (%rax,%rax,2), %rax
; CHECK-O3-NEXT: subq %rax, %rcx
; CHECK-O3-NEXT: movq %rcx, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = urem i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_urem2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_urem2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorl %ecx, %ecx
; CHECK-O0-NEXT: movl %ecx, %edx
; CHECK-O0-NEXT: divq %rsi
; CHECK-O0-NEXT: movq %rdx, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_urem2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: movq %rax, %rcx
; CHECK-O3-NEXT: orq %rsi, %rcx
; CHECK-O3-NEXT: shrq $32, %rcx
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: je .LBB80_1
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # %bb.2:
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divq %rsi
; CHECK-O3-NEXT: movq %rdx, (%rdi)
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: retq
[AtomicExpand] Fix a crash bug when lowering unordered loads to cmpxchg Add tests for wider atomic loads and stores. In the process, fix a crasher where we appearently handled unorder stores, but not loads, when lowering to cmpxchg idioms. llvm-svn: 356482
2019-03-20 01:20:49 +08:00
; CHECK-O3-NEXT: .LBB80_1:
[Tests] Update to newer ISA There are some issues w/missed opts on older platforms, but that's not the purpose of this test. Using a newer API points out that some TODOs are already handled, and allows addition of tests to exercise other issues (future patch.) llvm-svn: 356473
2019-03-20 00:46:56 +08:00
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: xorl %edx, %edx
; CHECK-O3-NEXT: divl %esi
; CHECK-O3-NEXT: # kill: def $edx killed $edx def $rdx
; CHECK-O3-NEXT: movq %rdx, (%rdi)
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = urem i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal to fold (TODO)
define void @rmw_fold_shl1(i64* %p, i64 %v) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-LABEL: rmw_fold_shl1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: shlq $15, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-CUR-LABEL: rmw_fold_shl1:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rdi), %rax
; CHECK-O3-CUR-NEXT: shlq $15, %rax
; CHECK-O3-CUR-NEXT: movq %rax, (%rdi)
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O3-EX-LABEL: rmw_fold_shl1:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: shlq $15, (%rdi)
; CHECK-O3-EX-NEXT: retq
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
%prev = load atomic i64, i64* %p unordered, align 8
%val = shl i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal to fold (TODO)
define void @rmw_fold_shl2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_shl2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: # kill: def $sil killed $sil killed $rsi
; CHECK-O0-NEXT: # implicit-def: $rcx
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O0-NEXT: movb %sil, %cl
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: shlxq %rcx, %rax, %rax
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-CUR-LABEL: rmw_fold_shl2:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: shlxq %rsi, (%rdi), %rax
; CHECK-O3-CUR-NEXT: movq %rax, (%rdi)
; CHECK-O3-CUR-NEXT: retq
Introduce infrastructure for an incremental port of SelectionDAG atomic load/store handling This is the first patch in a large sequence. The eventual goal is to have unordered atomic loads and stores - and possibly ordered atomics as well - handled through the normal ISEL codepaths for loads and stores. Today, there handled w/instances of AtomicSDNodes. The result of which is that all transforms need to be duplicated to work for unordered atomics. The benefit of the current design is that it's harder to introduce a silent miscompile by adding an transform which forgets about atomicity. See the thread on llvm-dev titled "FYI: proposed changes to atomic load/store in SelectionDAG" for further context. Note that this patch is NFC unless the experimental flag is set. The basic strategy I plan on taking is: introduce infrastructure and a flag for testing (this patch) Audit uses of isVolatile, and apply isAtomic conservatively* piecemeal conservative* update generic code and x86 backedge code in individual reviews w/tests for cases which didn't check volatile, but can be found with inspection flip the flag at the end (with minimal diffs) Work through todo list identified in (2) and (3) exposing performance ops (*) The "conservative" bit here is aimed at minimizing the number of diffs involved in (4). Ideally, there'd be none. In practice, getting it down to something reviewable by a human is the actual goal. Note that there are (currently) no paths which produce LoadSDNode or StoreSDNode with atomic MMOs, so we don't need to worry about preserving any behaviour there. We've taken a very similar strategy twice before with success - once at IR level, and once at the MI level (post ISEL). Differential Revision: https://reviews.llvm.org/D66309 llvm-svn: 371441
2019-09-10 03:23:22 +08:00
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-LABEL: rmw_fold_shl2:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq %rsi, %rcx
; CHECK-O3-EX-NEXT: # kill: def $cl killed $cl killed $rcx
; CHECK-O3-EX-NEXT: shlq %cl, (%rdi)
; CHECK-O3-EX-NEXT: retq
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
%prev = load atomic i64, i64* %p unordered, align 8
%val = shl i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal to fold (TODO)
define void @rmw_fold_lshr1(i64* %p, i64 %v) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-LABEL: rmw_fold_lshr1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: shrq $15, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-CUR-LABEL: rmw_fold_lshr1:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rdi), %rax
; CHECK-O3-CUR-NEXT: shrq $15, %rax
; CHECK-O3-CUR-NEXT: movq %rax, (%rdi)
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O3-EX-LABEL: rmw_fold_lshr1:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: shrq $15, (%rdi)
; CHECK-O3-EX-NEXT: retq
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
%prev = load atomic i64, i64* %p unordered, align 8
%val = lshr i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal to fold (TODO)
define void @rmw_fold_lshr2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_lshr2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: # kill: def $sil killed $sil killed $rsi
; CHECK-O0-NEXT: # implicit-def: $rcx
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O0-NEXT: movb %sil, %cl
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: shrxq %rcx, %rax, %rax
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-CUR-LABEL: rmw_fold_lshr2:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: shrxq %rsi, (%rdi), %rax
; CHECK-O3-CUR-NEXT: movq %rax, (%rdi)
; CHECK-O3-CUR-NEXT: retq
Introduce infrastructure for an incremental port of SelectionDAG atomic load/store handling This is the first patch in a large sequence. The eventual goal is to have unordered atomic loads and stores - and possibly ordered atomics as well - handled through the normal ISEL codepaths for loads and stores. Today, there handled w/instances of AtomicSDNodes. The result of which is that all transforms need to be duplicated to work for unordered atomics. The benefit of the current design is that it's harder to introduce a silent miscompile by adding an transform which forgets about atomicity. See the thread on llvm-dev titled "FYI: proposed changes to atomic load/store in SelectionDAG" for further context. Note that this patch is NFC unless the experimental flag is set. The basic strategy I plan on taking is: introduce infrastructure and a flag for testing (this patch) Audit uses of isVolatile, and apply isAtomic conservatively* piecemeal conservative* update generic code and x86 backedge code in individual reviews w/tests for cases which didn't check volatile, but can be found with inspection flip the flag at the end (with minimal diffs) Work through todo list identified in (2) and (3) exposing performance ops (*) The "conservative" bit here is aimed at minimizing the number of diffs involved in (4). Ideally, there'd be none. In practice, getting it down to something reviewable by a human is the actual goal. Note that there are (currently) no paths which produce LoadSDNode or StoreSDNode with atomic MMOs, so we don't need to worry about preserving any behaviour there. We've taken a very similar strategy twice before with success - once at IR level, and once at the MI level (post ISEL). Differential Revision: https://reviews.llvm.org/D66309 llvm-svn: 371441
2019-09-10 03:23:22 +08:00
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-LABEL: rmw_fold_lshr2:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq %rsi, %rcx
; CHECK-O3-EX-NEXT: # kill: def $cl killed $cl killed $rcx
; CHECK-O3-EX-NEXT: shrq %cl, (%rdi)
; CHECK-O3-EX-NEXT: retq
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
%prev = load atomic i64, i64* %p unordered, align 8
%val = lshr i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal to fold (TODO)
define void @rmw_fold_ashr1(i64* %p, i64 %v) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-LABEL: rmw_fold_ashr1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: sarq $15, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-CUR-LABEL: rmw_fold_ashr1:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rdi), %rax
; CHECK-O3-CUR-NEXT: sarq $15, %rax
; CHECK-O3-CUR-NEXT: movq %rax, (%rdi)
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O3-EX-LABEL: rmw_fold_ashr1:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: sarq $15, (%rdi)
; CHECK-O3-EX-NEXT: retq
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
%prev = load atomic i64, i64* %p unordered, align 8
%val = ashr i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal to fold (TODO)
define void @rmw_fold_ashr2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_ashr2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: # kill: def $sil killed $sil killed $rsi
; CHECK-O0-NEXT: # implicit-def: $rcx
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O0-NEXT: movb %sil, %cl
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: sarxq %rcx, %rax, %rax
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-CUR-LABEL: rmw_fold_ashr2:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: sarxq %rsi, (%rdi), %rax
; CHECK-O3-CUR-NEXT: movq %rax, (%rdi)
; CHECK-O3-CUR-NEXT: retq
Introduce infrastructure for an incremental port of SelectionDAG atomic load/store handling This is the first patch in a large sequence. The eventual goal is to have unordered atomic loads and stores - and possibly ordered atomics as well - handled through the normal ISEL codepaths for loads and stores. Today, there handled w/instances of AtomicSDNodes. The result of which is that all transforms need to be duplicated to work for unordered atomics. The benefit of the current design is that it's harder to introduce a silent miscompile by adding an transform which forgets about atomicity. See the thread on llvm-dev titled "FYI: proposed changes to atomic load/store in SelectionDAG" for further context. Note that this patch is NFC unless the experimental flag is set. The basic strategy I plan on taking is: introduce infrastructure and a flag for testing (this patch) Audit uses of isVolatile, and apply isAtomic conservatively* piecemeal conservative* update generic code and x86 backedge code in individual reviews w/tests for cases which didn't check volatile, but can be found with inspection flip the flag at the end (with minimal diffs) Work through todo list identified in (2) and (3) exposing performance ops (*) The "conservative" bit here is aimed at minimizing the number of diffs involved in (4). Ideally, there'd be none. In practice, getting it down to something reviewable by a human is the actual goal. Note that there are (currently) no paths which produce LoadSDNode or StoreSDNode with atomic MMOs, so we don't need to worry about preserving any behaviour there. We've taken a very similar strategy twice before with success - once at IR level, and once at the MI level (post ISEL). Differential Revision: https://reviews.llvm.org/D66309 llvm-svn: 371441
2019-09-10 03:23:22 +08:00
;
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-LABEL: rmw_fold_ashr2:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq %rsi, %rcx
; CHECK-O3-EX-NEXT: # kill: def $cl killed $cl killed $rcx
; CHECK-O3-EX-NEXT: sarq %cl, (%rdi)
; CHECK-O3-EX-NEXT: retq
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
%prev = load atomic i64, i64* %p unordered, align 8
%val = ashr i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_and1(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_and1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O0-NEXT: andl $15, %eax
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O0-NEXT: movl %eax, %ecx
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: movq %rcx, (%rdi)
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_and1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: andq $15, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = and i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_and2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_and2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: andq %rsi, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_and2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: andq %rsi, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = and i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_or1(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_or1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: orq $15, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_or1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: orq $15, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = or i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_or2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_or2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: orq %rsi, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_or2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: orq %rsi, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = or i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_xor1(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_xor1:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorq $15, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_xor1:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: xorq $15, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = xor i64 %prev, 15
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
; Legal, as expected
define void @rmw_fold_xor2(i64* %p, i64 %v) {
; CHECK-O0-LABEL: rmw_fold_xor2:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: xorq %rsi, %rax
; CHECK-O0-NEXT: movq %rax, (%rdi)
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: rmw_fold_xor2:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: xorq %rsi, (%rdi)
; CHECK-O3-NEXT: retq
%prev = load atomic i64, i64* %p unordered, align 8
%val = xor i64 %prev, %v
store atomic i64 %val, i64* %p unordered, align 8
ret void
}
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
;; The next batch test truncations, in combination w/operations which could
;; be folded against the memory operation.
; Legal to reduce the load width (TODO)
define i32 @fold_trunc(i64* %p) {
[SDAG] Update generic code to conservatively check for isAtomic in addition to isVolatile This is the first sweep of generic code to add isAtomic bailouts where appropriate. The intention here is to have the switch from AtomicSDNode to LoadSDNode/StoreSDNode be close to NFC; that is, I'm not looking to allow additional optimizations at this time. That will come later. See D66309 for context. Differential Revision: https://reviews.llvm.org/D66318 llvm-svn: 371786
2019-09-13 06:49:17 +08:00
; CHECK-LABEL: fold_trunc:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-NEXT: retq
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
%v = load atomic i64, i64* %p unordered, align 8
%ret = trunc i64 %v to i32
ret i32 %ret
}
; Legal to reduce the load width and fold the load (TODO)
define i32 @fold_trunc_add(i64* %p, i32 %v2) {
; CHECK-O0-LABEL: fold_trunc_add:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O0-NEXT: addl %esi, %eax
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: fold_trunc_add:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
; CHECK-O3-NEXT: addl %esi, %eax
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%trunc = trunc i64 %v to i32
%ret = add i32 %trunc, %v2
ret i32 %ret
}
; Legal to reduce the load width and fold the load (TODO)
define i32 @fold_trunc_and(i64* %p, i32 %v2) {
; CHECK-O0-LABEL: fold_trunc_and:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O0-NEXT: andl %esi, %eax
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: fold_trunc_and:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
; CHECK-O3-NEXT: andl %esi, %eax
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%trunc = trunc i64 %v to i32
%ret = and i32 %trunc, %v2
ret i32 %ret
}
; Legal to reduce the load width and fold the load (TODO)
define i32 @fold_trunc_or(i64* %p, i32 %v2) {
; CHECK-O0-LABEL: fold_trunc_or:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O0-NEXT: orl %esi, %eax
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: fold_trunc_or:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
; CHECK-O3-NEXT: orl %esi, %eax
; CHECK-O3-NEXT: # kill: def $eax killed $eax killed $rax
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%trunc = trunc i64 %v to i32
%ret = or i32 %trunc, %v2
ret i32 %ret
}
; It's tempting to split the wide load into two smaller byte loads
; to reduce memory traffic, but this would be illegal for a atomic load
define i32 @split_load(i64* %p) {
; CHECK-O0-LABEL: split_load:
; CHECK-O0: # %bb.0:
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: movb %al, %cl
; CHECK-O0-NEXT: shrq $32, %rax
RegAllocFast: Improve hinting heuristic Trace through multiple COPYs when looking for a physreg source. Add hinting for vregs that will be copied into physregs (we only hinted for vregs getting copied to a physreg previously). Give hinted a register a bonus when deciding which value to spill. This is part of my rewrite regallocfast series. In fact this one doesn't even have an effect unless you also flip the allocation to happen from back to front of a basic block. Nonetheless it helps to split this up to ease review of D52010 Patch by Matthias Braun llvm-svn: 360887
2019-05-16 20:50:39 +08:00
; CHECK-O0-NEXT: # kill: def $al killed $al killed $rax
; CHECK-O0-NEXT: orb %al, %cl
RegAllocFast: Remove early selection loop, the spill calculation will report cost 0 anyway for free regs The 2nd loop calculates spill costs but reports free registers as cost 0 anyway, so there is little benefit from having a separate early loop. Surprisingly this is not NFC, as many register are marked regDisabled so the first loop often picks up later registers unnecessarily instead of the first one available in the allocation order... Patch by Matthias Braun llvm-svn: 356499
2019-03-20 03:01:34 +08:00
; CHECK-O0-NEXT: movzbl %cl, %eax
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: split_load:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq (%rdi), %rax
; CHECK-O3-NEXT: movq %rax, %rcx
; CHECK-O3-NEXT: shrq $32, %rcx
; CHECK-O3-NEXT: orl %eax, %ecx
; CHECK-O3-NEXT: movzbl %cl, %eax
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* %p unordered, align 8
%b1 = trunc i64 %v to i8
%v.shift = lshr i64 %v, 32
%b2 = trunc i64 %v.shift to i8
%or = or i8 %b1, %b2
%ret = zext i8 %or to i32
ret i32 %ret
}
;; A collection of simple memory forwarding tests. Nothing particular
;; interesting semantic wise, just demonstrating obvious missed transforms.
[Tests] Add tests for reordering of unordered atomics on invariant locations llvm-svn: 356172
2019-03-15 01:36:58 +08:00
@Zero = constant i64 0
; TODO: should return constant
define i64 @constant_folding(i64* %p) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: constant_folding:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: retq
[Tests] Add tests for reordering of unordered atomics on invariant locations llvm-svn: 356172
2019-03-15 01:36:58 +08:00
%v = load atomic i64, i64* %p unordered, align 8
ret i64 %v
}
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
; Legal to forward and fold (TODO)
define i64 @load_forwarding(i64* %p) {
Fix a *nasty* miscompile in experimental unordered atomic lowering This is an omission in rL371441. Loads which happened to be unordered weren't being added to the PendingLoad set, and thus weren't be ordered w/respect to side effects which followed before the end of the block. Included test case is how I spotted this. We had an atomic load being folded into a using instruction after a fence that load was supposed to be ordered with. I'm sure it showed up a bunch of other ways as well. Spotted via manual inspecting of assembly differences in a corpus w/and w/o the new experimental mode. Finding this with testing would have been "unpleasant". llvm-svn: 373814
2019-10-05 08:32:10 +08:00
; CHECK-LABEL: load_forwarding:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: orq (%rdi), %rax
; CHECK-NEXT: retq
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
%v = load atomic i64, i64* %p unordered, align 8
%v2 = load atomic i64, i64* %p unordered, align 8
%ret = or i64 %v, %v2
ret i64 %ret
}
; Legal to forward (TODO)
define i64 @store_forward(i64* %p, i64 %v) {
[SDAG] Update generic code to conservatively check for isAtomic in addition to isVolatile This is the first sweep of generic code to add isAtomic bailouts where appropriate. The intention here is to have the switch from AtomicSDNode to LoadSDNode/StoreSDNode be close to NFC; that is, I'm not looking to allow additional optimizations at this time. That will come later. See D66309 for context. Differential Revision: https://reviews.llvm.org/D66318 llvm-svn: 371786
2019-09-13 06:49:17 +08:00
; CHECK-LABEL: store_forward:
; CHECK: # %bb.0:
; CHECK-NEXT: movq %rsi, (%rdi)
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: retq
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
store atomic i64 %v, i64* %p unordered, align 8
%ret = load atomic i64, i64* %p unordered, align 8
ret i64 %ret
}
; Legal to kill (TODO)
define void @dead_writeback(i64* %p) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: dead_writeback:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: movq %rax, (%rdi)
; CHECK-NEXT: retq
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
%v = load atomic i64, i64* %p unordered, align 8
store atomic i64 %v, i64* %p unordered, align 8
ret void
}
; Legal to kill (TODO)
define void @dead_store(i64* %p, i64 %v) {
[SDAG] Update generic code to conservatively check for isAtomic in addition to isVolatile This is the first sweep of generic code to add isAtomic bailouts where appropriate. The intention here is to have the switch from AtomicSDNode to LoadSDNode/StoreSDNode be close to NFC; that is, I'm not looking to allow additional optimizations at this time. That will come later. See D66309 for context. Differential Revision: https://reviews.llvm.org/D66318 llvm-svn: 371786
2019-09-13 06:49:17 +08:00
; CHECK-LABEL: dead_store:
; CHECK: # %bb.0:
; CHECK-NEXT: movq $0, (%rdi)
; CHECK-NEXT: movq %rsi, (%rdi)
; CHECK-NEXT: retq
[Tests] More unordered atomic lowering tests This time, focused around narrowing and widening transformations. Also, include a few simple memory optimization tests to highlight missed oppurtunities. This is part of building up the test base for D57601. llvm-svn: 353972
2019-02-14 03:49:17 +08:00
store atomic i64 0, i64* %p unordered, align 8
store atomic i64 %v, i64* %p unordered, align 8
ret void
}
[Tests] Add negative folding tests w/fences as requested in D59345 llvm-svn: 356165
2019-03-15 01:05:18 +08:00
;; The next batch of tests ensure that we don't try to fold a load into a
;; use where the code motion implied for the load is prevented by a fence.
;; Note: We're checking that the load doesn't get moved below the fence as
;; part of folding, but is technically legal to lift the add above the fence.
;; If that were to happen, please rewrite the test to ensure load movement
;; isn't violated.
define i64 @nofold_fence(i64* %p) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: nofold_fence:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: mfence
; CHECK-NEXT: addq $15, %rax
; CHECK-NEXT: retq
[Tests] Add negative folding tests w/fences as requested in D59345 llvm-svn: 356165
2019-03-15 01:05:18 +08:00
%v = load atomic i64, i64* %p unordered, align 8
fence seq_cst
%ret = add i64 %v, 15
ret i64 %ret
}
define i64 @nofold_fence_acquire(i64* %p) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: nofold_fence_acquire:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: #MEMBARRIER
; CHECK-NEXT: addq $15, %rax
; CHECK-NEXT: retq
[Tests] Add negative folding tests w/fences as requested in D59345 llvm-svn: 356165
2019-03-15 01:05:18 +08:00
%v = load atomic i64, i64* %p unordered, align 8
fence acquire
%ret = add i64 %v, 15
ret i64 %ret
}
define i64 @nofold_stfence(i64* %p) {
Use the handle --check-prefixes mechanism to de-verbosify a couple atomics tests [NFC] llvm-svn: 370256
2019-08-29 04:27:39 +08:00
; CHECK-LABEL: nofold_stfence:
; CHECK: # %bb.0:
; CHECK-NEXT: movq (%rdi), %rax
; CHECK-NEXT: #MEMBARRIER
; CHECK-NEXT: addq $15, %rax
; CHECK-NEXT: retq
[Tests] Add negative folding tests w/fences as requested in D59345 llvm-svn: 356165
2019-03-15 01:05:18 +08:00
%v = load atomic i64, i64* %p unordered, align 8
fence syncscope("singlethread") seq_cst
%ret = add i64 %v, 15
ret i64 %ret
}
[Tests] Add tests for reordering of unordered atomics on invariant locations llvm-svn: 356172
2019-03-15 01:36:58 +08:00
;; Next, test how well we can fold invariant loads.
@Constant = external constant i64
define i64 @fold_constant(i64 %arg) {
; CHECK-O0-LABEL: fold_constant:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: addq Constant, %rdi
; CHECK-O0-NEXT: movq %rdi, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: fold_constant:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movq %rdi, %rax
; CHECK-O3-NEXT: addq {{.*}}(%rip), %rax
; CHECK-O3-NEXT: retq
%v = load atomic i64, i64* @Constant unordered, align 8
%ret = add i64 %v, %arg
ret i64 %ret
}
define i64 @fold_constant_clobber(i64* %p, i64 %arg) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-LABEL: fold_constant_clobber:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq {{.*}}(%rip), %rax
; CHECK-O0-NEXT: movq $5, (%rdi)
; CHECK-O0-NEXT: addq %rsi, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-CUR-LABEL: fold_constant_clobber:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq {{.*}}(%rip), %rax
; CHECK-O3-CUR-NEXT: movq $5, (%rdi)
; CHECK-O3-CUR-NEXT: addq %rsi, %rax
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O3-EX-LABEL: fold_constant_clobber:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq %rsi, %rax
; CHECK-O3-EX-NEXT: addq {{.*}}(%rip), %rax
Fix a *nasty* miscompile in experimental unordered atomic lowering This is an omission in rL371441. Loads which happened to be unordered weren't being added to the PendingLoad set, and thus weren't be ordered w/respect to side effects which followed before the end of the block. Included test case is how I spotted this. We had an atomic load being folded into a using instruction after a fence that load was supposed to be ordered with. I'm sure it showed up a bunch of other ways as well. Spotted via manual inspecting of assembly differences in a corpus w/and w/o the new experimental mode. Finding this with testing would have been "unpleasant". llvm-svn: 373814
2019-10-05 08:32:10 +08:00
; CHECK-O3-EX-NEXT: movq $5, (%rdi)
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-NEXT: retq
[Tests] Add tests for reordering of unordered atomics on invariant locations llvm-svn: 356172
2019-03-15 01:36:58 +08:00
%v = load atomic i64, i64* @Constant unordered, align 8
store i64 5, i64* %p
%ret = add i64 %v, %arg
ret i64 %ret
}
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
[Tests] Add tests for reordering of unordered atomics on invariant locations llvm-svn: 356172
2019-03-15 01:36:58 +08:00
define i64 @fold_constant_fence(i64 %arg) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-LABEL: fold_constant_fence:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq {{.*}}(%rip), %rax
; CHECK-O0-NEXT: mfence
; CHECK-O0-NEXT: addq %rdi, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-CUR-LABEL: fold_constant_fence:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq {{.*}}(%rip), %rax
; CHECK-O3-CUR-NEXT: mfence
; CHECK-O3-CUR-NEXT: addq %rdi, %rax
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O3-EX-LABEL: fold_constant_fence:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq %rdi, %rax
; CHECK-O3-EX-NEXT: addq {{.*}}(%rip), %rax
Fix a *nasty* miscompile in experimental unordered atomic lowering This is an omission in rL371441. Loads which happened to be unordered weren't being added to the PendingLoad set, and thus weren't be ordered w/respect to side effects which followed before the end of the block. Included test case is how I spotted this. We had an atomic load being folded into a using instruction after a fence that load was supposed to be ordered with. I'm sure it showed up a bunch of other ways as well. Spotted via manual inspecting of assembly differences in a corpus w/and w/o the new experimental mode. Finding this with testing would have been "unpleasant". llvm-svn: 373814
2019-10-05 08:32:10 +08:00
; CHECK-O3-EX-NEXT: mfence
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-NEXT: retq
[Tests] Add tests for reordering of unordered atomics on invariant locations llvm-svn: 356172
2019-03-15 01:36:58 +08:00
%v = load atomic i64, i64* @Constant unordered, align 8
fence seq_cst
%ret = add i64 %v, %arg
ret i64 %ret
}
Allow unordered loads to be considered invariant in CodeGen The actual code change is fairly straight forward, but exercising it isn't. First, it turned out we weren't adding the appropriate flags in SelectionDAG. Second, it turned out that we've got some optimization gaps, so obvious test cases don't work. My first attempt (in atomic-unordered.ll) points out a deficiency in our peephole-opt folding logic which I plan to fix separately. Instead, I'm exercising this through MachineLICM. Differential Revision: https://reviews.llvm.org/D59375 llvm-svn: 356494
2019-03-20 02:27:18 +08:00
define i64 @fold_invariant_clobber(i64* dereferenceable(8) %p, i64 %arg) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-LABEL: fold_invariant_clobber:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: movq $5, (%rdi)
; CHECK-O0-NEXT: addq %rsi, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-CUR-LABEL: fold_invariant_clobber:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rdi), %rax
; CHECK-O3-CUR-NEXT: movq $5, (%rdi)
; CHECK-O3-CUR-NEXT: addq %rsi, %rax
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O3-EX-LABEL: fold_invariant_clobber:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq %rsi, %rax
; CHECK-O3-EX-NEXT: addq (%rdi), %rax
Fix a *nasty* miscompile in experimental unordered atomic lowering This is an omission in rL371441. Loads which happened to be unordered weren't being added to the PendingLoad set, and thus weren't be ordered w/respect to side effects which followed before the end of the block. Included test case is how I spotted this. We had an atomic load being folded into a using instruction after a fence that load was supposed to be ordered with. I'm sure it showed up a bunch of other ways as well. Spotted via manual inspecting of assembly differences in a corpus w/and w/o the new experimental mode. Finding this with testing would have been "unpleasant". llvm-svn: 373814
2019-10-05 08:32:10 +08:00
; CHECK-O3-EX-NEXT: movq $5, (%rdi)
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-NEXT: retq
[Tests] Add tests for reordering of unordered atomics on invariant locations llvm-svn: 356172
2019-03-15 01:36:58 +08:00
%v = load atomic i64, i64* %p unordered, align 8, !invariant.load !{}
store i64 5, i64* %p
%ret = add i64 %v, %arg
ret i64 %ret
}
Allow unordered loads to be considered invariant in CodeGen The actual code change is fairly straight forward, but exercising it isn't. First, it turned out we weren't adding the appropriate flags in SelectionDAG. Second, it turned out that we've got some optimization gaps, so obvious test cases don't work. My first attempt (in atomic-unordered.ll) points out a deficiency in our peephole-opt folding logic which I plan to fix separately. Instead, I'm exercising this through MachineLICM. Differential Revision: https://reviews.llvm.org/D59375 llvm-svn: 356494
2019-03-20 02:27:18 +08:00
define i64 @fold_invariant_fence(i64* dereferenceable(8) %p, i64 %arg) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-LABEL: fold_invariant_fence:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movq (%rdi), %rax
; CHECK-O0-NEXT: mfence
; CHECK-O0-NEXT: addq %rsi, %rax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-CUR-LABEL: fold_invariant_fence:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movq (%rdi), %rax
; CHECK-O3-CUR-NEXT: mfence
; CHECK-O3-CUR-NEXT: addq %rsi, %rax
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O3-EX-LABEL: fold_invariant_fence:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: movq %rsi, %rax
; CHECK-O3-EX-NEXT: addq (%rdi), %rax
Fix a *nasty* miscompile in experimental unordered atomic lowering This is an omission in rL371441. Loads which happened to be unordered weren't being added to the PendingLoad set, and thus weren't be ordered w/respect to side effects which followed before the end of the block. Included test case is how I spotted this. We had an atomic load being folded into a using instruction after a fence that load was supposed to be ordered with. I'm sure it showed up a bunch of other ways as well. Spotted via manual inspecting of assembly differences in a corpus w/and w/o the new experimental mode. Finding this with testing would have been "unpleasant". llvm-svn: 373814
2019-10-05 08:32:10 +08:00
; CHECK-O3-EX-NEXT: mfence
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O3-EX-NEXT: retq
[Tests] Add tests for reordering of unordered atomics on invariant locations llvm-svn: 356172
2019-03-15 01:36:58 +08:00
%v = load atomic i64, i64* %p unordered, align 8, !invariant.load !{}
fence seq_cst
%ret = add i64 %v, %arg
ret i64 %ret
}
[Tests] RMW folding tests w/unordered atomic operations We get a suprising number of these today actually, but some are missed. The main point of this is strengthen the test set for D57601. llvm-svn: 353966
2019-02-14 02:41:54 +08:00
[Tests] Add anyextend tests for unordered atomics Motivated by work on changing our representation of unordered atomics in SelectionDAG, but as an aside, all our lowerings for O3 are terrible. Even the ones which ignore the atomicity. llvm-svn: 371449
2019-09-10 04:26:52 +08:00
; Exercise a few cases involving any extend idioms
define i16 @load_i8_anyext_i16(i8* %ptr) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-CUR-LABEL: load_i8_anyext_i16:
; CHECK-O0-CUR: # %bb.0:
; CHECK-O0-CUR-NEXT: movb (%rdi), %al
; CHECK-O0-CUR-NEXT: movzbl %al, %ecx
; CHECK-O0-CUR-NEXT: # kill: def $cx killed $cx killed $ecx
; CHECK-O0-CUR-NEXT: movw %cx, %ax
; CHECK-O0-CUR-NEXT: retq
;
; CHECK-O3-CUR-LABEL: load_i8_anyext_i16:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movzbl (%rdi), %eax
; CHECK-O3-CUR-NEXT: # kill: def $ax killed $ax killed $eax
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O0-EX-LABEL: load_i8_anyext_i16:
; CHECK-O0-EX: # %bb.0:
; CHECK-O0-EX-NEXT: vpbroadcastb (%rdi), %xmm0
; CHECK-O0-EX-NEXT: vmovd %xmm0, %eax
; CHECK-O0-EX-NEXT: # kill: def $ax killed $ax killed $eax
; CHECK-O0-EX-NEXT: retq
;
; CHECK-O3-EX-LABEL: load_i8_anyext_i16:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: vpbroadcastb (%rdi), %xmm0
; CHECK-O3-EX-NEXT: vmovd %xmm0, %eax
; CHECK-O3-EX-NEXT: # kill: def $ax killed $ax killed $eax
; CHECK-O3-EX-NEXT: retq
[Tests] Add anyextend tests for unordered atomics Motivated by work on changing our representation of unordered atomics in SelectionDAG, but as an aside, all our lowerings for O3 are terrible. Even the ones which ignore the atomicity. llvm-svn: 371449
2019-09-10 04:26:52 +08:00
%v = load atomic i8, i8* %ptr unordered, align 2
%vec = insertelement <2 x i8> undef, i8 %v, i32 0
%res = bitcast <2 x i8> %vec to i16
ret i16 %res
}
define i32 @load_i8_anyext_i32(i8* %ptr) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-CUR-LABEL: load_i8_anyext_i32:
; CHECK-O0-CUR: # %bb.0:
; CHECK-O0-CUR-NEXT: movb (%rdi), %al
; CHECK-O0-CUR-NEXT: movzbl %al, %eax
; CHECK-O0-CUR-NEXT: retq
;
; CHECK-O3-CUR-LABEL: load_i8_anyext_i32:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movzbl (%rdi), %eax
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O0-EX-LABEL: load_i8_anyext_i32:
; CHECK-O0-EX: # %bb.0:
; CHECK-O0-EX-NEXT: vpbroadcastb (%rdi), %xmm0
; CHECK-O0-EX-NEXT: vmovd %xmm0, %eax
; CHECK-O0-EX-NEXT: retq
;
; CHECK-O3-EX-LABEL: load_i8_anyext_i32:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: vpbroadcastb (%rdi), %xmm0
; CHECK-O3-EX-NEXT: vmovd %xmm0, %eax
; CHECK-O3-EX-NEXT: retq
[Tests] Add anyextend tests for unordered atomics Motivated by work on changing our representation of unordered atomics in SelectionDAG, but as an aside, all our lowerings for O3 are terrible. Even the ones which ignore the atomicity. llvm-svn: 371449
2019-09-10 04:26:52 +08:00
%v = load atomic i8, i8* %ptr unordered, align 4
%vec = insertelement <4 x i8> undef, i8 %v, i32 0
%res = bitcast <4 x i8> %vec to i32
ret i32 %res
}
define i32 @load_i16_anyext_i32(i16* %ptr) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-CUR-LABEL: load_i16_anyext_i32:
; CHECK-O0-CUR: # %bb.0:
; CHECK-O0-CUR-NEXT: movw (%rdi), %ax
; CHECK-O0-CUR-NEXT: # implicit-def: $ecx
; CHECK-O0-CUR-NEXT: movw %ax, %cx
; CHECK-O0-CUR-NEXT: movl %ecx, %eax
; CHECK-O0-CUR-NEXT: retq
;
; CHECK-O3-CUR-LABEL: load_i16_anyext_i32:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movzwl (%rdi), %eax
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O0-EX-LABEL: load_i16_anyext_i32:
; CHECK-O0-EX: # %bb.0:
; CHECK-O0-EX-NEXT: vpbroadcastw (%rdi), %xmm0
; CHECK-O0-EX-NEXT: vmovd %xmm0, %eax
; CHECK-O0-EX-NEXT: retq
;
; CHECK-O3-EX-LABEL: load_i16_anyext_i32:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: vpbroadcastw (%rdi), %xmm0
; CHECK-O3-EX-NEXT: vmovd %xmm0, %eax
; CHECK-O3-EX-NEXT: retq
[Tests] Add anyextend tests for unordered atomics Motivated by work on changing our representation of unordered atomics in SelectionDAG, but as an aside, all our lowerings for O3 are terrible. Even the ones which ignore the atomicity. llvm-svn: 371449
2019-09-10 04:26:52 +08:00
%v = load atomic i16, i16* %ptr unordered, align 4
%vec = insertelement <2 x i16> undef, i16 %v, i64 0
%res = bitcast <2 x i16> %vec to i32
ret i32 %res
}
define i64 @load_i16_anyext_i64(i16* %ptr) {
[Test] Restructure check lines to show differences between modes more clearly With the landing of the previous patch (in particular D66318) there are a lot fewer diffs now. I added an experimental O0 line, and updated all the tests to group experimental and non-experimental O0/O3 together. Skimming the remaining diffs, there's only a few which are obviously incorrect. There's a large number which are questionable, so more todo. llvm-svn: 371790
2019-09-13 07:22:37 +08:00
; CHECK-O0-CUR-LABEL: load_i16_anyext_i64:
; CHECK-O0-CUR: # %bb.0:
; CHECK-O0-CUR-NEXT: movw (%rdi), %ax
; CHECK-O0-CUR-NEXT: # implicit-def: $ecx
; CHECK-O0-CUR-NEXT: movw %ax, %cx
; CHECK-O0-CUR-NEXT: vmovd %ecx, %xmm0
; CHECK-O0-CUR-NEXT: vmovq %xmm0, %rax
; CHECK-O0-CUR-NEXT: retq
;
; CHECK-O3-CUR-LABEL: load_i16_anyext_i64:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movzwl (%rdi), %eax
; CHECK-O3-CUR-NEXT: vmovd %eax, %xmm0
; CHECK-O3-CUR-NEXT: vmovq %xmm0, %rax
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O0-EX-LABEL: load_i16_anyext_i64:
; CHECK-O0-EX: # %bb.0:
; CHECK-O0-EX-NEXT: vpbroadcastw (%rdi), %xmm0
; CHECK-O0-EX-NEXT: vmovq %xmm0, %rax
; CHECK-O0-EX-NEXT: retq
;
; CHECK-O3-EX-LABEL: load_i16_anyext_i64:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: vpbroadcastw (%rdi), %xmm0
; CHECK-O3-EX-NEXT: vmovq %xmm0, %rax
; CHECK-O3-EX-NEXT: retq
[Tests] Add anyextend tests for unordered atomics Motivated by work on changing our representation of unordered atomics in SelectionDAG, but as an aside, all our lowerings for O3 are terrible. Even the ones which ignore the atomicity. llvm-svn: 371449
2019-09-10 04:26:52 +08:00
%v = load atomic i16, i16* %ptr unordered, align 8
%vec = insertelement <4 x i16> undef, i16 %v, i64 0
%res = bitcast <4 x i16> %vec to i64
ret i64 %res
}
[Tests] Add a unordered atomic load combine test llvm-svn: 373659
2019-10-04 04:28:59 +08:00
; TODO: Would be legal to combine for legal atomic wider types
define i16 @load_combine(i8* %p) {
; CHECK-O0-LABEL: load_combine:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movb (%rdi), %al
; CHECK-O0-NEXT: movb 1(%rdi), %cl
; CHECK-O0-NEXT: movzbl %al, %edx
; CHECK-O0-NEXT: # kill: def $dx killed $dx killed $edx
; CHECK-O0-NEXT: movzbl %cl, %esi
; CHECK-O0-NEXT: # kill: def $si killed $si killed $esi
; CHECK-O0-NEXT: shlw $8, %si
; CHECK-O0-NEXT: orw %si, %dx
; CHECK-O0-NEXT: movw %dx, %ax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-LABEL: load_combine:
; CHECK-O3: # %bb.0:
; CHECK-O3-NEXT: movzbl (%rdi), %ecx
; CHECK-O3-NEXT: movzbl 1(%rdi), %eax
; CHECK-O3-NEXT: shll $8, %eax
; CHECK-O3-NEXT: orl %ecx, %eax
; CHECK-O3-NEXT: # kill: def $ax killed $ax killed $eax
; CHECK-O3-NEXT: retq
%v1 = load atomic i8, i8* %p unordered, align 2
%p2 = getelementptr i8, i8* %p, i64 1
%v2 = load atomic i8, i8* %p2 unordered, align 1
%v1.ext = zext i8 %v1 to i16
%v2.ext = zext i8 %v2 to i16
%v2.sht = shl i16 %v2.ext, 8
%res = or i16 %v1.ext, %v2.sht
ret i16 %res
}
Fix a *nasty* miscompile in experimental unordered atomic lowering This is an omission in rL371441. Loads which happened to be unordered weren't being added to the PendingLoad set, and thus weren't be ordered w/respect to side effects which followed before the end of the block. Included test case is how I spotted this. We had an atomic load being folded into a using instruction after a fence that load was supposed to be ordered with. I'm sure it showed up a bunch of other ways as well. Spotted via manual inspecting of assembly differences in a corpus w/and w/o the new experimental mode. Finding this with testing would have been "unpleasant". llvm-svn: 373814
2019-10-05 08:32:10 +08:00
define i1 @fold_cmp_over_fence(i32* %p, i32 %v1) {
; CHECK-O0-LABEL: fold_cmp_over_fence:
; CHECK-O0: # %bb.0:
; CHECK-O0-NEXT: movl (%rdi), %eax
; CHECK-O0-NEXT: mfence
; CHECK-O0-NEXT: cmpl %eax, %esi
; CHECK-O0-NEXT: jne .LBB116_2
; CHECK-O0-NEXT: # %bb.1: # %taken
; CHECK-O0-NEXT: movb $1, %al
; CHECK-O0-NEXT: retq
; CHECK-O0-NEXT: .LBB116_2: # %untaken
; CHECK-O0-NEXT: xorl %eax, %eax
; CHECK-O0-NEXT: # kill: def $al killed $al killed $eax
; CHECK-O0-NEXT: retq
;
; CHECK-O3-CUR-LABEL: fold_cmp_over_fence:
; CHECK-O3-CUR: # %bb.0:
; CHECK-O3-CUR-NEXT: movl (%rdi), %eax
; CHECK-O3-CUR-NEXT: mfence
; CHECK-O3-CUR-NEXT: cmpl %eax, %esi
; CHECK-O3-CUR-NEXT: jne .LBB116_2
; CHECK-O3-CUR-NEXT: # %bb.1: # %taken
; CHECK-O3-CUR-NEXT: movb $1, %al
; CHECK-O3-CUR-NEXT: retq
; CHECK-O3-CUR-NEXT: .LBB116_2: # %untaken
; CHECK-O3-CUR-NEXT: xorl %eax, %eax
; CHECK-O3-CUR-NEXT: retq
;
; CHECK-O3-EX-LABEL: fold_cmp_over_fence:
; CHECK-O3-EX: # %bb.0:
; CHECK-O3-EX-NEXT: cmpl (%rdi), %esi
; CHECK-O3-EX-NEXT: mfence
; CHECK-O3-EX-NEXT: jne .LBB116_2
; CHECK-O3-EX-NEXT: # %bb.1: # %taken
; CHECK-O3-EX-NEXT: movb $1, %al
; CHECK-O3-EX-NEXT: retq
; CHECK-O3-EX-NEXT: .LBB116_2: # %untaken
; CHECK-O3-EX-NEXT: xorl %eax, %eax
; CHECK-O3-EX-NEXT: retq
%v2 = load atomic i32, i32* %p unordered, align 4
fence seq_cst
%cmp = icmp eq i32 %v1, %v2
br i1 %cmp, label %taken, label %untaken
taken:
ret i1 true
untaken:
ret i1 false
}