maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/clang/test/CodeGen/complex-math.c

633 lines
19 KiB
C
Raw Normal View History

[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
// RUN: %clang_cc1 %s -O1 -emit-llvm -triple x86_64-unknown-unknown -o - | FileCheck %s --check-prefix=X86
[complex] Use the much more powerful EmitCall routine to call libcalls for complex math. This should fix the windows build bots that started having trouble here and generally fix complex libcall emission on targets which use sret for complex data types. It also makes the code a bit simpler (despite calling into a much more complex bucket of code). llvm-svn: 219565
2014-10-11 17:24:41 +08:00
// RUN: %clang_cc1 %s -O1 -emit-llvm -triple x86_64-pc-win64 -o - | FileCheck %s --check-prefix=X86
// RUN: %clang_cc1 %s -O1 -emit-llvm -triple i686-unknown-unknown -o - | FileCheck %s --check-prefix=X86
complex long double support for PowerPC llvm-svn: 220034
2014-10-17 19:51:19 +08:00
// RUN: %clang_cc1 %s -O1 -emit-llvm -triple powerpc-unknown-unknown -o - | FileCheck %s --check-prefix=PPC
[CodeGen][ARM] ARM runtime helper functions are not always soft-fp Re-commit r309257 with less precise register checks in arm-float-helpers.c test. llvm-svn: 309263
2017-07-27 18:43:53 +08:00
// RUN %clang_cc1 %s -O1 -emit-llvm -triple armv7-none-linux-gnueabi -o - | FileCheck %s --check-prefix=ARM
// RUN: %clang_cc1 %s -O1 -emit-llvm -triple armv7-none-linux-gnueabihf -o - | FileCheck %s --check-prefix=ARMHF
ARMv7k: implement ABI changes for watchOS from standard iOS. llvm-svn: 251710
2015-10-31 00:30:45 +08:00
// RUN: %clang_cc1 %s -O1 -emit-llvm -triple thumbv7k-apple-watchos2.0 -o - -target-abi aapcs16 | FileCheck %s --check-prefix=ARM7K
[Complex] Don't use __div?c3 when building with fast-math. Summary: Plant an inline version of "((ac+bd)/(cc+dd)) + i((bc-ad)/(cc+dd))" instead. Patch by Paul Walker. Reviewed By: hfinkel Differential Revision: https://reviews.llvm.org/D40299 llvm-svn: 321183
2017-12-20 23:50:52 +08:00
// RUN: %clang_cc1 %s -O1 -emit-llvm -triple aarch64-unknown-unknown -ffast-math -o - | FileCheck %s --check-prefix=AARCH64-FASTMATH
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
float _Complex add_float_rr(float a, float b) {
// X86-LABEL: @add_float_rr(
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
float _Complex add_float_cr(float _Complex a, float b) {
// X86-LABEL: @add_float_cr(
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
float _Complex add_float_rc(float a, float _Complex b) {
// X86-LABEL: @add_float_rc(
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
float _Complex add_float_cc(float _Complex a, float _Complex b) {
// X86-LABEL: @add_float_cc(
// X86: fadd
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
float _Complex sub_float_rr(float a, float b) {
// X86-LABEL: @sub_float_rr(
// X86: fsub
// X86-NOT: fsub
// X86: ret
return a - b;
}
float _Complex sub_float_cr(float _Complex a, float b) {
// X86-LABEL: @sub_float_cr(
// X86: fsub
// X86-NOT: fsub
// X86: ret
return a - b;
}
float _Complex sub_float_rc(float a, float _Complex b) {
// X86-LABEL: @sub_float_rc(
// X86: fsub
// X86: fsub float -0.{{0+}}e+00,
// X86-NOT: fsub
// X86: ret
return a - b;
}
float _Complex sub_float_cc(float _Complex a, float _Complex b) {
// X86-LABEL: @sub_float_cc(
// X86: fsub
// X86: fsub
// X86-NOT: fsub
// X86: ret
return a - b;
}
float _Complex mul_float_rr(float a, float b) {
// X86-LABEL: @mul_float_rr(
// X86: fmul
// X86-NOT: fmul
// X86: ret
return a * b;
}
float _Complex mul_float_cr(float _Complex a, float b) {
// X86-LABEL: @mul_float_cr(
// X86: fmul
// X86: fmul
// X86-NOT: fmul
// X86: ret
return a * b;
}
float _Complex mul_float_rc(float a, float _Complex b) {
// X86-LABEL: @mul_float_rc(
// X86: fmul
// X86: fmul
// X86-NOT: fmul
// X86: ret
return a * b;
}
float _Complex mul_float_cc(float _Complex a, float _Complex b) {
// X86-LABEL: @mul_float_cc(
[complex] Teach the complex math IR gen to emit direct math and a NaN-test prior to the call to the library function. This should automatically make fastmath (including just non-NaNs) able to avoid the expensive libcalls and also open the door to more advanced folding in LLVM based on the rules for complex math. Two important notes to remember: first is that this isn't yet a proper limited range mode, it's still just improving the unlimited range mode. Also, it isn't really perfecet w.r.t. what an unlimited range mode should be doing because it isn't quite handling the flags produced by all the operations in the way desirable for that mode, but then neither is compiler-rt's libcall. When the compiler-rt libcall is improved to carefully manage flags, the code emitted here should be improved correspondingly. And it is still a long-term desirable thing to add a limited range mode to Clang that would be able to use direct math without library calls here. Special thanks to Steve Canon for the careful review on this patch and teaching me about these issues. =D Differential Revision: http://reviews.llvm.org/D5756 llvm-svn: 220167
2014-10-20 03:13:49 +08:00
// X86: %[[AC:[^ ]+]] = fmul
// X86: %[[BD:[^ ]+]] = fmul
// X86: %[[AD:[^ ]+]] = fmul
// X86: %[[BC:[^ ]+]] = fmul
// X86: %[[RR:[^ ]+]] = fsub float %[[AC]], %[[BD]]
// X86: %[[RI:[^ ]+]] = fadd float
// X86-DAG: %[[AD]]
// X86-DAG: ,
// X86-DAG: %[[BC]]
// X86: fcmp uno float %[[RR]]
// X86: fcmp uno float %[[RI]]
[complex] Use the much more powerful EmitCall routine to call libcalls for complex math. This should fix the windows build bots that started having trouble here and generally fix complex libcall emission on targets which use sret for complex data types. It also makes the code a bit simpler (despite calling into a much more complex bucket of code). llvm-svn: 219565
2014-10-11 17:24:41 +08:00
// X86: call {{.*}} @__mulsc3(
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
// X86: ret
return a * b;
}
float _Complex div_float_rr(float a, float b) {
// X86-LABEL: @div_float_rr(
// X86: fdiv
// X86-NOT: fdiv
// X86: ret
return a / b;
}
float _Complex div_float_cr(float _Complex a, float b) {
// X86-LABEL: @div_float_cr(
// X86: fdiv
// X86: fdiv
// X86-NOT: fdiv
// X86: ret
return a / b;
}
float _Complex div_float_rc(float a, float _Complex b) {
// X86-LABEL: @div_float_rc(
// X86-NOT: fdiv
[complex] Use the much more powerful EmitCall routine to call libcalls for complex math. This should fix the windows build bots that started having trouble here and generally fix complex libcall emission on targets which use sret for complex data types. It also makes the code a bit simpler (despite calling into a much more complex bucket of code). llvm-svn: 219565
2014-10-11 17:24:41 +08:00
// X86: call {{.*}} @__divsc3(
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
// X86: ret
[Complex] Don't use __div?c3 when building with fast-math. Summary: Plant an inline version of "((ac+bd)/(cc+dd)) + i((bc-ad)/(cc+dd))" instead. Patch by Paul Walker. Reviewed By: hfinkel Differential Revision: https://reviews.llvm.org/D40299 llvm-svn: 321183
2017-12-20 23:50:52 +08:00
// a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
// AARCH64-FASTMATH-LABEL: @div_float_rc(float %a, [2 x float] %b.coerce)
// A = a
// B = 0
// AARCH64-FASTMATH: [[C:%.*]] = extractvalue [2 x float] %b.coerce, 0
// AARCH64-FASTMATH: [[D:%.*]] = extractvalue [2 x float] %b.coerce, 1
//
// AARCH64-FASTMATH: [[AC:%.*]] = fmul fast float [[C]], %a
// BD = 0
// ACpBD = AC
//
// AARCH64-FASTMATH: [[CC:%.*]] = fmul fast float [[C]], [[C]]
// AARCH64-FASTMATH: [[DD:%.*]] = fmul fast float [[D]], [[D]]
// AARCH64-FASTMATH: [[CCpDD:%.*]] = fadd fast float [[CC]], [[DD]]
//
// BC = 0
// AARCH64-FASTMATH: [[AD:%.*]] = fmul fast float [[D]], %a
// AARCH64-FASTMATH: [[BCmAD:%.*]] = fsub fast float -0.000000e+00, [[AD]]
//
// AARCH64-FASTMATH: fdiv fast float [[AC]], [[CCpDD]]
// AARCH64-FASTMATH: fdiv fast float [[BCmAD]], [[CCpDD]]
// AARCH64-FASTMATH: ret
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
return a / b;
}
float _Complex div_float_cc(float _Complex a, float _Complex b) {
// X86-LABEL: @div_float_cc(
// X86-NOT: fdiv
[complex] Use the much more powerful EmitCall routine to call libcalls for complex math. This should fix the windows build bots that started having trouble here and generally fix complex libcall emission on targets which use sret for complex data types. It also makes the code a bit simpler (despite calling into a much more complex bucket of code). llvm-svn: 219565
2014-10-11 17:24:41 +08:00
// X86: call {{.*}} @__divsc3(
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
// X86: ret
[Complex] Don't use __div?c3 when building with fast-math. Summary: Plant an inline version of "((ac+bd)/(cc+dd)) + i((bc-ad)/(cc+dd))" instead. Patch by Paul Walker. Reviewed By: hfinkel Differential Revision: https://reviews.llvm.org/D40299 llvm-svn: 321183
2017-12-20 23:50:52 +08:00
// a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
// AARCH64-FASTMATH-LABEL: @div_float_cc([2 x float] %a.coerce, [2 x float] %b.coerce)
// AARCH64-FASTMATH: [[A:%.*]] = extractvalue [2 x float] %a.coerce, 0
// AARCH64-FASTMATH: [[B:%.*]] = extractvalue [2 x float] %a.coerce, 1
// AARCH64-FASTMATH: [[C:%.*]] = extractvalue [2 x float] %b.coerce, 0
// AARCH64-FASTMATH: [[D:%.*]] = extractvalue [2 x float] %b.coerce, 1
//
// AARCH64-FASTMATH: [[AC:%.*]] = fmul fast float [[C]], [[A]]
// AARCH64-FASTMATH: [[BD:%.*]] = fmul fast float [[D]], [[B]]
// AARCH64-FASTMATH: [[ACpBD:%.*]] = fadd fast float [[AC]], [[BD]]
//
// AARCH64-FASTMATH: [[CC:%.*]] = fmul fast float [[C]], [[C]]
// AARCH64-FASTMATH: [[DD:%.*]] = fmul fast float [[D]], [[D]]
// AARCH64-FASTMATH: [[CCpDD:%.*]] = fadd fast float [[CC]], [[DD]]
//
// AARCH64-FASTMATH: [[BC:%.*]] = fmul fast float [[C]], [[B]]
// AARCH64-FASTMATH: [[AD:%.*]] = fmul fast float [[D]], [[A]]
// AARCH64-FASTMATH: [[BCmAD:%.*]] = fsub fast float [[BC]], [[AD]]
//
// AARCH64-FASTMATH: fdiv fast float [[ACpBD]], [[CCpDD]]
// AARCH64-FASTMATH: fdiv fast float [[BCmAD]], [[CCpDD]]
// AARCH64-FASTMATH: ret
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
return a / b;
}
double _Complex add_double_rr(double a, double b) {
// X86-LABEL: @add_double_rr(
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
double _Complex add_double_cr(double _Complex a, double b) {
// X86-LABEL: @add_double_cr(
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
double _Complex add_double_rc(double a, double _Complex b) {
// X86-LABEL: @add_double_rc(
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
double _Complex add_double_cc(double _Complex a, double _Complex b) {
// X86-LABEL: @add_double_cc(
// X86: fadd
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
double _Complex sub_double_rr(double a, double b) {
// X86-LABEL: @sub_double_rr(
// X86: fsub
// X86-NOT: fsub
// X86: ret
return a - b;
}
double _Complex sub_double_cr(double _Complex a, double b) {
// X86-LABEL: @sub_double_cr(
// X86: fsub
// X86-NOT: fsub
// X86: ret
return a - b;
}
double _Complex sub_double_rc(double a, double _Complex b) {
// X86-LABEL: @sub_double_rc(
// X86: fsub
// X86: fsub double -0.{{0+}}e+00,
// X86-NOT: fsub
// X86: ret
return a - b;
}
double _Complex sub_double_cc(double _Complex a, double _Complex b) {
// X86-LABEL: @sub_double_cc(
// X86: fsub
// X86: fsub
// X86-NOT: fsub
// X86: ret
return a - b;
}
double _Complex mul_double_rr(double a, double b) {
// X86-LABEL: @mul_double_rr(
// X86: fmul
// X86-NOT: fmul
// X86: ret
return a * b;
}
double _Complex mul_double_cr(double _Complex a, double b) {
// X86-LABEL: @mul_double_cr(
// X86: fmul
// X86: fmul
// X86-NOT: fmul
// X86: ret
return a * b;
}
double _Complex mul_double_rc(double a, double _Complex b) {
// X86-LABEL: @mul_double_rc(
// X86: fmul
// X86: fmul
// X86-NOT: fmul
// X86: ret
return a * b;
}
double _Complex mul_double_cc(double _Complex a, double _Complex b) {
// X86-LABEL: @mul_double_cc(
[complex] Teach the complex math IR gen to emit direct math and a NaN-test prior to the call to the library function. This should automatically make fastmath (including just non-NaNs) able to avoid the expensive libcalls and also open the door to more advanced folding in LLVM based on the rules for complex math. Two important notes to remember: first is that this isn't yet a proper limited range mode, it's still just improving the unlimited range mode. Also, it isn't really perfecet w.r.t. what an unlimited range mode should be doing because it isn't quite handling the flags produced by all the operations in the way desirable for that mode, but then neither is compiler-rt's libcall. When the compiler-rt libcall is improved to carefully manage flags, the code emitted here should be improved correspondingly. And it is still a long-term desirable thing to add a limited range mode to Clang that would be able to use direct math without library calls here. Special thanks to Steve Canon for the careful review on this patch and teaching me about these issues. =D Differential Revision: http://reviews.llvm.org/D5756 llvm-svn: 220167
2014-10-20 03:13:49 +08:00
// X86: %[[AC:[^ ]+]] = fmul
// X86: %[[BD:[^ ]+]] = fmul
// X86: %[[AD:[^ ]+]] = fmul
// X86: %[[BC:[^ ]+]] = fmul
// X86: %[[RR:[^ ]+]] = fsub double %[[AC]], %[[BD]]
// X86: %[[RI:[^ ]+]] = fadd double
// X86-DAG: %[[AD]]
// X86-DAG: ,
// X86-DAG: %[[BC]]
// X86: fcmp uno double %[[RR]]
// X86: fcmp uno double %[[RI]]
[complex] Use the much more powerful EmitCall routine to call libcalls for complex math. This should fix the windows build bots that started having trouble here and generally fix complex libcall emission on targets which use sret for complex data types. It also makes the code a bit simpler (despite calling into a much more complex bucket of code). llvm-svn: 219565
2014-10-11 17:24:41 +08:00
// X86: call {{.*}} @__muldc3(
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
// X86: ret
return a * b;
}
double _Complex div_double_rr(double a, double b) {
// X86-LABEL: @div_double_rr(
// X86: fdiv
// X86-NOT: fdiv
// X86: ret
return a / b;
}
double _Complex div_double_cr(double _Complex a, double b) {
// X86-LABEL: @div_double_cr(
// X86: fdiv
// X86: fdiv
// X86-NOT: fdiv
// X86: ret
return a / b;
}
double _Complex div_double_rc(double a, double _Complex b) {
// X86-LABEL: @div_double_rc(
// X86-NOT: fdiv
[complex] Use the much more powerful EmitCall routine to call libcalls for complex math. This should fix the windows build bots that started having trouble here and generally fix complex libcall emission on targets which use sret for complex data types. It also makes the code a bit simpler (despite calling into a much more complex bucket of code). llvm-svn: 219565
2014-10-11 17:24:41 +08:00
// X86: call {{.*}} @__divdc3(
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
// X86: ret
[Complex] Don't use __div?c3 when building with fast-math. Summary: Plant an inline version of "((ac+bd)/(cc+dd)) + i((bc-ad)/(cc+dd))" instead. Patch by Paul Walker. Reviewed By: hfinkel Differential Revision: https://reviews.llvm.org/D40299 llvm-svn: 321183
2017-12-20 23:50:52 +08:00
// a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
// AARCH64-FASTMATH-LABEL: @div_double_rc(double %a, [2 x double] %b.coerce)
// A = a
// B = 0
// AARCH64-FASTMATH: [[C:%.*]] = extractvalue [2 x double] %b.coerce, 0
// AARCH64-FASTMATH: [[D:%.*]] = extractvalue [2 x double] %b.coerce, 1
//
// AARCH64-FASTMATH: [[AC:%.*]] = fmul fast double [[C]], %a
// BD = 0
// ACpBD = AC
//
// AARCH64-FASTMATH: [[CC:%.*]] = fmul fast double [[C]], [[C]]
// AARCH64-FASTMATH: [[DD:%.*]] = fmul fast double [[D]], [[D]]
// AARCH64-FASTMATH: [[CCpDD:%.*]] = fadd fast double [[CC]], [[DD]]
//
// BC = 0
// AARCH64-FASTMATH: [[AD:%.*]] = fmul fast double [[D]], %a
// AARCH64-FASTMATH: [[BCmAD:%.*]] = fsub fast double -0.000000e+00, [[AD]]
//
// AARCH64-FASTMATH: fdiv fast double [[AC]], [[CCpDD]]
// AARCH64-FASTMATH: fdiv fast double [[BCmAD]], [[CCpDD]]
// AARCH64-FASTMATH: ret
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
return a / b;
}
double _Complex div_double_cc(double _Complex a, double _Complex b) {
// X86-LABEL: @div_double_cc(
// X86-NOT: fdiv
[complex] Use the much more powerful EmitCall routine to call libcalls for complex math. This should fix the windows build bots that started having trouble here and generally fix complex libcall emission on targets which use sret for complex data types. It also makes the code a bit simpler (despite calling into a much more complex bucket of code). llvm-svn: 219565
2014-10-11 17:24:41 +08:00
// X86: call {{.*}} @__divdc3(
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
// X86: ret
[Complex] Don't use __div?c3 when building with fast-math. Summary: Plant an inline version of "((ac+bd)/(cc+dd)) + i((bc-ad)/(cc+dd))" instead. Patch by Paul Walker. Reviewed By: hfinkel Differential Revision: https://reviews.llvm.org/D40299 llvm-svn: 321183
2017-12-20 23:50:52 +08:00
// a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
// AARCH64-FASTMATH-LABEL: @div_double_cc([2 x double] %a.coerce, [2 x double] %b.coerce)
// AARCH64-FASTMATH: [[A:%.*]] = extractvalue [2 x double] %a.coerce, 0
// AARCH64-FASTMATH: [[B:%.*]] = extractvalue [2 x double] %a.coerce, 1
// AARCH64-FASTMATH: [[C:%.*]] = extractvalue [2 x double] %b.coerce, 0
// AARCH64-FASTMATH: [[D:%.*]] = extractvalue [2 x double] %b.coerce, 1
//
// AARCH64-FASTMATH: [[AC:%.*]] = fmul fast double [[C]], [[A]]
// AARCH64-FASTMATH: [[BD:%.*]] = fmul fast double [[D]], [[B]]
// AARCH64-FASTMATH: [[ACpBD:%.*]] = fadd fast double [[AC]], [[BD]]
//
// AARCH64-FASTMATH: [[CC:%.*]] = fmul fast double [[C]], [[C]]
// AARCH64-FASTMATH: [[DD:%.*]] = fmul fast double [[D]], [[D]]
// AARCH64-FASTMATH: [[CCpDD:%.*]] = fadd fast double [[CC]], [[DD]]
//
// AARCH64-FASTMATH: [[BC:%.*]] = fmul fast double [[C]], [[B]]
// AARCH64-FASTMATH: [[AD:%.*]] = fmul fast double [[D]], [[A]]
// AARCH64-FASTMATH: [[BCmAD:%.*]] = fsub fast double [[BC]], [[AD]]
//
// AARCH64-FASTMATH: fdiv fast double [[ACpBD]], [[CCpDD]]
// AARCH64-FASTMATH: fdiv fast double [[BCmAD]], [[CCpDD]]
// AARCH64-FASTMATH: ret
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
return a / b;
}
long double _Complex add_long_double_rr(long double a, long double b) {
// X86-LABEL: @add_long_double_rr(
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
long double _Complex add_long_double_cr(long double _Complex a, long double b) {
// X86-LABEL: @add_long_double_cr(
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
long double _Complex add_long_double_rc(long double a, long double _Complex b) {
// X86-LABEL: @add_long_double_rc(
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
long double _Complex add_long_double_cc(long double _Complex a, long double _Complex b) {
// X86-LABEL: @add_long_double_cc(
// X86: fadd
// X86: fadd
// X86-NOT: fadd
// X86: ret
return a + b;
}
long double _Complex sub_long_double_rr(long double a, long double b) {
// X86-LABEL: @sub_long_double_rr(
// X86: fsub
// X86-NOT: fsub
// X86: ret
return a - b;
}
long double _Complex sub_long_double_cr(long double _Complex a, long double b) {
// X86-LABEL: @sub_long_double_cr(
// X86: fsub
// X86-NOT: fsub
// X86: ret
return a - b;
}
long double _Complex sub_long_double_rc(long double a, long double _Complex b) {
// X86-LABEL: @sub_long_double_rc(
// X86: fsub
// X86: fsub x86_fp80 0xK8{{0+}},
// X86-NOT: fsub
// X86: ret
return a - b;
}
long double _Complex sub_long_double_cc(long double _Complex a, long double _Complex b) {
// X86-LABEL: @sub_long_double_cc(
// X86: fsub
// X86: fsub
// X86-NOT: fsub
// X86: ret
return a - b;
}
long double _Complex mul_long_double_rr(long double a, long double b) {
// X86-LABEL: @mul_long_double_rr(
// X86: fmul
// X86-NOT: fmul
// X86: ret
return a * b;
}
long double _Complex mul_long_double_cr(long double _Complex a, long double b) {
// X86-LABEL: @mul_long_double_cr(
// X86: fmul
// X86: fmul
// X86-NOT: fmul
// X86: ret
return a * b;
}
long double _Complex mul_long_double_rc(long double a, long double _Complex b) {
// X86-LABEL: @mul_long_double_rc(
// X86: fmul
// X86: fmul
// X86-NOT: fmul
// X86: ret
return a * b;
}
long double _Complex mul_long_double_cc(long double _Complex a, long double _Complex b) {
// X86-LABEL: @mul_long_double_cc(
[complex] Teach the complex math IR gen to emit direct math and a NaN-test prior to the call to the library function. This should automatically make fastmath (including just non-NaNs) able to avoid the expensive libcalls and also open the door to more advanced folding in LLVM based on the rules for complex math. Two important notes to remember: first is that this isn't yet a proper limited range mode, it's still just improving the unlimited range mode. Also, it isn't really perfecet w.r.t. what an unlimited range mode should be doing because it isn't quite handling the flags produced by all the operations in the way desirable for that mode, but then neither is compiler-rt's libcall. When the compiler-rt libcall is improved to carefully manage flags, the code emitted here should be improved correspondingly. And it is still a long-term desirable thing to add a limited range mode to Clang that would be able to use direct math without library calls here. Special thanks to Steve Canon for the careful review on this patch and teaching me about these issues. =D Differential Revision: http://reviews.llvm.org/D5756 llvm-svn: 220167
2014-10-20 03:13:49 +08:00
// X86: %[[AC:[^ ]+]] = fmul
// X86: %[[BD:[^ ]+]] = fmul
// X86: %[[AD:[^ ]+]] = fmul
// X86: %[[BC:[^ ]+]] = fmul
// X86: %[[RR:[^ ]+]] = fsub x86_fp80 %[[AC]], %[[BD]]
// X86: %[[RI:[^ ]+]] = fadd x86_fp80
// X86-DAG: %[[AD]]
// X86-DAG: ,
// X86-DAG: %[[BC]]
// X86: fcmp uno x86_fp80 %[[RR]]
// X86: fcmp uno x86_fp80 %[[RI]]
[complex] Use the much more powerful EmitCall routine to call libcalls for complex math. This should fix the windows build bots that started having trouble here and generally fix complex libcall emission on targets which use sret for complex data types. It also makes the code a bit simpler (despite calling into a much more complex bucket of code). llvm-svn: 219565
2014-10-11 17:24:41 +08:00
// X86: call {{.*}} @__mulxc3(
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
// X86: ret
complex long double support for PowerPC llvm-svn: 220034
2014-10-17 19:51:19 +08:00
// PPC-LABEL: @mul_long_double_cc(
[complex] Teach the complex math IR gen to emit direct math and a NaN-test prior to the call to the library function. This should automatically make fastmath (including just non-NaNs) able to avoid the expensive libcalls and also open the door to more advanced folding in LLVM based on the rules for complex math. Two important notes to remember: first is that this isn't yet a proper limited range mode, it's still just improving the unlimited range mode. Also, it isn't really perfecet w.r.t. what an unlimited range mode should be doing because it isn't quite handling the flags produced by all the operations in the way desirable for that mode, but then neither is compiler-rt's libcall. When the compiler-rt libcall is improved to carefully manage flags, the code emitted here should be improved correspondingly. And it is still a long-term desirable thing to add a limited range mode to Clang that would be able to use direct math without library calls here. Special thanks to Steve Canon for the careful review on this patch and teaching me about these issues. =D Differential Revision: http://reviews.llvm.org/D5756 llvm-svn: 220167
2014-10-20 03:13:49 +08:00
// PPC: %[[AC:[^ ]+]] = fmul
// PPC: %[[BD:[^ ]+]] = fmul
// PPC: %[[AD:[^ ]+]] = fmul
// PPC: %[[BC:[^ ]+]] = fmul
// PPC: %[[RR:[^ ]+]] = fsub ppc_fp128 %[[AC]], %[[BD]]
// PPC: %[[RI:[^ ]+]] = fadd ppc_fp128
// PPC-DAG: %[[AD]]
// PPC-DAG: ,
// PPC-DAG: %[[BC]]
// PPC: fcmp uno ppc_fp128 %[[RR]]
// PPC: fcmp uno ppc_fp128 %[[RI]]
complex long double support for PowerPC llvm-svn: 220034
2014-10-17 19:51:19 +08:00
// PPC: call {{.*}} @__multc3(
// PPC: ret
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
return a * b;
}
long double _Complex div_long_double_rr(long double a, long double b) {
// X86-LABEL: @div_long_double_rr(
// X86: fdiv
// X86-NOT: fdiv
// X86: ret
return a / b;
}
long double _Complex div_long_double_cr(long double _Complex a, long double b) {
// X86-LABEL: @div_long_double_cr(
// X86: fdiv
// X86: fdiv
// X86-NOT: fdiv
// X86: ret
return a / b;
}
long double _Complex div_long_double_rc(long double a, long double _Complex b) {
// X86-LABEL: @div_long_double_rc(
// X86-NOT: fdiv
[complex] Use the much more powerful EmitCall routine to call libcalls for complex math. This should fix the windows build bots that started having trouble here and generally fix complex libcall emission on targets which use sret for complex data types. It also makes the code a bit simpler (despite calling into a much more complex bucket of code). llvm-svn: 219565
2014-10-11 17:24:41 +08:00
// X86: call {{.*}} @__divxc3(
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
// X86: ret
complex long double support for PowerPC llvm-svn: 220034
2014-10-17 19:51:19 +08:00
// PPC-LABEL: @div_long_double_rc(
// PPC-NOT: fdiv
// PPC: call {{.*}} @__divtc3(
// PPC: ret
[Complex] Don't use __div?c3 when building with fast-math. Summary: Plant an inline version of "((ac+bd)/(cc+dd)) + i((bc-ad)/(cc+dd))" instead. Patch by Paul Walker. Reviewed By: hfinkel Differential Revision: https://reviews.llvm.org/D40299 llvm-svn: 321183
2017-12-20 23:50:52 +08:00
// a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
// AARCH64-FASTMATH-LABEL: @div_long_double_rc(fp128 %a, [2 x fp128] %b.coerce)
// A = a
// B = 0
// AARCH64-FASTMATH: [[C:%.*]] = extractvalue [2 x fp128] %b.coerce, 0
// AARCH64-FASTMATH: [[D:%.*]] = extractvalue [2 x fp128] %b.coerce, 1
//
// AARCH64-FASTMATH: [[AC:%.*]] = fmul fast fp128 [[C]], %a
// BD = 0
// ACpBD = AC
//
// AARCH64-FASTMATH: [[CC:%.*]] = fmul fast fp128 [[C]], [[C]]
// AARCH64-FASTMATH: [[DD:%.*]] = fmul fast fp128 [[D]], [[D]]
// AARCH64-FASTMATH: [[CCpDD:%.*]] = fadd fast fp128 [[CC]], [[DD]]
//
// BC = 0
// AARCH64-FASTMATH: [[AD:%.*]] = fmul fast fp128 [[D]], %a
// AARCH64-FASTMATH: [[BCmAD:%.*]] = fsub fast fp128 0xL00000000000000008000000000000000, [[AD]]
//
// AARCH64-FASTMATH: fdiv fast fp128 [[AC]], [[CCpDD]]
// AARCH64-FASTMATH: fdiv fast fp128 [[BCmAD]], [[CCpDD]]
// AARCH64-FASTMATH: ret
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
return a / b;
}
long double _Complex div_long_double_cc(long double _Complex a, long double _Complex b) {
// X86-LABEL: @div_long_double_cc(
// X86-NOT: fdiv
[complex] Use the much more powerful EmitCall routine to call libcalls for complex math. This should fix the windows build bots that started having trouble here and generally fix complex libcall emission on targets which use sret for complex data types. It also makes the code a bit simpler (despite calling into a much more complex bucket of code). llvm-svn: 219565
2014-10-11 17:24:41 +08:00
// X86: call {{.*}} @__divxc3(
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
// X86: ret
complex long double support for PowerPC llvm-svn: 220034
2014-10-17 19:51:19 +08:00
// PPC-LABEL: @div_long_double_cc(
// PPC-NOT: fdiv
// PPC: call {{.*}} @__divtc3(
// PPC: ret
[Complex] Don't use __div?c3 when building with fast-math. Summary: Plant an inline version of "((ac+bd)/(cc+dd)) + i((bc-ad)/(cc+dd))" instead. Patch by Paul Walker. Reviewed By: hfinkel Differential Revision: https://reviews.llvm.org/D40299 llvm-svn: 321183
2017-12-20 23:50:52 +08:00
// a / b = (A+iB) / (C+iD) = ((AC+BD)/(CC+DD)) + i((BC-AD)/(CC+DD))
// AARCH64-FASTMATH-LABEL: @div_long_double_cc([2 x fp128] %a.coerce, [2 x fp128] %b.coerce)
// AARCH64-FASTMATH: [[A:%.*]] = extractvalue [2 x fp128] %a.coerce, 0
// AARCH64-FASTMATH: [[B:%.*]] = extractvalue [2 x fp128] %a.coerce, 1
// AARCH64-FASTMATH: [[C:%.*]] = extractvalue [2 x fp128] %b.coerce, 0
// AARCH64-FASTMATH: [[D:%.*]] = extractvalue [2 x fp128] %b.coerce, 1
//
// AARCH64-FASTMATH: [[AC:%.*]] = fmul fast fp128 [[C]], [[A]]
// AARCH64-FASTMATH: [[BD:%.*]] = fmul fast fp128 [[D]], [[B]]
// AARCH64-FASTMATH: [[ACpBD:%.*]] = fadd fast fp128 [[AC]], [[BD]]
//
// AARCH64-FASTMATH: [[CC:%.*]] = fmul fast fp128 [[C]], [[C]]
// AARCH64-FASTMATH: [[DD:%.*]] = fmul fast fp128 [[D]], [[D]]
// AARCH64-FASTMATH: [[CCpDD:%.*]] = fadd fast fp128 [[CC]], [[DD]]
//
// AARCH64-FASTMATH: [[BC:%.*]] = fmul fast fp128 [[C]], [[B]]
// AARCH64-FASTMATH: [[AD:%.*]] = fmul fast fp128 [[D]], [[A]]
// AARCH64-FASTMATH: [[BCmAD:%.*]] = fsub fast fp128 [[BC]], [[AD]]
//
// AARCH64-FASTMATH: fdiv fast fp128 [[ACpBD]], [[CCpDD]]
// AARCH64-FASTMATH: fdiv fast fp128 [[BCmAD]], [[CCpDD]]
// AARCH64-FASTMATH: ret
[complex] Teach Clang to preserve different-type operands to arithmetic operators where one type is a C complex type, and to emit both the efficient and correct implementation for complex arithmetic according to C11 Annex G using this extra information. For both multiply and divide the old code was writing a long-hand reduced version of the math without any of the special handling of inf and NaN recommended by the standard here. Instead of putting more complexity here, this change does what GCC does which is to emit a libcall for the fully general case. However, the old code also failed to do the proper minimization of the set of operations when there was a mixed complex and real operation. In those cases, C provides a spec for much more minimal operations that are valid. Clang now emits the exact suggested operations. This change isn't *just* about performance though, without minimizing these operations, we again lose the correct handling of infinities and NaNs. It is critical that this happen in the frontend based on assymetric type operands to complex math operations. The performance implications of this change aren't trivial either. I've run a set of benchmarks in Eigen, an open source mathematics library that makes heavy use of complex. While a few have slowed down due to the libcall being introduce, most sped up and some by a huge amount: up to 100% and 140%. In order to make all of this work, also match the algorithm in the constant evaluator to the one in the runtime library. Currently it is a broken port of the simplifications from C's Annex G to the long-hand formulation of the algorithm. Splitting this patch up is very hard because none of this works without the AST change to preserve non-complex operands. Sorry for the enormous change. Follow-up changes will include support for sinking the libcalls onto cold paths in common cases and fastmath improvements to allow more aggressive backend folding. Differential Revision: http://reviews.llvm.org/D5698 llvm-svn: 219557
2014-10-11 08:57:18 +08:00
return a / b;
}
[complex] Teach the other two binary operators on complex numbers (== and !=) to support mixed complex and real operand types. This requires removing an assert from SemaChecking, and adding support both to the constant evaluator and the code generator to synthesize the imaginary part when needed. This seemed somewhat cleaner than having just the comparison operators force real-to-complex conversions. I've added test cases for these operations. I'm really terrified that there were *no* tests in-tree which exercised this. This turned up when trying to build R after my change to the complex type lowering. llvm-svn: 219570
2014-10-11 19:03:30 +08:00
// Comparison operators don't rely on library calls or have interseting math
// properties, but test that mixed types work correctly here.
_Bool eq_float_cr(float _Complex a, float b) {
// X86-LABEL: @eq_float_cr(
// X86: fcmp oeq
// X86: fcmp oeq
// X86: and i1
// X86: ret
return a == b;
}
_Bool eq_float_rc(float a, float _Complex b) {
// X86-LABEL: @eq_float_rc(
// X86: fcmp oeq
// X86: fcmp oeq
// X86: and i1
// X86: ret
return a == b;
}
_Bool eq_float_cc(float _Complex a, float _Complex b) {
// X86-LABEL: @eq_float_cc(
// X86: fcmp oeq
// X86: fcmp oeq
// X86: and i1
// X86: ret
return a == b;
}
_Bool ne_float_cr(float _Complex a, float b) {
// X86-LABEL: @ne_float_cr(
// X86: fcmp une
// X86: fcmp une
// X86: or i1
// X86: ret
return a != b;
}
_Bool ne_float_rc(float a, float _Complex b) {
// X86-LABEL: @ne_float_rc(
// X86: fcmp une
// X86: fcmp une
// X86: or i1
// X86: ret
return a != b;
}
_Bool ne_float_cc(float _Complex a, float _Complex b) {
// X86-LABEL: @ne_float_cc(
// X86: fcmp une
// X86: fcmp une
// X86: or i1
// X86: ret
return a != b;
}
Fix invalid calling convention used for libcalls on ARM. ARM ABI specifies that all the libcalls use soft FP ABI (even hard FP binaries). These days clang emits _mulsc3 / _muldc3 calls with default (C) calling convention which would be translated into AAPCS_VFP LLVM calling and thus the result of complex multiplication will be bogus. Introduce a way for a target to specify explicitly calling convention for libcalls. Right now this is temporary correctness fix. Ultimately, we'll end with intrinsic for complex multiplication and all calling convention decisions for libcalls will be put into backend. llvm-svn: 223123
2014-12-03 00:04:58 +08:00
// Check that the libcall will obtain proper calling convention on ARM
_Complex double foo(_Complex double a, _Complex double b) {
[CodeGen][ARM] ARM runtime helper functions are not always soft-fp Re-commit r309257 with less precise register checks in arm-float-helpers.c test. llvm-svn: 309263
2017-07-27 18:43:53 +08:00
// These functions are not defined as floating point helper functions in
// Run-time ABI for the ARM architecture document so they must not always
// use the base AAPCS.
Fix invalid calling convention used for libcalls on ARM. ARM ABI specifies that all the libcalls use soft FP ABI (even hard FP binaries). These days clang emits _mulsc3 / _muldc3 calls with default (C) calling convention which would be translated into AAPCS_VFP LLVM calling and thus the result of complex multiplication will be bogus. Introduce a way for a target to specify explicitly calling convention for libcalls. Right now this is temporary correctness fix. Ultimately, we'll end with intrinsic for complex multiplication and all calling convention decisions for libcalls will be put into backend. llvm-svn: 223123
2014-12-03 00:04:58 +08:00
// ARM-LABEL: @foo(
[CodeGen][ARM] ARM runtime helper functions are not always soft-fp Re-commit r309257 with less precise register checks in arm-float-helpers.c test. llvm-svn: 309263
2017-07-27 18:43:53 +08:00
// ARM: call void { double, double } @__muldc3
// ARMHF-LABEL: @foo(
// ARMHF: call { double, double } @__muldc3
ARMv7k: implement ABI changes for watchOS from standard iOS. llvm-svn: 251710
2015-10-31 00:30:45 +08:00
// ARM7K-LABEL: @foo(
// ARM7K: call { double, double } @__muldc3
Fix invalid calling convention used for libcalls on ARM. ARM ABI specifies that all the libcalls use soft FP ABI (even hard FP binaries). These days clang emits _mulsc3 / _muldc3 calls with default (C) calling convention which would be translated into AAPCS_VFP LLVM calling and thus the result of complex multiplication will be bogus. Introduce a way for a target to specify explicitly calling convention for libcalls. Right now this is temporary correctness fix. Ultimately, we'll end with intrinsic for complex multiplication and all calling convention decisions for libcalls will be put into backend. llvm-svn: 223123
2014-12-03 00:04:58 +08:00
return a*b;
}