Alexey Bataev 0d74fd3fdf [SLP][COST][X86]Improve cost model for masked gather. ... Revived D101297 in its original form + added some changes in X86 legalization cehcking for masked gathers. This solution is the most stable and the most correct one. We have to check the legality before trying to build the masked gather in SLP. Without this check we have incorrect cost (for SLP) in case if the masked gather is not legal/slower than the gather. And we're missing some vectorization opportunities. This can be fixed in the cost model, but in this case we need to add special checks for the cost of GEPs for ScatterVectorize node, add special check for small trees, etc., i.e. there are a lot of corner cases here and there, which insrease code base and make it harder to maintain the code. > Can't we rely on cost model to deal with this? This can be profitable for futher vectorization, when we can start from such gather loads as seed. The question from D101297. Actually, no, it can't. Actually, simple gather may give us better result, especially after we started vectorization of insertelements. Plus, like I said before, the cost for non-legal masked gathers leads to missed vectorization opportunities. Differential Revision: https://reviews.llvm.org/D105042		2021-07-08 11:53:30 -07:00
..
AArch64	[LV] Prevent vectorization with unsupported element types.	2021-07-06 13:06:21 +01:00
AMDGPU	[LV] Update more target-specific tests after 23c2f2e6b2.	2021-06-07 12:13:21 +01:00
ARM	[LV] Unconditionally branch from middle to scalar preheader if the scalar loop must execute (try 4)	2021-07-07 07:44:35 -07:00
Hexagon	[LV] Fix crash when computing max VF too early	2021-02-01 12:14:59 +00:00
PowerPC	[LV] Update more target-specific tests after 23c2f2e6b2.	2021-06-07 12:13:21 +01:00
RISCV	[LV] Update more target-specific tests after 23c2f2e6b2.	2021-06-07 12:13:21 +01:00
SystemZ	[SLP,LV] Use poison constant vector for shufflevector/initial insertelement	2021-01-06 11:22:50 +09:00
VE	[VE][TTI] don't advertise vregs/vops	2020-11-06 11:12:10 +01:00
X86	[SLP][COST][X86]Improve cost model for masked gather.	2021-07-08 11:53:30 -07:00
XCore	…
12-12-11-if-conv.ll	…
2012-10-20-infloop.ll	…
2012-10-22-isconsec.ll	…
2016-07-27-loop-vec.ll	…
alias-set-with-uncomputable-bounds.ll	…
align.ll	…
assume.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
bsd_regex.ll	…
bzip_reverse_loops.ll	…
calloc.ll	…
cast-induction.ll	…
check-prof-info.ll	Revert "Use uint64_t for branch weights instead of uint32_t"	2020-10-31 00:25:32 -07:00
conditional-assignment.ll	…
consec_no_gep.ll	…
consecutive-ptr-uniforms.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
control-flow.ll	Reapply "[LV] Vectorize (some) early and multiple exit loops"" w/fix for builder	2020-12-28 10:13:28 -08:00
cpp-new-array.ll	…
dbg.value.ll	…
dead_instructions.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
debugloc.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
demanded-bits-of-pointer-instruction.ll	…
diag-missing-instr-debug-loc.ll	Revert "[NFC] remove explicit default value for strboolattr attribute in tests"	2021-05-24 19:43:40 +02:00
diag-with-hotness-info-2.ll	Revert "[NFC] remove explicit default value for strboolattr attribute in tests"	2021-05-24 19:43:40 +02:00
diag-with-hotness-info.ll	Revert "[NFC] remove explicit default value for strboolattr attribute in tests"	2021-05-24 19:43:40 +02:00
disable_nonforced.ll	…
disable_nonforced_enable.ll	…
discriminator.ll	…
dont-fold-tail-for-const-TC.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
dont-fold-tail-for-divisible-TC.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
ee-crash.ll	…
exact.ll	…
explicit_outer_detection.ll	[LV] Consider Loop Unroll Hints When Making Interleave Decisions	2021-04-28 17:27:52 -04:00
explicit_outer_nonuniform_inner.ll	…
explicit_outer_uniform_diverg_branch.ll	…
extract-last-veclane.ll	[SVE][LoopVectorize] Add support for extracting the last lane of a scalable vector	2021-03-05 09:57:56 +00:00
fcmp-vectorize.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
first-order-recurrence-complex.ll	[LV] Unconditionally branch from middle to scalar preheader if the scalar loop must execute (try 4)	2021-07-07 07:44:35 -07:00
first-order-recurrence-multiply-recurrences.ll	…
first-order-recurrence-sink-replicate-region.ll	Recommit "[VPlan] Add VPReductionPHIRecipe (NFC)." and follow-ups.	2021-07-06 14:15:42 +01:00
first-order-recurrence.ll	[LV] Fix crash when target instruction for sinking is dead.	2021-06-29 13:31:22 +01:00
fix-reduction-dbg.ll	[Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics	2021-02-12 08:13:50 -05:00
flags.ll	[Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics	2021-02-12 08:13:50 -05:00
float-induction.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
float-minmax-instruction-flag.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
float-reduction.ll	Enable support for floating-point division reductions	2020-11-23 20:00:58 -05:00
fneg.ll	…
followup.ll	[LV] Collect dead induction truncates	2020-10-08 08:28:58 +01:00
funcall.ll	…
gcc-examples.ll	…
gep_with_bitcast.ll	…
global_alias.ll	…
hints-trans.ll	[SimplifyCFG] TryToSimplifyUncondBranchFromEmptyBlock() already knows how to preserve DomTree	2020-12-17 01:03:49 +03:00
hoist-loads.ll	…
i8-induction.ll	…
icmp-uniforms.ll	[VPlan] Include name when printing after 93a9d2de8f.	2021-03-23 09:50:14 +00:00
if-conv-crash.ll	…
if-conversion-edgemasks.ll	…
if-conversion-nest.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
if-conversion-reduction.ll	…
if-conversion.ll	…
if-pred-non-void.ll	[VPlan] Merge predicated-triangle regions, after sinking.	2021-06-28 11:10:38 +01:00
if-pred-not-when-safe.ll	[SimplifyCFG] TryToSimplifyUncondBranchFromEmptyBlock() already knows how to preserve DomTree	2020-12-17 01:03:49 +03:00
if-pred-stores.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
if-reduction.ll	[LoopVectorize] Fix VPRecipeBuilder::createEdgeMask to correctly generate the mask	2021-02-14 21:12:34 +09:00
incorrect-dom-info.ll	Revert "[NFC] remove explicit default value for strboolattr attribute in tests"	2021-05-24 19:43:40 +02:00
increment.ll	…
induction-step.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
induction.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
induction_plus.ll	…
infiniteloop.ll	…
int_sideeffect.ll	…
interleaved-accesses-1.ll	[IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder	2020-12-30 04:21:04 +09:00
interleaved-accesses-2.ll	…
interleaved-accesses-3.ll	…
interleaved-accesses-alias.ll	…
interleaved-accesses-masked-group.ll	…
interleaved-accesses-pred-stores.ll	[IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder	2020-12-30 04:21:04 +09:00
interleaved-accesses-uniform-load.ll	…
interleaved-accesses.ll	[LV] Unconditionally branch from middle to scalar preheader if the scalar loop must execute (try 4)	2021-07-07 07:44:35 -07:00
interleaved-acess-with-remarks.ll	…
intrinsic.ll	Update @llvm.powi to handle different int sizes for the exponent	2021-06-17 09:38:28 +02:00
invariant-store-vectorization.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
irregular_type.ll	[LoopVectorize] Refine hasIrregularType predicate	2021-03-17 17:03:47 +01:00
iv_outside_user.ll	…
lcssa-crash.ll	…
libcall-remark.ll	…
lifetime.ll	…
loop-form.ll	[LV] Unconditionally branch from middle to scalar preheader if the scalar loop must execute (try 4)	2021-07-07 07:44:35 -07:00
loop-legality-checks.ll	Reapply "[LV] Vectorize (some) early and multiple exit loops"" w/fix for builder	2020-12-28 10:13:28 -08:00
loop-scalars.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
loop-vect-memdep.ll	…
loop-vect-option.ll	…
memdep-fold-tail.ll	…
memdep.ll	…
metadata-unroll.ll	…
metadata-width.ll	Revert "[NFC] remove explicit default value for strboolattr attribute in tests"	2021-05-24 19:43:40 +02:00
metadata.ll	…
middle-block-dbg.ll	…
miniters.ll	…
minmax_reduction.ll	[LoopVectorizer] Require no-signed-zeros-fp-math=true for fmin/fmax	2021-02-15 13:47:05 +00:00
mixed-precision-remarks.ll	[LV] Add analysis remark for mixed precision conversions	2021-02-17 21:37:08 -05:00
multi-use-reduction-bug.ll	…
multiple-address-spaces.ll	Revert "[NFC] remove explicit default value for strboolattr attribute in tests"	2021-05-24 19:43:40 +02:00
multiple-exits-versioning.ll	[LV] Add test cases with multiple exits which require versioning.	2021-01-20 11:48:48 +00:00
multiple-strides-vectorization.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
no-interleave-up-front.ll	…
no_array_bounds.ll	…
no_array_bounds_scalable.ll	[SVE] Add support for scalable vectors with vectorize.scalable.enable loop attribute	2020-12-02 13:23:43 +00:00
no_idiv_reduction.ll	…
no_int_induction.ll	…
no_outside_user.ll	…
no_switch.ll	…
no_switch_disable_vectorization.ll	…
noalias-md-licm.ll	…
noalias-md.ll	…
noalias-scope-decl.ll	[noalias.decl] Look through llvm.experimental.noalias.scope.decl	2021-01-19 20:09:42 +01:00
nofloat-report.ll	…
nofloat.ll	…
non-const-n.ll	…
nontemporal.ll	…
nounroll.ll	Follow up of rGddb3b26a1269: added 'requires asserts' to test case.	2021-04-29 08:34:24 +01:00
novect-lcssa-cfg-invalidation.ll	…
nsw-crash.ll	…
nuw.ll	…
opt.ll	…
optimal-epilog-vectorization-limitations.ll	[LV] Legalize scalable VF hints	2021-01-08 10:49:44 +00:00
optimal-epilog-vectorization-liveout.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
optimal-epilog-vectorization-scalable.ll	[LV] Add -scalable-vectorization=<option> flag.	2021-05-19 10:40:56 +01:00
optimal-epilog-vectorization.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
optsize.ll	[IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder	2020-12-30 04:21:04 +09:00
outer-loop-vec-phi-predecessor-order.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
outer_loop_test1.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
outer_loop_test2.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
partial-lcssa.ll	…
phi-cost.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
phi-hang.ll	…
pointer-induction.ll	[ScalarEvolution] Ensure backedge-taken counts are not pointers.	2021-06-21 16:24:16 -07:00
pr25281.ll	…
pr28541.ll	…
pr30654-phiscev-sext-trunc.ll	…
pr30806-phi-scev.ll	…
pr30806.ll	…
pr31098.ll	…
pr31190.ll	…
pr32859.ll	…
pr33706.ll	…
pr34681.ll	[test, LoopVectorize] Fix use of var defined in CHECK-NOT	2021-04-09 10:01:57 +01:00
pr35743.ll	…
pr35773.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
pr36311.ll	…
pr36983.ll	…
pr37248.ll	…
pr37515.ll	…
pr38697.ll	[SCEV] Properly guard reasoning about infinite loops being UB on mustprogress	2021-06-07 14:47:36 -07:00
pr38800.ll	…
pr39099.ll	…
pr39417-optsize-scevchecks.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
pr43166-fold-tail-by-masking.ll	…
pr44488-predication.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
pr45259.ll	[ScalarEvolution] Ensure backedge-taken counts are not pointers.	2021-06-21 16:24:16 -07:00
pr45525.ll	…
pr45679-fold-tail-by-masking.ll	…
pr46525-expander-insertpoint.ll	[IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder	2020-12-30 04:21:04 +09:00
pr47343-expander-lcssa-after-cfg-update.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
pr48832.ll	[LoopVectorize] Fix VPRecipeBuilder::createEdgeMask to correctly generate the mask	2021-02-14 21:12:34 +09:00
pr50686.ll	[LoopVectorize] precommit pr50686.ll for D104148	2021-06-14 13:58:25 +08:00
preserve-dbg-loc-and-loop-metadata.ll	…
ptr-induction.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
ptr_loops.ll	…
read-only.ll	…
reduction-inloop-pred.ll	[VPlan] Merge predicated-triangle regions, after sinking.	2021-06-28 11:10:38 +01:00
reduction-inloop-uf4.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
reduction-inloop.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
reduction-order.ll	[LV] Tail folded inloop reductions.	2020-10-11 16:58:34 +01:00
reduction-predselect.ll	[VPlan] Merge predicated-triangle regions, after sinking.	2021-06-28 11:10:38 +01:00
reduction-ptr.ll	[Analysis][LoopVectorize] do not form reductions of pointers	2021-02-19 14:01:57 -05:00
reduction-small-size.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
reduction.ll	[VPlan] Register recipe for instr if the simplified value is recipe.	2021-05-11 14:32:34 +01:00
remarks-multi-exit-loops.ll	[LV] Fix crash when generating remarks with multi-exit loops.	2021-01-01 13:54:41 +00:00
remove_metadata.ll	…
reverse_induction.ll	…
reverse_iter.ll	…
runtime-check-address-space.ll	Revert "[NFC] remove explicit default value for strboolattr attribute in tests"	2021-05-24 19:43:40 +02:00
runtime-check-needed-but-empty.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
runtime-check-pointer-element-type.ll	[LAA] Use DL to get element size for bound computation.	2020-10-07 18:57:07 +01:00
runtime-check-readonly-address-space.ll	Revert "[NFC] remove explicit default value for strboolattr attribute in tests"	2021-05-24 19:43:40 +02:00
runtime-check-readonly.ll	…
runtime-check-small-clamped-bounds.ll	[LV] Extend FIXME in test add in 91ee1e3799.	2021-07-05 15:56:03 +01:00
runtime-check.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
runtime-drop-crash.ll	[LV] Generate RT checks up-front and remove them if required.	2021-03-01 10:48:04 +00:00
safegep.ll	…
same-base-access.ll	…
scalable-first-order-recurrence.ll	[LV] Add -scalable-vectorization=<option> flag.	2021-05-19 10:40:56 +01:00
scalable-loop-unpredicated-body-scalar-tail.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
scalable-reduction-inloop.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
scalable-trunc-min-bitwidth.ll	[SVE] Fix ShuffleVector cast<FixedVectorType> in truncateToMinimalBitwidths	2021-07-07 15:30:10 +01:00
scalable-vf-hint.ll	[LV] Add -scalable-vectorization=<option> flag.	2021-05-19 10:40:56 +01:00
scalar-select.ll	…
scalar_after_vectorization.ll	…
scalarized-bitcast.ll	[LoopVectorize] Simplify scalar cost calculation in getInstructionCost	2021-04-28 13:41:07 +01:00
scev-during-mutation.ll	[LV] Workaround PR49900 (a crash due to analyzing partially mutated IR)	2021-05-05 09:56:28 -07:00
scev-exitlim-crash.ll	Revert "[NFC] remove explicit default value for strboolattr attribute in tests"	2021-05-24 19:43:40 +02:00
select-reduction.ll	[Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics	2021-02-12 08:13:50 -05:00
simple-unroll.ll	…
single-value-blend-phis.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
skeleton-lcssa-crash.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
skip-iterations.ll	…
small-loop.ll	…
start-non-zero.ll	…
store-shuffle-bug.ll	…
struct_access.ll	…
tail-folding-counting-down.ll	…
tail-folding-vectorization-factor-1.ll	…
tbaa-nodep.ll	[SimplifyCFG] TryToSimplifyUncondBranchFromEmptyBlock() already knows how to preserve DomTree	2020-12-17 01:03:49 +03:00
tripcount.ll	Revert "Use uint64_t for branch weights instead of uint32_t"	2020-10-31 00:25:32 -07:00
undef-inst-bug.ll	…
uniform-blend.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
unroll-novec-memcheck-metadata.ll	…
unroll.ll	…
unroll_nonlatch.ll	[LV] Unconditionally branch from middle to scalar preheader if the scalar loop must execute (try 4)	2021-07-07 07:44:35 -07:00
unroll_novec.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
unsafe-dep-remark.ll	…
unsafe-vf-hint-remark.ll	[LV] Clamp VF hint when unsafe	2020-12-01 11:30:34 +00:00
unsized-pointee-crash.ll	…
use-scalar-epilogue-if-tp-fails.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
value-ptr-bug.ll	…
vect-phiscev-sext-trunc.ll	…
vect.omp.persistence.ll	…
vect.stats.ll	…
vector-geps.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
vector-intrinsic-call-cost.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
vectorize-once.ll	[SimplifyCFG] TryToSimplifyUncondBranchFromEmptyBlock() already knows how to preserve DomTree	2020-12-17 01:03:49 +03:00
vectorize-pointer-phis.ll	Revert "[LAA] Support pointer phis in loop by analyzing each incoming pointer."	2021-05-28 10:33:52 +01:00
vectorizeVFone.ll	…
version-mem-access.ll	…
vplan-dot-printing.ll	[VPlan] Add plain text (not DOT's digraph) dumps	2021-03-19 10:50:12 -07:00
vplan-outer-loop-uncomputable-trip-count.ll	…
vplan-printing.ll	Recommit "[VPlan] Add VPReductionPHIRecipe (NFC)." and follow-ups.	2021-07-06 14:15:42 +01:00
vplan-sink-scalars-and-merge-vf1.ll	[VPlan] Do not sink uniform recipes in sinkScalarOperands.	2021-05-27 14:07:48 +01:00
vplan-sink-scalars-and-merge.ll	[VPlan] Merge predicated-triangle regions, after sinking.	2021-06-28 11:10:38 +01:00
vplan-stress-test-no-explict-vf.ll	…
vplan-vectorize-inner-loop-reduction.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
vplan-widen-call-instruction.ll	[LV] Mark increment of main vector loop induction variable as NUW.	2021-06-07 10:47:52 +01:00
vplan-widen-select-instruction.ll	[VPlan] Support to widen select intructions in VPlan native path	2021-03-10 20:59:53 +00:00
vplan_hcfg_stress_test.ll	…
write-only.ll	…
zero-sized-pointee-crash.ll	…