Repository navigation
Conversation
…d terminating condition An aligned pointer with a power-of-two stride must visit null before it repeats. If it is dereferenced on every iteration, and the alignment is at least the stride, those iterations cannot all execute in a defined program unless null pointer is defined in the address space. Relax the hasNoSelfWrap check in LoopTermFold to recognize pointer IVs with dereferenceable memory accesses dominating the loop latch. This eliminates redundant loop counter IVs when pointer accesses already advance monotonically, such as the inner loop of CoreMark's matrix_mul_matrix_bitextract. Fixes llvm#163757
|
Hello @VedaantS 👋 Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description. Frequently asked questionsHow do I add reviewers? This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically. You can also add reviewers manually using the Reviewers section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using What if there are no comments? If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers. Are any special GitHub settings required to contribute to LLVM? We only require contributors to have a public email address associated with their GitHub commits, see this section of LLVM Developer Policy for details. If you have questions, feel free to leave a comment on this PR, or ask on LLVM Discord or LLVM Discourse. Thank you, |
Author
|
I have read and agree to the LLVM contribution policies, including the LLVM AI Tool Use Policy, and have noted tool usage in the PR description. |
|
@llvm/pr-subscribers-llvm-transforms Author: Vedaant Srivastava (VedaantS) ChangesFixes #163757 An aligned pointer with a power-of-two stride must visit null before it repeats. If it is dereferenced on every iteration, and the alignment is at least the stride, those iterations cannot all execute in a defined program unless null pointer is defined in the address space. Relax the hasNoSelfWrap check in LoopTermFold to recognize pointer IVs with dereferenceable memory accesses dominating the loop latch. This eliminates redundant loop counter IVs when pointer accesses already advance monotonically, such as the inner loop of CoreMark's matrix_mul_matrix_bitextract. Fixes #163757 Tool disclosure: The patch and unit tests were authored manually; Google Antigravity was used to review the diff, run local tests, and write this PR description. Patch is 27.33 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/240066.diff 4 Files Affected:
diff --git a/llvm/lib/Transforms/Scalar/LoopTermFold.cpp b/llvm/lib/Transforms/Scalar/LoopTermFold.cpp
index e6eccdd7d616e2..8da886f1da5ee5 100644
--- a/llvm/lib/Transforms/Scalar/LoopTermFold.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopTermFold.cpp
@@ -21,7 +21,9 @@
#include "llvm/Analysis/ValueTracking.h"
#include "llvm/Config/llvm-config.h"
#include "llvm/IR/BasicBlock.h"
+#include "llvm/IR/DataLayout.h"
#include "llvm/IR/Dominators.h"
+#include "llvm/IR/Function.h"
#include "llvm/IR/IRBuilder.h"
#include "llvm/IR/InstrTypes.h"
#include "llvm/IR/Instruction.h"
@@ -48,6 +50,47 @@ using namespace llvm;
STATISTIC(NumTermFold,
"Number of terminating condition fold recognized and performed");
+// An aligned pointer with a power-of-two stride must visit null before it
+// repeats. If it is dereferenced on every iteration, and the alignment is at
+// least the stride, those iterations cannot all execute in a defined program.
+static bool hasNoSelfWrapViaDereference(PHINode &PN,
+ const SCEVAddRecExpr *AddRec,
+ ScalarEvolution &SE, DominatorTree &DT,
+ Instruction *LatchTerm) {
+ auto *PtrTy = dyn_cast<PointerType>(PN.getType());
+ if (!PtrTy || AddRec->getLoop()->getHeader() != PN.getParent() ||
+ NullPointerIsDefined(PN.getFunction(), PtrTy->getAddressSpace()))
+ return false;
+
+ const auto *Step = dyn_cast<SCEVConstant>(AddRec->getStepRecurrence(SE));
+ if (!Step)
+ return false;
+ APInt Stride = Step->getAPInt().abs();
+ if (!Stride.isPowerOf2() || Stride.getActiveBits() > 64)
+ return false;
+
+ const DataLayout &DL = PN.getDataLayout();
+ // GEP arithmetic must cover the entire pointer representation for its cycle
+ // to include null.
+ if (DL.isNonIntegralPointerType(PtrTy) ||
+ DL.getIndexSizeInBits(PtrTy->getAddressSpace()) !=
+ DL.getPointerSizeInBits(PtrTy->getAddressSpace()))
+ return false;
+
+ for (User *U : PN.users()) {
+ if (getLoadStorePointerOperand(U) != &PN)
+ continue;
+ // Volatile accesses may legally use address zero (for example, MMIO).
+ if (cast<Instruction>(U)->isVolatile())
+ continue;
+ if (getLoadStoreAlignment(U).value() >= Stride.getZExtValue() &&
+ DL.getTypeStoreSize(getLoadStoreType(U)).getKnownMinValue() != 0 &&
+ DT.dominates(cast<Instruction>(U), LatchTerm))
+ return true;
+ }
+ return false;
+}
+
static std::optional<std::tuple<PHINode *, PHINode *, const SCEV *, bool>>
canFoldTermCondOfLoop(Loop *L, ScalarEvolution &SE, DominatorTree &DT,
const LoopInfo &LI, const TargetTransformInfo &TTI) {
@@ -152,7 +195,9 @@ canFoldTermCondOfLoop(Loop *L, ScalarEvolution &SE, DominatorTree &DT,
// iteration. The simplest case to consider is a candidate IV which is
// narrower than the trip count (and thus original IV), but this can
// also happen due to non-unit strides on the candidate IVs.
- if (!AddRec->hasNoSelfWrap() ||
+ if ((!AddRec->hasNoSelfWrap() &&
+ !hasNoSelfWrapViaDereference(PN, AddRec, SE, DT,
+ LoopLatch->getTerminator())) ||
!SE.isKnownNonZero(AddRec->getStepRecurrence(SE)))
continue;
diff --git a/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold.ll b/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold.ll
index 3c53befa67e230..1d4167414c9977 100644
--- a/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold.ll
+++ b/llvm/test/Transforms/LoopStrengthReduce/lsr-term-fold.ll
@@ -230,20 +230,20 @@ for.end: ; preds = %for.body
ret void
}
-; In this case, the integer IV has a larger bitwidth than the pointer IV.
-; This means that the smaller IV may wrap around multiple times before
-; the original loop exit is taken.
+; The integer IV is wider than the pointer IV. However, the aligned store
+; would dereference null before the pointer repeats, so it is safe to fold.
define void @iv_size(ptr %a, i128 %N) {
; CHECK-LABEL: @iv_size(
; CHECK-NEXT: entry:
+; CHECK-NEXT: [[TMP0:%.*]] = trunc i128 [[N:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = shl i64 [[TMP0]], 2
+; CHECK-NEXT: [[SCEVGEP:%.*]] = getelementptr i8, ptr [[A:%.*]], i64 [[TMP1]]
; CHECK-NEXT: br label [[FOR_BODY:%.*]]
; CHECK: for.body:
-; CHECK-NEXT: [[LSR_IV1:%.*]] = phi ptr [ [[UGLYGEP2:%.*]], [[FOR_BODY]] ], [ [[A:%.*]], [[ENTRY:%.*]] ]
-; CHECK-NEXT: [[LSR_IV:%.*]] = phi i128 [ [[LSR_IV_NEXT:%.*]], [[FOR_BODY]] ], [ [[N:%.*]], [[ENTRY]] ]
+; CHECK-NEXT: [[LSR_IV1:%.*]] = phi ptr [ [[UGLYGEP2:%.*]], [[FOR_BODY]] ], [ [[A]], [[ENTRY:%.*]] ]
; CHECK-NEXT: store i32 1, ptr [[LSR_IV1]], align 4
-; CHECK-NEXT: [[LSR_IV_NEXT]] = add nsw i128 [[LSR_IV]], -1
; CHECK-NEXT: [[UGLYGEP2]] = getelementptr i8, ptr [[LSR_IV1]], i64 4
-; CHECK-NEXT: [[EXITCOND_NOT:%.*]] = icmp eq i128 [[LSR_IV_NEXT]], 0
+; CHECK-NEXT: [[EXITCOND_NOT:%.*]] = icmp eq ptr [[UGLYGEP2]], [[SCEVGEP]]
; CHECK-NEXT: br i1 [[EXITCOND_NOT]], label [[FOR_END:%.*]], label [[FOR_BODY]]
; CHECK: for.end:
; CHECK-NEXT: ret void
diff --git a/llvm/test/Transforms/LoopStrengthReduce/term-fold-dereference-narrow-index.ll b/llvm/test/Transforms/LoopStrengthReduce/term-fold-dereference-narrow-index.ll
new file mode 100644
index 00000000000000..c86326b2619e6a
--- /dev/null
+++ b/llvm/test/Transforms/LoopStrengthReduce/term-fold-dereference-narrow-index.ll
@@ -0,0 +1,34 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -passes=loop-term-fold -S < %s | FileCheck %s
+
+; The unchanged high pointer bits can prevent the recurrence from visiting null.
+target datalayout = "e-p:64:64:64:32"
+
+define void @narrow_pointer_index(ptr %a, i32 %n) {
+; CHECK-LABEL: @narrow_pointer_index(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr [ [[A:%.*]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[COUNT:%.*]] = phi i32 [ [[N:%.*]], [[ENTRY]] ], [ [[COUNT_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store i8 1, ptr [[P]], align 2
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr [[P]], i32 2
+; CHECK-NEXT: [[COUNT_NEXT]] = add i32 [[COUNT]], -1
+; CHECK-NEXT: [[DONE:%.*]] = icmp eq i32 [[COUNT_NEXT]], 0
+; CHECK-NEXT: br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr [ %a, %entry ], [ %p.next, %loop ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %loop ]
+ store i8 1, ptr %p, align 2
+ %p.next = getelementptr i8, ptr %p, i32 2
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
diff --git a/llvm/test/Transforms/LoopStrengthReduce/term-fold-dereference.ll b/llvm/test/Transforms/LoopStrengthReduce/term-fold-dereference.ll
new file mode 100644
index 00000000000000..1045476589001e
--- /dev/null
+++ b/llvm/test/Transforms/LoopStrengthReduce/term-fold-dereference.ll
@@ -0,0 +1,515 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -passes=loop-term-fold -S < %s | FileCheck %s
+
+target datalayout = "e-p:32:32-p1:32:32"
+
+define void @aligned_load(ptr %a, i32 %n, i1 %do_store) {
+; CHECK-LABEL: @aligned_load(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[TMP0:%.*]] = shl i32 [[N:%.*]], 1
+; CHECK-NEXT: [[SCEVGEP:%.*]] = getelementptr i8, ptr [[A:%.*]], i32 [[TMP0]]
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr [ [[A]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[V:%.*]] = load i16, ptr [[P]], align 2
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr [[P]], i32 2
+; CHECK-NEXT: [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[P_NEXT]], [[SCEVGEP]]
+; CHECK-NEXT: br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr [ %a, %entry ], [ %p.next, %loop ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %loop ]
+ %v = load i16, ptr %p, align 2
+ %p.next = getelementptr i8, ptr %p, i32 2
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
+
+define void @aligned_store(ptr %a, i32 %n, i1 %do_store) {
+; CHECK-LABEL: @aligned_store(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[TMP0:%.*]] = shl i32 [[N:%.*]], 1
+; CHECK-NEXT: [[SCEVGEP:%.*]] = getelementptr i8, ptr [[A:%.*]], i32 [[TMP0]]
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr [ [[A]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store i8 1, ptr [[P]], align 2
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr [[P]], i32 2
+; CHECK-NEXT: [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[P_NEXT]], [[SCEVGEP]]
+; CHECK-NEXT: br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr [ %a, %entry ], [ %p.next, %loop ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %loop ]
+ store i8 1, ptr %p, align 2
+ %p.next = getelementptr i8, ptr %p, i32 2
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
+
+define void @negative_stride(ptr %a, i32 %n, i1 %do_store) {
+; CHECK-LABEL: @negative_stride(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[TMP0:%.*]] = mul i32 [[N:%.*]], -2
+; CHECK-NEXT: [[SCEVGEP:%.*]] = getelementptr i8, ptr [[A:%.*]], i32 [[TMP0]]
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr [ [[A]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store i8 1, ptr [[P]], align 2
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr [[P]], i32 -2
+; CHECK-NEXT: [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[P_NEXT]], [[SCEVGEP]]
+; CHECK-NEXT: br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr [ %a, %entry ], [ %p.next, %loop ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %loop ]
+ store i8 1, ptr %p, align 2
+ %p.next = getelementptr i8, ptr %p, i32 -2
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
+
+define void @unit_stride(ptr %a, i32 %n, i1 %do_store) {
+; CHECK-LABEL: @unit_stride(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[SCEVGEP:%.*]] = getelementptr i8, ptr [[A:%.*]], i32 [[N:%.*]]
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr [ [[A]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store i8 1, ptr [[P]], align 1
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr [[P]], i32 1
+; CHECK-NEXT: [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[P_NEXT]], [[SCEVGEP]]
+; CHECK-NEXT: br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr [ %a, %entry ], [ %p.next, %loop ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %loop ]
+ store i8 1, ptr %p, align 1
+ %p.next = getelementptr i8, ptr %p, i32 1
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
+
+define void @larger_stride(ptr %a, i32 %n, i1 %do_store) {
+; CHECK-LABEL: @larger_stride(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[TMP0:%.*]] = shl i32 [[N:%.*]], 2
+; CHECK-NEXT: [[SCEVGEP:%.*]] = getelementptr i8, ptr [[A:%.*]], i32 [[TMP0]]
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr [ [[A]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store i8 1, ptr [[P]], align 4
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr [[P]], i32 4
+; CHECK-NEXT: [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND:%.*]] = icmp eq ptr [[P_NEXT]], [[SCEVGEP]]
+; CHECK-NEXT: br i1 [[LSR_FOLD_TERM_COND_REPLACED_TERM_COND]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr [ %a, %entry ], [ %p.next, %loop ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %loop ]
+ store i8 1, ptr %p, align 4
+ %p.next = getelementptr i8, ptr %p, i32 4
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
+
+; The pointer can stay in a nonzero residue class and never visit null.
+define void @insufficient_alignment(ptr %a, i32 %n, i1 %do_store) {
+; CHECK-LABEL: @insufficient_alignment(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr [ [[A:%.*]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[COUNT:%.*]] = phi i32 [ [[N:%.*]], [[ENTRY]] ], [ [[COUNT_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store i8 1, ptr [[P]], align 2
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr [[P]], i32 4
+; CHECK-NEXT: [[COUNT_NEXT]] = add i32 [[COUNT]], -1
+; CHECK-NEXT: [[DONE:%.*]] = icmp eq i32 [[COUNT_NEXT]], 0
+; CHECK-NEXT: br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr [ %a, %entry ], [ %p.next, %loop ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %loop ]
+ store i8 1, ptr %p, align 2
+ %p.next = getelementptr i8, ptr %p, i32 4
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
+
+; Only power-of-two strides support the no-self-wrap bound used here.
+define void @odd_stride(ptr %a, i32 %n, i1 %do_store) {
+; CHECK-LABEL: @odd_stride(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr [ [[A:%.*]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[COUNT:%.*]] = phi i32 [ [[N:%.*]], [[ENTRY]] ], [ [[COUNT_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store i8 1, ptr [[P]], align 4
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr [[P]], i32 3
+; CHECK-NEXT: [[COUNT_NEXT]] = add i32 [[COUNT]], -1
+; CHECK-NEXT: [[DONE:%.*]] = icmp eq i32 [[COUNT_NEXT]], 0
+; CHECK-NEXT: br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr [ %a, %entry ], [ %p.next, %loop ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %loop ]
+ store i8 1, ptr %p, align 4
+ %p.next = getelementptr i8, ptr %p, i32 3
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
+
+; A nonzero stride is required for a pointer exit test.
+define void @zero_stride(ptr %a, i32 %n, i1 %do_store) {
+; CHECK-LABEL: @zero_stride(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr [ [[A:%.*]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[COUNT:%.*]] = phi i32 [ [[N:%.*]], [[ENTRY]] ], [ [[COUNT_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store i8 1, ptr [[P]], align 2
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr [[P]], i32 0
+; CHECK-NEXT: [[COUNT_NEXT]] = add i32 [[COUNT]], -1
+; CHECK-NEXT: [[DONE:%.*]] = icmp eq i32 [[COUNT_NEXT]], 0
+; CHECK-NEXT: br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr [ %a, %entry ], [ %p.next, %loop ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %loop ]
+ store i8 1, ptr %p, align 2
+ %p.next = getelementptr i8, ptr %p, i32 0
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
+
+; Dereferencing null does not imply undefined behavior in this function.
+define void @null_is_valid(ptr %a, i32 %n, i1 %do_store) null_pointer_is_valid {
+; CHECK-LABEL: @null_is_valid(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr [ [[A:%.*]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[COUNT:%.*]] = phi i32 [ [[N:%.*]], [[ENTRY]] ], [ [[COUNT_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store i8 1, ptr [[P]], align 2
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr [[P]], i32 2
+; CHECK-NEXT: [[COUNT_NEXT]] = add i32 [[COUNT]], -1
+; CHECK-NEXT: [[DONE:%.*]] = icmp eq i32 [[COUNT_NEXT]], 0
+; CHECK-NEXT: br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr [ %a, %entry ], [ %p.next, %loop ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %loop ]
+ store i8 1, ptr %p, align 2
+ %p.next = getelementptr i8, ptr %p, i32 2
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
+
+; Do not assume null is invalid in other address spaces.
+define void @other_address_space(ptr addrspace(1) %a, i32 %n, i1 %do_store) {
+; CHECK-LABEL: @other_address_space(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr addrspace(1) [ [[A:%.*]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: [[COUNT:%.*]] = phi i32 [ [[N:%.*]], [[ENTRY]] ], [ [[COUNT_NEXT:%.*]], [[LOOP]] ]
+; CHECK-NEXT: store i8 1, ptr addrspace(1) [[P]], align 2
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr addrspace(1) [[P]], i32 2
+; CHECK-NEXT: [[COUNT_NEXT]] = add i32 [[COUNT]], -1
+; CHECK-NEXT: [[DONE:%.*]] = icmp eq i32 [[COUNT_NEXT]], 0
+; CHECK-NEXT: br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr addrspace(1) [ %a, %entry ], [ %p.next, %loop ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %loop ]
+ store i8 1, ptr addrspace(1) %p, align 2
+ %p.next = getelementptr i8, ptr addrspace(1) %p, i32 2
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
+
+; The pointer may visit null on iterations that skip the store.
+define void @conditional_store(ptr %a, i32 %n, i1 %do_store) {
+; CHECK-LABEL: @conditional_store(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: br label [[LOOP:%.*]]
+; CHECK: loop:
+; CHECK-NEXT: [[P:%.*]] = phi ptr [ [[A:%.*]], [[ENTRY:%.*]] ], [ [[P_NEXT:%.*]], [[LATCH:%.*]] ]
+; CHECK-NEXT: [[COUNT:%.*]] = phi i32 [ [[N:%.*]], [[ENTRY]] ], [ [[COUNT_NEXT:%.*]], [[LATCH]] ]
+; CHECK-NEXT: br i1 [[DO_STORE:%.*]], label [[ACCESS:%.*]], label [[LATCH]]
+; CHECK: access:
+; CHECK-NEXT: store i8 1, ptr [[P]], align 2
+; CHECK-NEXT: br label [[LATCH]]
+; CHECK: latch:
+; CHECK-NEXT: [[P_NEXT]] = getelementptr i8, ptr [[P]], i32 2
+; CHECK-NEXT: [[COUNT_NEXT]] = add i32 [[COUNT]], -1
+; CHECK-NEXT: [[DONE:%.*]] = icmp eq i32 [[COUNT_NEXT]], 0
+; CHECK-NEXT: br i1 [[DONE]], label [[EXIT:%.*]], label [[LOOP]]
+; CHECK: exit:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+loop:
+ %p = phi ptr [ %a, %entry ], [ %p.next, %latch ]
+ %count = phi i32 [ %n, %entry ], [ %count.next, %latch ]
+ br i1 %do_store, label %access, label %latch
+access:
+ store i8 1, ptr %p, align 2
+ br label %latch
+latch:
+ %p.next = getelementptr i8, ptr %p, i32 2
+ %count.next = add i32 %count, -1
+ %done = icmp eq i32 %count.next, 0
+ br i1 %done, label %exit, label %loop
+exit:
+ ret void
+}
+
+; A zero-size access does not establish that null is invalid.
+define void @empty_store(ptr %a, i32 %n, i1 %do_store) {
+; CHE...
[truncated]
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #163757
An aligned pointer with a power-of-two stride must visit null before it repeats. If it is dereferenced on every iteration, and the alignment is at least the stride, those iterations cannot all execute in a defined program unless null pointer is defined in the address space.
Relax the hasNoSelfWrap check in LoopTermFold to recognize pointer IVs with dereferenceable memory accesses dominating the loop latch. This eliminates redundant loop counter IVs when pointer accesses already advance monotonically, such as the inner loop of CoreMark's matrix_mul_matrix_bitextract.
Fixes #163757
Tool disclosure: The patch and unit tests were authored manually; Google Antigravity was used to review the diff, run local tests, and write this PR description.