Permit to re-run the failed run's #8
Labels
No labels
ai::retry
ai::retry_once
ai::retry_until_green
ai::validate_plan
status::blocked
status::done
status::functional_analysis
status::in_ci
status::in_development
status::in_review
status::technical_analysis
status::waiting_info
status::waiting_validation
type::analysis
type::bug
type::question
type::task
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Alchemic.Studio/Alfred#8
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Add a button on the top right of the page to re-run the "run" on failed run.
Functional analysis
Expected behaviour
When an operator opens a run that ended in failed, they can restart it from the admin UI without going to the forge to add the
retrycommand label. A Re-run button sits in the top-right corner of the run detail page (/runs/{id}), next to the "Run #id · phase" title. Clicking it, and then confirming, starts the same flow as theretrycommand label.All five defaults were accepted by @SebastienDenooz on 2026-10-28:
run_detail: the button is only on the run detail page.same_as_retry_command: a re-run works exactly like theretrycommand label.latest_failed_only: only the latest run of a mailbox can be re-run, and only when it failed.failed_only: cancelled runs are excluded.authorized_with_confirm: only users who can manage the project may re-run, and the click must be confirmed.Acceptance criteria
failed;pendingorrunning.pending,running,succeededorcancelled. It also does not appear for a failed run that a newer run of the same mailbox has superseded.retrycommand label does for that mailbox:retrycommand, and the rejection of stale requests.Thanks @SebastienDenooz, I am on it. I will post my analysis here.
The status label status::functional_analysis is managed by me: I restored it. Use the command labels to steer my work.
I need a few answers before going further. Reply in a comment; without an answer by 2026-10-28 23:51 UTC, I will apply the defaults.
q1 — Which page should show the "Re-run" button in its top-right corner? The run detail page (
/runs/{id}, title "Run #id · phase") is the only page about a single run. The runs list and the mailbox page show many runs.q2 — What should a re-run do? The engine already has a retry flow: the
retrycommand label on a blocked issue resets the counters, plans the failed phase again, posts a "resumed" comment on the forge and restores the status label. Should the button use this same flow, or should it only start a new run of the same phase with the same trigger events, without touching the forge?retrycommand label: reset counters, plan the failed phase again, post the resumption comment and restore the status label (default)q3 — Which failed runs can be re-run? A mailbox can have several failed runs over time, and newer runs may already have moved the issue on.
q4 — Should the button also appear on cancelled runs, or only on failed ones?
failed, as the issue says (default)failedorcancelledq5 — Who may re-run, and should the click be confirmed first?
You can also answer every question at once in a comment of your own.
Ok for me, you can go forward
I need a few answers before going further. Reply in a comment; without an answer by 2026-10-29 08:55 UTC, I will apply the defaults.
q1 — A failed run does not always leave its issue blocked, and the
retrycommand label only acts on a blocked issue (TriggerResolverline 49 andIssueLifecycleline 106 both require statusblockedwith a failed phase). An issue becomes blocked only when the LLM retries run out (or because of a type conflict or a closed MR). For the other failures the issue stays where it was. Non-retryable failures (auth_error,setup_failed,provider_not_allowed,unhandled_phase, host or read-only violations) keep the issue in its status (for examplein_development) withfailed_phaseset, and nothing restarts it; adding theretrylabel does nothing today either. Transient failures already have an automaticinternal.retry_duequeued. Stale or lease-expired runs are sent back to the dispatcher automatically. What should the Re-run button do for a failed run whose issue is not blocked?blockedwith a failed phase, which is exactly where theretrylabel works. Hide it on every other failed run, so the behaviour stays exactly that of the label. (default)llm_retries, posts the "resumed" comment and restores the status label, but changes no status (this goes beyond what the label does).You can also answer every question at once in a comment of your own.
I just want a retry button on
run'sthat is in statefailed. Because i have arunthat it is in a statefailed" where the error of the commit islease_expired. And i just want thisrunto run again. Is it that you understand and talking about or not?Plan
Summary
This plan adds a Re-run button to the run detail page (
/runs/{id}). It goes through the same machinery as theretrycommand label, and it also covers the case @SebastienDenooz asked about: a run that failed withlease_expiredand that nothing restarts today.Why the button cannot just add the
retrylabelThe exploration confirmed the gap behind the last question. When
LeaseReclaimerorAbortStaleRunsinterrupts a run:failedwithlease_expiredorstale_run.RunExecutor::fail()seeswasAborted()and returns early. Noretry_dueevent is created andfailed_phaseis not set.RunDispatcher::plan()finds an empty buffer and sets the mailbox back toidle.The issue keeps its working status (for example
in_development) with no pending run. Theretrylabel only acts onblockedissues, so it cannot help here.What the button does
The button follows the same mechanism as the label and adds one case:
RetryCommandtransition and is exactly what theretrylabel does (q2 and the first q1 default).lease_expired,stale_run,auth_error,setup_failed, …): a newRerunRequestedtrigger does the same work but keeps the status. It resets the counters of the failed phase, plans the failed run's phase again with its trigger events, posts the "Retryingphase." comment and restores the status label. This matches the user's request: "I just want this run to run again."The eligibility rules stay as accepted (latest run of the mailbox, state
failed, nothing pending or running), and so do the authorization and confirmation rules.Approach
1. A new internal event:
internal.rerun_requestedapp/Enums/EventKind.php: addcase InternalRerunRequested = 'internal.rerun_requested';.inbox_events.kindis a free string, so no migration is needed.Coalescerdefault branch already puts unknown kinds inotherEvents.RunIntent::hasTrigger()already countsotherEvents, so the event is always consumed.EventKind::inresources/viewsandlang/).{failed_run_id, phase, trigger_event_ids, requested_by}, whererequested_byis the user id.MailboxWaker::wake($mailbox, EventKind::InternalRerunRequested, "rerun:run-{id}", $payload). It moves an idle mailbox tobuffering, due now, and dispatchesProcessMailbox. The key makes a double click idempotent.2. Lifecycle
app/Lifecycle/LifecycleTrigger.php: addRerunRequested.app/Lifecycle/TriggerResolver.php: add two arms right after the retry command arms (line 49). They come afterMrMerged,MrClosed,MrReopenedand type conflicts, so those keep priority.RERUNNABLEisFunctionalAnalysis,TechnicalAnalysis,InDevelopment,InCiandInReview.waiting_reasonandresume_phaseinstatusAttributes().app/Lifecycle/IssueLifecycle.php:app/Providers/LifecycleServiceProvider.php(lines 80-82): registerRetryCommandHandlerforRerunRequestedas well. It already does everything needed:retry_resumed, and its idempotency key is derived from the new event id, so the comment is posted again for each re-run;nextStatusas the status label.Update the handler docblock.
3. Dispatcher:
app/Scheduling/RunDispatcher.php$retry(line 100), read$rerun = $intent->otherEvents[EventKind::InternalRerunRequested->value] ?? null;.transition(): forRerunRequested, passfailedPhase: Phase::tryFrom(payload.phase)through the existing$failedPhaseoverride (line 127). ForRetryCommandcoming from a re-run, keep$mailbox->failed_phase; it is set on a blocked issue.triggerEventIds($intent, $retry ?? $rerun): the new run gets the failed run'strigger_event_ids, so the agent sees the same context. This mirrors theretry_duereplay.attempt: when$rerunis present, usefailed_run.attempt + 1. Read it fromfailed_run_id, with a fallback to the current rule.statusAttributes(): no change.ResetCounterresets the counter of$mailbox->failed_phasewhen it is set (non-retryable failures), then clears it. Forlease_expired,failed_phaseis already null and nothing is reset, which is correct because no counter was incremented.directPhase(), MR mailboxes: MR mailboxes have no issue status and only runReview. Accept$rerun !== nullas a review request (closed_at === null && ($requested || $rerun !== null)). That case posts no comment and applies no label, because the MR has no status label.4. Domain service:
app/Runs/RerunFailedRun.phpThe view and the controller both use this service, so there is a single source of truth.
refusal(Run $run): ?stringreturns null when the run is eligible, otherwise a translated reason. The run is eligible when all of these hold:state === Failed;max(id));pendingorrunning;stateis neitherRunningnorCancelling;closed_at === nulland the status isBlockedwithfailed_phase, or the status is inRERUNNABLE;closed_at === null.rerun(Run $run, User $user): void:DB::transaction, lock the mailbox (lockForUpdate) and checkrefusal()again. If the run is no longer eligible, throwRunNotRerunnable(new, inapp/Runs/, carrying the reason).MailboxWaker::wake(...).Concurrency is handled by the lock plus
MailboxVersion(the waker already retries onStaleMailboxVersion).5. Authorization:
app/Policies/RunPolicy.phpAdd
rerun(User $user, Run $run): bool, which returns$user->can('update', $run->mailbox->project). That is the existing "manage project" ability, owners only. Viewers (members) still passviewand see the page without the button. Direct POST calls by other users get 403 fromGate::authorize.6. HTTP
routes/web.php: next toruns.show(around line 86), addRoute::post('/runs/{run}/rerun', [RunController::class, 'rerun'])->whereNumber('run')->name('runs.rerun');.RunController::rerun(Request, Run, RerunFailedRun), following theMailboxController::replaypattern:CurrentOrganization::abortUnlessOwns, thenGate::authorize('rerun', $run).rerun(). OnRunNotRerunnable, redirect back toruns.show($run)witherrorset to the reason (acceptance criterion 8).$run->id(the sync queue in tests, or a fast worker), redirect to it withstatus"Re-run started.". Otherwise redirect back to the run withstatus"Re-run requested: the new run will appear on the mailbox shortly." (criterion 7).RunController::data(): addstatus,state,failed_phaseandclosed_atto the eager-loaded mailbox columns. PasscanRerun = $request->user()->can('rerun', $run) && $rerun->refusal($run) === nullto the view.7. View
resources/views/runs/show.blade.php:12: wrap the header inflex items-start justify-between gap-4, as inmailboxes/show.blade.php:10. When$canRerunis true, put on the right:resources/views/components/confirm.blade.php: add an optionalvariantprop. It defaults todanger, so existing uses are unchanged, and lets the re-run use a non-destructive style throughx-form.button.resources/js/app.jsalready drives the<dialog>. The live polling only refreshesruns/partials/live.blade.php, so the button stays until a reload. A click after the run changed is caught by the server check of criterion 8.lang/JSON files, if any exist beyonden.8. Docs
docs/specification.md:RerunRequestedtrigger.Tests
Unit
tests/Unit/Lifecycle/TriggerResolverTest.php:RetryCommand;RERUNNABLEstatus it givesRerunRequested;MrMergedandMrClosedtake priority.tests/Unit/Lifecycle/IssueLifecycleTest.php:RerunRequestedkeeps the status, runsfailedPhase, and carries[ResetCounter, ApplyStatusLabel]. It is invalid without a failed phase or on waiting or blocked statuses.Feature
tests/Feature/Scheduling/RunDispatcherTest.php:lease_expiredrun on anin_developmentmailbox, with a re-run event: onependingDevelopment run is created with the failed run'strigger_event_idsandattempt + 1. The mailbox isrunningwith its status unchanged.fakeForgerecords the "Retryingdevelopment." comment and the status label.auth_error,failed_phaseset,llm_retries> 0):llm_retriesis 0 andfailed_phaseis null.retrylabel produce the same mailbox columns, run phase, comment and label calls. Reuse theblockedSetup()andblockedEvent()helpers fromtests/Feature/Lifecycle/Blocked/BlockedTest.php.Review.tests/Feature/Admin/Runs/RunRerunTest.php(new; reuserunMember()andtracedRun()fromRunScreensTest):pending,running,succeededandcancelled, for a failed run superseded by a newer run, when another run is pending or running, when the mailbox is running or cancelling, and when the issue is closed or done.session('error').E2E:
tests/Browser/Runs/RunRerunTest.phpUses Pest browser, which runs Playwright. The path:
lease_expiredrun.Use
fakeForge,fakeContainerRuntimeandfakeLlm, so the executed run does not touch the host.CI
.gitlab-ci.yml,.github/workflows/ci.ymland.forgejo/workflows/ci.ymlalready runphp artisan test(Unit, Feature and Browser with Playwright chromium), plus pint, phpstan and the build. The new tests run in all three pipelines without changes. Check that each pipeline picks up the new Browser test.Risks and notes
retrylabel: for non-blocked issues the button does more than the label does today. This is deliberate, following @SebastienDenooz's comment. Theretrylabel behaviour itself is unchanged.waiting_info,waiting_validation) are not re-runnable. Re-applying them would clearwaiting_reasonandresume_phase. Failed runs there are rare, since the human answer triggers the next run anyway.retry_duealready queued: the mailbox isbuffering, so the button shows. A re-run makes it due now, and both events are consumed by a single planned run. That is acceptable. Hiding the button in that case is an alternative if preferred.ProjectPolicy::update("manage project"), while mailbox cancel and replay are member-level. This follows criterion 4. If members should re-run too, it is a one-line change inRunPolicy::rerun.The implementation plan is ready for validation. Add the ai::validate_plan label to let me start.
Plan validated by @SebastienDenooz. Starting the development.
The pipeline failure comes from the CI runner, not from this merge request, so no change to the repository can fix it.
Cause. Forgejo Actions runs #17 (push) and #18 (pull_request) both stopped at the Install dependencies step, while running
npx playwright install --with-deps chromium. They never reached Pint, Larastan, the Vite build or Pest:The Docker host behind the
dockerrunner label is out of disk space.apt-getcan't download the system libraries Chromium needs, so the job fails before any test runs. The earlier runs onmainused the same workflow and passed, which fits a runner that has filled up since.The code itself passes every check. I ran the same steps as the workflow on commit
52b5266:vendor/bin/pint --test: passedvendor/bin/phpstan analyse: passed, 0 errorsnpm run build: passedphp artisan testagainst PostgreSQL, including the Playwright browser tests: all tests passed, no failuresWhat's needed. Someone with access to the runner host has to free up disk space, then re-run the pipeline. On that host, run
docker system pruneto remove dangling images, stopped containers and build cache, and clear old job workspaces and caches. No code change is required.I am blocked during
ci_fix: the problem cannot be fixed from the repository (see my previous comment). Add the ai::retry label once the cause is fixed to let me retry.Retrying
ci_fix.The pipeline failed because the CI runner ran out of disk space, not because of the code.
Both failed runs on commit
52b5266(run 17 from the push, run 18 from the pull request, around 13:52–13:58 UTC) stopped at the same step, before any test ran:The Forgejo runner did not have enough free disk space to install Chromium's system dependencies. Both runs were re-run on the same commit around 19:55–20:08 UTC and passed (
CI / checks (push)andCI / checks (pull_request)are now green).Running the checks locally on this branch, everything passes:
tests/Browser/Runs/RunRerunTest.php.php artisan test, as CI runs it: 2139 passed.Nothing can fix this from the repository. The fix is to free disk space on the Forgejo runner with the
dockerlabel: prune old images, containers and volumes, clean the apt caches, and maybe add disk space.I am blocked during
ci_fix: the problem cannot be fixed from the repository (see my previous comment). Add the ai::retry label once the cause is fixed to let me retry.Retrying
ci_fix.I couldn't reproduce the pipeline failure, and no failure logs came with the request.
I ran the same steps as the CI jobs, in the same order, on this branch:
vendor/bin/pint --test: passedvendor/bin/phpstan analyse: passed, 0 errorsnpm run build(Node 24): passedphp artisan testagainst PostgreSQL 16, Playwright/Chromium included: 2139 / 2139 tests passed, including the newtests/Feature/Admin/Runs/RunRerunTest.php,tests/Feature/Lifecycle/RerunTest.phpandtests/Browser/Runs/RunRerunTest.phpSince the code passes every check, the failure most likely comes from the runner: toolchain install (apt, NodeSource, Composer or Playwright downloads), the PostgreSQL service, or a transient browser/network issue. Please re-run the failed job. If it fails again, share the job log so the actual error can be looked at.
I am blocked during
ci_fix: the problem cannot be fixed from the repository (see my previous comment). Add the ai::retry label once the cause is fixed to let me retry.