Skip to content
All posts

When Engineers Should Use Automations

Use read-only automations to review CI failures, release candidates, and regression risks with clear evidence, narrow permissions, and stop criteria.

Engineers should automate checks when the question stays stable but the evidence keeps changing. “Did this release candidate pass its required checks?” is a useful recurring question. “Is this architecture good?” usually needs a focused review with more context.

Start with read-only access to a named repository and a small report. An automation should reduce the time between a meaningful change and your next decision without creating another stream of warnings to dismiss.

Three checks worth repeating

Watch a release candidate, not every build

Trigger: A scheduled check during an agreed release window. Use an event trigger only if your account and connected source support it.

Authorized data: Check results and release criteria for the exact candidate commit in the named repository.

Decision: Report which required checks passed, failed, or remain pending. Link the evidence and identify the commit. A passing result for yesterday’s commit does not clear today’s candidate. Missing or stale results should leave the release unverified.

This helps the release owner decide whether to investigate or continue their checklist. It must not merge, deploy, or waive a failed check. End the watch when the candidate is replaced or the release window closes.

Catch repeated CI failures

Trigger: A weekday review of newly completed CI runs.

Authorized data: Run metadata and permitted, sanitized failure logs from one repository. Avoid fetching logs that could expose credentials or customer data.

Decision: Group repeated failure signatures, distinguish new failures from existing ones, and link the affected runs. Describe a suspected flaky test as a hypothesis. One passing rerun does not establish that a failure is harmless.

For a hypothetical pilot, report a signature only after it appears in three runs, while surfacing a newly failing required check immediately. Tune that rule against actual noise and missed failures. Do not automatically rerun jobs, disable tests, or file issues unless those actions have separate authorization.

Review regression risks in a bounded diff

Trigger: A daily review of commits since the last successful review.

Authorized data: The named project’s Git history, changed files, and relevant tests. Record the reviewed commit range so a missed run does not silently leave a gap.

Decision: Return a few concrete risks with file references, an explanation of the possible failure, and a suggested test. Distinguish a code-reading concern from a reproduced defect. Report unavailable history or missing test evidence directly.

A separate, explicitly authorized workflow can run selected tests. Keep dependency installation, network access, and code changes outside the initial review.

A prompt you can adapt

Every weekday at 9 AM America/New_York, review new completed CI runs for [repository] since the last successful report. Read only the permitted run metadata and sanitized logs. Group repeated failures, identify newly failing required checks, and link each finding to its commit and run. Separate observations from suspected causes. Do not rerun jobs, edit files, post comments, merge, or deploy. Report access failures instead of assuming a healthy state. Keep each report in this chat. End this pilot after five weekday reports so I can review its usefulness.

Replace the placeholder, test manually, and verify the saved schedule. The automation setup guide covers environment availability and permissions, including local projects that require an awake computer.

Keep judgment and sensitive details with the engineer

Avoid automating ambiguous architecture decisions, incident command, security exceptions, or release approval. Stop or revise checks that repeatedly produce unactionable findings. Never widen access simply to make a scheduled run succeed.

Agent Alerts can be an optional reporting surface when the running environment has the tools and permission. Send a neutral status such as “Release review needs attention,” with details kept in the approved source. Review notification behavior and Live Activity examples, then test visible delivery before relying on it.