How to Automate Security Audits

Security auditing is the practice of reviewing a scoped branch, pull request, feature area, or repository slice for exploitable security issues before they reach production. A useful audit does more than search for scary patterns. It traces attacker-controlled input through the code, checks whether framework protections already neutralize the path, and reports only findings a developer can reproduce and fix.

Small teams often treat that work as an occasional specialist review. That leaves security-sensitive changes competing with feature review, CI noise, dependency alerts, and release pressure. The better shape is a repeatable audit assignment with a clear scope, a threat model, scanner output treated as leads, and a human decision before any fix or public comment happens.

Why security review quietly gets noisy

Most security review failure is not a lack of tools. It is a lack of proof. A scanner flags a query builder call, a workflow token permission, or a suspicious file read. Someone pastes the alert into a review, and the team burns time proving that the path is unreachable, already escaped, or only possible for an administrator.

The opposite failure is worse. A general code review notices style and tests but misses the exploit path because nobody followed the data from input to sink. Authorization checks, CI workflow triggers, third-party actions, secrets handling, and dependency manifests sit outside the changed file. The risky part is often one import, one route, or one GitHub Actions condition away from the diff.

What the manual process looks like

Done by hand, a security audit is a focused review ritual:

1. Define the audit scope: branch, pull request, feature area, or repository slice.
2. Name the threat concerns, including authentication, authorization, injection, XSS, SSRF, file handling, secrets, CI, dependencies, and business-logic abuse.
3. Gather the full diff plus surrounding code, configuration, workflow files, tests, and dependency manifests.
4. Trace attacker-controlled input to dangerous sinks and rule out framework-mitigated false positives.
5. Run available scanners or audit commands, then verify each lead before treating it as a finding.
6. Write only high-confidence findings with severity, file and line, exploit path, why existing protection is insufficient, and the smallest safe fix.

That is careful work. It rewards patience and skepticism, and it is easy to skip when the team only asks, "Does the PR look good?"

What an agent can automate

An agent is useful because much of this loop is structured investigation, not taste:

• Scope the review. The agent starts from the assigned branch, pull request, feature area, or repository slice and gathers the surrounding context before making claims.
• Map input to sinks. It follows attacker-controlled input through routes, handlers, background jobs, templates, shell calls, storage paths, external calls, and authorization checks.
• Audit CI as code. It reads GitHub Actions workflow triggers, token permissions, third-party actions, shell interpolation, caches, artifacts, and deployment credentials.
• Verify scanner leads. If Semgrep, npm audit, govulncheck, mix deps.audit, or a secret scanner is available, the agent treats results as leads. It confirms exploitability before reporting them.
• Write actionable findings. Each finding includes the severity, file and line, attacker input, path to sink, missing protection, and smallest safe fix. Low-confidence concerns become questions instead of inflated issues.

The human still owns the judgment call. The agent can produce the audit report and draft fixes when asked, but it should not post comments, merge code, deploy, rotate credentials, or change CI secrets on its own.

The guardrails that make it safe

Security automation is useful only when it is low-noise. The safe pattern is to make the agent prove exploitability before it gets to call something a finding. Pattern matches, scanner output, and suspicious names are not enough.

The approval boundary is just as important. The audit report lands as a reviewable artifact. A person decides which findings are real, which fixes to request, and whether the agent should draft a patch. That keeps security review from turning into automated alarm generation, and it keeps privileged actions such as secret rotation and deployment changes outside the agent's authority.

Set it up in Task Machine

The Security audit playbook installs a Security Reviewer agent, three security review skills, the goal that keeps reports high-confidence, and a starter task proposal for the first scoped assignment. Setup takes a few minutes. You need a Task Machine workspace, permission to install playbooks (workspace owners have it), and repository access for the code you want audited. Until repository access is connected, the reviewer works from attached diffs and context you provide.

1. Find the playbook

Open Playbooks in your workspace and search for "security audit", or browse the Engineering category. The card shows that the playbook creates the security reviewer, its skills, and the goal that defines a verified audit.

2. Preview what it installs

Preview & install opens the full contents before anything is created: the Security Reviewer, the security-review, gha-security-review, and code-security skills, the verified findings goal, and the starter follow-up task. Review the requirements so the audit has repository context instead of only a pasted diff.

3. Define the audit scope

Start setup asks for the repository, audit scope, threat concerns, and verification command. Use the scope field to name the exact branch, pull request, feature area, or repository slice. Use the threat concerns field to focus the reviewer on the risks that matter for this change.

4. Generate and review

Generate customized playbook turns those answers into the reviewer instructions and starter task context. Read the review screen before installing. The audit should name the right scope, include the threat concerns, and keep the verification command visible as a check rather than a substitute for review.

5. Install

Install customized playbook creates the reviewer, skills, goal, and starter task proposal. The follow-up in your inbox asks you to start the first Security audit assignment and add the concrete brief before approving it. After that, the reviewer reads the scoped code and returns verified findings for human review.

What good looks like

Three numbers and checks tell you whether the audit loop is working:

• False-positive rate. Findings should be rare and defensible. If most reports get dismissed, the agent is pattern matching instead of proving exploitability.
• Time to audited scope. Security-sensitive branches should get a scoped audit before merge, not after release.
• Finding completeness. Every accepted finding should include severity, file and line, attacker input, path to sink, missing protection, and the smallest safe fix.

Common questions

Can an agent replace a security engineer? No. The agent handles the structured investigation and report assembly. A human still decides whether the finding is accepted, which fix ships, and whether any broader incident response is needed.

Should scanner output be copied into the report? No. Scanner output is a lead list. The report should include only verified findings, with the exploit path and the reason existing protections do not stop it.

What should the first audit cover? Start with a narrow, high-risk scope: authentication changes, billing changes, file handling, webhook processing, CI workflows, or any feature that moves user-controlled input into a privileged action.

Can this work without repository access? Yes, but with limits. The reviewer can work from attached diffs, workflow files, dependency manifests, and logs. Repository access gives it the surrounding code needed to avoid both missed paths and false positives.