Agency OS Radar

Daily open-source discoveries for a one-person AI software agency

Report date: Sunday, November 1, 2026 · Agency OS Edition · Addis Ababa (EAT)

Start Here (plain English)

The problem: an AI coding agent can write code, but a one-person agency can't sit and watch it. If nobody is watching, who stops the wrong PR from merging, the same job from being paid for twice, or a customer being told "it shipped" when it didn't? Chat-agent guardrails only work while you're reading the chat.

The solution in today's radar: cromford-ai/cromford is an open-source software factory that runs the whole loop unattended — ticket in, triaged, planned, built, adversarially reviewed, preview-QA'd, merged, deploy-verified, and production-checked. Its whole design idea is: anything a script can decide, a script decides — the expensive, unreliable model is only asked the questions that genuinely need reading comprehension. Deterministic gates, per-call cost accounting, and a human tier for anything risky.

Format note: single-featured-factory layout (standing focus set Oct 25, 2026); the Ethiopia business-opportunity section remains removed. Prior reports were available through Oct 31 (most recently KbWen/agentic-os; earlier: nexu-io/looper, jmanico/grim-prompter, coleam00/ai-software-factory, obra/superpowers). Cromford has not been featured before. All facts below verified today from the repo's README and commit history via GitHub; star/fork counts are as returned by the GitHub API today. This is a very young project — treat it as a pattern to study, not a tool to trust blindly.

The Featured Factory: cromford-ai/cromford INVESTIGATE

Link: github.com/cromford-ai/cromford · homepage cromford.dev
What it is (simple English): the "open-source software factory" — you put tickets on a product board (e.g. GitHub Issues); a fleet of watcher boxes picks them up and runs each one through fixed stages: triage → plan → build → adversarial review → preview QA → merge → deploy-verify → post-merge QA. It refuses at each stage for concrete reasons (an unanswered customer question, a plan with no failing test, weakened tests, a verify script that doesn't prove its claim, CI not green, no deployment within the window, the shipped behaviour missing in production).
What changed recently and when (VERIFIED from commit history): 7 commits today, Nov 1, 2026 (versioned nightly as "Cromford 2026.11.01"), including a merge-tier fix ("a green verdict carries across" in the pre-queue gate, Oct 31), a spec-trace parser fix, nightly test-tier ratchets, and an honest engineering note that an "Execute" run burned its whole context ceiling with zero file writes. It has no GitHub releases/tags yet — nightly commit versioning is the release mechanism.
Why it matters to a vibe coder: this is the closest public blueprint to Agency OS itself — an unattended pipeline with per-stage models, per-stage budgets, evidence-based refusals, human-tier merges, cost rows for every LLM call, and a rule-miner that turns repeated review findings into CI guardrails. Even if you never run it, its README and docs (SELF-HOST, OPERATING, LESSONS, ARCHITECTURE, CONTEXT-COMPOSITION) are a masterclass in making agent loops safe.
License & costs (verified): Apache License 2.0 (verified from the GitHub API today). The code is free; running it is not: it calls paid model APIs (a fixed failover ladder: claude → grok → codex-configured openrouter → fireworks), and self-hosting uses your own Cloudflare, GitHub and optionally Fly accounts. Every call is priced into logs/cost.csv; a model missing from catalog/model-prices.json raises a daily alert instead of silently costing $0.
Repo stats (GitHub API today): created Oct 13, 2026 · 28 stars · 12 forks · 3 open issues · primary language Shell. Young — expect churn.
Windows compatibility & install: the README states plainly: "The suite is written for Linux. On anything else, run it in the container" (bin/run-tests-in-docker.sh, Debian image, needs Docker + network for the first image pull). Windows is not mentioned in the README — running it on your PC means Docker Desktop + WSL2: plausible, untested. Self-hosting additionally expects bash, jq, optionally semgrep/codegraph, and a Linux "watcher box" (a VM).
Verdict: INVESTIGATE — exceptional as a design reference and safe to study/run read-only, but a 3-week-old project requiring Linux, cloud accounts and paid models is not something to wire into a client pipeline today.

The Factory Map — onto the Agency OS Lifecycle

Stage names, scripts, artifacts and refusal conditions verified from the repo README today.

Agency OS stage	Factory module	Artifact it produces	Which gate consumes it
LEAD / PROPOSAL	— not covered	—	— (tickets must already exist on a board)
DISCOVERY / PROBLEM_DEFINITION	triage — `bin/run-triage.sh` + deterministic `bin/triage-gate.sh`	RUN/SKIP decision with reasons	Refuses to start when the ticket's last word is an unanswered customer question
SPECIFICATION	plan — `bin/run-build.sh`	A written change plan ending in named acceptance tests	Refuses when it cannot name a test that would fail today
ARCHITECTURE	— no separate stage (planning absorbs design; docs cover context composition)	—	— (gap)
DEVELOPMENT	build — `bin/run-build.sh`, on a watcher box or optional ephemeral Fly machine (`bin/fly-pool.sh`)	Implementation against the plan, on its own box/budget	Refuses when tests were weakened rather than made to pass
QA	review (adversarial, never the author's session, `bin/review-prs.sh`), preview QA (`bin/e2e-smoke.sh` clicks the PR's own preview), post-merge QA (`bin/postmerge-qa.sh` drives production)	Review findings; preview smoke result; production check result	Refuses when the verify script doesn't prove its claim, the route 404s, or the shipped behaviour isn't there
SECURITY	— no dedicated stage; `semgrep` only as an optional CI check (`SEMGREP_REQUIRED=1`), not a merge gate	—	— (gap — see Honest Gaps)
CEO_REVIEW	merge tiers — tier 1 auto-merges when review approves + CI green; tier 2 waits for a human; `flags/automerge-disabled` forces everything to tier 2 (`bin/run-build.sh:1592`)	PR in one named state with an age budget (`bin/merge-queue.sh`, `bin/merge-sweep.sh`)	Refuses to merge when CI is not green, base/head moved after approval, or the tier is wrong
RELEASE	deploy-verify — `bin/deploy-train.sh`	Confirmation a production deployment followed the merge	Refuses when no deployment appeared within the window
MAINTENANCE	Rule-miner (`bin/rule-miner.sh`) turns repeated review findings into `guardrails/` that fail next time in CI — "only tightens, never loosens"; restartable stages (state on disk + DB); `docs/RESTORE.md` box recovery; `docs/LESSONS.md`	New guardrail rules; recoverable box state	Load-bearing tests protect the fence (`iron-law-guard-test.sh`, `lane-fence-test.sh`, `alert-loop-test.sh`)
REUSABLE_COMPONENT	Closest analog: guardrails + repo-managed `skills/` (one `SKILL.md` per competency, deployed to every box)	Tested procedures, not a component library	Partial (gap)

Honest Gaps

No LEAD / PROPOSAL: it consumes tickets that already exist; client-facing discovery, pricing and proposals stay with Agency OS.
No dedicated SECURITY stage: secret scanning, dependency auditing and authn/authz review are not merge gates — semgrep is an optional CI check only. Biggest gap, and the one that matters most to you.
Gap filler (verified): Amal-David/keyleak-detector — MIT (verified from the GitHub API today), Python CLI + Chrome extension, finds exposed API keys and secrets in JS bundles at runtime and validates Supabase/Firebase RLS misconfigurations; last pushed Oct 30, 2026; 274 stars (API today). Verdict: TEST NOW — local, free, Windows-friendly (Python CLI), no subscription.
Self-host is manual: the installer's --apply step "is not built yet" (README's own words) — resources are still built by hand; the dry-run only prints what would be created.
Platform: the test suite is Linux-only; the README honestly reports its own macOS failure count (96 pass / 28 fail) and 3 container suites that fail for not-yet-understood reasons. Windows is untested territory.
Very young: created Oct 13, 2026, nightly-versioned, no tagged releases — the project itself says its commits may report "nightly test tier red on main". Do not build a business on it this month.

Why It Fits Agency OS

Deterministic before LLM: "anything a script can decide, a script decides" — the same principle as Agency OS's gate engine being authoritative instead of trusting the agent's self-report.
Evidence before advancing: each stage has its own definition of done and refuses with a named reason — Agency OS's validated-evidence rule, implemented as pipelines.
Human approval at the boundary: tier-2 merges wait for a human, and flags/automerge-disabled can force every build to human review — matching your rule that release and acceptance need human approval.
Living project state: stages talk through files and a database, not one long conversation; a dead stage is restartable because its inputs are on disk — philosophically the same job as project-state.json being authoritative. It complements, never replaces, your gate engine.
Anti-drift tests: tests/evals-suite-test.sh checks that a prompt change still makes the same triage decisions — an honest answer to "did my agent regress when I changed the prompt?"
Agency OS compatibility: plausible — untested. Nothing here is proven on Windows or alongside your existing gate_engine.py stack; the integration proposal below stays additive.

Limitations, Risks & Human Review

Unattended by design: it opens PRs and can auto-merge tier-1 builds with nobody watching. For client work, keep flags/automerge-disabled on — and treat any auto-merge adoption as a commercial decision requiring your CEO_REVIEW approval.
Real money, metered but real: every LLM call costs; the cost ledger makes it visible but doesn't make it small. Budget stops and provider failover are built in, but you must set the caps in selfhost.conf.
Cloud-account surface: Cloudflare (D1, R2), GitHub and optionally Fly credentials sit in a config file — treat selfhost.conf as a secret; never commit it.
It is not a security tool: nothing in the pipeline scans secrets or dependencies before merge (see gaps). Keep your own security stage.
Spoofable self-reports remain possible: its gates are strong because most are deterministic, but plan/review stages still read text the model wrote; the rule-miner "only tightens" guardrails, which is good, but human review of artifacts stays mandatory.
Never run the nightly of a 3-week-old project unattended on accounts you care about. Read-only study first; scratch experiments only.

Test Today: the read-only dry-run + suite in a container (~30 min)

Why this one: zero cost, zero cloud accounts, zero risk — the self-host installer's --dry-run prints every resource, env line, spend cap and secret name and creates nothing. It's the best available preview of what running an unattended factory actually requires.
Prerequisites: Windows PC with Docker Desktop (WSL2 backend) and Git. The repo is Shell/bash — use the Docker container route the README itself prescribes for non-Linux.
Official steps: gh repo fork cromford-ai/cromford --clone (or plain git clone https://github.com/cromford-ai/cromford.git); cp env-templates/selfhost.conf.example selfhost.conf; then bin/selfhost-install.sh --config selfhost.conf --dry-run. Optionally also bin/run-tests-in-docker.sh to run the project's own 121-test suite in a Debian container.
Practical test: read the dry-run output top to bottom and tick off what Agency OS would demand: spend caps present? secrets named? which stages run unattended? Then run the test suite in the container and watch deterministic gates fail/pass on their own merits.
Expected results: a printed inventory of resources, env lines, spend caps and secret names (nothing created); a container test run with passing suites and honest skips (or failures — the README documents 3 not-yet-understood ones). No network calls from the suite itself, no secrets, no fixtures.
Safety precautions: scratch clone only; never fill in real secrets for a dry-run; do not run the watcher, cron or --apply; do not enable any auto-merge flag; read any script before executing it. The dry-run is read-only by design — keep it that way.

Agency OS Integration Proposal: a deterministic triage gate + per-call cost ledger

Problem solved: today your agents can start work on an under-specified client request, and you have no per-call record of what each stage's LLM usage cost a client project.
Workflow/agent improved: the discovery agent and orchestrator — adding one deterministic check and one log format, feeding — never replacing — gate_engine.py.
Lifecycle stage: DISCOVERY (gate) and MAINTENANCE (cost evidence), with a cost view at CEO_REVIEW.
Inputs: the ticket/request thread (from project-state.json); every agent LLM call (command, model, tokens, stage). Outputs: a BLOCKED-with-reason verdict when the last word of a request is an unanswered client question; a CSV row per call into a per-project costs.csv that rolls up per delivery.
How results are validated: both are script-decided, not LLM-decided — a status-field comparison for the unanswered-question gate (testable, like Cromford's triage-gate.sh), and a priced-catalogue lookup that alerts when a model has no price row instead of logging $0 (Cromford's exact pattern).
What happens on failure: unanswered question → DISCOVERY gate returns BLOCKED with the question quoted; unpriced model call → daily alert, cost row marked UNKNOWN rather than silently zero.
Gate or human approval? Neither replaces anything: gate-engine transitions and rollback stay exactly as they are; auto-starting billable work on a client ticket remains a human-approved commercial decision.
Smallest safe experiment: in a scratch Agency OS project, add one PowerShell check to advance_gate.ps1's DISCOVERY precondition: fail if the latest client-message thread in project-state.json ends in a question with no recorded answer. One fake ticket that ends in a question, one BLOCKED verdict, one passing case — then decide whether to keep it.

Daily Learning: deterministic before LLM

Concept (simple English): for any decision, ask first "could a plain script with no AI make this call?" — comparing a status field, counting, checking CI green, looking up a price. If yes, write the script: it costs nothing, never drifts, and can be unit-tested. Save the LLM for the questions that genuinely need reading comprehension.
How an AI agent might get it wrong: when asked "is this ticket ready to build?", it answers with a confident paragraph of judgment — which can differ run to run, costs tokens, and can't be tested. Worse, it may answer "yes" on a ticket whose last line is an unanswered customer question, because the judgment was vibes, not a comparison.
How you recognize the failure: the same input gives different verdicts on different runs; the verdict cites no concrete field or value you can check; gating logic lives in a prompt instead of a script, so there's no test that pins it.
What detects it: Cromford demonstrates the fix and tests it: bin/triage-gate.sh is a status-field comparison, and tests/evals-suite-test.sh fails if a prompt change alters triage decisions. In Agency OS, the same failure class is caught by gate_engine.py making transitions from project-state.json fields — and by your precommit scripts re-running checks instead of trusting the agent's summary.

Final Summary

Best discovery today: cromford-ai/cromford — Apache-2.0, created Oct 13, 2026, 7 commits today (Nov 1): an unattended software factory whose safety comes from deterministic gates, not better prompts. Verdict: INVESTIGATE — study the design, don't adopt the tool yet.
Best security/reliability improvement: Cromford's deterministic refusal conditions (unanswered question → no start; no failing test → no plan; no deployment in window → no "shipped" claim) — plus verified gap-filler keyleak-detector (MIT) for the security stage it lacks.
Best Agency OS integration: a script-decided DISCOVERY gate that blocks tickets ending in unanswered client questions, plus a per-call cost ledger with alerts on unpriced models — additive evidence for gate_engine.py, replacing nothing.
One practical action today: clone Cromford into a scratch folder and run bin/selfhost-install.sh --config selfhost.conf --dry-run (Docker/WSL2) — read what an unattended factory needs before it touches a real account. Then add the one-line unanswered-question check to your DISCOVERY gate.