Orchestration¶
PROTEA the platform is just one repository in a larger working tree.
The same machine usually also hosts the LightGBM lab
(protea-reranker-lab), the LaTeX manuscript (thesis/), and a
parallel orchestration system that drives long-running engineering
work via automated coding agents (agent-farm).
This page describes how those pieces relate to PROTEA at runtime: the shared filesystem layout, the boundaries the orchestration system respects, and the contract surface the platform exposes for agent consumption.
Working tree layout¶
The canonical working tree at ~/Thesis2/ looks like this:
~/Thesis2/
├── repositories/ # 8 git repos cloned at default branches
│ ├── PROTEA/ # this repo (develop trunk)
│ ├── protea-method/ # inference layer
│ ├── protea-backends/ # PLM embedding plugins
│ ├── protea-reranker-lab/ # offline LightGBM lab
│ ├── protea-runners/ # experiment runner plugins
│ ├── protea-sources/ # annotation source plugins
│ ├── protea-contracts/ # ABCs / payloads / schema
│ └── cafaeval-protea/ # cafaeval fork
├── agent-farm/ # agent orchestration system
├── thesis/ # LaTeX manuscript
├── storage/ # local artefact store
├── backups/ # Postgres + MinIO + lab snapshots
└── CLAUDE.md # root-level project context
Every cross-repo path that PROTEA scripts emit is anchored at this root.
Moving the tree (e.g. ~/Thesis2/ → /srv/protea/) requires
updating protea/config/system.yaml and the
PROTEA_DEPLOY_PATH env var; nothing else.
The agent-farm orchestration system¶
agent-farm is a separate public repository, intentionally agnostic to PROTEA: it ships a prompt + yaml registry for a dozen automated coding agents and a sqlite-backed task / heartbeat / result state. The conductor session spawns subagents in natural language, persistent services run as headless tmux windows that escalate to a human operator only on failure, and one-shot batch jobs invoke the same code paths a human would call from the CLI.
Why it lives outside PROTEA¶
PROTEA is a research deliverable; the orchestration system that drove its construction is engineering tooling. Splitting the repos:
keeps PROTEA’s build, CI, and citations focused on the platform
lets the orchestration system evolve and be reused on other projects (the prompts mostly say
read your project's CLAUDE.md, notread PROTEA)keeps PROTEA’s CI matrix small (the agent-farm Python is bash + a few hundred lines of helpers, no production code)
What the orchestration system reads from PROTEA¶
Agents are clients of the public API surface, not of the internals:
GET /openapi.jsonfor the API contractPOST /jobsto dispatch operations from the registry (insert_proteins,compute_embeddings,predict_go_terms,run_cafa_evaluation,export_research_dataset, …)GET /jobs/{id}and/eventsto poll progress and surface failuresGET /datasets/{id_or_name}andGET /scoring/rerankersfor the artefact registryprotea/config/system.yamlandapps/web/.env.localto discover the local URLs
What the orchestration system writes to PROTEA¶
Only via PRs. Direct edits to ~/Thesis2/repositories/PROTEA/ are
forbidden; agents work in ephemeral worktrees under
~/Thesis2/worktrees/<task-id>/ so the developer’s tree is never
disturbed by an automated session.
Hard boundaries¶
The orchestration system imposes the same hard constraints PROTEA does
(no force push, no --no-verify, no git stash, no third-party
co-author trailers on commits). It additionally refuses to:
modify branch-protected refs (
main/develop)restart the dev stack without explicit user permission
bypass the smell budget (
scripts/check_smells.py) on any PR
Plan tracking¶
The agent-farm repo carries the canonical plan store at
agent-farm/plans/: one Markdown file per loop with YAML
frontmatter for every slice. The frontmatter encodes id,
phase, status, deps, acceptance_criteria and
priority; the master PLAN.md is auto-generated by
plans/render.py from those frontmatters.
PROTEA-side slices live in agent-farm/plans/executor/PLAN.md.
The convention for spawning an executor task is that
spawn_args.slice MUST match a slice id whose status is
pending and whose deps are all done. The
plans/render.py --next helper prints the next pickable slice per
loop.
Re-ranker lab boundary¶
protea-reranker-lab is the second non-PROTEA repository the
machine hosts. It owns LightGBM training and offline study work. The
contract between PROTEA and the lab is the artefact store:
PROTEA publishes a
Datasetrow +train.parquet/eval.parquet/manifest.jsonviaexport_research_datasetthe lab pulls via
pull_dataset.py, trains a booster, writesruns/<run_id>/{model.txt, spec.yaml, run.json}the lab then calls
POST /reranker-models/import(multipart) orPOST /reranker-models/import-by-reference(JSON, booster already in MinIO) and PROTEA inserts theRerankerModelrow
The lab never reads PROTEA’s database; PROTEA never reads the lab’s
runs/ directory. The artefact store is the only shared substrate
(MinIO in prod, local FS in dev).
The methodological champion as of 2026-05-25 is the binary-label LightGBM
booster trained on the bench-v1-K5-v226-lineage-prostt5 dataset (axis
tuple: plm=prostt5, k=5, rr=binary, feat=generalist, eval=v226, prop=lineage,
ens=none; NK+LK Fmax 0.7291 +/- 0.0028, three-seed replication). The next
active loop is F-DATA-PACK, which packages the 24-dataset grid produced by
FARM-EXP.13 as FAIR-compliant research artefacts. See
ADR-D38: Defer neural-head champion; pivot to curated dataset packaging for the decision to
defer the neural-head track and prioritise dataset packaging.
Thesis manuscript boundary¶
thesis/ is a local-only git repo with the LaTeX manuscript.
PROTEA neither reads nor writes it at runtime. The thesis-writer
orchestration agent imports numbers from EvaluationResult rows and
figure scripts from protea-reranker-lab/runs/ but does so by
reading exported artefacts (CSV, parquet, PNG), never the live DB,
so a thesis build is reproducible from a frozen snapshot.
Plugin contract surface¶
PROTEA discovers runtime capabilities through four distinct plugin
layers, all mediated by Python entry_points so the platform code
never hardcodes backend or source names.
┌───────────────────────────────────────────────────────────────────────┐
│ PROTEA platform (this repo) │
│ │
│ OperationRegistry ← protea.core.operation_catalog (live list) │
│ PluginLoader ← entry_points discovery at import time │
│ │
│ entry_points group discovered from example keys │
│ ───────────────────── ──────────────────── ───────────────── │
│ protea.backends ← protea-backends esm, t5, ankh, │
│ (PLM embedding) esm3c │
│ │
│ protea.sources ← protea-sources goa, quickgo, │
│ (annotation source) uniprot │
│ │
│ protea.runners ← protea-runners lightgbm, knn, │
│ (experiment runner) baseline │
│ │
│ (method delegation) ← protea-method protea_method │
│ (pure inference lib) .pipeline.predict() │
│ │
└───────────────────────────────────────────────────────────────────────┘
┌─────────────────────┐ ┌─────────────────────┐ ┌──────────────────────┐
│ protea-contracts │ │ protea-backends │ │ protea-sources │
│ (shared ABCs + │◀───│ ESM / T5 / Ankh / │ │ GOA / QuickGO / │
│ payloads + │ │ ESM3-C wrappers │ │ UniProt plugins │
│ feature schema + │◀───│ │◀───│ │
│ schema_sha) │ └─────────────────────┘ └──────────────────────┘
└──────────┬──────────┘
│ imported by all four plugin repos + PROTEA platform
▼
┌─────────────────────┐ ┌─────────────────────┐
│ protea-method │ │ protea-runners │
│ pure inference: │ │ experiment runners: │
│ KNN, feature │◀───│ LightGBM, KNN │
│ compute, reranker │ │ baseline │
│ apply │ └─────────────────────┘
└─────────────────────┘
protea-contracts is the only cross-cutting import. Every other
repo is a leaf: it imports protea-contracts but nothing else from
the PROTEA tree. This keeps CI matrices small and prevents circular
dependencies.
F-LAFA container deployment topology¶
Three containers were submitted to the LAFA benchmark, all built on
top of the protea-method-runtime base image (ADR-D15). Each
container is self-contained: embeddings, reference annotations, and
optionally a LightGBM booster are frozen into a bundle at build time.
┌───────────────────────────────────────────────────────────────────────┐
│ protea-method-runtime (base image) │
│ python + protea-method + protea-contracts + PLM weights │
│ entrypoint: protea_predict.py │
└──────────────────────┬────────────────────────────────────────────────┘
│ FROM protea-method-runtime
┌───────────────┼───────────────┐
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ lafa_knn_v1 │ │lafa_knn_8plm│ │ lafa_v18 │
│ │ │ │ │ │
│ 1 PLM │ │ 8 PLMs │ │ 1 PLM │
│ KNN only │ │ ensemble │ │ KNN + v6_x │
│ no reranker │ │ mean agg. │ │ + reranker │
│ no v6_feats │ │ no v6_feats │ │ (full pipe) │
│ aspect-sep │ │ aspect-sep │ │ aspect-sep │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
└───────────────┴───────────────┘
│ bind-mount at container run time
┌───────────────────────▼──────────────────────────────────────────────┐
│ /bundle frozen reference (embeddings .parquet + GO map) │
│ /input/ queries.fasta (LAFA evaluator supplies) │
│ /output/ predictions.tsv (written by entrypoint) │
│ /hf-cache HuggingFace model weight cache │
└──────────────────────────────────────────────────────────────────────┘
Container source directories:
apps/lafa_knn_v1/(single-PLM KNN baseline)apps/lafa_knn_8plm/(8-PLM ensemble viaensemble_driver.py)apps/lafa_v18/(full PROTEAlafa_v18pipeline with reranker)