2027 Orchestrator Awards · Judging

The Judging Methodology

How the Orchestrator Awards evaluates AI-native work — and why it's built this way.

Recognition is a governance act

An award is not a reaction or a matter of taste. It is a decision about who is accountable for a piece of work and whether that work meets a defined standard. We treat every recognition as a governed determination — made by qualified people, against published criteria, on the basis of evidence, with a record of how the decision was reached. If we can't show how a verdict was made, we don't consider it credible.

Human authority is permanent

Final judgment is always the act of named, unconflicted human evaluators. No model decides eligibility, nomination, finalist status, or recognition. If model assistance is ever introduced, it may only help organize information — never exercise judgment — and its scope and limits would be published and separately authorized first. The firewall between assistance and authority does not move.

Evidence before recognition

A work is judged on what can be shown, not what is claimed. Every entry gives the evidence floor on the entry form: the link to the work (its final master); the entrant's name, as the accountable person; the AI systems used; how the entrant directed the AI systems; a production workflow summary; a link to a version-locked evidence dossier, holding the workflow records, prompts and settings, intermediate outputs, and the provenance of every source; and the confirmations that the entrant has the right to submit the work and that rights, likeness, voice and consent are cleared. Each change an entrant makes before entries close is kept as a new version, and judges see the last version made before the close. That evidence lets an evaluator assess craft rather than take a result on faith. Where achievement can't be evidenced, it can't be scored.

Admissibility before scoring

The five eligibility gates

Before any score is calculated, a competitive work must pass five gates. Gates decide whether a work may be evaluated at all; they are pass/fail, and failing any one stops evaluation:

  1. Named human accountability — a named person or disclosed team accepts responsibility for the work.
  2. Material AI-use disclosure — all material AI use is disclosed completely and truthfully. Incomplete or untruthful disclosure is disqualifying.
  3. Rights, likeness, voice & consent — all rights, likeness, voice, and consent are cleared and attested.
  4. Minimum workflow evidence — the complete evidence floor is present and version-locked.
  5. Category eligibility — the work meets this category's specific eligibility statement.

Scoring begins only after all five gates pass.

The ten scoring pillars

Once admitted, a work is scored across ten weighted pillars:

Orchestration Craft

16%

Story and Intent

14%

Identity and Continuity

11%

Technical Execution

10%

Transparency and Disclosure

10%

Human Creative Control

10%

Originality and Creative Novelty

9%

Safety and Rights Discipline

8%

Workflow Evidence

8%

Field Contribution

4%

Total: 100%.

Why disclosure is tested twice

Disclosure appears as both a gate and a scored pillar, and this is deliberate — they measure different things and are not double-counted. As a gate, disclosure is about honesty: did the entrant disclose material AI use completely and truthfully? Incomplete or untruthful disclosure is disqualifying, regardless of the quality of the work. As the Transparency and Disclosure pillar, it's about quality: once disclosure is given, how precise, versioned, and auditable is it? In a medium where the method is part of the art, hiding the method is itself a failure of the work.

Who is qualified to judge

Judging AI-native work requires literacy the work itself demands — the ability to tell a workflow that genuinely solved a hard problem from one that got lucky once. Evaluators are selected for demonstrated competence in the relevant domain. Literacy in one domain is not treated as authority in another; each domain requires its own qualified evaluators.

Conflicts and recusal

Every evaluator confirms confidentiality and discloses conflicts before reviewing. Where a conflict exists — including any relationship to the institution's ownership — the evaluator recuses. These rules are published before operations, and the recusal record is part of the audit trail.

Independent scoring and aggregation

Evaluators score independently. One evaluator cannot see or influence another's assessment before it is locked. Individual scores are then aggregated, so no single voice determines an outcome, and the record shows how the aggregate was formed.

Review, challenge, and correction

Because recognition is a governed act, it is open to challenge. There is a defined process to raise evidence disputes, flag errors, and correct the record where warranted. A verdict that can't withstand scrutiny isn't one we want to stand behind.

The Pioneer Award is different

The Pioneer Award is an honor conferred by the Awards and does not pass through the five gates or the competitive rubric. It is governed by a separate published honorary methodology, with its own conflict record, recusal record, and evidence dossier.