writing-eval: Local Quality Control for AI Writing

A draft can be grammatically correct and still sound wrong.

Sentences may be clean and the facts intact, yet the piece no longer sounds like the author, publication, or company readers recognize. A prompt edit, model update, or new agent can cause that drift without producing an obvious failure.

Most teams catch it through memory and taste: an editor reads the draft, senses that something is off, and starts marking lines. That process can work, but it doesn't give an agent or CI job a stable definition of acceptable work or make regressions easy to compare across versions.

writing-eval is a local, deterministic editorial regression tool. It turns authorized reference prose and configurable rules into repeatable style-alignment evidence for people, agents, and CI before a draft ships. A person still decides what the evidence means.

What writing-eval checks

The current public checkout supports the four workflows below.

1. Check one draft

A single-draft check scans Markdown or plain text, and reports configured findings. You can also pass - to read from standard input.

This is the fastest way to inspect a draft before review:

./writing-eval check path/to/draft.md

Use it when you want a local report without first building a style profile. The result can identify rule findings and review candidates, but it doesn't compare the draft with a specific author's corpus unless you supply a profile or references.

The pinned checkout's built-in rules contain 34 rules: 16 warn and 18 info. No builtin rule blocks by itself.

Add --format json to write structured output to standard output, or use --json PATH to write the same result to a file. The file form is usually easier for automation because it separates the report artifact from other process output.

The check command accepts Markdown, plain text, or - for standard input. The --style and --references options are mutually exclusive: a named profile uses stored profile data, while direct references supply a corpus for the current check.

2. Compare a draft with a style profile

A style profile records statistical patterns from an authorized body of prose. A check can then compare a new draft with that baseline:

./writing-eval check draft.md --style acme

The profile check produces a heuristic score with four sections:

• clarity and directness
• readability
• rhythm and structure
• vocabulary and style

Profile-relative allowances prevent a generic rule from overruling patterns that the source corpus repeatedly demonstrates. Findings inside the profile-derived allowance don't become excess issues, so the checker doesn't flatten every author into the same generic prose.

3. Apply a custom rule overlay

The current checkout includes an optional anti-AI overlay at rules/anti-ai.yaml and supports custom rule overlays such as the example below.

Save this complete overlay as rules/acme-editorial.yaml:

extends: builtin
rules:
  - id: company_jargon
    severity: warn
    detector: '(?i)\b(?:synergy|ideate)\b'
    message: Use plain language.

  - id: passive_voice
    enabled: false

Each new rule needs id, severity, detector, and message. Existing rules can override selected fields or be disabled with enabled: false. This expanded example was executed successfully against the pinned checkout.

Run the overlay with:

./writing-eval check draft.md --rules rules/acme-editorial.yaml --format json

Keep the file under version control as editorial policy. A team can add agreed phrases or remove a generic warning that conflicts with its voice. Review every change as policy, not as a trick for raising a score.

Don't tune the overlay after seeing a weak draft just to make that draft pass. Version the rule file, explain changes in review, and compare runs only when they use the same rules.

4. Evaluate a corpus

Corpus evaluation compares a set of outputs with a reference corpus and writes Markdown and JSON reports:

./writing-eval eval \
  --outputs runs/release-candidate \
  --references data/reference-corpus.jsonl \
  --report /tmp/writing-eval-report.md \
  --json /tmp/writing-eval-report.json

Use this workflow to compare a release candidate, prompt revision, or model change across many examples. The explicit eval subcommand is preferred, although the repository retains a backward-compatible flat invocation.

Corpus evaluation can also report literal-preservation diagnostics. Those checks can flag changed numbers, dates, URLs, or quoted spans. They can't decide whether an unsupported claim is true or whether a paraphrase preserves the original meaning.

Install the public checkout

writing-eval requires Python 3.11 or newer and uv. PyYAML is its only runtime dependency.

git clone https://github.com/majesticlabs-dev/writing-eval.git
cd writing-eval
uv sync
./writing-eval --version

Build a private style profile

The profile command accepts one non-empty article, but that is only enough to build a profile. For a style baseline, use at least 25 articles and prefer 40 or more.

A smaller profile can still be useful in a controlled comparative benchmark. Keep the profile, rules, drafts, and tool version fixed, then compare systems or prompt versions against the same baseline. The result measures relative change under that setup, not a stable estimate of the author's full style.

This recommendation comes from a study of 101 long-form articles. With 25 articles, 70.8% of held-out drafts had score noise at or below the rubric's 2-point resolution. With 40 articles, coverage reached 87.5%. Treat score differences under 3 points as noise.

Article count matters more than total word count. At the same 40,000-word budget, 28 shorter articles produced about half the score noise of 11 longer articles. The earlier 15,000-word minimum was not measured, and the current study supports no numeric word floor. The shortest tested article was 517 words, so profiles made from very short posts remain unmeasured.

Use prose from the same author, genre, and register. Mixing blog posts, support tickets, keynote transcripts, sales emails, or several authors creates a blended baseline that is harder to interpret.

Build and use a profile with these commands:

./writing-eval profile build acme --from posts/acme
./writing-eval profile list
./writing-eval check draft.md --style acme

A rebuild replaces the existing profile because there is no incremental append command. When the authoritative corpus changes, rebuild from the complete corpus rather than adding only the newest files.

After a rule change, profile cache can refresh precomputed reference statistics. A stale or missing cache can slow checks, but it doesn't make the result incorrect.

Keep the corpus authorized and private

Only process prose your team is allowed to use. A public URL doesn't automatically grant permission to copy an author's full body of work into an internal style system.

Keep private drafts, third-party source prose, and private profiles under the repository's git-ignored data/ tree. Don't commit them. Check staged changes before every push, especially after moving or renaming corpus files.

A profile is a statistical baseline, but its references.jsonl retains the full corpus text verbatim. It isn't model training, author cloning, or permission to reproduce another person's work. Authorization comes first.

Reports can contain matched spans, excerpts, and source paths. Restrict access to local machines and CI runners, keep report artifacts private, and use short retention periods. Redact sensitive text before evaluation, or don't upload profiles and reports when policy or regulation requires data to remain on an approved system.

Read the report without worshipping the score

The score summarizes configured rules and profile alignment. Read the findings behind it.

A profile report groups evidence into clarity and directness, readability, rhythm and structure, and vocabulary and style. Review candidates appear separately from scored issues, and very short or non-prose inputs can be unscored.

Under profile-alignment-v2, each of the four sections starts at 25 points. Each excess warn occurrence deducts 2 points. An info finding deducts 0. Profile-relative metric gaps deduct points only after their documented tolerances. Each section is capped at 25 points. Treat the score as context after reviewing the findings and deltas.

The underlying measurements can include:

• rule findings and normalized tell rates
• average sentence length and sentence-length variance
• repeated sentence openings
• Flesch reading ease and Flesch-Kincaid grade
• MTLD lexical diversity
• token 1-gram L2 distance and overrepresented terms when references exist
• paragraph statistics
• literal-preservation diagnostics in corpus evaluation

Flesch scores estimate reading difficulty, but this implementation uses a syllable heuristic. Token and sentence boundaries are simpler than those from a full linguistic parser. MTLD describes lexical diversity, not precision or insight. Token distance can reveal vocabulary drift, but a small or mixed corpus can make that signal noisy.

Output length matters too: a 300-word release note and a 3,000-word tutorial don't create directly comparable distributions. Genre, register, sample size, and corpus composition can materially change the numbers.

Deterministic means the same version produces repeatable results for the same inputs. Repeatable results can still reflect biased or ill-suited rules.

The normal check and corpus-analysis paths are local and don't call hosted models. The optional historical benchmark generation script can invoke an authenticated external OpenAI Codex CLI.

Add writing-eval to an agent review loop

The repository includes skills/writing-eval/SKILL.md. Let your Agent system read it and help you integrate with your workflow.

A safe agent loop has five steps:

1. Draft against the approved brief and source material.
2. Run writing-eval with the pinned profile and rule version.
3. Read the report or JSON, then select findings that match team policy.
4. Edit the draft while preserving facts, commands, links, and required language.
5. Run the same check again and send the result to a person for acceptance.

Keep the target fixed during the loop. The agent may edit the draft. It must not edit the profile, corpus, or rules to improve its own result.

Define your own CI policy over JSON

The pinned public checkout's check behavior requires careful handling. Exit code 0 means the analysis completed, even when warn or info findings exist. Exit code 1 covers usage or input errors.

An exit code of 0 isn't an editorial pass. Your automation must inspect the report.

A safe wrapper writes the current run to a fresh file and checks the command before parsing. The policy below is only an example: it fails when an assessed report contains a high-priority assessment issue. Each team must choose its own policy.

#!/usr/bin/env bash
set -eu

report="$(mktemp "${TMPDIR:-/tmp}/writing-eval-check.XXXXXX.json")"
trap 'rm -f "$report"' EXIT

set +e
./writing-eval check draft.md --style acme --json "$report"
writing_eval_status=$?
set -e

if [ "$writing_eval_status" -ne 0 ]; then
  printf 'writing-eval could not analyze the input\n' >&2
  exit "$writing_eval_status"
fi

# Example policy: fail on high-priority assessment issues.
python3 - "$report" <<'PY'
import json
import sys
from pathlib import Path

path = Path(sys.argv[1])

try:
    report = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeError, json.JSONDecodeError) as error:
    raise SystemExit(f"current writing-eval report is absent or malformed: {error}")

if not isinstance(report, dict):
    raise SystemExit("current writing-eval report must be a JSON object")

assessment = report.get("assessment")
if not isinstance(assessment, dict):
    raise SystemExit("current writing-eval report has no assessment object")

if assessment.get("status") != "scored":
    raise SystemExit("writing-eval did not produce a scored result")

issues = assessment.get("issues")
if not isinstance(issues, list) or not all(isinstance(item, dict) for item in issues):
    raise SystemExit("current writing-eval report has malformed assessment issues")

blocking = [item for item in issues if item.get("priority") == "high"]
if blocking:
    for item in blocking:
        print(item.get("summary", "high-priority assessment issue"), file=sys.stderr)
    raise SystemExit(2)
PY

mktemp prevents a previous report from being mistaken for the current run. When writing-eval fails, the shell exits before parsing. The Python block rejects absent, malformed, or unscored output, and the trap deletes the report on every exit path.

This separates four states that are easy to collapse by accident:

1. The CLI or input failed.
2. The analysis completed.
3. Team policy found something that requires review.
4. A person or separate approved policy accepted the draft.

The team owns steps three and four. writing-eval supplies evidence. It doesn't decide which warn findings block a release, and the current CLI doesn't claim that it does.

For stable CI comparisons, pin the tool version, profile, corpus, and rules. If CI stores a report, keep the artifact private and retain it only as long as the review requires. When any evaluation input changes, record that change and establish a new baseline.

Know the limits before adopting it

writing-eval fits workflows that need local, repeatable writing checks and have enough authorized prose or explicit editorial rules to define a target. It is especially useful when several people, agents, prompts, or models produce drafts that should share one review standard.

It may be a poor fit when the team has no coherent source corpus, no agreement about editorial policy, or no person responsible for interpreting findings. In that case, the tool can produce precise numbers around an undefined target.

Keep these boundaries visible:

• It isn't an AI detector and can't prove that prose is human-written.
• It doesn't check factuality, originality, or plagiarism.
• It doesn't measure argument strength, reader preference, or overall writing quality.
• It isn't a model, training system, or hosted judge.
• Its scoring reflects configured rules and corpus choices, not a universal standard.
• It supplies evidence for editorial review but doesn't replace acceptance by a person or an approved team policy.

Also review the license before embedding the tool in a product or service. writing-eval uses the Elastic License 2.0. Subject to that license, public source may be used locally, copied, modified, redistributed, and self-hosted. The license restricts providing a hosted or managed service that exposes a substantial set of the software's features.

Read the license itself and get qualified advice for a commercial use case. This article isn't legal advice.

Start with one representative draft

After installation, pin the inspected version and run one draft that your editors know well. Compare the report with their existing review notes.

Then ask a concrete question: did the tool expose a repeatable problem the team wants to check again?

If yes, define that problem in a versioned rule or a coherent private profile. Run the same check on several drafts. Keep human acceptance separate from automated analysis.

One representative draft is enough for a useful first evaluation: a local check against a target your team controls.

If your team needs shared profiles, organization-specific CI policy, or managed deployment, tell Majestic Labs which editorial decision you need to make repeatable.

Resources

https://github.com/majesticlabs-dev/writing-eval

https://majesticlabs.dev/open-source

https://majesticlabs.dev/blog/202607/building-private-ai-evals