Automated Content Checks: Rules a Machine Can Enforce Before Review

A check is a rule that a machine can answer with yes or no, every time, on every draft, before a person spends a minute on it. Automated content checks are not a quality score and not a second opinion from a model: each one tests a single thing that is either true of the draft or not, and stops the draft when it is not. This post covers which rules a machine can enforce and which it cannot, how to write each check from a real rejection, how to prove a check can fail, how to keep false alarms from teaching people to ignore it, and what our own blog's checks caught when we built them.

Automated content checks: one rule, pass or fail

An automated content check takes one draft and one rule and returns pass or fail, with the reason when it fails. It does not weigh the rule against anything else, and it does not give partial credit. "Every link on the page leads somewhere that exists" is a check. "The article is persuasive" is not, because two careful people could disagree about it, and a rule people can disagree about cannot be enforced by code.

That is what separates a check from a score. Our post on what a content score cannot see treats quality as a person's judgement across several dimensions, rated before publishing. Checks sit underneath that judgement. They clear away everything that has a right answer, so that the person scoring or approving the draft spends their attention on the parts that do not.

The idea is older than software. W. Edwards Deming's fourteen points for management include "Cease dependence on inspection to achieve quality", followed by building quality "into the product in the first place". For content, that means a reviewer should never be the first to notice a broken link or a banned phrase. By the time a draft reaches a person, every mechanical rule has already been tested, and the draft that failed one has gone back without wasting anyone's time.

Pre-publish content checks are one layer of content quality control, not all of it. They run on every draft the system produces, in the same order, without getting tired, and they never decide whether a draft is good enough to publish. That decision belongs at the approval gate, which our guide to designing human approval gates covers in full.

What a machine can check, and what it cannot

A machine can check anything that can be stated as a fact about the text and verified against a list, a file or the page itself. That is a longer list than most teams expect, and it covers most of the mistakes that make a reviewer sigh. The useful test for any rule is whether you could write down, in advance, the exact condition under which it fails.

Rules that pass that test include: no phrase from the banned list appears; the length sits inside the band for its format; every link resolves and every outside link goes to a source on the approved list; every number in the text has a matching line in the source notes; no passage repeats a passage already published elsewhere on the site; the headings run in order with none skipped; every image carries a real text alternative; and the structured data describes the same page the reader sees. None of those needs judgement. Each has a list or a file to check against. The source notes behind the numbers come from listing every claim in a draft by type, which is what makes "has a source" checkable at all.

Some come straight from published standards. The W3C's explanation of the non-text content criterion says "Text alternatives are a primary way for making information accessible", and whether an image has one is a yes or no question a script can answer. Google's help page on the Rich Results Test describes a tool that reports errors in a page's structured data, which is a check someone else has already written. Checks like these are the everyday work of website SEO automation, run on every page rather than once a year.

Duplication deserves its own line. Google's spam policies describe scaled content abuse as many pages made mainly to manipulate rankings rather than help people, and an automated system is exactly the kind of thing that can repeat itself without anyone noticing. A check that compares each new draft against everything already published catches the repeated paragraph before a reader or a search engine does.

What a machine cannot check is whether a claim is true in the world, whether an opinion is one the business holds, whether a joke lands, or whether the piece is worth a reader's time. It is tempting to hand those to a language model and call the answer a check. The 2023 paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, by Lianmin Zheng and colleagues, found that models used as judges show "position, verbosity, and self-enhancement biases". A model's opinion can be a useful flag for a reviewer to look at. It is not pass or fail, and a system that treats it as one has quietly moved a judgement out of a person's hands.

Write each check from a rejection

The best source of checks is the list of reasons a reviewer has already sent drafts back. Every rejection that could have been predicted by a rule is a check waiting to be written, and every one you write is a rejection the reviewer never has to make again. Starting from rejections also keeps the list honest: you build checks for mistakes your system actually makes, not for mistakes you imagine.

Write the rule in a sentence before writing any code. "Every percentage in a draft must appear, with its source, in the brief's notes" is a sentence a reviewer can confirm is what they meant. Only then turn it into a test. Skipping the sentence is how checks end up enforcing something slightly different from what the business wanted, which is worse than having no check, because people stop looking.

Anthropic's guide to defining success criteria and building evaluations offers a useful order of preference for how to test: it lists exact-match, string-match and code-graded tests alongside graded ones, and advises structuring questions "to allow for automated grading". For content, a check that compares strings or reads a file beats one that asks a model, whenever the rule allows it, because it gives the same answer twice.

Some rejections will not become checks, and that is information too. If reviewers keep rejecting drafts for being "flat", no code will catch that, and the fix belongs upstream, in the voice rules or the examples the drafting step reads. Our post on the generic-content check in a repurposing workflow shows the middle ground: a critic step that flags drafts for a person rather than failing them outright.

Prove the check can fail before you trust it

A check that has never failed is not known to work. Before a new check goes into the system, feed it a draft you know breaks its rule and watch it fail, then feed it one you know is clean and watch it pass. Only a check that does both has been shown to work. Teams skip this because a quiet check looks like good news, and that is exactly the danger: a check that reads nothing and a check that finds nothing look identical from outside.

Keep those known-bad drafts. Each check gets at least one example that must fail and one that must pass, stored beside the check and run whenever the check changes. The n8n documentation on why to test a workflow describes the same habit for AI steps: "The foundation of evaluation is running a test dataset through your workflow." A small set of drafts with known answers is that dataset, and it costs an afternoon to build.

Then watch for checks that stop reading. A check that searches the page for a pattern will report nothing if the page's structure changes and the pattern stops matching. The fix is a floor: if the check reads fewer items than it should, it refuses to run instead of passing. A refusal is a problem someone will notice. A silent pass is not, and it can last for months.

This is the part of building checks that turns a script into a control. It is unglamorous, and it is the only reason anyone should believe a green result.

False alarms: calibrate on what you have already published

Before a check goes live, run it on the content you have already published and approved, and read every finding. Your archive is the best calibration set you have, because a person has already decided each of those pieces was good enough. If a new check flags a large share of them, either the rule is wrong or your standard has changed, and you need to know which before the check starts blocking drafts.

False alarms are not a minor cost. A check that flags good work trains everyone to override it, and once overriding is a habit, the check catches nothing even when it is right. Jakob Nielsen's point about confirmation dialogs applies here: people who are warned too often stop reading warnings. The aim is a check that fires rarely enough to be taken seriously every time it does.

Calibrating usually means narrowing what the check reads, not lowering the bar. A rule about link text that reads only the link's own words will flag honest citations whose sentence names the source; reading the sentence as well removes most of those without letting a vague link through. Read the findings, find the pattern behind the false ones, and teach the check that pattern. Then run it on the archive again.

Order, speed, and a message that names the fix

Run the cheapest, most certain checks first and stop at the first group that fails. Banned phrases, length and broken links take a moment and fail often in early drafts; comparing a draft against the whole archive takes longer and is worth running only on a draft that has passed the rest. A pipeline that runs everything every time, slowest first, makes drafting feel slow and tempts people to skip the checks.

A failed check should stop the line, in the plain sense. Toyota's page on the Toyota Production System describes equipment that halts when it detects a problem: "When equipment stops, the andon (problem display board) lights up to notify workers of the abnormality." In a content workflow the equivalent is a draft that does not move on, with the failure shown to whoever can fix it.

The message matters as much as the stop. Nielsen Norman Group's error-message guidelines, by Tim Neusesser and Evan Sunwall, put it simply: "Merely stating the problem is also not enough; offer some potential remedies." "Link check failed" sends someone hunting. "The third link in section two goes to a page that no longer exists; replace it or remove the sentence" gets it fixed.

Where the checks run depends on the tool. In n8n, the workflow tool we use for routing, an If node can "split a workflow conditionally", sending a failing draft back with its reasons and a passing one on to review.

In Claude Code, where drafts are written, checks can run every time a draft file changes. Its guide to automating actions with hooks describes commands that run "at specific points in its lifecycle", which "gives you deterministic control: certain actions always happen rather than relying on the LLM to choose to run them." That is the property a check needs, and our Claude Code guides for content work show the rest of such a project.

What checks automate, and what stays with the reviewer

Checks automate every rule that has a right answer, and leave every rule that needs judgement to a person. Drawn that way, the line is clear on paper and blurs in practice, because a check that works well tempts people to add one more that almost works.

Automated: banned phrases, length bands, link health, approved sources, numbers matched to notes, repeated passages, heading order, text alternatives, structured data, frozen dates. Also running the known-bad examples whenever a check changes, and refusing to run when a check reads less than it should.

Human: whether a claim is true, whether an opinion belongs to the business, whether a piece is worth publishing, which new checks to write, and whether a finding is a real fault or a rule that needs changing. A person also decides when a check is wrong and the draft is right, and that decision should change the check, not just let the draft through once. Our post on the publishing chores a machine does well draws the same line for the SEO layer.

A worked example: the checks behind this blog

This blog runs on its own checks, so the clearest example we can give is ours. Every post, this one included, passes a set of automated content checks before it is published: one that compares it with every post already on the site, one that counts its links and where they go, one that compares its figures with other posts' figures, and several that read its structured data and its registration in the sitemap.

The overlap check is the one built most directly from a rejection: "this reads like a post we already have". When we first ran it across the existing posts, it reported 62 errors, including shared passages of 20 to 38 words, seven posts opening with the same four words, and two posts with an identical main heading. Because those posts were already published and approved, we did not change them; the check now applies to new posts, and each new post must clear it with no errors. That is calibration on the archive in practice: the findings told us what our standard actually was.

The link check taught us about false alarms. One of its rules asks whether a link's words name where it goes. Its first version flagged 168 of the 1,123 links already on the blog, about 15 per cent, and most of those were honest citations whose surrounding sentence named the source. Reading the sentence, not just the link, brought it to 67, about 6 per cent, and what was left were genuinely vague links. A check flagging one link in seven would have been switched off within a month.

Two checks taught us to prove failure. A colour check on the figures printed nothing and exited as a pass on its first run: the line that decides whether the script should run compared two paths, one of them encoded because the folder name contains a space, so the check never ran at all. And while writing about internal links we found that our link checker confirms a linked page exists but ignores the part of a link that points to a section, so a link to a renamed section would pass. We said so in that post on what link automation can and cannot check rather than claim a check we do not run.

Every check on the blog now ships with examples that must fail, and each new rule was broken on purpose once and watched failing before it was trusted. None of this replaced the person who reads each post before it goes out. It changed what that person reads for.

What checks cost to keep honest

Checks are cheap to run and expensive to keep honest, and most of the cost is attention rather than build time. Writing a check from a clear rule takes an hour or two. Building its known-bad examples, calibrating it on the archive and writing a message that names the fix takes longer, and that is the part that decides whether anyone trusts it.

After that, the cost is maintenance, the same review habit that is keeping a content operating system from going stale. A check is code, and it can break as the drafts, the templates and the tools change. Each one needs an owner, a reason written beside it, and a look whenever it has been quiet for a long time, because a check that never fires has either solved its problem or stopped reading. Re-running its known-bad examples answers which in a minute. Published pages need the same care as drafts, since facts and links change underneath them; our post on catching the facts and links that changed covers that side.

There is also a cost in saying no. Every check you add is one more thing a draft can fail, and a long list of marginal checks slows the system and buries the important failures among trivial ones. Keep the list to rules that came from real rejections, delete checks whose rule the business no longer holds, and resist the check that "might be useful". The pillar page on content automation as a specialist builds it covers where checks sit in the larger system.

Frequently asked questions

What are automated content checks?

They are rules a machine tests on every draft before a person reviews it, each returning pass or fail with a reason. Examples include banned phrases, length, broken links, unsourced numbers, repeated passages and missing image descriptions. They handle anything with a right answer, so the reviewer can focus on judgement.

How are checks different from content scoring?

A score is a person's judgement of quality across several dimensions, given as a rating. A check tests one rule and passes or fails. Checks run first and clear away mechanical mistakes; scoring and approval come after, on the parts of a draft that need a person.

Can an AI model be used as a check?

Only as a flag, not as pass or fail. Research on models used as judges has found position, verbosity and self-enhancement biases, so a model's opinion should send a draft to a person for a look, not decide on its own whether the draft passes.

How do you know a check actually works?

Give it a draft that breaks its rule and confirm it fails, and a clean draft and confirm it passes. Keep those examples and run them whenever the check changes. A check that reads less than it should must refuse to run rather than pass.

What should you do when a check flags good content?

Treat it as a fault in the check. Run new checks on content you have already approved before they go live, read every finding, and narrow what the check reads until it stops flagging good work. A check that raises false alarms trains people to ignore it.

Which checks should come first?

Start with the reasons reviewers already reject drafts for, and turn the ones that can be stated as exact rules into checks. Run the fast, certain ones first, such as banned phrases, length and links, and the slower ones only on drafts that pass them.

Where to start

Pull the last month of rejection reasons and sort them into two piles: the ones you could have predicted with a written rule, and the ones that needed judgement. Write the first pile as sentences, choose the one that comes up most often, and build that check with one draft that must fail and one that must pass. Run it on what you have already published before it touches a new draft.

Then add the next, one at a time. If you would rather have the checks, the gate and the system around them built with you, that is what our work on building content automation for a business covers. For where checks sit in the larger design, see our Content OS guides and our systems thinking guides.

Keep reading

• Content scoring: how to evaluate quality before publishing — the judgement that comes after the checks have cleared the mechanical faults.
• How to design human approval gates — the gate a checked draft arrives at, and the rejections new checks are written from.
• How to build a content repurposing workflow — a critic step that flags generic drafts for a person rather than failing them.