Website operations · Fairlead Systems

How Fairlead's own site publishes and checks itself

Each weekday morning the site rebuilds, proves the day's article is really live, tries one recovery rebuild if it is not, and gathers material for the next one.

Swimlane diagram. At 06:45 UK n8n calls the hosting platform's build hook, with up to three tries, waits for the build and counts the live articles, reporting on Telegram. At 07:15 it works out which article is due today and treats it as live only if the page returns success and carries that article's own canonical reference; if not, it triggers one recovery rebuild, waits six minutes, checks again and reports live, recovered or failed. At 07:30 it reads nine official and trade feeds, scores each story against Fairlead's themes, skips anything already stored, saves new stories with a heartbeat record and sends a top-ten digest. The release check reads that heartbeat to spot a missed run.
How the information moves. Colours show what travels along each line; numbers match the steps below. Select the diagram to open it full size; on a small screen it also scrolls sideways.

The problem

This website is static. An article dated for today only appears after a fresh build, and a successful build does not prove that a particular article is online: a static host can answer a missing address with a normal-looking page. Finding fresh, citable material for the next article meant reading many publisher feeds every day, with the same story turning up from several sources.

How it works

  1. Release. At 06:45 UK time n8n calls the hosting platform’s build hook, with up to three tries. After the build it fetches the live articles index, bypassing caches, counts the distinct article links and reports on Telegram. A rejected build request gets its own alert.
  2. Verify. At 07:15 it reads the release schedule, works out which article is due today, and treats it as live only if the page returns success and carries that article’s own canonical reference. If not, it triggers one fresh build, waits six minutes, checks again and reports live, recovered or failed, with what to check next.
  3. Gather. At 07:30 it reads nine official and trade-press feeds, drops items older than four days, scores each story against Fairlead’s themes and skips anything already stored. New stories are saved and a top-ten digest goes to Telegram. Nothing is published: a person decides which stories to research.

The engineering that matters

The liveness test checks the page’s content, not just its status code, so a fallback page served with a success code is never mistaken for the article.

Each scan writes a heartbeat record of how many feeds it reached. The next morning’s release check reads that heartbeat, which catches the quietest failure of all: a scheduled job that simply did not run. A failing feed does not stop the scan, and de-duplication works both within a run and against everything already stored.

Have a process like this?

Tell us what it looks like today. We will say whether this approach would work for you.