How Fairlead's own site publishes and checks itself
Each weekday morning the site rebuilds, proves the day's article is really live, tries one recovery rebuild if it is not, and gathers material for the next one.
The problem
This website is static. An article dated for today only appears after a fresh build, and a successful build does not prove that a particular article is online: a static host can answer a missing address with a normal-looking page. Finding fresh, citable material for the next article meant reading many publisher feeds every day, with the same story turning up from several sources.
How it works
- Release. At 06:45 UK time n8n calls the hosting platform’s build hook, with up to three tries. After the build it fetches the live articles index, bypassing caches, counts the distinct article links and reports on Telegram. A rejected build request gets its own alert.
- Verify. At 07:15 it reads the release schedule, works out which article is due today, and treats it as live only if the page returns success and carries that article’s own canonical reference. If not, it triggers one fresh build, waits six minutes, checks again and reports live, recovered or failed, with what to check next.
- Gather. At 07:30 it reads nine official and trade-press feeds, drops items older than four days, scores each story against Fairlead’s themes and skips anything already stored. New stories are saved and a top-ten digest goes to Telegram. Nothing is published: a person decides which stories to research.
The engineering that matters
The liveness test checks the page’s content, not just its status code, so a fallback page served with a success code is never mistaken for the article.
Each scan writes a heartbeat record of how many feeds it reached. The next morning’s release check reads that heartbeat, which catches the quietest failure of all: a scheduled job that simply did not run. A failing feed does not stop the scan, and de-duplication works both within a run and against everything already stored.
Have a process like this?
Tell us what it looks like today. We will say whether this approach would work for you.