Pipelines

The site ships with 10 pipelines that keep the directory fresh: pulling news, crawling tool sites, writing editorial, generating comparisons, and auditing data. Each is a POST /api/cron/* endpoint. Nothing calls them for you — you schedule them (see Deployment) or trigger them by hand.

Pipelines are optional. The site works as a manually curated directory with none of them running. Turn on the ones that fit how much automation you want. The ones marked DeepSeek need a DEEPSEEK_API_KEY (or the key set in /admin/settings → Integrations); without it they skip their editorial step. Every pipeline needs an API key with the publish scope, and every run is logged at /admin/pipelines.

Running a pipeline

• From the console — /admin/pipelines has a "Run now" control for each, plus a force option that ignores the enabled flag.
• By curl:

curl -X POST https://your-domain/api/cron/news \
  -H "Authorization: Bearer $API_KEY"

Each pipeline can be disabled in its config; a disabled run returns early unless you pass ?force=1.

The pipelines

news — the news desk

Fetches every configured RSS source, deduplicates against what's already stored, and drops items older than 45 days. A source is only disabled after 10 consecutive failures, so one flaky fetch is normal. Needs: RSS sources (seeded/managed in /admin/news/sources). No AI. Cadence: hourly.

news-enrich — editorial for news

Ranks all fresh candidates in one pass — clustering duplicate stories and scoring newsworthiness, with source authority breaking ties — then writes a digest, key points, why-it-matters, a category, and related tool links for each. Also fetches an OG image. Needs: DeepSeek. Cadence: every 2 hours.

tools-crawl — liveness and freshness

Sweeps tools whose crawl window has elapsed (hot = daily, normal = weekly, cold = monthly), fetches each site, stores a snapshot, and does status bookkeeping: 3 consecutive failures → offline, a permanent redirect to another host → renamed, a recovery → back to active. Needs: nothing (no AI). Cadence: daily.

tools-enrich — editorial for tools

Reads fresh crawl snapshots and writes each tool's tagline, description, category, tags, features, and structured pricing, plus a quality score. Entries the model judges not to be a real, working product (score below threshold) are kept out of listings rather than published wrong. Needs: DeepSeek. Cadence: daily.

data-audit — consistency checks

A cross-entity audit over the whole catalog. Report-only by default; call it with ?fix=1 to also apply the single-correct-outcome repairs (like dropping references to tools that are no longer publishable). Findings land in the run record, so /admin/pipelines is your audit inbox. Needs: nothing. Cadence: nightly.

tool-discovery — coverage-gap detector

Mines published news stories for AI product mentions, matches each against the catalog deterministically (slug, name, website host), and aggregates the misses. A name that keeps appearing across independent stories is the "hot tool we don't carry yet" signal; once it crosses a threshold it enters the normal submission review funnel. It never invents entries and nothing public renders from it directly. Needs: DeepSeek (to extract names). Cadence: daily.

compare-generate — head-to-head pages

Generates comparison pages for high-value tool pairs — a verdict, when-to-choose-each guidance, and an FAQ — from live tool data. It regenerates only when the underlying tool data changed (or when forced), so it's cheap to run often. Needs: DeepSeek. Cadence: weekly.

stacks-refresh — workflow pages

Refreshes stacks (curated tool workflows) with an intro, ordered steps with handoffs, and an FAQ, keyed off the current member tools. Needs: DeepSeek. Cadence: weekly.

roles-sync — pages by profession

Maps the catalog to job roles, picking the right tools per profession and writing an intro, the picks with reasons, and an FAQ. This is what grows the /for role pages beyond the three the seed provides. Needs: DeepSeek. Cadence: weekly.

tags-sync — topic hubs

Two jobs in one pass: discover tags from live content (post and tool tags), upsert them and refresh cached counts; then write an original intro for tags that lack one. Only enriched tags become indexable, which is what separates a topic hub from a thin taxonomy page. Needs: DeepSeek (for intros; discovery/counts run without it). Cadence: daily.