Building a Production-Grade AI Trends Pipeline in n8n: What Actually Breaks

Collecting "what's trending in AI" sounds like a weekend project — pull a few RSS feeds, hit a couple of APIs, dump it into a sheet. It stays simple for about a week, until Reddit rate-limits you mid-run, Product Hunt changes a response field, and your LLM step returns Markdown-wrapped JSON that silently breaks everything downstream.

This is a workflow that pulls AI-related content from eight sources — Reddit, Hacker News, Product Hunt, GitHub Trending, Google News RSS, Dev.to, Medium, and YouTube — normalizes the responses into one schema, deduplicates and scores the results, sends the top candidates to an LLM for structured summarization, validates the output, stores it, and emails a daily report. What follows is what actually failed in production and how it got hardened, not a node-by-node tutorial.

Workflow Architecture

The pipeline runs in five stages, each isolated behind a Merge node so one source failing doesn't take down the others:

Every collection branch runs with Continue On Fail enabled at the node level — the single most important decision here. A dead API should degrade the dataset, not kill the run. Every HTTP node downstream is built assuming its upstream call will eventually fail.

Data Collection

Each source needs different auth, pagination, and rate-limit handling — treating them uniformly is the first mistake.

Reddit's OAuth token expires hourly and the API silently returns HTML instead of JSON when the token is stale. The node requests the full response object (not just the body) so headers can be inspected before parsing:

{
  "method": "GET",
  "url": "https://oauth.reddit.com/r/artificial/top",
  "authentication": "predefinedCredentialType",
  "nodeCredentialType": "redditOAuth2Api",
  "queryParameters": { "parameters": [{ "name": "t", "value": "day" }, { "name": "limit", "value": "50" }] },
  "options": { "response": { "response": { "fullResponse": true } }, "timeout": 8000 }
}

If Content-Type isn't application/json, it's routed to token refresh, not the normalizer.

Hacker News has no auth but requires N+1 calls (story IDs, then one per story). Fetching 50 stories serially adds 10-15 seconds. IDs are fetched once, then processed with Split In Batches (size 10) plus a 500ms Wait between batches — HN doesn't publish a rate limit, but connection resets started above ~20 concurrent requests.

Product Hunt's GraphQL API uses a point budget (6250 per 15 min). Rather than count points manually, a Code node reads X-Rate-Limit-Remaining from response headers and short-circuits further calls in the same run below a threshold:

const remaining = parseInt($input.first().json.headers['x-rate-limit-remaining'] || '9999', 10);
if (remaining < 200) {
  return [{ json: { source: 'producthunt', skip: true, reason: `rate_limit_low:${remaining}` } }];
}
return $input.all();

GitHub Trending has no official API — this hits an unofficial proxy that's had multi-hour outages. Continue On Fail plus a fallback to GitHub's search API (repos created in 24h, sorted by stars) keeps the pipeline alive with a slightly worse trending signal.

Google News RSS wraps article URLs in a redirect that must be resolved before dedup compares them — otherwise every hit looks "unique" even when seen elsewhere. Medium's per-tag RSS caps at 10 items with no pagination, so four tags are queried and merged. YouTube's Data API has a hard daily quota; the workflow caches the last run's timestamp and uses publishedAfter to avoid re-scanning the same window rather than burning quota on overlapping searches.

Data Normalization

Eight sources means eight shapes. A single Code node maps everything to one schema after the merge:

function normalizeReddit(d) {
  return {
    id: `reddit_${d.id}`, source: 'reddit', title: d.title,
    url: d.url_overridden_by_dest || `https://reddit.com${d.permalink}`,
    author: d.author, publishedAt: new Date(d.created_utc * 1000).toISOString(),
    engagement: d.score + d.num_comments * 2
  };
}

const normalizers = { reddit: normalizeReddit /* ...one per source, same shape */ };

const results = [];
for (const item of $input.all()) {
  const fn = normalizers[item.json.source];
  if (!fn) continue; // unknown source: drop, don't crash
  try {
    results.push({ json: fn(item.json.data || item.json) });
  } catch (e) {
    results.push({ json: { source: item.json.source, error: e.message, skipped: true } });
  }
}
return results;

The try/catch is per item, not per batch. An early version wrapped the whole loop in one try/catch, and a single malformed Reddit item (a deleted post missing created_utc) dropped all 50 items from that source for the run.

Failure Scenarios

This is where most of the engineering time actually went. Each of these is a real incident.

API timeouts. Dev.to's node would hang for the full 30-second default roughly twice a week, stalling the whole run since downstream nodes waited on it — detected via execution-duration alerts. Fixed by dropping the timeout to 8000ms everywhere plus Continue On Fail: a slow API loses that run's data instead of blocking every other source. Retrying a server-side queueing timeout rarely helps inside the same window; the next scheduled run catches up.

Rate limits (429). Product Hunt returned bursts of 429s when a manual re-run overlapped the scheduled one — no distributed lock meant two executions double-counted against quota. Fixed with retry-with-backoff (maxTries: 3) plus a Wait node reading the actual Retry-After header instead of a guessed delay:

{ "retryOnFail": true, "maxTries": 3, "waitBetweenTries": 2000 }

Fixed backoff without reading Retry-After either wasted time or triggered another 429; the header is already provided, so using it is strictly better.

Invalid JSON. The LLM occasionally wrapped output in Markdown fences or added a trailing sentence after the closing brace. Detected by JSON.parse() throwing in validation. Fixed with fence-stripping plus structured output enforcement where supported:

function extractJson(raw) {
  const cleaned = raw.trim().replace(/^```(?:json)?\s*/i, '').replace(/```\s*$/i, '');
  const start = cleaned.indexOf('{'), end = cleaned.lastIndexOf('}');
  if (start === -1 || end === -1) throw new Error('no_json_object_found');
  return JSON.parse(cleaned.slice(start, end + 1));
}

Both layers stay in place — response_format reduces the problem but some models still wrap output under certain system prompts.

Missing fields. Stored records sometimes had null summaries because the LLM returned valid JSON missing the summary key on ambiguous or non-English titles — caught by schema validation. Fixed with one automatic regeneration attempt including the specific missing field, capped at one retry to control cost and avoid loops. Defaulting missing fields to empty strings was tried first and just produced garbage reports nobody trusted.

Duplicate records. The same story from Hacker News and a syndicated Dev.to repost appeared as two "trending" items — URL dedup missed reposts under different URLs. Fixed with a second pass: fuzzy title matching via normalized Levenshtein distance at an 85% similarity threshold:

function normalizeTitle(t) { return t.toLowerCase().replace(/[^\w\s]/g, '').replace(/\s+/g, ' ').trim(); }

function levenshtein(a, b) {
  const dp = Array.from({ length: a.length + 1 }, (_, i) =>
    Array.from({ length: b.length + 1 }, (_, j) => (i === 0 ? j : j === 0 ? i : 0)));
  for (let i = 1; i <= a.length; i++)
    for (let j = 1; j <= b.length; j++)
      dp[i][j] = a[i - 1] === b[j - 1] ? dp[i - 1][j - 1]
        : 1 + Math.min(dp[i - 1][j - 1], dp[i - 1][j], dp[i][j - 1]);
  return dp[a.length][b.length];
}

An LLM-based similarity check (embedding or pairwise call) was considered and rejected — too slow and expensive for string matching across roughly 400 items per run.

Partial workflow failures. A single failed branch (GitHub down) used to fail the whole execution, because Merge in Append mode still expects every inbound connection to fire something. Fixed by enabling Continue On Fail per source node and normalizing the error case to an empty array before it reaches the normalizer:

const item = $input.first().json;
if (item.error) return []; // no data this run, don't break the merge
return $input.all();

A hard dependency where every source must succeed makes the whole pipeline as reliable as its flakiest API — for a trends aggregator, partial data beats no data.

LLM hallucinations. The model occasionally invented a "key insight" not present in any source, or attributed a trend to a company never mentioned. Caught by spot-checking summaries against source URLs. Fixed by requiring every claim to reference a specific sourceId from the input batch, then validating that link mechanically:

function validateGrounding(summary, inputIds) {
  const unknown = summary.keyPoints.map(p => p.sourceId).filter(id => !inputIds.includes(id));
  if (unknown.length) throw new Error(`ungrounded_source_ids:${unknown.join(',')}`);
}

Asking the model to "be accurate" in the prompt didn't measurably help. Forcing a structural link between claim and source turns an unverifiable quality problem into a checkable schema problem.

Unexpected API responses. GitHub's proxy returned { "error": "rate limited upstream" } with HTTP 200 during an outage — a shape the normalizer had never seen, so it proceeded and stored records with title: undefined. Caught days later by downstream validation. Fixed with explicit shape validation, not just status-code checks:

function isValidGithubResponse(body) {
  return Array.isArray(body?.repositories) &&
    body.repositories.every(r => typeof r.name === 'string' && typeof r.url === 'string');
}

Status-code-only validation assumes well-behaved APIs. Every source normalizer now has an explicit shape guard, not just the ones that already broke once.

Expired credentials. The YouTube service account key rotated on Google's side and every call failed with 401 for six hours before anyone noticed — Continue On Fail masked it as "just no videos today." Fixed by distinguishing auth errors from data errors and routing 401/403 to a high-priority alert:

const status = $input.first().json.statusCode;
if ([401, 403].includes(status)) return [{ json: { alert: 'auth_failure', source: 'youtube', severity: 'high' } }];
return [{ json: { alert: 'data_failure', severity: 'low' } }];

Continue On Fail treats every failure the same; "credentials are dead" and "API had a bad day" need completely different responses.

Error Handling

Continue On Fail is set per-node, only where partial data is acceptable — every source collector, never storage or report nodes. Retry logic uses retryOnFail with maxTries: 3 for transient network errors, but not for 4xx errors other than 429 — retrying a malformed request wastes time and quota. Wait nodes sit between batched requests and before retrying rate-limited calls, reading Retry-After where available. An Error Trigger workflow receives execution metadata on failure and posts to Slack with the failed node and message. IF nodes gate every stage transition with a single condition each — chained single-condition IFs are easier to trace in the execution log than one compound-boolean IF, at the cost of a few extra nodes. Merge nodes run in Append mode when combining sources (order doesn't matter) but Combine-by-id mode when reattaching LLM output to its source record, where a mode mismatch would silently corrupt data. Validation happens at three points: after normalization (schema shape), after scoring (numeric sanity — scores must be finite, not NaN from a divide-by-zero on missing engagement data), and after the LLM call. Fallback branches exist for GitHub (search API) and for the LLM step — an item failing validation twice is stored with needsReview: true and a raw-text summary instead of being dropped.

Making LLM Output Reliable

The model is treated as an untrusted input source, same as any external API. Structured generation uses native structured-output mode where available, falling back to a prompt that enumerates the schema and forbids prose outside the JSON object — prompt instructions alone produced roughly a 4-6% malformed-output rate in testing.

Validation checks types and required keys, not just that JSON.parse() succeeds:

const schema = { title: 'string', summary: 'string', category: 'string', keyPoints: 'array', confidence: 'number' };

function validateSchema(obj, schema) {
  const errors = [];
  for (const [key, type] of Object.entries(schema)) {
    if (!(key in obj)) { errors.push(`missing:${key}`); continue; }
    const actual = Array.isArray(obj[key]) ? 'array' : typeof obj[key];
    if (actual !== type) errors.push(`type_mismatch:${key}:expected_${type}:got_${actual}`);
  }
  return errors;
}

Recovery distinguishes recoverable from unrecoverable failures — a missing optional field gets a default, a missing required field triggers regeneration, and completely unparseable output skips straight to needsReview rather than burning a retry on it. Regeneration sends the specific validation errors back, not a generic "try again":

const retryPrompt = `Previous response had these issues: ${errors.join(', ')}. Return ONLY valid JSON matching the schema. Original input: ${JSON.stringify(originalInput)}`;

Capped at one retry — an unbounded loop on a flaky output just burns API budget. Grounding verification (the sourceId check above) runs as a separate gate before storage, since output can be well-formed JSON and still be factually disconnected from its inputs.

Logging and Monitoring

Every run writes a compact log, not the full payload:

{
  "runId": "2026-07-27T06:00:00Z",
  "sources": {
    "reddit": { "fetched": 48, "errors": 0 },
    "producthunt": { "fetched": 0, "errors": 1, "reason": "rate_limit_low:150" },
    "devto": { "fetched": 0, "errors": 1, "reason": "timeout" }
  },
  "afterDedup": 187, "sentToLlm": 15, "llmRetries": 2, "needsReview": 1, "stored": 14
}

This lives in a run_logs table separate from the trend records — it's what gets checked first when a report looks thin. Full n8n execution history is kept 14 days for deep debugging; the compact log is kept indefinitely since it's cheap and answers "was this a bad day for source X" instantly. Slack alerts fire only on hard failures — auth errors, storage failures, execution-level errors — not on partial data from Continue On Fail branches, which would generate alert fatigue given how often at least one of eight external APIs has a rough five minutes. Partial-data days show up as a footnote in the daily report instead of paging anyone.

Lessons Learned

Almost none of the work above is visible in a happy-path diagram. Treating every external API as adversarial by default — rather than adding error handling reactively after each outage — would have prevented most of these incidents from becoming incidents at all. Building the happy path first and adding error handling "later" usually means later is after someone already noticed a gap in a report.

Continue On Fail is powerful but not free: it hid a real problem (the YouTube credential expiry) behind a mechanism meant to hide unimportant ones (a single flaky call). The fix wasn't removing it, it was adding a second signal — status-code-aware alerting — that Continue On Fail alone couldn't provide.

LLM output needs the same skepticism as any third-party API response: schema validation, grounding checks, bounded retries, not just a well-written prompt. The prompt lowers the error rate; the validation layer is what makes the remaining errors survivable.

Normalization and dedup logic accumulate special cases over time — the Google News redirect wrapper, Medium's per-tag pagination cap — and there's no way to anticipate all of them upfront. That ongoing cost is part of what it actually takes to run eight source integrations instead of one, and it's the part tutorials tend to skip.