Output Validation Strategies

Validation is your first line of defense against failures cascading through your chain. Robust validation catches problems early, before they corrupt downstream steps.

The Validation Pipeline

Every step's output should pass through a validation pipeline:

Step Output → Structure Check → Type Check → Range Check → Semantic Check → Next Step

Structural Validation

Schema Validation

Define expected structure and enforce it:

const outputSchema = {
  type: "object",
  required: ["sentiment", "confidence", "themes"],
  properties: {
    sentiment: {
      type: "string",
      enum: ["positive", "negative", "neutral", "mixed"]
    },
    confidence: {
      type: "number",
      minimum: 0,
      maximum: 1
    },
    themes: {
      type: "array",
      items: { type: "string" },
      minItems: 0,
      maxItems: 10
    }
  }
};

Extracting Structure from Unstructured Output

When AI outputs don't parse cleanly:

Handling Partial Structures

Sometimes outputs are partially valid:

function validateWithDefaults(output, schema) {
  const result = { ...output };
  const warnings = [];

  for (const [field, spec] of Object.entries(schema.properties)) {
    if (!(field in result)) {
      if (schema.required.includes(field)) {
        if ('default' in spec) {
          result[field] = spec.default;
          warnings.push(`Used default for missing required field: ${field}`);
        } else {
          throw new ValidationError(`Missing required field: ${field}`);
        }
      }
    }
  }

  return { data: result, warnings };
}

Type Validation

Runtime Type Checking

Verify types match expectations:

function checkTypes(output, schema) {
  const errors = [];

  for (const [field, value] of Object.entries(output)) {
    const expected = schema.properties[field];
    if (!expected) continue;

    const actualType = Array.isArray(value) ? 'array' : typeof value;
    if (actualType !== expected.type) {
      errors.push({
        field,
        expected: expected.type,
        actual: actualType,
        value
      });
    }
  }

  return errors;
}

Type Coercion

Sometimes flexible type handling is appropriate:

function coerceTypes(output, schema) {
  const coerced = { ...output };

  for (const [field, spec] of Object.entries(schema.properties)) {
    if (!(field in coerced)) continue;

    if (spec.type === 'number' && typeof coerced[field] === 'string') {
      const parsed = parseFloat(coerced[field]);
      if (!isNaN(parsed)) {
        coerced[field] = parsed;
      }
    }

    if (spec.type === 'array' && typeof coerced[field] === 'string') {
      // Try to parse comma-separated string as array
      coerced[field] = coerced[field].split(',').map(s => s.trim());
    }
  }

  return coerced;
}

Range and Constraint Validation

Numeric Ranges

function validateRanges(output, constraints) {
  const errors = [];

  for (const [field, range] of Object.entries(constraints)) {
    const value = output[field];
    if (typeof value !== 'number') continue;

    if ('min' in range && value < range.min) {
      errors.push(`${field} (${value}) below minimum (${range.min})`);
    }
    if ('max' in range && value > range.max) {
      errors.push(`${field} (${value}) above maximum (${range.max})`);
    }
  }

  return errors;
}

String Constraints

Semantic Validation

Relevance Checking

Verify output relates to input:

Consistency Checking

Ensure internal consistency:

function checkConsistency(output) {
  const errors = [];

  // Sentiment-score consistency
  if (output.sentiment === 'positive' && output.score < 0.4) {
    errors.push('Positive sentiment inconsistent with low score');
  }
  if (output.sentiment === 'negative' && output.score > 0.6) {
    errors.push('Negative sentiment inconsistent with high score');
  }

  // Count-array consistency
  if (output.count !== output.items?.length) {
    errors.push(`Count (${output.count}) doesn't match items length (${output.items?.length})`);
  }

  return errors;
}

Cross-Reference Validation

Check output against known facts:

async function crossReference(output, referenceData) {
  const warnings = [];

  // Check if mentioned entities exist
  for (const entity of output.entities) {
    if (entity.type === 'company') {
      const exists = await checkCompanyExists(entity.name);
      if (!exists) {
        warnings.push(`Could not verify company: ${entity.name}`);
      }
    }
  }

  return warnings;
}

Validation Strategies by Step Type

Step Type	Key Validations
Extraction	All expected fields present, confidence scores reasonable
Classification	Category is valid enum, confidence provided
Generation	Length constraints, format requirements, no forbidden content
Summarization	Length within bounds, key points preserved
Analysis	Conclusions supported by data, no contradictions

Building Validation Prompts

Use AI to validate complex outputs:

Exercise: Design a Validation Pipeline

Design a complete validation pipeline for this step:

Step: CustomerFeedbackAnalyzer Input: Raw customer feedback text Output: Structured analysis with sentiment, topics, and actionability

Key Takeaways

• Validation should happen after every step, not just at the end
• Structural validation catches format issues
• Type validation ensures correct data types
• Range validation enforces business constraints
• Semantic validation checks meaning and consistency
• Use AI-powered validation for complex semantic checks
• Design different validation strategies for different step types
• Always define what happens when validation fails

Next, we'll explore retry and fallback patterns for handling validation failures.