Fan-Out and Fan-In Patterns
Fan-out distributes work across multiple parallel paths. Fan-in collects and combines those results. Together, they enable powerful parallel processing patterns.
The Fan-Out / Fan-In Pattern
The pattern has four phases: split the input, run several workers in parallel, collect their results, then combine them into one output. Fan-out is the split step, fan-in is the collect step.
- Input
- SplitFan-out: divide the work
- WorkersRun several in parallel
- CollectFan-in: gather results
- OutputCombine into one result
Fan-Out Strategies
Even Distribution
Split work evenly across workers:
function evenSplit(items, numWorkers) {
const chunks = [];
const chunkSize = Math.ceil(items.length / numWorkers);
for (let i = 0; i < items.length; i += chunkSize) {
chunks.push(items.slice(i, i + chunkSize));
}
return chunks;
}
// Example: 100 items across 5 workers = 20 items each
const chunks = evenSplit(items, 5);
Content-Based Distribution
Route items to specialized workers:
Load-Aware Distribution
Consider current load when distributing:
class LoadBalancer {
constructor(workers) {
this.workers = workers.map(w => ({ ...w, currentLoad: 0 }));
}
getNextWorker() {
// Find worker with lowest load
return this.workers.reduce((min, w) =>
w.currentLoad < min.currentLoad ? w : min
);
}
async distribute(items) {
return Promise.all(items.map(async item => {
const worker = this.getNextWorker();
worker.currentLoad++;
try {
return await worker.process(item);
} finally {
worker.currentLoad--;
}
}));
}
}
Fan-In Strategies
Simple Aggregation
Collect all results into an array:
async function simpleAggregate(workers, input) {
const results = await Promise.all(
workers.map(worker => worker.process(input))
);
return results;
}
Merge and Deduplicate
Combine overlapping results:
function mergeResults(workerResults) {
const merged = {
entities: [],
seen: new Set()
};
for (const result of workerResults) {
for (const entity of result.entities) {
const key = `${entity.type}:${entity.name}`;
if (!merged.seen.has(key)) {
merged.seen.add(key);
merged.entities.push(entity);
}
}
}
return merged.entities;
}
Weighted Combination
Combine results with different weights:
Consensus Building
Find agreement among workers:
function buildConsensus(workerResults, threshold = 0.6) {
const votes = {};
// Count votes for each classification
for (const result of workerResults) {
const key = result.classification;
votes[key] = (votes[key] || 0) + 1;
}
// Find winner
const total = workerResults.length;
for (const [classification, count] of Object.entries(votes)) {
if (count / total >= threshold) {
return {
consensus: classification,
agreement: count / total,
votes
};
}
}
return { consensus: null, reason: 'No consensus reached', votes };
}
Map-Reduce Pattern
A common fan-out/fan-in application:
async function mapReduce(items, mapFn, reduceFn) {
// Fan-out: Map phase
const mapped = await Promise.all(
items.map(item => mapFn(item))
);
// Fan-in: Reduce phase
return reduceFn(mapped);
}
// Example: Analyze reviews and aggregate sentiment
const result = await mapReduce(
reviews,
review => analyzeSentiment(review), // Map: analyze each
sentiments => averageSentiment(sentiments) // Reduce: combine
);
Hierarchical Fan-Out
For very large workloads, use multiple levels:
┌→ Worker 1a
┌→ Tier 1 ─┼→ Worker 1b
│ └→ Worker 1c
│
Input ──┼→ Tier 2 ─┼→ Worker 2a ──┼→ Aggregate → Output
│ └→ Worker 2b
│
└→ Tier 3 ─┼→ Worker 3a
└→ Worker 3b
async function hierarchicalFanOut(items, config) {
const { tierSize, workersPerTier } = config;
// Split into tiers
const tiers = chunk(items, tierSize);
// Process each tier in parallel
const tierResults = await Promise.all(
tiers.map(tier =>
parallelProcess(tier, workersPerTier)
)
);
// Aggregate all tier results
return aggregateTiers(tierResults);
}
Handling Partial Failures
Collect Successful Results
async function fanOutWithFailures(workers, input) {
const results = await Promise.allSettled(
workers.map(w => w.process(input))
);
const successful = [];
const failed = [];
for (let i = 0; i < results.length; i++) {
if (results[i].status === 'fulfilled') {
successful.push({
worker: workers[i].name,
result: results[i].value
});
} else {
failed.push({
worker: workers[i].name,
error: results[i].reason
});
}
}
return {
successful,
failed,
successRate: successful.length / workers.length
};
}
Minimum Success Threshold
async function fanOutWithThreshold(workers, input, minSuccess = 0.5) {
const { successful, failed, successRate } = await fanOutWithFailures(workers, input);
if (successRate < minSuccess) {
throw new Error(`Only ${(successRate * 100).toFixed(0)}% succeeded (need ${minSuccess * 100}%)`);
}
return combine(successful.map(s => s.result));
}
Exercise: Design a Fan-Out/Fan-In System
Design a complete fan-out/fan-in system:
Key Takeaways
- Fan-out distributes work; fan-in combines results
- Use even distribution for homogeneous work
- Use content-based distribution for specialized workers
- Use load-aware distribution for variable processing times
- Simple aggregation works for collecting results
- Weighted combination uses worker confidence levels
- Consensus building finds agreement among workers
- Map-reduce is a common fan-out/fan-in pattern
- Handle partial failures by collecting successful results
- Set minimum success thresholds for reliable processing
Next, we'll explore strategies for aggregating and reconciling results from multiple workers.
التالي · الدرس 21 من 43
Result Aggregation Strategies
قراءة 3 دقيقة
أنهيت 20 من 43 درسًا

اسأل Byte عن هذا الدرس
محادثة خاصة حول هذا الدرس. قرأ Byte الدرس كاملًا، ويجيب خلال ثوانٍ، ويُظهر القسم الذي تأتي منه كل إجابة. يمكنك السؤال بأي لغة.
الأسئلة والأجوبة
أسئلة عامة حول هذا الدرس. يجيب AI Tutor فورًا بلغتك، ويراجع فريق FreeAcademy الإجابات، ويمكن لكل المتعلمين قراءتها.

