The integration challenge
Every automation project I work on involves connecting multiple APIs. And every API has its own quirks — different rate limits, authentication methods, error formats, and reliability levels.
After building hundreds of API integrations, I’ve settled on a set of patterns that make integrations reliable, maintainable, and easy to debug.
Pattern 1: Retry with exponential backoff
APIs fail. Network requests time out. Rate limits get hit. Your code needs to handle these gracefully.
async function withRetry(fn, maxRetries = 3, baseDelay = 1000) {
let lastError;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
return await fn();
} catch (error) {
lastError = error;
// Don't retry on 4xx errors (except 429)
if (error.status >= 400 && error.status < 500 && error.status !== 429) {
throw error;
}
// Exponential backoff with jitter
const delay = baseDelay * Math.pow(2, attempt) + Math.random() * 1000;
await new Promise(resolve => setTimeout(resolve, delay));
}
}
throw lastError;
}
// Usage
const result = await withRetry(() => fetchOrder(orderId));Pattern 2: Rate limiting
Most APIs have rate limits. Hitting them causes failures and can get your API key revoked.
Token bucket rate limiter
class RateLimiter {
constructor(maxRequests, windowMs) {
this.maxRequests = maxRequests;
this.windowMs = windowMs;
this.tokens = maxRequests;
this.lastRefill = Date.now();
}
async acquire() {
this.refill();
if (this.tokens < 1) {
const waitTime = this.windowMs - (Date.now() - this.lastRefill);
await new Promise(resolve => setTimeout(resolve, waitTime));
this.refill();
}
this.tokens -= 1;
}
refill() {
const now = Date.now();
const elapsed = now - this.lastRefill;
if (elapsed >= this.windowMs) {
this.tokens = this.maxRequests;
this.lastRefill = now;
}
}
}
// Usage: 100 requests per minute
const limiter = new RateLimiter(100, 60000);
async function callApi(fn) {
await limiter.acquire();
return fn();
}Pattern 3: Idempotency
If a request fails after the server has processed it but before you receive the response, retrying could cause duplicate operations. Idempotency keys solve this.
const crypto = require('crypto');
async function createOrder(orderData) {
const idempotencyKey = crypto.randomUUID();
const response = await fetch('/api/orders', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Idempotency-Key': idempotencyKey,
},
body: JSON.stringify(orderData),
});
return response.json();
}If the request times out and you retry with the same idempotency key, the server will return the original response instead of creating a duplicate order.
Pattern 4: Webhook handling
Webhooks are the backbone of real-time integrations. But they come with their own challenges.
Verify webhook signatures
Always verify that a webhook is from who it claims to be from:
import crypto from 'crypto';
function verifyWebhook(payload, signature, secret) {
const expectedSignature = crypto
.createHmac('sha256', secret)
.update(payload)
.digest('hex');
// Use timing-safe comparison to prevent timing attacks
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expectedSignature)
);
}
app.post('/webhook', (req, res) => {
const signature = req.headers['x-webhook-signature'];
const payload = JSON.stringify(req.body);
if (!verifyWebhook(payload, signature, process.env.WEBHOOK_SECRET)) {
return res.status(401).send('Invalid signature');
}
// Process webhook
processWebhook(req.body);
res.status(200).send('OK');
});Respond quickly, process asynchronously
Webhook senders expect a quick response (usually within 5 seconds). If your processing takes longer, respond immediately and process in the background:
app.post('/webhook', async (req, res) => {
// Verify signature
if (!verifyWebhook(req)) {
return res.status(401).send('Invalid');
}
// Queue for processing
await messageQueue.add('webhook', req.body);
// Respond immediately
res.status(200).send('OK');
});Handle duplicate webhooks
Webhook senders often retry if they don’t receive a 200 response, which means you might receive the same webhook multiple times. Use the event ID for deduplication:
const processedEvents = new Set();
app.post('/webhook', (req, res) => {
const eventId = req.headers['x-webhook-event-id'];
if (processedEvents.has(eventId)) {
return res.status(200).send('Already processed');
}
processedEvents.add(eventId);
processWebhook(req.body);
res.status(200).send('OK');
});In production, use Redis or a database instead of an in-memory set.
Pattern 5: Circuit breakers
When an API is down, you don’t want to keep sending requests and waiting for timeouts. A circuit breaker stops sending requests after a threshold of failures:
class CircuitBreaker {
constructor(threshold = 5, timeout = 60000) {
this.failures = 0;
this.threshold = threshold;
this.timeout = timeout;
this.lastFailure = null;
this.state = 'closed'; // closed, open, half-open
}
async call(fn) {
if (this.state === 'open') {
if (Date.now() - this.lastFailure > this.timeout) {
this.state = 'half-open';
} else {
throw new Error('Circuit breaker is open');
}
}
try {
const result = await fn();
this.failures = 0;
this.state = 'closed';
return result;
} catch (error) {
this.failures++;
this.lastFailure = Date.now();
if (this.failures >= this.threshold) {
this.state = 'open';
}
throw error;
}
}
}Pattern 6: Pagination handling
Most APIs paginate results. Here’s a clean pattern for handling pagination:
async function* getAllPages(fetchPage) {
let offset = 0;
let hasMore = true;
while (hasMore) {
const response = await fetchPage(offset);
for (const item of response.data) {
yield item;
}
hasMore = response.has_more || response.data.length > 0;
offset += response.data.length;
}
}
// Usage
for await (const contact of getAllPages(offset =>
hubspot.contacts.list({ offset })
)) {
console.log(contact);
}Conclusion
Reliable API integrations are the foundation of every automation system. By implementing these patterns — retry logic, rate limiting, idempotency, webhook verification, circuit breakers, and pagination — you can build integrations that handle the real world’s unpredictability.
The key insight is that every API call can fail, and your code should expect and handle that failure gracefully. Build for failure, and your integrations will be reliable.
