Decoding Invisible Integration Failures: A High-Integrity Incident Report for ServiceTrade API Owners

Executive Summary

A mid-market fire protection firm serving venues from arenas to schools hit a silent snag in its ServiceTrade integration. Using Zapier, Make.com, and AWS Lambda, every API call returned HTTP 200 OK—but no jobs appeared. Dispatchers and technicians worked unaware until a quarterly audit revealed dozens of missing service orders. This report unpacks how silent API failures can erode trust and compliance and shares concrete steps to detect and prevent them.

Incident Timeline

07:45 AM – Technician Clock-In

Technician clocks in at a warehouse, triggering a Make.com scenario to create a ServiceTrade job.

07:46 AM – Payload Mapping

Make.com → AWS API Gateway → Lambda. The script maps fields:

• jobDate
• clientCode
• assignedTech

07:47 AM – Empty jobId

API Response (Failure)

{
  "status": 200,
  "jobId": "",
  "message": "Job not created: missing required fields."
}

Contrast with success:

{
  "status": 200,
  "jobId": "234987",
  "message": "Job created successfully."
}

08:00 AM – Dispatch Continues

Dispatchers schedule calls normally, unaware that jobs never landed. 40%

03:00 PM – Audit Mismatch

Operations manager flags 30 missing jobs during a quarterly audit—field_value_was_empty_no_error.

04:15 PM – Timezone Discovery

Engineer finds Lambda defaulted to UTC, while dispatch runs CST.

Environment Configuration

// Node.js Lambda snippet
process.env.TZ = 'America/Chicago'
        

Root Cause Analysis

• Silent API Validation: The createJob endpoint drops incomplete payloads (missing jobTitle or scheduledStart) without error, returning 200 OK and empty jobId.
• Schema Drift: Internal Make.com templates didn’t enforce required fields, relying on API silence.
• Timezone Tripwire: UTC default vs. CST dispatch times caused window checks to fail.
• Dispatch Filter Mishap: Inverted filter (“dispatch_status != closed”) routed new jobs to inactive queues.
• Invoice Logic Flaw: Billing scripts assumed job presence; with silent drops, accounts receivable gaps emerged. Similar to Honeywell UK’s 2021 silent middleware issue.

Resolution and Best Practices

1. Preflight Validation: Use AJV in Lambda to enforce JSON schema locally—catch missing jobTitle, serviceCategory, or scheduledStart early.
2. Idempotency Protection: Generate UUID keys in Make.com as externalReference to avoid duplicate jobs.
3. CloudWatch Detection: Subscription filters for zero-length jobId values, forwarding failures to an SQS DLQ.
4. Dead-Letter Queue: Retain evidence on 200 OK/no jobId for ops review before next billing cycle.
5. Enhanced Observability: Integrate Datadog, Sentry, or AWS X-Ray with correlation IDs for full traceability.
6. Slack/Triage Integration: Send DLQ alerts to Slack or Teams, modeled on PAIY timesheet exception workflows.
7. Automated Postman Tests: Nightly collections against staging and production to verify “it just worked” confidence.

Strategic Recommendations

• Observability First: Roll out end-to-end tracing (AWS X-Ray, New Relic) to surface response_code_200_but_nothing_happened instantly.
• Incident Playbook: Maintain a runbook for silent failures (200 OK + empty body) with SLA-aligned on-call steps.
• Training & Governance: Quarterly workshops on JSON schema, timezones, and root-cause drills—reference NFPA’s digital transformation guidelines.
• Continuous Feedback: Track metrics like field value was empty no error and invoice fired before job was closed to guide process improvements.
• Shape the API: Join ServiceTrade user forums or user groups to share logs of silent failures and advocate for explicit error responses.