Reliability in calls to external services
Four mechanisms
Timeout. On every call, always. A call with no time limit holds a resource until the system chokes.
Retry with growing delay. Only for actions safe to repeat, and only for transient errors. Retrying a validation error is a waste.
Circuit breaker. After a run of failures, stop trying for a while. That protects both you and the service already struggling.
Degraded mode. What the system shows when the service is unavailable: cached data, partial functionality, or a clear message.
The duplication problem
When a call fails on timeout, it's unknown whether the action was performed. On every action with impact — a charge, a send, a creation — send a unique key that lets the other side detect a repeat. Without it, a retry may charge twice.
What to monitor
Error rate and response time per external service separately, and the circuit breaker's state. A vendor's outage that appears on your side as “the system is slow” wastes hours of needless investigation.
Going deeper
Separate a critical dependency from a secondary one, and make sure a secondary dependency can't bring down a primary path. An analytics or enrichment service on the synchronous path is a failure point with no business justification at all.