Skip to content
Fabian Finalé Franqui
Selected work

Technical deep dive

Tracing an outage from the API edge to a stale firewall rule

External users were getting HTTP 503s while the application looked healthy. The cause was a network rule that no longer matched where the workload lived.

My role
Investigation and root-cause analysis
Scope
Application, API gateway, private networking, container platform
Focus
  • Cloud networking
  • Incident response
  • Root-cause analysis
HTTP 503HealthyClientExternalAPI gatewayEdgePrivate linkIntegrationFirewallEgress rulesLoad balancerNew subnetsWorkloadKubernetesOld ranges ≠ new subnetsAPI infraNetworkPlatform
Fig. 1The request path, hop by hop. Every layer answered except one: an egress rule still scoped to the workload’s old address ranges.

Context

A public API path began returning HTTP 503s to external users. Requests traveled through a managed API gateway, across a private network link, and on to a load-balanced workload running on Kubernetes.

Problem

A 503 is an unhelpful signal. It says something between the client and the service couldn’t complete the request, not which hop failed. With the application itself looking healthy, the obvious explanation was ruled out, and the answer had to be somewhere in the path.

Constraints

  • External users were affected while the investigation ran.
  • The path crossed several ownership boundaries: application, API infrastructure, networking, and platform.
  • The downstream workload had moved to new subnets as part of a platform change.

My role

I took part in the investigation, following the request path layer by layer until the failure had a precise explanation and a targeted fix.

How it was narrowed down

  1. 01Follow the request, not the loudest dashboard

    Instead of starting from whichever component had the noisiest metrics, the investigation followed the path in order: client, API gateway, private link, firewall rules, load balancer, workload.

  2. 02Find where “healthy” stops

    With the application layer healthy, the question became which hop between the gateway and the workload was refusing traffic. That moved the search from the code to the network path.

  3. 03State the root cause precisely

    The firewall rules on the private link (a security group, in AWS terms) only allowed outbound traffic to specific address ranges. When the workload moved to new subnets behind its load balancer, its addresses fell outside those ranges. The gateway could no longer reach it and reported the only thing it could: service unavailable.

Execution

  1. Rule out the service

    Confirm the application is healthy before chasing infrastructure.

  2. Walk the path

    Gateway, private link, firewall rules, load balancer, workload — in order.

  3. Isolate the hop

    Identify the rule that no longer matched the workload’s addresses.

  4. Remediate

    Update the egress rules to allow the new subnet ranges.

  5. Prevent

    Document the drift and add synthetic checks on the path.

Outcome

A specific root cause and a targeted remediation — allowing the new subnet ranges in the egress rules — rather than a round of restarts and hope.

The root-cause analysis also documented why the rule had drifted from the architecture and what would keep future infrastructure changes from causing the same class of failure, and more synthetic checks were added to the app to watch that path.

A cause that could be stated in one sentence to people outside engineering, which matters as much as the fix when users are affected.

Lessons

  • When a platform change moves a workload, every rule that encoded its old location becomes a latent incident.
  • Status codes describe symptoms at the edge. The cause is usually a hop or two further in.
  • Cross-layer incidents need someone willing to cross layers — application, gateway, network, platform — without waiting for each owner in turn.