Troubleshooting

Symptom-to-Cause Reference

Use this table to identify the likely cause of your deployment issue:

Symptom	Likely Cause	Where to Look
Service spinning / rollback loops	New tasks fail ALB health checks repeatedly, triggering RollbackAlarm RollbackAlarm A CloudWatch metric alarm created by Express Mode that triggers automatic deployment rollback when new tasks fail ALB health checks.	EC2 → Target Groups → Targets tab (health reason)
503 errors from Application URL Application URL The unique Express-provisioned HTTPS URL on *.ecs.<region>.on.aws used for all normal application traffic.	No healthy targets registered in the target group	EC2 → Target Groups → Targets tab
Private DNS resolution failure	VPC VPC Virtual Private Cloud — an isolated virtual network within AWS where resources are deployed. DNS support or hostnames not enabled	VPC settings: enable_dns_support, enable_dns_hostnames
Health status “Unknown” on Tasks tab	Normal and expected — Express Mode does not define container-level healthCheck	No action needed (see explanation below)
Target group unhealthy	Security group Security Group A virtual firewall for EC2 instances and other resources that controls inbound and outbound traffic. missing ingress, container crash, wrong health path, or port mismatch	EC2 → Target Groups → health reason column
Deployment rollback	ALB health checks fail within the deregistration delay, RollbackAlarm fires	ECS → Service → Events/Timeline, CloudWatch Alarms

Deployment Rollback Flow

When a deployment fails, Express Mode Express Gateway Service
The aws_ecs_express_gateway_service Terraform resource that provisions an ECS service with managed ALB, auto scaling, and simplified configuration for HTTP/HTTPS web applications and APIs. follows a predictable rollback cycle:

1. New tasks start with the updated task definition
2. The ALB Shared ALB
   The Application Load Balancer shared by up to 25 Express services in the same VPC using Host header routing rules. health checks the new targets on the configured health_check_path
3. Health checks fail (target returns non-200 or connection refused)
4. The RollbackAlarm RollbackAlarm
   A CloudWatch metric alarm created by Express Mode that triggers automatic deployment rollback when new tasks fail ALB health checks. fires after the evaluation period
5. Express Mode rolls back to the previous stable deployment
6. The cycle repeats on the next deployment attempt until the root cause is fixed

Caution

Rollback loops will continue indefinitely. Express Mode does not stop retrying — you must fix the underlying issue before the next deployment succeeds.

Debug Order for Failed Deployments

When your service fails to deploy, follow this order to find the root cause quickly:

Step 1: EC2 Target Groups health reason

Navigate to EC2 → Target Groups → your target group → Targets tab. The health reason column tells you exactly why targets are unhealthy:

• Request timeout — container is not responding on the expected port
• Health checks failed — container responded with non-200 status
• Target not in service — task was deregistered or stopped

Step 2: CloudWatch Logs

Check the ECS task log group for container startup errors. Look for:

• Application crash stack traces
• Missing environment variables
• Port binding failures
• Permission denied errors

Step 3: ECS stopped task reason and exit code

In the ECS console, go to Cluster → Service → Tasks (stopped) and check:

• Stopped reason — why ECS stopped the task
• Exit code — the container’s exit code (non-zero indicates a crash)
  • Exit code 1 — application error
  • Exit code 137 — killed by OOM (out of memory)
  • Exit code 139 — segmentation fault

Step 4: curl the health endpoint

From a resource within the same VPC VPC
Virtual Private Cloud — an isolated virtual network within AWS where resources are deployed. , curl the health check path directly:

curl -v http://<task-private-ip>:<container-port>/health

This confirms whether the container is actually serving traffic on the expected port and path.

Tip

If curl succeeds from inside the VPC but the target group shows unhealthy, the issue is likely a security group rule — the ALB cannot reach the task on the container port.

Common Root Causes

These are the most frequent reasons for Express Mode health check failures:

Missing security group ingress

The task security group Security Group
A virtual firewall for EC2 instances and other resources that controls inbound and outbound traffic. must allow inbound traffic on the container port from the VPC VPC
Virtual Private Cloud — an isolated virtual network within AWS where resources are deployed. CIDR block. The shared ALB Shared ALB
The Application Load Balancer shared by up to 25 Express services in the same VPC using Host header routing rules. lives within the same VPC and routes health check probes to task IPs.

main.tf

ingress {
  from_port   = var.container_port
  to_port     = var.container_port
  protocol    = "tcp"
  cidr_blocks = [aws_vpc.this.cidr_block]
}

Container exits on startup

The container crashes before it can respond to health checks. Common causes:

• Missing required environment variables
• Database or dependency connection failures
• Incorrect entrypoint command
• Insufficient memory allocation

Wrong health_check_path

The health_check_path in the Express Gateway Service Express Gateway Service
The aws_ecs_express_gateway_service Terraform resource that provisions an ECS service with managed ALB, auto scaling, and simplified configuration for HTTP/HTTPS web applications and APIs. configuration must match an endpoint that returns HTTP 200. If your application uses /healthz but the config says /health, checks will fail with 404.

Port mismatch

The container_port in the task definition must match the port your application actually listens on. If the app binds to port 8080 but container_port is set to 3000, the ALB cannot reach it.

Danger

Port mismatches produce “Request timeout” in the target group health reason because the ALB sends probes to a port with no listener. The container appears healthy from its own perspective but is unreachable by the ALB.

Health Status “Unknown” Is Expected

When viewing the ECS Tasks tab, you may see Health status: Unknown for your Express Mode tasks. This is normal and expected behavior.

Express Mode Express Gateway Service
The aws_ecs_express_gateway_service Terraform resource that provisions an ECS service with managed ALB, auto scaling, and simplified configuration for HTTP/HTTPS web applications and APIs. does not define a container-level healthCheck in the task definition. Instead, it relies entirely on ALB target group health checks to determine whether tasks are healthy. The “Unknown” status simply means ECS has no container-level health information — it does not indicate a problem.

Note

Operational health for Express Mode services is determined by the ALB target group state, not the ECS Health status column. Check EC2 → Target Groups → Targets for the actual health of your tasks.

To confirm your service is healthy:

1. Navigate to EC2 → Target Groups → Targets
2. Verify targets show status healthy
3. The Application URL Application URL
   The unique Express-provisioned HTTPS URL on *.ecs.<region>.on.aws used for all normal application traffic. is accessible and returns expected responses