Common Issues

Operational issues grouped by component. Commands assume the demo repo demo repo
event-driven-serverless-platform-demo — source repository for Terraform, Lambda services, portal, and scripts. Commands run from its root with `terraform -chdir=terraform` unless stated otherwise. root with terraform -chdir=terraform.

Terraform and deploy

Init or apply fails with provider errors

Verify Terraform Terraform
HashiCorp Terraform — provisions all platform infrastructure through five modules in `terraform/`. >= 1.6 and AWS AWS
Amazon Web Services — cloud platform for ingest, processing, storage, API, and portal hosting in this walkthrough. credentials:

terraform --version
aws sts get-caller-identity
terraform -chdir=terraform init -upgrade

See Terraform Init.

Processor Lambda fails on apply — image not found

Mirror the image before full apply:

terraform -chdir=terraform apply -target=module.processing[0].aws_ecr_repository.processor_image
./scripts/sync-processor-image.sh

See ECR and Processor Image.

Amplify deploy skipped during apply

Re-deploy manually:

export AMPLIFY_APP_ID="$(terraform -chdir=terraform output -raw amplify_app_id)"
export VITE_API_URL="$(terraform -chdir=terraform output -raw api_url)"
export WEB_SOURCE_DIR="$(pwd)/web"
./scripts/deploy-amplify.sh

See Amplify Portal.

Ingest and scheduler

Scheduler not listed

Confirm stage 2 of staged apply ran:

terraform -chdir=terraform apply -var="region=ap-southeast-2" -target=module.ingest_scheduler
aws scheduler list-schedules --query "Schedules[?contains(Name, 'ingest-sync')]"

See EventBridge Scheduler.

No raw files in data lake

Trigger manual ingest:

aws lambda invoke --function-name ingest-sync --payload '{}' /dev/stdout
aws s3 ls "s3://$(terraform -chdir=terraform output -raw bucket_name)/raw/rinexhourly/" --recursive | head

See Manual Ingest.

ingest-sync-errors alarm in ALARM

Check CloudWatch CloudWatch
Amazon CloudWatch — alarms, dashboard (`event-driven-platform`), and structured Lambda logs for observability. Logs for the ingest-sync function. Common causes: invalid LOOKBACK_HOURS, GeoNet bucket access, or IAM IAM
AWS Identity and Access Management — execution roles and policies for Lambdas, Scheduler, API Gateway, and S3. permission gaps.

SQS queues

Process_Queue depth growing

Check processor processor
Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR. health and concurrency:

aws lambda get-function-configuration --function-name processor \
  --query '{State:State,LastUpdateStatus:LastUpdateStatus}'

aws cloudwatch describe-alarms --alarm-names processor-lambda-throttles \
  --query 'MetricAlarms[].StateValue'

Request a Lambda Lambda
AWS Lambda — serverless compute for ingest-sync, query-api, reprocess-api, and the processor container image. concurrency quota increase if throttles persist. See Processor Lambda and SQS Queues.

DLQ messages visible

Inspect DLQ DLQ
Dead-letter queue — SQS queue receiving messages that exceeded maxReceiveCount (5). CloudWatch alarms fire when visible count ≥ 1. messages and processor logs. Messages arrive after 5 failed processing attempts:

aws cloudwatch describe-alarms --alarm-names dlq-messages-visible reprocess-dlq-messages-visible

See Alarms and Dashboard.

Processor Lambda

LastUpdateStatus Failed

Verify ECR ECR
Amazon Elastic Container Registry — stores the mirrored processor Lambda container image at deploy time. image exists and tag matches:

terraform -chdir=terraform output -raw processor_image_uri
./scripts/sync-processor-image.sh
terraform -chdir=terraform apply -var="region=ap-southeast-2"

No processed output in S3

Confirm raw files exist and processor is consuming queues:

aws s3 ls "s3://$(terraform -chdir=terraform output -raw bucket_name)/raw/rinexhourly/" --recursive | head
aws sqs get-queue-attributes \
  --queue-url "$(terraform -chdir=terraform output -raw queue_url)" \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

See S3 Processed Output.

REST API

curl works but portal shows CORS errors

Apply CORS lockdown after first deploy:

terraform -chdir=terraform apply \
  -var="region=ap-southeast-2" \
  -var="amplify_domain=$(terraform -chdir=terraform output -raw cors_domain)"

See CORS Lockdown.

Empty catalog

Catalog lists processed output only — not raw ingest keys. Wait for processor to write under processed/tec/ or submit a reprocess job. See S3 Key Patterns.

Query returns truncated results

Narrow time range or add sv filter. Default cap is 2000 rows. See REST API Usage.

Reprocess returns HTTP 400

Only allowlisted parameter keys are accepted. See SQS Message Schemas.

Portal

Blank portal page

Rebuild and redeploy:

cd web && npm install && npm run build
export WEB_SOURCE_DIR="$(pwd)/web"
export AMPLIFY_APP_ID="$(terraform -chdir=terraform output -raw amplify_app_id)"
export VITE_API_URL="$(terraform -chdir=terraform output -raw api_url)"
./scripts/deploy-amplify.sh

See Portal verification.

Observability

Alarms not notifying

Subscribe to the SNS SNS
Amazon Simple Notification Service — publishes CloudWatch alarm notifications; subscribe an email endpoint after deploy. topic:

TOPIC_ARN="$(terraform -chdir=terraform output -raw alarm_topic_arn)"
aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol email --notification-endpoint "you@example.com"

Confirm subscription from email. See Alarms and Dashboard.

Tip

All Lambdas emit structured JSON logs with trace_id trace_id
UUID v4 correlation ID — propagated in SQS messages and structured logs for end-to-end request tracing. for cross-service correlation in CloudWatch Logs.

Teardown issues

If terraform destroy fails on the S3 S3
Amazon Simple Storage Service — data lake bucket for raw ingest and processed TEC output; S3 events trigger processing. bucket, empty it first — see Teardown.