SQS Queues

SQS SQS
Amazon Simple Queue Service — buffers ingest and reprocess messages between S3/API and the processor Lambda. queues decouple ingest notifications from processor processor
Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR. execution. Verify messages flow into Process_Queue Process_Queue
SQS standard queue — receives S3 ObjectCreated notifications for new raw RINEX files; triggers processor Lambda ingest path. after raw RINEX RINEX
Receiver Independent Exchange Format — standard GNSS observation file format synced from GeoNet and calibrated by the processor. files appear in the data lake data lake
Private S3 bucket holding raw RINEX under `raw/rinexhourly/` and processed TEC output under `processed/tec/`. .

Check Process_Queue depth

From the demo repo root, after new objects land under raw/rinexhourly/:

aws sqs get-queue-attributes \
  --queue-url "$(terraform -chdir=terraform output -raw queue_url)" \
  --attribute-names ApproximateNumberOfMessages

Expect ApproximateNumberOfMessages to decrease as the processor consumes messages. A non-zero count immediately after ingest is normal.

Check in-flight messages

aws sqs get-queue-attributes \
  --queue-url "$(terraform -chdir=terraform output -raw queue_url)" \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

ApproximateNumberOfMessagesNotVisible reflects messages currently being processed (visibility timeout 900s).

Check Reprocess_Queue

aws sqs get-queue-attributes \
  --queue-url "$(terraform -chdir=terraform output -raw reprocess_queue_url)" \
  --attribute-names ApproximateNumberOfMessages

Reprocess_Queue stays empty until you submit a job via the Reprocess API Reprocess API
Reprocess API Lambda (`services/reprocess-api/`) — accepts `/reprocess` jobs, writes DynamoDB records, enqueues Reprocess_Queue. . See Reprocess Workflow.

Queue configuration

Queue	Trigger	Concurrency cap
Process_Queue	S3 S3 Amazon Simple Storage Service — data lake bucket for raw ingest and processed TEC output; S3 events trigger processing. ObjectCreated on raw keys	processor_maximum_concurrency (default 15)
Reprocess_Queue	Reprocess API	reprocess_maximum_concurrency (default 2)

Both use batch_size=1 and dead-letter queues dead-letter queues
Dead-letter queue — SQS queue receiving messages that exceeded maxReceiveCount (5). CloudWatch alarms fire when visible count ≥ 1. after 5 failed receives.

Troubleshooting

If messages accumulate without draining, check Processor Lambda and Common Issues — SQS.