SQS Message Schemas

The processor Lambda processor Lambda
Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR. container consumes messages from two SQS SQS
Amazon Simple Queue Service — buffers ingest and reprocess messages between S3/API and the processor Lambda. queues with separate concurrency limits. Message body shapes are the same structurally; routing differs by queue.

Queue routing

Queue	Accepts	Source
Reprocess_Queue Reprocess_Queue SQS standard queue — buffers reprocessing job messages from the Reprocess API with lower concurrency than ingest.	Shape 1 only	Reprocess API Reprocess API Reprocess API Lambda (`services/reprocess-api/`) — accepts `/reprocess` jobs, writes DynamoDB records, enqueues Reprocess_Queue. direct payload
Process_Queue Process_Queue SQS standard queue — receives S3 ObjectCreated notifications for new raw RINEX files; triggers processor Lambda ingest path.	Shape 2 and Shape 3	S3 S3 Amazon Simple Storage Service — data lake bucket for raw ingest and processed TEC output; S3 events trigger processing. event notifications

Shape 1 — Reprocess API (direct payload)

Sent by the Reprocess_API Lambda when a job is submitted:

{
  "key": "raw/rinexhourly/2026/179/auck1790.26o",
  "job_id": "uuid-v4",
  "trace_id": "uuid-v4",
  "parameters": {
    "NAV_DAY_OFFSET": 2
  }
}

Field	Type	Required	Description
key	String	Yes	Full S3 object key of the raw RINEX RINEX Receiver Independent Exchange Format — standard GNSS observation file format synced from GeoNet and calibrated by the processor. file
job_id	String	Yes	UUID v4 identifying the reprocessing job
trace_id	String	Yes	UUID v4 for end-to-end correlation
parameters	Object	No	Override processing parameters (see allowlist below)

Shape 2 — S3 Event Notification (ingest-triggered)

Standard AWS S3 event notification forwarded from the data lake bucket to Process_Queue. The processor reads Records[0].s3 from the message body to extract bucket and key.

Shape 3 — SNS-Wrapped S3 Event

SNS SNS
Amazon Simple Notification Service — publishes CloudWatch alarm notifications; subscribe an email endpoint after deploy. notification wrapping a Shape 2 S3 event in a Message field. The processor unwraps and processes as Shape 2.

Note

S3 TestEvent bodies are acknowledged and silently skipped.

Allowed parameters keys

Only the following parameter names are accepted. Any other key is rejected by the Reprocess API before the message is enqueued.

Parameter	Type	Default	Description
NAV_DAY_OFFSET	Integer	1	Days before observation DOY DOY Day of year — three-digit value (001–366) used in RINEX paths and processed output keys. to fetch nav files
SAVE_PARQUET	Boolean	true	Write Parquet Parquet Columnar file format — primary processed output for efficient Query API range scans; JSON is a fallback when `pyarrow` is unavailable. output
SAVE_CSV	Boolean	true	Write CSV output
SAVE_JSON	Boolean	true	Write JSON output
SAVE_STATIC_PLOTS	Boolean	false	Generate static plot images (.png)
SAVE_INTERACTIVE_PLOTS	Boolean	false	Generate interactive HTML plots

Related pages

• DynamoDB Jobs Table — job status tracking
• Reprocess Workflow — end-to-end verification