Infrastructure Layers

Runtime behavior within each layer — how components interact during normal operation. For Terraform modules and design decisions, see System Overview. For schemas and key layouts, see Data Contract.

Ingest layer

Copies recent RINEX RINEX
Receiver Independent Exchange Format — standard GNSS observation file format synced from GeoNet and calibrated by the processor. from GeoNet GeoNet
GeoNet open data — public S3 bucket `geonet-open-data` with GNSS RINEX hourly files under `gnss/rinexhourly/`. Not owned by this stack; no push notification when new files arrive. into the private data lake data lake
Private S3 bucket holding raw RINEX under `raw/rinexhourly/` and processed TEC output under `processed/tec/`. under raw/rinexhourly/. A new object in the data lake is what triggers processing ( S3 S3
Amazon Simple Storage Service — data lake bucket for raw ingest and processed TEC output; S3 events trigger processing. ObjectCreated → SQS SQS
Amazon Simple Queue Service — buffers ingest and reprocess messages between S3/API and the processor Lambda. ).

Note

Scheduled polling (workaround): GeoNet publishes to a public bucket this stack does not own. There is no webhook, feed subscription, or S3 event when a new hourly file lands. Ingest-sync Ingest-sync
Ingest-sync Lambda (`services/ingest-sync/`) — polls GeoNet on schedule and copies recent RINEX into the data lake (workaround for no source-bucket event subscription). runs on a recurring schedule, lists the source prefix, and copies files within LOOKBACK_HOURS LOOKBACK_HOURS
Ingest configuration — UTC rolling window (default 1 hour) defining how far back ingest-sync scans for new RINEX files. . Lookback and schedule cadence must overlap so files are not missed. For an on-demand pull, use Manual Ingest.

Component	Runtime role
EventBridge Scheduler EventBridge Scheduler AWS EventBridge Scheduler — triggers ingest-sync Lambda on a recurring UTC schedule (default: hourly).	Poll cadence (default rate(1 hour)); invokes ingest-sync
Ingest_Sync_Lambda Ingest_Sync_Lambda Ingest-sync Lambda (`services/ingest-sync/`) — polls GeoNet on schedule and copies recent RINEX into the data lake (workaround for no source-bucket event subscription).	List GeoNet prefix, copy new files into raw/rinexhourly/{year}/{doy}/

Per invocation: compute rolling UTC window from LOOKBACK_HOURS (default 1) → list s3://geonet-open-data/gnss/rinexhourly/ → skip objects already in the data lake → copy the rest. Ingest-sync does not enqueue SQS SQS
Amazon Simple Queue Service — buffers ingest and reprocess messages between S3/API and the processor Lambda. directly.

Verify: EventBridge Scheduler · Manual Ingest · S3 key patterns (raw layout)

Processing layer

Calibrates raw RINEX into TEC TEC
Total Electron Content — integrated electron density along the GNSS signal path; primary calibrated output of the processor. with PyTECGg PyTECGg
PyTECGg — Python library performing TEC calibration on GNSS RINEX observations inside the processor container. inside an SQS-triggered Lambda container SQS-triggered Lambda container
Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR. (see design notes for why not Batch on Fargate).

Queue	Fed by	Concurrency (default)	Path
Process_Queue Process_Queue SQS standard queue — receives S3 ObjectCreated notifications for new raw RINEX files; triggers processor Lambda ingest path.	S3 ObjectCreated on raw/ keys	15	Scheduled ingest
Reprocess_Queue Reprocess_Queue SQS standard queue — buffers reprocessing job messages from the Reprocess API with lower concurrency than ingest.	Reprocess API Reprocess API Reprocess API Lambda (`services/reprocess-api/`) — accepts `/reprocess` jobs, writes DynamoDB records, enqueues Reprocess_Queue.	2	Operator jobs

Both queues: visibility timeout 900s, batch_size=1, ReportBatchItemFailures, DLQ DLQ
Dead-letter queue — SQS queue receiving messages that exceeded maxReceiveCount (5). CloudWatch alarms fire when visible count ≥ 1. after 5 failed receives.

Processor invocation: read one SQS message → load raw RINEX from raw/rinexhourly/ → run PyTECGg (nav files from BKG) → write Parquet Parquet
Columnar file format — primary processed output for efficient Query API range scans; JSON is a fallback when `pyarrow` is unavailable. /JSON to processed/tec/station={station}/year={year}/doy={doy}/ → update Jobs table Jobs table
DynamoDB table keyed by `job_id` — stores reprocess job metadata, parameters, status, and output references. when job_id is present. Timeout 900s; memory 2048 MiB. Throughput is capped by per-queue concurrency and account Lambda concurrency.

Verify: SQS Queues · Processor Lambda · S3 Processed Output · Parquet output

Presentation layer

Exposes processed TEC through API Gateway API Gateway
Amazon API Gateway — REST API exposing `/catalog`, `/query`, and `/reprocess`. Browser CORS is locked to the Amplify hostname; CLI clients call the API Gateway URL directly. and an Amplify Amplify
AWS Amplify — hosts the Vite portal SPA. This walkthrough deploys via manual zip upload (no Git connection required). portal. Clients never read the data lake or DynamoDB DynamoDB
Amazon DynamoDB — Jobs table tracks reprocessing job status by `job_id` (queued → processing → completed/failed). directly.

Endpoint	Lambda	Runtime role
GET /catalog	Query API	Prefix-list stations and (year, doy) under processed/tec/
GET /query	Query API	Range-read Parquet/JSON from S3; sanitize non-finite floats
POST /reprocess	Reprocess API	Validate params, create job record, enqueue Reprocess_Queue
GET /reprocess/{job_id}	Reprocess API	Poll job status from DynamoDB

Browser CORS locks to the Amplify hostname after CORS Lockdown. The portal (Vite SPA in web/) polls /catalog and /query for charts and IPP IPP
Ionospheric Pierce Point — geographic location where the satellite signal intersects the ionospheric shell; plotted on the portal IPP map. maps; the reprocess panel calls /reprocess.

Request/response shapes, query limits (QUERY_MAX_ROWS=2000, 7-day window), and reprocess allowlists are in API Response Schemas and DynamoDB Jobs Table.

Verify: REST API · Reprocess Workflow · Portal

Observability layer

Owned by the observability Terraform module. Monitors:

• DLQs — visible messages on ingest and reprocess dead-letter queues
• Processor — Lambda errors and throttles
• Queues — depth and oldest-message age on Process_Queue and Reprocess_Queue
• Ingest — ingest-sync Lambda errors

Alarms publish to an SNS SNS
Amazon Simple Notification Service — publishes CloudWatch alarm notifications; subscribe an email endpoint after deploy. topic. The event-driven-platform dashboard aggregates alarm status, queue metrics, and Lambda throughput.

Tip

Structured logs include trace_id trace_id
UUID v4 correlation ID — propagated in SQS messages and structured logs for end-to-end request tracing. for correlating ingest, processing, and API requests across CloudWatch log groups.

Alarm names, SNS subscription, and dashboard verification: Alarms and Dashboard.

Related pages

• System Overview — Terraform modules, staged apply, design decisions
• Project Overview — end-to-end flow and walkthrough path
• Staged Apply — processing before scheduler
• Data Contract — SQS, S3 keys, Parquet, API schemas