System Overview

This page is the technical architecture reference for the demo repo. For the end-to-end data flow and walkthrough path, start with Project Overview. For per-component runtime behavior, see Infrastructure Layers.

The platform is a single monorepo provisioned entirely through Terraform Terraform
HashiCorp Terraform — provisions all platform infrastructure through five modules in `terraform/`. . Five modules compose the stack; each owns shared resources in its layer rather than duplicating infrastructure definitions.

Monorepo layout

Path	Contents
terraform/	Root module and five submodules under terraform/modules/
services/ingest-sync/	Ingest-sync Lambda Ingest-sync Lambda Ingest-sync Lambda (`services/ingest-sync/`) — polls GeoNet on schedule and copies recent RINEX into the data lake (workaround for no source-bucket event subscription). — GeoNet → data lake sync
services/query-api/	Query API Query API Query API Lambda (`services/query-api/`) — serves `/catalog` and `/query` from processed S3 keys and Parquet/JSON files. — catalog and time-range queries
services/reprocess-api/	Reprocess API Reprocess API Reprocess API Lambda (`services/reprocess-api/`) — accepts `/reprocess` jobs, writes DynamoDB records, enqueues Reprocess_Queue. — job submission and status
Container image (external)	Processor Lambda Processor Lambda Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR. — ghcr.io/platformfuzz/tec-processor-image, mirrored to ECR ECR Amazon Elastic Container Registry — stores the mirrored processor Lambda container image at deploy time. at deploy
web/	Vite SPA — charts, IPP IPP Ionospheric Pierce Point — geographic location where the satellite signal intersects the ionospheric shell; plotted on the portal IPP map. maps, reprocess panel
scripts/	Deploy helpers (ECR mirror, Amplify zip upload)

Zip Lambdas use Python 3.14. The processor runs as a container image, not a zip deployment.

Five Terraform modules

Module	Owns	Depends on
ingest	S3 S3 Amazon Simple Storage Service — data lake bucket for raw ingest and processed TEC output; S3 events trigger processing. data lake, ingest-sync Lambda ingest-sync Lambda Ingest-sync Lambda (`services/ingest-sync/`) — polls GeoNet on schedule and copies recent RINEX into the data lake (workaround for no source-bucket event subscription).	—
ingest-scheduler	EventBridge Scheduler EventBridge Scheduler AWS EventBridge Scheduler — triggers ingest-sync Lambda on a recurring UTC schedule (default: hourly). , IAM IAM AWS Identity and Access Management — execution roles and policies for Lambdas, Scheduler, API Gateway, and S3. invoke role	ingest (Lambda ARN/name)
processing	SQS SQS Amazon Simple Queue Service — buffers ingest and reprocess messages between S3/API and the processor Lambda. / DLQ DLQ Dead-letter queue — SQS queue receiving messages that exceeded maxReceiveCount (5). CloudWatch alarms fire when visible count ≥ 1. , S3→SQS notification, DynamoDB DynamoDB Amazon DynamoDB — Jobs table tracks reprocessing job status by `job_id` (queued → processing → completed/failed). Jobs, ECR ECR Amazon Elastic Container Registry — stores the mirrored processor Lambda container image at deploy time. , processor Lambda	ingest (bucket)
presentation	Query Query Query API Lambda (`services/query-api/`) — serves `/catalog` and `/query` from processed S3 keys and Parquet/JSON files. / reprocess API reprocess API Reprocess API Lambda (`services/reprocess-api/`) — accepts `/reprocess` jobs, writes DynamoDB records, enqueues Reprocess_Queue. Lambdas, API Gateway API Gateway Amazon API Gateway — REST API exposing `/catalog`, `/query`, and `/reprocess`. Browser CORS is locked to the Amplify hostname; CLI clients call the API Gateway URL directly. , Amplify	ingest (bucket read), processing (queues, jobs table)
observability	CloudWatch CloudWatch Amazon CloudWatch — alarms, dashboard (`event-driven-platform`), and structured Lambda logs for observability. alarms, SNS SNS Amazon Simple Notification Service — publishes CloudWatch alarm notifications; subscribe an email endpoint after deploy. topic, dashboard	ingest, processing (and presentation when enabled)

Layer enable flags gate module creation:

Variable	Module
enable_ingest	ingest
enable_processing	processing
enable_presentation	presentation

Processing requires ingest. Presentation requires processing. The scheduler and observability modules follow the enabled layers they monitor.

The modules table above maps each resource to its owning module.

Staged apply order

Processing must exist before automated ingest starts — otherwise raw RINEX RINEX
Receiver Independent Exchange Format — standard GNSS observation file format synced from GeoNet and calibrated by the processor. files land in S3 S3
Amazon Simple Storage Service — data lake bucket for raw ingest and processed TEC output; S3 events trigger processing. with no consumer.

Recommended sequence:

1. Processing — SQS, processor Lambda, DynamoDB Jobs table (creates ingest bucket dependencies)
2. Ingest scheduler — EventBridge Scheduler EventBridge Scheduler
   AWS EventBridge Scheduler — triggers ingest-sync Lambda on a recurring UTC schedule (default: hourly). once processing is healthy
3. Full apply — presentation, observability, and reconciliation

Commands and Amplify options are in Staged Apply.

Key design decisions

Provisioning and compute

Decision	Rationale
Terraform-only IaC	Single workflow; no drift between tools
Processor as Lambda container	SQS-triggered container image — see design notes below
External container image	Built independently; mirrored to ECR at deploy time
Python 3.14 zip Lambdas	Ingest, query, and reprocess use managed runtime python3.14

Eventing and queues

Decision	Rationale
EventBridge Scheduler EventBridge Scheduler AWS EventBridge Scheduler — triggers ingest-sync Lambda on a recurring UTC schedule (default: hourly).	Recurring UTC triggers with dedicated invoke roles
SQS standard queues	Processing order irrelevant; higher throughput than FIFO
Dual queues (ingest vs reprocess)	Separate concurrency caps — operator jobs do not starve ingest
Visibility timeout 900s	Covers processor execution and SQS retry window

Data layout and API

Decision	Rationale
Parquet Parquet Columnar file format — primary processed output for efficient Query API range scans; JSON is a fallback when `pyarrow` is unavailable. primary output	Efficient range scans; JSON fallback when pyarrow unavailable
Deterministic output keys	Idempotent reprocessing via safe overwrites
Prefix-based catalog	Stations and dates from processed/tec/station=.../year=.../doy=... layout — not S3 Annotations
DynamoDB DynamoDB Amazon DynamoDB — Jobs table tracks reprocessing job status by `job_id` (queued → processing → completed/failed). for jobs	Low-latency job_id lookups; no joins required
NaN sanitization in Query API	Non-finite floats become JSON null for browser compatibility

Note

Also considered — AWS Batch on Fargate: An earlier design used Batch to scale parallel calibration workers as ingest volume grew. A limit-increase request for higher Batch/Fargate parallel execution was rejected, so the same container image (ghcr.io/platformfuzz/tec-processor-image) runs as an SQS-triggered Lambda container Lambda container
Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR. instead — each invocation pulls one message from Process_Queue Process_Queue
SQS standard queue — receives S3 ObjectCreated notifications for new raw RINEX files; triggers processor Lambda ingest path. or Reprocess_Queue Reprocess_Queue
SQS standard queue — buffers reprocessing job messages from the Reprocess API with lower concurrency than ingest. . Parallelism comes from SQS event-source concurrency, not a Batch compute environment. Trade-off: 900s Lambda timeout and Lambda concurrency caps vs Batch worker scaling.

Note

Also removed — S3 Annotations: S3 Annotations and annotations-indexer were explored for catalog discovery. The current deployment uses prefix-based listing on processed output keys under processed/tec/.

Observability

All Lambdas emit structured JSON logs with trace_id trace_id
UUID v4 correlation ID — propagated in SQS messages and structured logs for end-to-end request tracing. , duration_ms, outcome, and error fields on failure. The observability module provisions alarms (DLQ depth, processor health, queue backlog, ingest-sync errors), an SNS topic, and the event-driven-platform dashboard.

Alarm names, subscription steps, and dashboard verification are in Alarms and Dashboard.

Related pages

• Project Overview — what you build and walkthrough path
• Infrastructure Layers — ingest, processing, presentation runtime detail
• Staged Apply — recommended Terraform apply order
• Terraform Variables and Terraform Outputs — configuration reference
• Data Contract — SQS, DynamoDB, S3 keys, and API schemas