Project Overview

What you will build

This walkthrough documents the event-driven-serverless-platform-demo — a consolidated AWS AWS
Amazon Web Services — cloud platform for ingest, processing, storage, API, and portal hosting in this walkthrough. event-driven serverless platform for GNSS GNSS
Global Navigation Satellite System — source of RINEX observation data ingested and calibrated in this platform. RINEX RINEX
Receiver Independent Exchange Format — standard GNSS observation file format synced from GeoNet and calibrated by the processor. ingest, PyTECGg PyTECGg
PyTECGg — Python library performing TEC calibration on GNSS RINEX observations inside the processor container. TEC TEC
Total Electron Content — integrated electron density along the GNSS signal path; primary calibrated output of the processor. calibration, and interactive visualization.

By the end you will have:

• Ingest — hourly RINEX RINEX
  Receiver Independent Exchange Format — standard GNSS observation file format synced from GeoNet and calibrated by the processor. synced from GeoNet GeoNet
  GeoNet open data — public S3 bucket `geonet-open-data` with GNSS RINEX hourly files under `gnss/rinexhourly/`. Not owned by this stack; no push notification when new files arrive. into a private data lake data lake
  Private S3 bucket holding raw RINEX under `raw/rinexhourly/` and processed TEC output under `processed/tec/`.
• Processing — SQS SQS
  Amazon Simple Queue Service — buffers ingest and reprocess messages between S3/API and the processor Lambda. -driven processor Lambda processor Lambda
  Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR. container writing calibrated TEC output
• Presentation — REST API REST API
  Amazon API Gateway — REST API exposing `/catalog`, `/query`, and `/reprocess`. Browser CORS is locked to the Amplify hostname; CLI clients call the API Gateway URL directly. and an Amplify Amplify
  AWS Amplify — hosts the Vite portal SPA. This walkthrough deploys via manual zip upload (no Git connection required). portal for charts, maps, and reprocess jobs
• Operations — CloudWatch CloudWatch
  Amazon CloudWatch — alarms, dashboard (`event-driven-platform`), and structured Lambda logs for observability. alarms, SNS SNS
  Amazon Simple Notification Service — publishes CloudWatch alarm notifications; subscribe an email endpoint after deploy. notifications, and the event-driven-platform dashboard

Everything lives in one monorepo and is provisioned through Terraform Terraform
HashiCorp Terraform — provisions all platform infrastructure through five modules in `terraform/`. .

Architecture and data flow

Data moves left to right across three layers — ingest, processing, and presentation. New raw files in S3 S3
Amazon Simple Storage Service — data lake bucket for raw ingest and processed TEC output; S3 events trigger processing. enqueue the process queue process queue
SQS standard queue — receives S3 ObjectCreated notifications for new raw RINEX files; triggers processor Lambda ingest path. ; operator reprocess jobs use a separate reprocess queue reprocess queue
SQS standard queue — buffers reprocessing job messages from the Reprocess API with lower concurrency than ingest. with lower concurrency so scheduled ingest is not starved.

The browser and CLI reach processed data only through API Gateway API Gateway
Amazon API Gateway — REST API exposing `/catalog`, `/query`, and `/reprocess`. Browser CORS is locked to the Amplify hostname; CLI clients call the API Gateway URL directly. — not S3 or DynamoDB DynamoDB
Amazon DynamoDB — Jobs table tracks reprocessing job status by `job_id` (queued → processing → completed/failed). directly. Infrastructure Layers covers each component; System Overview covers Terraform modules and design decisions.

Three layers

Layer	Responsibility	Key components
Ingest	Sync recent RINEX from GeoNet into the data lake	EventBridge Scheduler EventBridge Scheduler AWS EventBridge Scheduler — triggers ingest-sync Lambda on a recurring UTC schedule (default: hourly). , Ingest_Sync_Lambda Ingest_Sync_Lambda Ingest-sync Lambda (`services/ingest-sync/`) — polls GeoNet on schedule and copies recent RINEX into the data lake (workaround for no source-bucket event subscription). , S3 S3 Amazon Simple Storage Service — data lake bucket for raw ingest and processed TEC output; S3 events trigger processing. Data_Lake_Bucket Data_Lake_Bucket Private S3 bucket holding raw RINEX under `raw/rinexhourly/` and processed TEC output under `processed/tec/`.
Processing	Calibrate raw observations into TEC output	Process_Queue Process_Queue SQS standard queue — receives S3 ObjectCreated notifications for new raw RINEX files; triggers processor Lambda ingest path. , Reprocess_Queue Reprocess_Queue SQS standard queue — buffers reprocessing job messages from the Reprocess API with lower concurrency than ingest. , Processor Lambda Processor Lambda Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR. container, DynamoDB DynamoDB Amazon DynamoDB — Jobs table tracks reprocessing job status by `job_id` (queued → processing → completed/failed). Jobs table Jobs table DynamoDB table keyed by `job_id` — stores reprocess job metadata, parameters, status, and output references.
Presentation	Query, visualize, and submit reprocessing jobs	API Gateway API Gateway Amazon API Gateway — REST API exposing `/catalog`, `/query`, and `/reprocess`. Browser CORS is locked to the Amplify hostname; CLI clients call the API Gateway URL directly. , Query Query Query API Lambda (`services/query-api/`) — serves `/catalog` and `/query` from processed S3 keys and Parquet/JSON files. / Reprocess API Reprocess API Reprocess API Lambda (`services/reprocess-api/`) — accepts `/reprocess` jobs, writes DynamoDB records, enqueues Reprocess_Queue. Lambdas, Amplify Amplify AWS Amplify — hosts the Vite portal SPA. This walkthrough deploys via manual zip upload (no Git connection required). Portal

What you will see

Hosted portal

The portal browses processed stations stations
Four-character GNSS receiver site identifier (e.g. `auck`) parsed from RINEX filenames and used in processed key paths. , charts TEC metrics over time, and maps ionospheric pierce points ionospheric pierce points
Ionospheric Pierce Point — geographic location where the satellite signal intersects the ionospheric shell; plotted on the portal IPP map. . See Portal Usage and Amplify Portal.

Operations dashboard

Observability spans all layers. The event-driven-platform dashboard aggregates alarm health, queue backlog, Lambda throughput, and API activity. Subscribe to SNS and verify alarms in Alarms and Dashboard.

Key terms

Term	Definition
Demo Repo Demo Repo event-driven-serverless-platform-demo — source repository for Terraform, Lambda services, portal, and scripts. Commands run from its root with `terraform -chdir=terraform` unless stated otherwise.	Source repository event-driven-serverless-platform-demo — run commands from its root with terraform -chdir=terraform unless stated otherwise
RINEX RINEX Receiver Independent Exchange Format — standard GNSS observation file format synced from GeoNet and calibrated by the processor.	GNSS observation files synced from GeoNet and calibrated by the processor
TEC TEC Total Electron Content — integrated electron density along the GNSS signal path; primary calibrated output of the processor.	Total Electron Content — primary calibrated output served by the Query API and portal
Data Lake Data Lake Private S3 bucket holding raw RINEX under `raw/rinexhourly/` and processed TEC output under `processed/tec/`.	Private S3 bucket with raw/rinexhourly/ ingest keys and processed/tec/ output
Process vs reprocess queues Process vs reprocess queues SQS standard queue — receives S3 ObjectCreated notifications for new raw RINEX files; triggers processor Lambda ingest path.	Ingest path ( Process_Queue Process_Queue SQS standard queue — receives S3 ObjectCreated notifications for new raw RINEX files; triggers processor Lambda ingest path. ) vs operator jobs ( Reprocess_Queue Reprocess_Queue SQS standard queue — buffers reprocessing job messages from the Reprocess API with lower concurrency than ingest. ) — separate concurrency so reprocess does not starve ingest
Processor Lambda Processor Lambda Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR.	Container image (ghcr.io/platformfuzz/tec-processor-image, mirrored to ECR ECR Amazon Elastic Container Registry — stores the mirrored processor Lambda container image at deploy time. ) — SQS-triggered Lambda container (see design notes)
Terraform stack Terraform stack HashiCorp Terraform — provisions all platform infrastructure through five modules in `terraform/`.	Five modules in terraform/ — ingest, ingest-scheduler, processing, presentation, observability

Walkthrough path

The sidebar follows a deploy → verify → use order.

Before you deploy

1. Prerequisites — Tools and Accounts
2. Architecture — System Overview and Infrastructure Layers

Deploy

Provision the stack in order: Terraform Init → ECR and Processor Image → Staged Apply → Amplify Portal → CORS Lockdown.

Verify

Confirm each stage of the pipeline:

• Scheduler and ingest — EventBridge Scheduler, Manual Ingest
• Processing — SQS Queues, Processor Lambda, S3 Processed Output
• Presentation — REST API, Reprocess Workflow, Portal
• Operations — Alarms and Dashboard

Use and reference

• Usage — REST API, Portal, Manual Ingest
• Data Contract — SQS, DynamoDB, S3 keys, Parquet, and API schemas
• Development, Troubleshooting, and Reference — local setup, common issues, Terraform outputs

Next steps

1. Confirm Tools and Accounts before cloning the demo repo.
2. Read System Overview for module layout and design decisions.
3. Follow Terraform Init to deploy the stack.