Skip to content
Event-Driven Serverless Walkthrough

Manual Ingest

Manual ingest invokes the Ingest_Sync_Lambda
Ingest-sync Lambda (`services/ingest-sync/`) — polls GeoNet on schedule and copies recent RINEX into the data lake (workaround for no source-bucket event subscription). directly, bypassing the EventBridge Scheduler
AWS EventBridge Scheduler — triggers ingest-sync Lambda on a recurring UTC schedule (default: hourly). . Use it for immediate data population or recovery.

Invoke ingest-sync

Section titled “Invoke ingest-sync”

From the demo repo root:

Terminal window
aws lambda invoke \
  --function-name ingest-sync \
  --payload '{}' \
  /dev/stdout

What it does

Section titled “What it does”

On each invocation the Lambda
AWS Lambda — serverless compute for ingest-sync, query-api, reprocess-api, and the processor container image. :

  1. Reads LOOKBACK_HOURS
    Ingest configuration — UTC rolling window (default 1 hour) defining how far back ingest-sync scans for new RINEX files.     (default 1) to compute a rolling UTC window
  2. Lists objects from s3://geonet-open-data/gnss/rinexhourly/
  3. Copies new files to raw/rinexhourly/{year}/{doy}/{filename} in the data lake
    Private S3 bucket holding raw RINEX under `raw/rinexhourly/` and processed TEC output under `processed/tec/`.
  4. Skips files already present

When to use manual ingest

Section titled “When to use manual ingest”
ScenarioAction
First deploy, before scheduler firesInvoke once to seed raw data
Testing after config change

Invoke after updating lookback_hours

Missed scheduler run

Default hourly schedule + 1h lookback covers one miss; invoke if longer gap

Verify results

Section titled “Verify results”
Terminal window
aws s3 ls "s3://$(terraform -chdir=terraform output -raw bucket_name)/raw/rinexhourly/" --recursive | tail -5


aws sqs get-queue-attributes \
  --queue-url "$(terraform -chdir=terraform output -raw queue_url)" \
  --attribute-names ApproximateNumberOfMessages

New raw files trigger S3
Amazon Simple Storage Service — data lake bucket for raw ingest and processed TEC output; S3 events trigger processing. notifications to Process_Queue
SQS standard queue — receives S3 ObjectCreated notifications for new raw RINEX files; triggers processor Lambda ingest path. . The processor Lambda
Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR. consumes messages and writes processed output.

Section titled “Related pages”