Manual Ingest

Use manual ingest to populate the data lake data lake
Private S3 bucket holding raw RINEX under `raw/rinexhourly/` and processed TEC output under `processed/tec/`. immediately instead of waiting for the next EventBridge Scheduler EventBridge Scheduler
AWS EventBridge Scheduler — triggers ingest-sync Lambda on a recurring UTC schedule (default: hourly). invocation.

Invoke ingest-sync

From the demo repo root:

aws lambda invoke \
  --function-name ingest-sync \
  --payload '{}' \
  /dev/stdout

The Lambda Lambda
AWS Lambda — serverless compute for ingest-sync, query-api, reprocess-api, and the processor container image. returns a JSON response with sync statistics. Check CloudWatch Logs for structured output including trace_id trace_id
UUID v4 correlation ID — propagated in SQS messages and structured logs for end-to-end request tracing. and outcome.

When to use manual ingest

• First deploy verification before the hourly schedule fires
• Testing after changing lookback_hours lookback_hours
  Ingest configuration — UTC rolling window (default 1 hour) defining how far back ingest-sync scans for new RINEX files. or source_prefix
• Recovering from a missed scheduler run (though default lookback_hours=1 with hourly schedule covers one missed run)

Verify files copied

After invocation, list raw objects in the data lake:

aws s3 ls "s3://$(terraform -chdir=terraform output -raw bucket_name)/raw/rinexhourly/" --recursive | head

New keys follow the pattern raw/rinexhourly/{year}/{doy}/{filename}.

Downstream effects

New S3 S3
Amazon Simple Storage Service — data lake bucket for raw ingest and processed TEC output; S3 events trigger processing. objects trigger ObjectCreated notifications to Process_Queue Process_Queue
SQS standard queue — receives S3 ObjectCreated notifications for new raw RINEX files; triggers processor Lambda ingest path. . Continue to SQS Queues and Processor Lambda verification.

Related usage

For operational ingest outside the scheduler, see Manual Ingest usage.