S3 Processed Output
After the processor Lambda
Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR. completes, TEC
Total Electron Content — integrated electron density along the GNSS signal path; primary calibrated output of the processor. output appears under the processed/tec/ prefix with Hive-style partition keys in the data lake
Private S3 bucket holding raw RINEX under `raw/rinexhourly/` and processed TEC output under `processed/tec/`. .
List processed output
Section titled “List processed output”From the demo repo root:
aws s3 ls "s3://$(terraform -chdir=terraform output -raw bucket_name)/processed/tec/" --recursiveExpected key pattern
Section titled “Expected key pattern”processed/tec/station={station}/year={year}/doy={doy}/{filename}.parquetprocessed/tec/station={station}/year={year}/doy={doy}/{filename}.jsonExample:
processed/tec/station=auck/year=2026/doy=179/auck1790.parquetprocessed/tec/station=auck/year=2026/doy=179/auck1790.jsonKey derivation
Section titled “Key derivation”Output keys are deterministically derived from raw input keys:
- Parse
raw/rinexhourly/{year}/{doy}/{filename} - Extract station
Four-character GNSS receiver site identifier (e.g. `auck`) parsed from RINEX filenames and used in processed key paths. (first 4 characters of filename, lowercased) - Write to
processed/tec/station={station}/year={year}/doy={doy}/{source_stem}.{ext}
See S3 Key Patterns and Parquet Output.
Catalog dependency
Section titled “Catalog dependency”Stations and dates appear in /catalog only after processed output exists. Raw ingest keys alone do not populate the catalog.