S3 Key Patterns

S3 S3
Amazon Simple Storage Service — data lake bucket for raw ingest and processed TEC output; S3 events trigger processing. keys follow deterministic patterns that enable idempotent reprocessing and prefix-based catalog discovery.

Raw ingest keys

Pattern: raw/rinexhourly/{year}/{doy}/{filename}

Component	Format	Example
year	4-digit integer	2026
doy	3-digit zero-padded integer	179
filename	GeoNet hourly RINEX RINEX Receiver Independent Exchange Format — standard GNSS observation file format synced from GeoNet and calibrated by the processor. 3 observation file — {station}{doy}{hour}.{YY}o (e.g. .26o for 2026)	auck1790.26o

Full example: raw/rinexhourly/2026/179/auck1790.26o

Ingest copies from GeoNet GeoNet
GeoNet open data — public S3 bucket `geonet-open-data` with GNSS RINEX hourly files under `gnss/rinexhourly/`. Not owned by this stack; no push notification when new files arrive. ‘s public bucket (geonet-open-data/gnss/rinexhourly/) into the private data lake data lake
Private S3 bucket holding raw RINEX under `raw/rinexhourly/` and processed TEC output under `processed/tec/`. under this layout.

Processed output keys

Pattern: processed/tec/station={station}/year={year}/doy={doy}/{source_stem}.{ext}

Component	Format	Example
station	4-character lowercase alphanumeric	auck
year	4-digit integer	2026
doy	3-digit zero-padded integer	179
source_stem	Input filename without extension	auck1790
ext	Format extension	parquet, json, csv, png, html

Full examples:

• processed/tec/station=auck/year=2026/doy=179/auck1790.parquet
• processed/tec/station=auck/year=2026/doy=179/auck1790.json

Key derivation

The output key is deterministically derived from the input key:

1. Parse raw/rinexhourly/{year}/{doy}/{filename} to extract year, doy, and filename
2. Extract station (first 4 characters of filename, lowercased)
3. Construct processed/tec/station={station}/year={year}/doy={doy}/{source_stem}.{ext} where source_stem is the input filename without its extension

This ensures idempotent reprocessing — the same input always produces the same output key per format, allowing safe overwrites.

Catalog lookup contract

Catalog endpoints list processed output only:

• GET /catalog — Lists processed/tec/station=*/ prefixes; returns unique station codes uppercased
• GET /catalog?station=AUCK — Lists keys under processed/tec/station=auck/; parses (year, doy) pairs sorted descending

Raw ingest keys under raw/rinexhourly/ are not included in the catalog.

The Reprocess API Reprocess API
Reprocess API Lambda (`services/reprocess-api/`) — accepts `/reprocess` jobs, writes DynamoDB records, enqueues Reprocess_Queue. resolves raw input keys by listing objects under raw/rinexhourly/ for the requested station, year, and DOY.

Verify keys in S3

aws s3 ls "s3://$(terraform -chdir=terraform output -raw bucket_name)/raw/rinexhourly/" --recursive | head
aws s3 ls "s3://$(terraform -chdir=terraform output -raw bucket_name)/processed/tec/" --recursive | head

Related pages

• Parquet Output — column schema
• S3 Processed Output verification