Parquet Output

The processor processor
Processor Lambda container — runs PyTECGg calibration per SQS message; adopted when a Batch on Fargate parallel-execution quota increase was rejected. Image from `ghcr.io/platformfuzz/tec-processor-image` mirrored to ECR. writes one or more output formats per input file. All enabled formats share the same partition path and differ only by file extension.

Output formats

Format	Extension	Enabled by default
Parquet Parquet Columnar file format — primary processed output for efficient Query API range scans; JSON is a fallback when `pyarrow` is unavailable. (Snappy)	.parquet	Yes (SAVE_PARQUET=true)
CSV	.csv	Yes (SAVE_CSV=true)
JSON (array of rows)	.json	Yes (SAVE_JSON=true)
Static plot	.png	No
Interactive plot	.html	No

Query API preference

The Query API Query API
Query API Lambda (`services/query-api/`) — serves `/catalog` and `/query` from processed S3 keys and Parquet/JSON files. reads .parquet files when pyarrow is available in the Lambda Lambda
AWS Lambda — serverless compute for ingest-sync, query-api, reprocess-api, and the processor container image. runtime. It falls back to .json when pyarrow is absent. Both formats contain the same 11-column schema.

Output data schema

All data formats (Parquet, CSV, JSON) contain 11 fields:

Column	Type	Description
epoch	Timestamp	Observation epoch in UTC
sv	String	Satellite vehicle identifier (e.g., G01, R05)
id_arc	Int32	Arc identifier for continuous signal tracking
lat_ipp	Float64 / null	IPP IPP Ionospheric Pierce Point — geographic location where the satellite signal intersects the ionospheric shell; plotted on the portal IPP map. latitude in degrees
lon_ipp	Float64 / null	IPP longitude in degrees
azi	Float64	Azimuth in degrees
ele	Float64	Elevation angle in degrees
bias	Float64	Receiver bias estimate
stec	Float64	Slant TEC TEC Total Electron Content — integrated electron density along the GNSS signal path; primary calibrated output of the processor. in TECU
vtec	Float64	Vertical TEC in TECU
veq	Float64	Equivalent vertical TEC

NaN handling

The processor may emit NaN for fields such as lat_ipp and lon_ipp in edge cases. The Query API sanitizes these to JSON null before returning them, ensuring spec-compliant responses for browser clients.

Example query response row

{
  "epoch": "2024-05-29T01:00:00Z",
  "sv": "G01",
  "id_arc": 1,
  "lat_ipp": -36.85,
  "lon_ipp": 174.76,
  "azi": 45.2,
  "ele": 30.1,
  "bias": 0.5,
  "stec": 12.3,
  "vtec": 8.7,
  "veq": 9.1
}

List output files

aws s3 ls "s3://$(terraform -chdir=terraform output -raw bucket_name)/processed/tec/" --recursive | grep parquet

Related pages

• S3 Key Patterns — output key layout
• API Response Schemas — query response format
• REST API Usage