Satellite and IoT Data Architecture

Why Satellite and IoT Data Workloads Are Different

Telemetry is generated by physical systems that do not behave like a stable business application. A device can lose connectivity, buffer observations, restart with an incorrect clock, resend a batch, or change payload shape after a firmware release. A ground station can receive a concentrated burst during a contact window. Sensor readings can be valid technically but implausible physically. The architecture must preserve enough evidence to distinguish those conditions.

Design around event time and ingestion time separately. Event time describes when the observation occurred at the source; ingestion time records when the platform received it. Keeping both makes late-arrival policies, replay, latency measurement, and sequence analysis possible. Device or spacecraft identity, firmware or schema version, unit of measure, quality flags, and location context should travel with the observation or be resolvable through governed reference data.

Satellite and IoT Reference Architecture

Define an Event Contract Before Scaling

A contract makes producer and consumer assumptions explicit. It should define identity, timestamps, version, units, required fields, valid ranges, compatibility, and ownership. The example below is an illustrative envelope, not a universal standard; each mission or device domain should define the fields and guarantees it can actually support.

{
  "event_id": "globally-unique-id",
  "source_id": "device-or-spacecraft-id",
  "schema_version": "2.1",
  "event_time": "ISO-8601 timestamp from source",
  "ingested_at": "ISO-8601 platform timestamp",
  "sequence": 184223,
  "location": {"longitude": -63.58, "latitude": 44.65},
  "measurements": [
    {"name": "temperature", "value": 18.4, "unit": "celsius"}
  ],
  "quality_flags": [],
  "trace_id": "end-to-end-correlation-id"
}

Quality Controls by Failure Mode

• Duplicate delivery: deduplicate on stable event identity or a documented composite key.
• Out-of-order events: process with event-time windows and preserve sequence evidence.
• Clock drift: compare source time, receive time, and known clock health before treating latency as real.
• Schema drift: validate version compatibility and route unsupported payloads without discarding them.
• Impossible values: apply device-state, calibration, unit, and physical-range rules with clear severity.
• Missing context: quarantine or enrich observations that lack device identity, unit, location, or required reference data.

Choose Streaming, Micro-Batch, or Batch by Decision Latency

Not every telemetry workload needs continuous stream processing. Use the latency of the business or operational decision, not the presence of sensors, to choose the pattern. A single architecture can support multiple paths from the same durable event and raw history.

Model Time, Position, and Observation Context Explicitly

Telemetry commonly combines a moving source, a timestamp, a measured phenomenon, and spatial context. Keep the coordinate reference system and units explicit. Longitude and latitude order, altitude support, planar versus spherical calculations, and geometry validity can change analytical results. Store or transform spatial values into native geospatial types where the platform and workload benefit, while preserving the original representation for traceability.

Separate immutable observations from changing reference data such as device ownership, calibration, orbit metadata, firmware, deployment site, and administrative region. Use effective dates so a historical event resolves to the context that was valid when it occurred rather than today’s device configuration.

• Keep event time, ingestion time, and processing time distinct
• Document coordinate reference system, axis order, altitude, and precision
• Use effective-dated reference data for device and calibration history
• Preserve raw payload, parser version, and transformed representation
• Define spatial indexing and clustering only after observing query patterns

Implementation Sequence

1. Classify operational decisions. Identify which outcomes require seconds, minutes, or batch latency and what happens when data is missing or late.
2. Define identity and time semantics. Standardize source identity, event identity, sequence, event time, ingestion time, units, and location representation.
3. Publish versioned contracts. Document required fields, compatibility, ownership, quality ranges, retention, and deprecation behavior.
4. Build durable ingestion and raw history. Decouple producers and consumers, preserve replayable source-aligned data, and verify recovery before adding many downstream products.
5. Add validation and quarantine paths. Route malformed or unsupported events with enough context for diagnosis; do not silently discard evidence.
6. Create purpose-built serving products. Separate operational state, historical analytics, geospatial products, and AI features according to their grains and service levels.
7. Operate the control plane. Monitor lag, loss, duplicates, late arrivals, schema versions, quality, storage growth, access, and end-to-end lineage.

Metrics for a Reliable Telemetry Platform

• Events accepted, rejected, quarantined, duplicated, and replayed by source and schema version
• End-to-end latency percentiles using both event and ingestion time
• Consumer lag and backlog recovery time after a burst or outage
• Percentage of observations with valid identity, unit, time, and spatial context
• Schema compatibility failures and unsupported firmware versions
• Raw-to-serving reconciliation and missing-sequence rates
• Storage, compute, and network cost per useful telemetry product