Pixels to Planning: Geospatial Data Platforms on AWS

Most multispectral satellite image files arrive as Cloud-Optimized GeoTIFF (COG) or HDF5, ranging from 500 MB to 5 GB per scene depending on resolution and band count. Sentinel-2 alone produces roughly 1.6 TB per day globally. When you start ingesting multiple constellations — Sentinel, Landsat, Planet, SAR radar data — the volume grows to tens of terabytes daily before you even compute derived indices like NDVI, NDWI, or land surface temperature.

The mistake I see most often is treating this data like log files: dump everything into S3 with date-based prefixes and expect Athena to handle it. It does not. The problem is that geospatial queries have two conflicting partitioning axes: time (when the image was captured) and space (which tile/bounding box it covers). A typical analytical query — 'show me forest cover change in the Amazon between 2022 and 2024' — needs to cross hundreds of tiles over two years. With naive date-only partitioning, Athena scans time partitions without spatial filtering, generating S3 scan costs that can reach tens of dollars per query.

The correct solution is to use a table format with geospatial predicate pushdown — Apache Iceberg with GeoParquet extension, stored on S3, with the catalog managed via AWS Glue Data Catalog. GeoParquet encodes geometries as WKB binary columns with bounding box statistics per row group, allowing Athena (via Trino) to skip row groups that do not intersect the query polygon. In internal benchmarks I have run, this reduced scanned data volume by 60–80% for typical spatial window queries.

After resolving the storage format, the second critical decision is the partitioning strategy in S3 and Iceberg. For geospatial data, I use a three-level hierarchy: year/month as the first-level partition (for temporal pruning), H3 grid resolution 4 as the second-level partition (covering approximately 1,770 km² per cell, generating ~5,000 cells for global coverage), and sensor as the third level.

H3 (Uber's hierarchical hexagonal grid library) has a critical property for geospatial data: cells at the same resolution level have approximately equal area, meaning partition file sizes are predictable — something rectangular bounding box partitioning does not guarantee. By indexing each scene's geometries with h3.polyfill() in the Glue Job, you can join tables from different sensors using the H3 index as a key, without expensive geometric intersection operations at query time.

An important operational detail: the Glue Job converting COG to GeoParquet should be configured with --conf spark.sql.shuffle.partitions=200 and G.2X workers (8 vCPU, 32 GB RAM) to process multispectral bands without disk spill. For Sentinel-2 10m resolution scenes (13 bands, ~800 MB per scene), processing a full day of global ingestion takes approximately 45 minutes with 20 G.2X workers — a cost of roughly USD 12 per execution. That number matters when you are planning a continuous operations budget.

The Iceberg table should be configured with write.target-file-size-bytes=268435456 (256 MB) and compaction scheduled via Glue Workflow every 6 hours to avoid the small files problem that degrades Athena performance.

When Earth AI platform outputs feed financial decisions — carbon credit valuation, climate risk scoring for credit portfolios, parametric insurance pricing based on vegetation indices — lineage traceability stops being a best practice and becomes a regulatory requirement. In Brazil, BACEN already requires financial institutions to demonstrate the full methodology behind climate risk models (CMN Resolution 4.945/2021). In Europe, the EU AI Act classifies environmental risk scoring systems as 'high-risk', requiring training data documentation.

In practice, this means each model prediction must be traceable to: (1) the exact Iceberg dataset snapshot used in training, (2) the version of the Glue Job code that processed the raw data, (3) the model version in SageMaker Model Registry, and (4) the feature engineering repository commit. This is what I call quadruple traceability — and most implementations I see cover only (3).

The technical implementation uses SageMaker Lineage Tracking to capture the model → dataset → processing chain, with the Iceberg snapshot ID as the immutable anchor. For integration with external governance tools (Collibra, Alation, DataHub), export lineage events via EventBridge to a Lambda that converts them to OpenLineage format and sends them to the Marquez API. This pattern allows external auditors to query the full lineage without needing direct access to the AWS account.

A critical detail: Lake Formation must have audit logging enabled for all data access operations, with logs sent to a separate S3 bucket with Object Lock. In carbon credit audits I have participated in, the absence of data access audit logs was the primary blocker for certification.

There is a dangerous tendency to treat geospatial data as 'just map data' — public, non-sensitive, requiring no rigorous controls. This is a serious mistake. High-resolution location data combined with satellite image time series can reveal: troop movements, industrial facility production capacity, crop conditions before public reports (material non-public information for insider trading purposes), and land occupation patterns with legal implications.

The security architecture I implement for these systems follows Zero Trust with four layers: network (VPC with private endpoints for S3, SageMaker, and Glue — no traffic leaving to the public internet), identity (IAM roles with aws:RequestedRegion and aws:SourceVpc conditions to ensure only services within the VPC access the data), data (dedicated KMS CMK per data classification — one CMK for raw data, another for processed data, another for model outputs — with key policies requiring kms:ViaService for access only via specific AWS services), and application (Lake Formation with Row/Column-level security).

A specific pattern I use to protect highly sensitive data: the processed data S3 bucket has a bucket policy with aws:PrincipalOrgPaths that restricts access only to specific OUs within the AWS Organization. This ensures that even if an IAM role is compromised in another account in the organization, it will not have access to the sensitive geospatial data.

For LGPD and GDPR compliance with data that may contain personal information (urban area imagery at sub-meter resolution), implement a face/license plate detection and anonymization step as part of the Glue transformation Job, before any data reaches the curated zone.

Computer vision models for geospatial data have a type of drift that does not appear in standard statistical monitors: sensor drift. When a satellite provider updates radiometric calibration processing, or when you add a new constellation to the pipeline, the pixel value distribution changes even though the physical world has not. A model trained on Sentinel-2 L2A can silently degrade when it starts receiving data from a new sensor without retraining.

The solution is to implement drift monitoring at two levels: (1) feature drift using SageMaker Model Monitor with a baseline calculated separately by sensor and by season (summer vs. winter vegetation has completely different distributions), and (2) concept drift by monitoring the output prediction distribution against ground truth collected periodically via crowdsourcing or field validation.

For retraining, I use a continuous training with human approval pattern: a Step Functions workflow triggered when the drift score exceeds a threshold (e.g., PSI > 0.2 for any input feature), executes retraining with the last 90 days of Iceberg data, registers the new model in Model Registry with 'PendingApproval' status, and notifies the data science team via SNS. Manual approval is mandatory before production deployment — not for bureaucratic reasons, but because in financial decisions, a silently degrading model can cause systemic damage before detection.

A concrete capacity number: for land cover segmentation models (U-Net with ResNet-50 backbone, ~25M parameters), full retraining on 90 days of Sentinel-2 data for Brazil (~180K scenes) takes approximately 8 hours on an ml.p3.8xlarge instance (4 V100 GPUs) — a cost of ~USD 90. This is acceptable for monthly or drift-triggered retraining.