OlmoEarth v1.3: Attention Pooling, Matryoshka Distillation, and Searchlight

AI disclosure: This article was prepared with generative AI assistance. Editorial responsibility: Florian Zimmermeister (flozi00). Legal notice and contact.

Ai2 released OlmoEarth v1.3 Base on October 22, 2026. It is a downloadable multimodal Earth-observation model, not an API-only research preview: the release includes a 785 MB checkpoint, training and inference code, a technical report, and annual 10 m embeddings that can be read directly from object storage. The model card describes a 128M-parameter ViT, a 14.3M-parameter attention pooler, and a roughly 0.1M-parameter linear student.¹²

The useful architecture is the whole production path. A 12-layer ViT combines Sentinel-1 radar with Sentinel-2 and Landsat multispectral time series. A learned spatial query grid reads those tokens, a detached student compresses every location from 768 to 128 dimensions, the first 64 coordinates are trained as a usable prefix, and a signed-power int8 format makes the result distributable. Searchlight inference then changes the spatial attention execution to remove crop seams at continental scale.

This article audits the released configuration at revision 9e9e918a369701c105dde7ef56789b0ff1d47f50 and source at 7353d1b5105a3ad662809a700c98065fc711f4ab. The weight payload was not downloaded or executed. Equations derived from those artifacts are labelled as such; Ai2's evaluation and throughput numbers remain vendor measurements.³⁴

The released inference graph

During pretraining, six map modalities—such as WorldCover, elevation and OpenStreetMap rasters—supply reconstruction or direct supervision targets. They are not required inputs to the released observation encoder. The shipped embedding path consumes the three satellite modalities and returns the student's location vectors; the decoder and distillation back-projection heads are training machinery.⁵³

The encoder configuration also applies random 20% band dropout to Sentinel-2 and Landsat during training. That is robustness training, not permission to reorder or silently omit arbitrary bands in production: the saved tokenization configuration fixes the expected band groups and order.

Token geometry: why patch size two matters

For a simplified input where all $M$ modalities have $T$ timesteps over an $H\times W$ pixel window and use one band group, a spatial patch size $p$ produces

encoder tokens. The released global pipeline uses $T=12$, $M=3$, $H=W=16$ and $p=2$, so $N=2{,}304$. A patch covers $20\times20$ m, but the pooler emits one query per $10\times10$ m pixel. With output patch size $s=1$, the query grid has

Per attention head, the pooler's cross-attention score matrix is $Q\times N=256\times2{,}304$ and its spatial self-attention matrix is $Q\times Q=256\times256$. The encoder itself processes $N^2=5{,}308{,}416$ token pairs per full-attention layer. These pair counts omit projections, MLPs, masks and memory traffic, but they expose the design trade: keep the expensive ViT at 20 m tokens, then recover a 10 m output grid with cheaper learned queries.

At $p=1$, the encoder would see four times as many tokens and sixteen times as many attention pairs. End-to-end work does not fall by 16× because linear layers and the pooler remain. The report measures 2,378 GMACs for the 10 m ViT path and 308 GMACs for the 20 m ViT plus 10 m pooler path on this window—7.7×, under that exact setup.⁵

The attention pooler is spatial, not a mean

The pooler begins with one learned vector $u\in\mathbb{R}^{768}$ and repeats it $Q$ times. Content alone cannot distinguish these identical queries. Their identity comes from 2D RoPE coordinates. For query index $k$ on one axis, input patch size $p$, output patch size $s$ and ground-sampling-distance ratio $g$, the implementation places its center at

When $s=p$, query and patch centers coincide. When $s=1<p$, additional queries sit at pixel centers inside each encoder patch. The query tensor $Z_0\in\mathbb{R}^{B\times Q\times768}$ cross-attends to visible ViT tokens $X\in\mathbb{R}^{B\times N\times768}$, then self-attends across the query grid:

The notation suppresses the blocks' residual and MLP branches; $L\in\mathbb{R}^{B\times Q\times768}$ is the location-token grid. The read uses the ViT's final visible tokens as keys and values, so its softmax can weight cloudy optical observations differently from radar or another month. The self-attention step lets neighboring output locations exchange context. This is learned, data-dependent aggregation—not arithmetic averaging.⁶

The encoder uses mixed 3D RoPE because its tokens have space and time coordinates. The location grid has no modality or time axis, so the pooler deliberately uses only axial 2D RoPE. Repeating a single learned query also avoids baking one training-window size into the parameters.

Detached Matryoshka distillation: preserving points and geometry

The served student is intentionally prevented from changing the teacher. The implementation detaches $L$, applies a linear projection and then LayerNorm:

Two prefixes are trained: $S^{(128)}=S$ and $S^{(64)}=S_{:,:,:64}$. Each width $d\in\{128,64\}$ has its own two-layer back-projection $g_d:\mathbb{R}^{d}\rightarrow\mathbb{R}^{768}$ with hidden width 256. Its pointwise term is

Pointwise agreement alone does not explicitly preserve which locations are mutually similar. The Gram term flattens batch and location axes, samples at most $n=2{,}048$ tokens, L2-normalizes rows, and matches teacher and student cosine matrices:

The code sums cosine and Gram terms for both widths without extra weights. The $2{,}048$-token cap bounds the quadratic Gram matrix to about 4.2 million entries. Gradients update the student and its training-only back-projections, but not the ViT or pooler. Teacher training continues through masked reconstruction and map supervision in the same forward pass; the student learns against that moving teacher without a second teacher execution.⁷

One subtle contract matters in deployment: LayerNorm is applied to the full 128-vector before slicing the first 64 coordinates. The 64-vector is trained as a prefix, but it is not independently LayerNormed or L2-normalized. Use exactly the exported prefix; do not take arbitrary widths or silently renormalize it and assume equivalent classifier behavior.

Signed-power int8 is not ordinary linear quantization

The global embeddings are stored as int8 with $-128$ reserved for nodata. For scale $a=127.5$ and exponent $\gamma=\tfrac12$, the released transform is

Squaring during dequantization makes representable values denser near zero and coarser near magnitude one. The first positive level is approximately $(1/127.5)^2=6.15\times10^{-5}$; the spacing near one is roughly $2/127.5\approx0.0157$. This is useful when many LayerNorm coordinates are small, but it is a model-specific numerical contract, not a generic vector-database int8 setting.

Ai2 explicitly quantizes the emitted LayerNorm vector without L2-normalizing it because downstream classifiers use magnitude. The report gives a median cosine similarity of 0.99997 between float and dequantized vectors. That measures angular preservation, not equality of every classifier output or change score; validate the actual downstream task.⁵

Storage layout makes the 64-dimensional prefix operational

A raw int8 chunk is $64\times256\times256=4{,}194{,}304$ bytes, or 4 MiB. The report measures 3.04 MB transferred for the first 64 dimensions and 6.05 MB for all 128 dimensions. Chunking the dimension axis at 64 is what turns Matryoshka training into an I/O saving: a 64-dimensional read fetches one chunk rather than two.

The reverse is also true. A one-pixel lookup still downloads the containing compressed chunk, roughly 3 MB, so this store is optimized for areas rather than random point traffic. Raw component arithmetic alone is not enough; chunk geometry determines bytes on the wire. The full 128-dimensional global product is approximately 150 TB per year.⁵

Searchlight fixes seams by changing the execution graph

Ordinary crop inference gives edge pixels less and directionally biased context, so adjacent crops can create visible boundaries. The released global product uses Searchlight: a large spatial domain is processed with neighborhood attention so every interior output sees the same centered $16\times16$ spatial window. For $P$ spatial positions and fixed neighborhood size $K=256$, the spatial score term grows as $O(PK)$ instead of $O(P^2)$.⁸

This is not bit-identical to the training graph. Training uses full attention on finite windows; Searchlight uses neighborhood attention on larger domains. Ai2 reports PASTIS validation mIoU of 60.4 with Searchlight versus 59.8 with tiled inference. At the remaining 20.48 km window boundaries, the production pipeline adds a 48-pixel margin and discards it after inference; the report says this adds 9.6% more computed pixels and improves its measured cross-edge cosine statistic from 0.960 to 0.994.⁵

Those values demonstrate one vendor-tested configuration, not a universal guarantee. A local deployment with different crop sizes, margins, temporal coverage or modality availability needs its own seam and downstream-quality tests.

Production boundaries

1. Pin both artifacts. The checkpoint and code are separate repositories. Record the Hugging Face revision, source commit, preprocessing, band order, patch size and output prefix.
2. Do not call every width Matryoshka. Only 128 and the first 64 coordinates receive explicit losses in the released configuration.
3. Preserve magnitude when reproducing the global product. Generic L2 normalization changes the representation used by the documented linear probes and signed-power quantizer.
4. Match time and modality coverage. The global product uses annual inputs with monthly least-cloudy imagery from all three satellite sources. A shorter window or missing Landsat path is a different input distribution.
5. Treat Searchlight as a serving variant. Test seam metrics and task accuracy; its local attention mask is not the same operation as unrestricted full attention.
6. Read Zarr by area. Random single-pixel calls amplify I/O because chunk decompression and transfer are the minimum unit.
7. Audit the current loader path. At the pinned source commit, ModelID lists models only through v1.2, so the convenience load_model_from_id cannot select v1.3. The generic local-path loader can build the downloaded config.json and weights.pth; pin hashes and treat PyTorch pickle weights as trusted-code artifacts.⁹
8. Check the license for the deployment. The OlmoEarth Artifact License restricts military, defense-related and extractive-industry uses. This is an engineering warning, not legal advice.⁵

OlmoEarth v1.3's main contribution is not simply “a smaller embedding.” It separates four budgets: encoder resolution, output resolution, vector width and storage transfer width. Learned location queries recover 10 m outputs from a 20 m ViT grid; detached relational distillation compresses 768 dimensions to trained 128/64 representations; and the Zarr layout makes that 64-dimensional prefix cheaper to move. Searchlight then addresses a deployment artifact that the checkpoint alone cannot solve.

For general vector-size and quantization arithmetic, continue with the RAG retrieval math guide. For a text-retrieval architecture where different-size query and document encoders share a space, see pplx-embed-v2.

Primary sources