Supervised learning for image segmentation typically requires spatially aligned image and label sets.
When images and labels originate from different sources, the pairing may be misaligned, which can significantly deteriorate the performance of the learned models.
This is especially common in remote sensing, when aerial or satellite images are co-registered with labels from another source (e.g., OpenStreetMap).
In this work, we propose a novel approach for training on misaligned labels, where we simultaneously learn the label alignment.
Our align and segment (AnS) approach builds on the spatial transformer module to transform the misaligned labels using an affine transformation to provide a better learning target for a canonical semantic segmentation network.
We prevent shortcut learning of misaligned labels in these semantic segmentation networks through a self-supervised regularization loss and show that it is complementary to data augmentation, especially for systematically misaligned training data.
A decisive characteristic of our AnS approach is that it learns without requiring any ``golden'' labels.
We experimentally show on both synthetic and real-world data from different cities that our approach enables high-quality building segmentation and precise label-image alignment at the same time.
5 min read
Motivation
Assume we want to segment buildings and classify their attributes, such as roof material and roof geometry.
Normally, we need to manually annotate buildings and their attributes.
To avoid tedious labelling, we could use existing building footprint data, for example, from Google Open Buildings and OpenStreetMap buildings (see Fig. 1).
The problem is that these building footprints may not align with the imagery of interest, as different imagery was used to create them.
The two examples from Fig. 1 show Google satellite imagery in the background with Google Open Buildings on the top and OpenStreetMap on the bottom.
The first data source is likely misaligned because they were created based on imagery from a different timestamp.
The OpenStreetMap buildings were likely created from different imagery sources.
To make full use of existing building data in downstream applications,
we need methods that can learn from misaligned labels and especially from systematically misaligned labels.
Fig. 1: Misaligned buildings from Google and OpenStreetMap.
Overlaid on top of Google satellite imagery
Methodology
To solve this problem, we introduce an approach called Aligne and Segment (AnS) inspired by the Spatial Transformer Network (Jaderberg et al., 2015).
This spatial transformer network (Note, it is not about the attention mechanism!) is a building block originally proposed for convolutional neural networks that predicts a spatial transformation that can be applied to transform an image input (see Fig. 2).
Note: Spatial Transformer Network is not based on attention.
We use a similar idea to align labels instead of the input.
The challenge is that any segmentation network with an encoder and decoder, for example, using U-Net, when trained with misaligned labels, the network potentially learns to predict the misaligned labels.
This issue is even more dominant if the labels are systematically misaligned, as it is often the case with building footprint data.
To overcome this issue, we propose to train segmentation networks (SNet) together with a transformation network (TNet) (See Fig. 3).
The TNet takes the output mask from any segmentation network and the misaligned labels as input.
It then predicts an affine transformation which is used to correct the misaligned labels.
(The TNet could be implemented to predict any kind of transformation, but in this work, we only use affine transformations.)
Fig. 3: Align and Segment (AnS) method.
These corrected labels are used to supervise the segmentation network with a cross-entropy loss.
We jointly train both networks, which allows the SNet to learn from the corrected labels.
However, there is still no guarantee that the SNet will learn the aligned labels, especially in the case of systematically misaligned labels.
Therefore, we regularize the learning process by freezing a pretrained encoder (DINOv3) and by using a consistency loss.
With this self-supervised regularization loss, we train the TNet without using the output of the SNet.
Instead, we sample random transformations Aθ2 which are applied to the misaligned labels to create new misaligned label masks (See Fig. 4).
The TNet is trained to predict this sampled transformation by optimizing the consistency loss.
Fig. 4: Illustration of our consistency loss.
Note: We did not use any golden labels in our loss functions.
Results
We evaluated our method on several synthetic and real-world datasets covering multiple cities.
A subset of results is shown in Tab. 1, presenting segmentation and alignment accuracies on two synthetic datasets and a real dataset comprising 41 cities.
The two synthetic datasets consist of synthetically misaligned labels with random and systematic misalignments.
In the latter case, all buildings were misaligned in a single direction.
The results on real-world data show performance when training on real misaligned labels from OpenStreetMap evaluated with manually annotated golden labels.
We compared our unsupervised AnS method against two supervised methods and three unsupervised methods.
Our method outperformed all unsupervised methods.
Tab. 1: Results on two synthetic datasets and one real-world dataset.
Predictions
Fig. 5 depicts example predictions.
The red mask shows the original OpenStreetMap labels.
Blue and green masks are segmentation and corrected masks from our method.
The segmentation boundaries are not very sharp because we did not use any edge-aware loss functions.
However, the corrected labels preserve the building geometry. We can clearly see how far the corrected labels have moved.
Fig. 5: Example predictions.
5 min video
Video Presentation
BibTeX
@inproceedings{Guthula_align2026,
title={Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels},
author={Venkanna Babu Guthula and Oswin Krause and Dimitri Gominski and Hui Zhang and Johan Mottelson and Ankit Kariryaa and Nico Lang and Christian Igel},
booktitle={European Conference on Computer Vision (ECCV)},
year={2026}
}
