Publications about the project
Multilingual Disaster-Relevance TwHIN-BERT
This is a multilingual Twhin-BERT-base model fine-tuned for relevance classification of social media posts in natural disaster response. The model was developed using a dataset of 4,574 manually labelled posts from five use case scenarios: the 2020 California wildfires, 2021 Ahr Valley floods, 2023 Chile wildfires, 2023 Turkey earthquake and 2023 Emilia-Romagna floods. 3,659 posts were used for model training and validation, and 915 were used for testing.
🏷️ Labels
The model classifies short texts using one of the following labels:
- Related and relevant: A post that is related to the respective natural disaster and relevant for emergency responders. It contains useful information for supporting disaster management (e.g., posts about destructions, in-situ information, critical infrastructure, affected individuals, affected areas, requests for help, caution or advice).
- Related but not relevant: A post that refers to the respective disaster but does not contain helpful or valuable information for supporting disaster management (e.g., declarations of solidarity, volunteering initiatives, appeals for donations, political or religious statements, bot-generated content, comparisons to past events, shared news articles).
- Not related: A post that has no relation to the disaster event in question.
⚙️ Example pipeline
Please make sure you have transformers==4.57.6 with PyTorch and zenodo-get==3.0.3 installed. The model can then be used as follows:
import zipfile
from zenodo_get import download
from transformers import pipeline
# Download files
download("10.5281/zenodo.21456026", output_dir="./")
# Extract the zip
with zipfile.ZipFile('multilingual-disaster-relevance-twhin-bert.zip', 'r') as zip_ref:
zip_ref.extractall('multilingual-disaster-relevance-twhin-bert')
# Run classifier
classifier = pipeline('text-classification', model='./multilingual-disaster-relevance-twhin-bert')
classifier('I can see fire and smoke from the nearby fire!')
Output:
[{'label': 'Related and relevant', 'score': 0.997657299041748}]
💡 Performance
The model achieved strong performance across our test data:
| Metric | Value |
|---|---|
| Macro F1 | 0.779 |
| Accuracy | 0.802 |
| ROC-AUC | 0.928 |
📃 Reference
If you want to use this model in your research, please cite it as follows:
@article{Hanny.2025c,
title = {A Multimodal {{GeoAI}} Approach to Combining Text with Spatiotemporal Features for Enhanced Relevance Classification of Social Media Posts in Disaster Response},
author = {Hanny, David and Schmidt, Sebastian and Gandhi, Shaily and Granitzer, Michael and Resch, Bernd},
year = {2025},
journal = {Big Earth Data},
volume = {0},
number = {0},
pages = {1--45},
publisher = {Taylor \& Francis},
issn = {2096-4471},
doi = {10.1080/20964471.2025.2572140},
}
Acknowledgements
This work has received funding from the European Commission - European Union under HORIZON EUROPE (HORIZON Research and Innovation Actions) under grant agreement 101093003 (HORIZON-CL4-2022-DATA-01-01).
Towards multimodal geospatial reasoning: a foundation model approach for disaster detection from social media, news, and weather data
The timely detection of disasters is essential for effective emergency response. Traditional satellite-based monitoring provides accurate hazard observations but suffers from acquisition delays and weather-dependent imaging conditions. Therefore, recent research increasingly uses rapidly available digital data such as social media, news, and weather observations. However, most approaches analyse these sources in isolation and lack standardised evaluation. We address this gap using a grid-based framework that quantifies disaster detection accuracy relative to satellite-derived reference data. Within this framework, we introduce a multimodal geospatial reasoning method that employs generative Language Models (LMs) to interpret heterogeneous information. The method integrates Bluesky social media posts, GDELT news headlines, and weather observations through structured prompts and relevance-based data retrieval, framing detection as a binary classification problem on an H3 grid. Across two case studies on the 2024 Central Europe floods and the 2025 Southern California wildfires, LM-based detection outperformed traditional hotspot and anomaly detection while requiring only ten content items per prediction. Results were robust across prompt variants, and Automatic Prompt Optimisation (APO) provided only moderate gains. Overall, this research offers the first systematic evaluation of Bluesky, GDELT, and weather data for disaster detection and shows that Foundation Models (FMs) can act as efficient zero-shot or few-shot detectors of natural-hazard-induced disasters.
Augmented Images Fire Scenarios
RGB dataset depicting 134 real drone high resolution images on the Montiferru region provided by "Regione Sardegna - Direzione Generale della Protezione Civile", and 134 augmented images where wildfire where added to the real images using AI methods.
DJI_202411051053_original.zip contains the original 134 jpg images with metadata from the drone.
DJI_202411051053_synthetic_fire.zip contains the augmented 134 jpg images with the same metadata
Augmented images Flood Scenario
RGB dataset depicting 218 real drone high resolution images on the town of Altenahr after the floods of 2021 provided by DLR, and 218 augmented images where floods, debris and trapped persons where added to the real images using AI methods.
Floods_original.zip contains the original 218 jpg images with metadata from the drone.
Floods_augmented.zip contains the augmented 218 jpg images with the same metadata.
Flood and Fire Real Description Generated Synthetic Dataset
Synthetic RGB image dataset depicting 550 images of forest fires and 500 images of floods generated with descriptions of real images with AI methods.
This data can be used to train/evaluate fire and floods detection models on RGB images.
FireDescription.txt contains the 11 descriptions from real images that were used as a prompt for the fire images
FloodsDescription.txt contains the 10 descriptions from real images that were usesd as a prompt for the flood images
DescriptionDataset/
├── FireJPG/ (fire1__00001_.jpg → fire11__00050_.jpg)
├── FloodsJPG/ (flood1__00001_.jpg → flood10__00050_.jpg)
├── FireDescription.txt
└── FloodsDescription.txt
Unsupervised Multimodal Graph-based Model for Geo-social Analysis
The systematic analysis of user-generated social media content, especially when enriched with geospatial context, plays a vital role in domains such as disaster management and public opinion monitoring. Although multimodal approaches have made significant progress, most existing models remain fragmented, processing each modality separately rather than integrating them into a unified end-to-end model. To address this, we propose an unsupervised, multimodal graph-based methodology that jointly embeds semantic and geographic information into a shared representation space. The proposed methodology comprises two architectural paradigms: a mono graph (MonoGrah) model that jointly encodes both modalities, and a multi graph (MultiGraph) model that separately models semantic and geographic relationships and subsequently integrates them through multi-head attention mechanisms. A composite loss, combining contrastive, coherence, and alignment objectives, guides the learning process to produce semantically coherent and spatially compact clusters. Experiments on four real-world disaster datasets demonstrate that our models consistently outperformexisting baselines in topic quality, spatial coherence, and interpretability. Inherently domain-independent, the framework can be readily extended to diverse forms of multimodal data and a wide range of downstream analysis tasks.
Enhancing satellite-based emergency mapping: Identifying wildfires through geo-social media analysis
When a disaster emerges, timely acquisition of information is crucial for a rapid situation assessment. Although automation in the standard satellite-based emergency mapping workflow has been advanced, delays still occur at crucial steps. In order to speed up the provision of satellite-based crisis products to emergency managers, this paper proposes a geo-social media-based approach that detects disaster events based on the spatio-temporal analysis of georeferenced, disaster-related Tweets. The proposed methodology is validated on the basis of two use cases: wildfires in Chile and British Columbia. The results show the general ability of Twitter to forecast events several days in advance, at least for the Chile use case. However, there are large spatial differences, as there is a correlation between population density and the reliability of Twitter data. Consequently, only few meaningful alerts could be generated for British Columbia, an area with very low population numbers.
Multimodal GeoAI: An integrated spatio-temporal topic-sentiment model for the analysis of geo-social media posts for disaster management
A multimodal GeoAI approach to combining text with spatiotemporal features for enhanced relevance classification of social media posts in disaster response
Geo-referenced social media data supports disaster management by offering real-time insights through user-generated content. To identify critical information amid high volumes of noise, classifying the relevance of posts is essential. Most existing methods primarily use textual features, neglecting spatial and temporal context despite its importance in determining relevance. This study proposes a multimodal approach that integrates text with spatiotemporal features for relevance classification of geo-referenced social media posts. We evaluate our method on 4,574 manually labelled posts from five disasters: the 2020 California wildfires, 2021 Ahr Valley floods, 2023 Chile wildfires, 2023 Turkey earthquake and 2023 Emilia-Romagna floods. Labels were assigned based on text, geographic location and time. Our spatiotemporal features include proximity to disaster impact sites, local co-occurrences with disaster-related posts, event type and geographic context. When utilised on their own, they achieved a macro F1 score of 0.713 with a random forest classifier. A fine-tuned TwHIN-BERT-base model using only text scored 0.779. For multimodal classification, we tested feature concatenation, in-context learning, stacking and partial stacking. Partial stacking produced the highest macro F1 score (0.814). Our multilingual, context-aware classification approach lays the groundwork for more integrated GeoAI applications in disaster management, the social sciences and beyond.
Clustering-Based Joint Topic-Sentiment Modeling of Social Media Data: A Neural Networks Approach
With the vast amount of social media posts available online, topic modeling and sentiment analysis have become central methods to better understand and analyze online behavior and opinion. However, semantic and sentiment analysis have rarely been combined for joint topic-sentiment modeling which yields semantic topics associated with sentiments. Recent breakthroughs in natural language processing have also not been leveraged for joint topic-sentiment modeling so far. Inspired by these advancements, this paper presents a novel framework for joint topic-sentiment modeling of short texts based on pre-trained language models and a clustering approach. The method leverages techniques from dimensionality reduction and clustering for which multiple algorithms were considered. All configurations were experimentally compared against existing joint topic-sentiment models and an independent sequential baseline. Our framework produced clusters with semantic topic quality scores of up to 0.23 while the best score among the previous approaches was 0.12. The sentiment classification accuracy increased from 0.35 to 0.72 and the uniformity of sentiments within the clusters reached up to 0.9 in contrast to the baseline of 0.56. The presented approach can benefit various research areas such as disaster management where sentiments associated with topics can provide practical useful information.