Stack of news articles
Entity recognition in news articles and free text
Context
News about extreme weather events are important to evaluate weather forecasts in retrospective and to correlate weather events with their consequences such as floodings or damage occurred. Moreover, it is difficult to quantify the severity of an extreme weather event. Weather forecasting services, such as the Deutscher Wetterdienst (DWD), therefore analyze press articles and releases to unveil insights on the date, location and severity of the weather event.
Challenges
If the primary sources of weather related articles are defined, the articles need to be retrieved from the different publishers. Press agencies such as dpa or Reuters deliver xml feeds that can easily integrated in the data pipeline. However, other publishers such as regional newspapers do not deliver comparable services, so that the articles need to be scraped from different websites. Depending on the scraping method, the formats can vary from pdf to txt.
When the data is preprocessed in a way that makes the articles readable for an algorithm, the challenge remains how to detect, classify and evaluate the different entities such as location, damages or type of weather event.
Potential solution approaches
Depending on the entities to be extracted and their diversity of inputs, different technical approaches can be chosen. In case of quite uniform formatted entities, such as date or time, regular expressions can be programmed in order to match with common data types. For date, this could be dd/mm/yyyy or similar.
For more complicated entities, a dictionary of synonyms and ontologies can be developed for classification and mapping of text to entities and topics. Topic modelling approaches such as Latent Dirichlet allocation (LDA) are modelled for measuring similarity between text components. Further advanced approaches, which can lead to more promising results, BERT or domain specific word embeddings (such as BioBERT for biomedical language) or supervised learning approaches based on labelled data might be chosen.