Blog - Computer Vision

In a fast paced and always changing global economy the ability to classify crop fields via remote sensing at the end of a growth cycle does not provide the much needed immediate insight required by decision makers. To address this problem we developed a model that allows continuous classification of crop fields at any point in time and improves predictions as more data becomes available. In practice, we developed a single model capable of delivering predictions about which crops are growing at any point in time based on satellite data. The data available at the time of inference could be a few images at the beginning of the year or a full time series of images from a complete growing cycle. This exceeds the capabilities of current deep learning solutions that either only offer predictions at the end of the growing cycle or have to use multiple models that are specialized to return results from pre-specified points in time. This article details the key changes we employed to the model described in a previous blog post “Classification of Crop fields through Satellite Image Time Series” that enlarges its functionality and performance. The results presented in this article are based on a research paper recently published by dida. For more detailed information about this topic and other experiments on this model please check out the original manuscript: “Early Crop Classification via Multi-Modal Satellite Data Fusion and Temporal Attention” .

Machine Learning has been solving complex problems for decades. Just think about how Computer Vision methods can reliably predict life-threatening diseases, self-driving cars are on their way to revolutionize traffic safety, or automatic translation gives us the ability to talk to just about anyone on the planet. The power of Machine Learning has been embraced by many branches of industry and science. There are some areas however where the potential of Machine Learning is harder to see and also less utilized. One of these is environmental protection. Protecting the natural environment is one of the biggest challenges our generation is facing, with pressing issues such as climate change, plastic pollution or resource depletion. Let us now look at how Machine Learning has been and can be used as a tool in environmental protection.

Probably the most common form of problem we tackle with machine learning is classification, that is taking new data points and putting them into one of a number of fixed sets or classes. But what if we don’t necessarily know all the classes when we train the model? A good example of this is face recognition where we want a system that can store faces and then identify if any new images it sees contain that face. Obviously, we can’t retrain the model every time we add someone new to the database so we need a better solution. One way to solve this problem is metric learning. In metric learning, our goal is to learn a metric or distance measure between different data points. If we train our model correctly then this distance measure will put examples of the same class close together and different classes further apart.

Recommendation systems are everywhere. We use them to buy clothes, find restaurants and choose which TV show to watch. In this blog post, I will give an overview of the underlying basic concepts, common use cases and discuss some limitations. This is the first of a series of articles about recommendation engines. Stay tuned for the follow-ups, where we will explore some of the mentioned concepts in much more detail! Already in 2010, 60 % of watch time on Youtube came from recommendations [1] and personalized recommendations are said to increase conversion rates on e-commerce sites by up to 5 times [2]. It is safe to say that if customers are presented with a nice pre-selection of products they will be less overwhelmed, more likely to consume something and have an overall better experience on the website. But how do recommendation engines work? Let's dive right in.

An estimated number of 906 Earth observation satellites are currently in orbit, providing science and industry with many terabytes of data every day. The satellites operate with both radar as well as optical sensors and cover different spectral ranges with varying spectral, spatial, and temporal resolutions. Due to this broad spectrum of geospatial data, it is possible to find new applications for remote sensing methods in many industrial and governmental institutions. On our website, you can find some projects in which we have successfully used satellite data and possible use cases of remote sensing methods for various industries . Well-known satellite systems and programs include Sentinel-1 (radar) and Sentinel-2 (optical) from ESA, Landsat (optical) from NASA, TerraSAR-X and TanDEM-X (both radar) from DLR, and PlanetScope (optical) from Planet. There are basically two types of geospatial data: raster data and vector data . Raster data Raster data are a grid of regularly spaced pixels, where each pixel is associated with a geographic location, and are represented as a matrix. The pixel values depend on the type of information that is stored, e.g., brightness values for digital images or temperature values for thermal images. The size of the pixels also determines the spatial resolution of the raster. Geospatial raster data are thus used to represent satellite imagery. Raster images usually contain several bands or channels, e.g. a red, green, and blue channel. In satellite data, there are also often infrared and/or ultraviolet bands. Vector data Vector data represent geographic features on the earth's surface, such as cities, country borders, roads, bodies of water, property rights, etc.. Such features are represented by one or more connected vertices, where a vertex defines a position in space by x-, y- and z-values. A single vertex is a point, multiple connected vertices are a line, and multiple (>3) connected and closed vertices are called polygons. The x-, y-, and z-values are always related to the corresponding coordinate reference system (CRS) that is stored in vector files as meta information. The most common file formats for vector data are GeoJSON, KML, and SHAPEFILE. In order to process and analyze these data, various tools are required. In the following, I will present the tools we at dida have had the best experience with and which are regularly used in our remote sensing projects. I present the tools one by one, grouped into the following sections: Requesting satellite data EOBrowser Sentinelsat Sentinelhub Processing raster data Rasterio Pyproj SNAP pyroSAR Rioxarray Processing vector data Shapely Python-geojson Geojson.io Geopandas Fiona Providing geospatial data QGIS GeoServer Leafmap Processing meteorological satellite data Wetterdienst Wradlib

One sees an image and easily tells what is happening in it because it is humans’ basic ability to grasp and describe details about an image by just having a glance. Can machines recognize different objects and their relationships in an image and describe them in a natural language just like humans do? This is the problem image captioning tries to solve. Image captioning is all about describing images in natural language (such as English), combining two core topics of artificial intelligence: computer vision and natural language processing . Image captioning is an incredible application of deep learning that evolved considerably in recent years. This article will provide a high-level overview of image captioning architecture and explore the attention mechanism – the most common approach proposed to solve this problem. The most recent image captioning works have shown benefits in using a transformer-based approach, which is based solely on attention and learns the relationships between elements of a sequence without using recurrent or convolutional layers. We will not be considering transformer-based architectures here, instead we will focus only on the attention-based approach.

'Hotdog' or 'not hotdog'? That could be the question — at least when performing an image classification task. To be able to address this or a similarly important question by means of a machine learning model, we first need to come up with a labeled dataset for training. That is, we sometimes have to manually look at hundreds or even thousands of images that do or do not contain hotdogs, and decide if they do. One way to do that would be to open up one image at a time and keep track of image classes in another file, e.g., a spreadsheet. However, such a heavy-handed approach sounds rather tedious and is likely prone to fat-fingering errors. Wouldn't it be great if there was a streamlined solution that makes this labeling process more efficient, even fun? That is exactly right and also what we set out to do in this article: Create a simple annotation tool to easily assign class labels to a set of images.

By 2021, there is little doubt that Machine Learning (ML) brings great potential to today’s world. In a study by Bitkom , 30% of companies in Germany state that they have planned or least discussed attempts to leverage the value of ML. But while the companies’ willingness to invest in ML is rising, Accenture estimates that 80% – 85% of these projects remain a proof of concept and are not brought into production. Therefore at dida, we made it our core mission to bridge that gap between proof of concept and production software, which we achieve by applying data-centric techniques, among other things. In this article, we will see why many ML Projects do not make it into production, introduce the concepts of model- and data-centric ML, and give examples how we at dida improve projects by applying data-centric techniques.

The field of remote sensing has been benefiting from the advancements made in Machine Learning (ML). In this article we explore a state of the art model architecture, the Transformer , initially developed for Natural Language Processing (NLP) problems, which is now widely used with many forms of sequential data. Following the paper by Garnot et al. , we utilize an altered version of this architecture to classify crop fields from time series of satellite images . With this, we achieve better results than traditional methods (e. g. random forests) and with less resources than recurrent networks.

Did you ever need to combine data about an object from two different sources, say, images and text? We are often facing such challenges during our work at dida. Here we present an example from the realm of technical drawings. Such drawings are used in many fields for specialists to share information. They consist of drawings that follow very specific guidelines so that every specialist can understand what is depicted on them. Normally, technical drawings are given in formats that allow indexing, such as svg, html, dwg, dwf, etc. but many, especially older ones, only exist in image format (jpeg, png, bmp, etc.), for example from book scans. This kind of drawings is hard to access automatically which makes its use hard and time consuming. In this regard, automatic detection tools could be used to facilitate the search. In this blogpost, we will demonstrate how both traditional and deep-learning based computer vision techniques can be applied for information extraction from exploded-view drawings. We assume that such a drawing is given together with some textual information for each object on the drawing. The objects can be identified by numbers connected to them. Here is a rather simple example of such a drawing: An electric drill machine. There are three key components on each drawing: The numbers, the objects and the auxiliary lines. The auxiliary lines are used to connect the objects to the numbers. The task at hand will be to find all objects of a certain kind / class over a large number of drawings , e.g. the socket with number 653 in the image above appears in several drawings and even in drawings from other manufacturers. This is a typical classification task, but with a caveat: Since there is additional information for each object accessible through the numbers, we need to assign each number on the image to the corresponding object first. Next we describe this auxiliary task can be solved by using traditional computer vision techniques.

Since its first introduction in late 2017, the Transformer has quickly become the state of the art architecture in the field of natural language processing (NLP). Recently, researchers started to apply the underlying ideas to the field of computer vision and the results suggest that the resulting Visual Transformers are outperforming their CNN-based predecessors in terms of both speed and accuracy. In this blogpost, we will have a closer look at how to apply transformers to computer vision tasks and what it means to tokenize an image.

Digitization and the internet in particular have not only provided us with a seemingly inexhaustible source of textual data, but also of images. In the case of texts, this treasure has been lifted in the form of task-agnostic pretraining by language models such as BERT or GPT-3. Contrastive Language-Image Pretraining (short: CLIP) now does a similar thing with images, or rather: the combination of images and texts. In this blog article I will give a rough non-technical outline of how CLIP works, and I will also show how you can try CLIP out yourself! If you are more technically minded and care about the details, then I recommend reading the original publication , which I think is well written and comprehensible.

In this blog article, I want to briefly describe the process of migrating labels from Planet Scope to Sentinel-2 images. I will explain the process step-by-step and provide the necessary lines of code. The code can easily be adapted to migrate labels between other satellites than in our example case here.

Automating processes using machine learning (ML) algorithms can increase the efficiency of a system beyond human capacity and thus becomes more and more popular in many industries. But between an idea and a well-defined project there are several points that need to be considered in order to properly assess the economic potential and technical complexity of the project. Especially for companies like dida that offer custom workflow automation software, a well-prepared project helps to quickly assess the feasibility and the overall technical complexity of the project goals -which, in turn, makes it possible to deliver software that fulfills the client's requirements. In this article, we discuss which topics should be considered in advance and why the questions we ask are important to start a successful ML software project.

Creating a high quality data set is a crucial part of any machine learning project . In practice, this often takes longer than the actual training and hyperparameter optimization. Thus choosing an appropriate tool for labeling is essential. Here we will have a closer look at some of the best image labeling tools for Computer Vision tasks: labelme labelImg CVAT hasty.ai Labelbox We will install and configure the tools and illustrate their capabilities by applying them to label real images for an object detection task. We will proceed by looking at the above tools one by one. Our collection of computer vision content also clearly shows how central the use of such labeling tools is for us as machine learning specialists.

This article provides an overview of our tech stack at dida. Of course we adapt the tools we use to what is required by a given project, but the ones we have listed here are our go-to tools whenever we are free to choose. I will first describe the tools that shape our software development process, and then our favourite Python libraries and software tools for machine and deep learning.

Unsurprisingly, a major requirement that makes mining endeavours successful is the right location - one where the enterprise knows with confidence that the soil bears high grade minerals of interest. Finding such a site, however, poses a significant challenge. Conventionally, when mining enterprises pursue greenfield exploration, field studies and drillings are conducted. As these are very expensive, they should only serve as a last assurance after potentially interesting regions have been identified. This is where Remote Sensing comes into play. In this article, we will have a look at the possibilities that spaceborne imaging provides for greenfield exploration. Let’s have a satellite scout promising spots.

Here I will a give a brief introduction to using Docker containers for software deployment. I assume that we want to deploy a Python application myapp that has a web API.

In this article, we will introduce the basic ideas behind graph neural networks (GNNs) through an analogy with convolutional neural networks (CNNs), which are very well known due to their prevalence in the field of computer vision. In fact, we'll see that convolutional nets are an example of GNNs, albeit one where the underlying graph is very simple, perhaps even boring. Once we see how to think of a convolutional net through this lens, it won't be hard to replace that boring graph with more interesting ones, and we'll arrive naturally at the general concept of GNN. After that, we will survey some applications of GNNs, including our use here at dida. But let's start with the basics.

Working with satellite data , one needs to understand and possibly convert the coordinates the data is given in. Sometimes, especially if released by official bodies, satellite data is provided in MGRS tiles , which are derived from the UTM coordinate system. For example, this is true for Sentinel-2 tiles. I want to answer the following three questions in this post, using the Python libraries mgrs and pyproj : What is the difference between MGRS and UTM? To which MGRS tile does a certain point referenced in latitude and longitude degrees belong to? How can I express a MGRS tile in Lat/Lon coordinates? Before we answer these questions, let's first look into what MGRS is.

Urbanisation on a global scale is happening at an ever increasing rate. In the year 2008, more than 50% of the worlds population lived in cities and it is predicted that by 2050 about 64% of the developing world and 86% of the developed world will be urbanised. This trend puts significant stress on infrastructure planning. Providing everything from sanitation, water systems and transportation to adequate housing for more than 1.1 billion new urbanites over the next 10 years will be an extraordinary challenge. In a research project for the European Space Agency's program "AI for social impact", dida assessed the use of state-of-the-art computer vision methods for monitoring urban development over time of three rapidly growing cities in west Africa: Lagos, Accra and Luanda. The population of these cities are expected to grow by 30-55% in size by the end of 2030 which means that in-situ data collection about how these cities develop is almost impossible given the available resources. Instead, we came up with a concept that would rely solely on satellite images and machine learning.

Throughout the globe, rain forests and other natural landscapes are endangered by illegal mining, which transforms areas formerly rich in flora and fauna into wasteland. In order for local governments to take countermeasures, they first need to know about the locations of illegal mines. In countries covered by vast areas of impenetrable rain forest, such as Brazil or Congo, obtaining this information is a difficult problem. In this blog post I describe an approach to detect illegal mines based on deep learning and remote sensing, that we have developed to support the conservation efforts of governments and NGOs. In particular, we use a U-Net for semantic segmentation , a branch of computer vision. As part of the project of automatic detection of illegal mines , we were also joined by scientists from the Institute of Mineral Resources Engineering of the RWTH Aachen University, who contributed their mining-specific expertise. The project was funded by the European Space Agency .

In the previous post we introduced the basics of reinforcement learning (RL) and the type of problem it can be applied to. The discussed setting was limited in the sense that we were dealing with a single agent acting in a stationary environment. Now we will take it one step further and discuss Multi-Agent Reinforcement Learning ( MARL ). Here we deal with multiple explicitly modeled agents in the same environment, hence every agent is part of the environment as it is perceived by all others. Since all agents learn over time and start to behave differently, the assumption of a stationary environment is violated.

In this blog post I will describe a number of pretraining tasks one can use either separately or in combination to get good “starting” weights before you train a model on your actual labelled dataset. Typically, remote sensing tasks come under the umbrella of semantic segmentation , so all the pretraining techniques described here are for tasks that output a prediction for each pixel and use a U-Net as the architecture.

Machine Learning concerns itself with solving complicated tasks by having a software learn the rules of a process from data. One can try to discover structure in an unknown data set (unsupervised learning) or one can try to learn a mathematical function between related quantities (supervised learning). But what if you wanted the algorithm to learn to react to its environment and to behave in a particular way? No worries, machine learning’s got you covered! This branch of Machine Learning (ML) is called Reinforcement Learning (RL). In this post we will give a quick introduction to the general framework and look at a few basic solution attempts in more detail. Finally, we will give a visual example of RL at work and discuss further approaches. In the second part of the blog post we will discuss Multi-Agent Reinforcement Learning (MARL).

It seems to be a common mistake to believe that machine learning is usually an unsupervised task : you have data (without pre-existing labels) that you train e.g. a neural network on for tasks like classification or image segmentation. The truth is that most models in machine learning are supervised , that is, they rely on labeled training data . But labeling often takes a lot of time and can be very tedious. In this blog post I want to find out if I am able to perform the same classification task once with labels, once without. For this task I will use the famous MNIST data set , which contains 60,000 training and 10,000 validation images of handwritten digits, all of them labeled. Every image consists of 28x28 greyscale pixels and contains only one digit, located in the center of the image. To make things easier, I use the CSV version of the data set.

Self-driving cars still have difficulties in detecting objects in front of them with sufficient reliability. In general, though, the performance of state-of-the-art object detection models is already very impressive - and they are not too difficult to apply. Here I will walk you through streaming a YouTube video into Python and then applying a pre-trained PyTorch model to it in order to detect objects. We'll be applying a model pre-trained on the object detection dataset COCO . (In reality, the model would of course be fine tuned to the task at hand.)

In my previous blog post I have started to explain how Bayesian Linear Regression works. So far, I have introduced Bayes' Theorem , the Maximum Likelihood Estimator (MLE) and Maximum A-Posteriori (MAP) . Now we will delve into the mathematical depths of the details behind Bayesian Linear Regression.

Bayesian regression methods are very powerful, as they not only provide us with point estimates of regression parameters, but rather deliver an entire distribution over these parameters. This can be understood as not only learning one model, but an entire family of models and giving them different weights according to their likelihood of being correct. As this weight distribution depends on the observed data, Bayesian methods can give us an uncertainty quantification of our predictions representing what the model was able to learn from the data. The uncertainty measure could be e.g. the standard deviation of the predictions of all the models, something that point estimators will not provide by default. Knowing what the model doesn't know helps to make AI more explainable. To clarify the basic idea of Bayesian regression, we will stick to discussing Bayesian Linear Regression (BLR). BLR is the Bayesian approach to linear regression analysis. We will start with an example to motivate the method. To make things clearer, we will then introduce a couple of non-Bayesian methods that the reader might already be familiar with and discuss how they relate to Bayesian regression. In the following I assume that you have elementary knowledge of linear algebra and stochastics. Let's get started!

This is the last post in the three part series covering machine learning approaches for time series and sequence modeling. In the first post , the basic principles and techniques for serial sequences in artificial neural networks were shown. The second post introduced a recent convolutional approach for time series called temporal convolutional network (TCN), which shows great performance on sequence-to-sequence tasks ( Bai, 2018 ). In this post, however, I will talk about a real world application which employs a machine learning model for time series analysis. To this end, I will present a beat tracking algorithm, which is a computational method for extracting the beat positions from audio signals. The presented beat tracking system ( Davies, 2019 ) is based on the TCN architecture which captures the sequential structure of audio input.

This post is the first part of a series of posts that are linked together as they all deal with the topic of time series and sequence modeling, respectively. In order to give a comprehensive piece of content easy to grasp, the series is segmented into three parts: How to deal with time series and serial sequences? A recurrent approach. Temporal Convolutional Networks (TCNs) for sequence modeling. Beat tracking in audio files as an application of sequence modeling.

Anyone wanting to train a Machine Learning (ML) model these days has a plethora of Python frameworks to choose from. However, when it comes to distributing your trained model to something other than a Python environment, the number of options quickly drops. Luckily there is Tensorflow.js , a JavaScript (JS) subset of the popular Python framework with the same name. By converting a model such that it can be loaded by the JS framework, the inference can be done effectively in a web browser or a mobile app. The goal of this article is to show how to train a model in Python and then deploy it as a JS app which can be distributed online.

Here I’m going to walk through how we approached the problem of detecting convective clouds in satellite data including what we are looking for (and why!) and the machine learning approach we used. This post will consist of four sections: First we will introduce convective clouds and give a brief overview of the problem. In section 2 we will discuss the satellite data we are working with. In section 3 we discuss how we go about manually labelling the data, which is a particularly difficult task requiring the use of some external data. Finally, in section 4 we will give a brief overview of the neural network architecture that we use, the U-Net, and how we go about training it. You can also have a look at my talk at 2020's Applied Machine Learning Days in Lausanne, Switzerland:

Since not only the complexity of Machine Learning (ML) models but also the size of data sets continue to grow, so does the need for computer power. While most laptops today can handle a significant workload, the performance is often simply not enough for our purposes at dida. In the following article, we walk you through some of the most common bottlenecks and show how cloud services can help to speed things up.

In Machine Learning, an insufficient amount of training data often hinders the performance of classification algorithms. Experience shows that shortage of training data is rather the rule than the exception, which is why people have come up with clever data augmentation methods. In this blog post I demonstrate how you can create new images of a distribution of images with a Generative Adversarial Network ( GAN ). This can be applied as a data augmentation method for problems such as defect detection in industrial production.

Artificial intelligence (AI) and in particular computer vision promise to be valuable aids for diagnosing diseases based on medical imaging techniques . For humans, it takes years of academic and on-the-job training to e.g. perform medical diagnosis from X-ray images. As we will see, it is also quite a challenge for intelligent algorithms. At this year's KIS-RIS-PACS and DICOM convention organized by the Department of Medicine at the University of Mainz, Germany, researchers from radiology and adjacent fields gathered to discuss the state-of-the-art of AI in their field. Philipp Jackmuth from dida was the speaker of choice for this topic and here we will discuss key points of his talk.

The terms Artificial Intelligence (AI) , Machine Learning (ML) and Deep Learning (DL) are used with staggering frequency. However, it is not obvious what they are supposed to describe and how they relate to each other. This blog article is meant to elucidate some of the terminology used in these fields for the tech-enthusiastic layman, as well as to give some high-level notion of how these technologies can be used in Computer Vision (CV) .

This post presents some key learnings from our project on identifying roofs on satellite images . Our aim was to develop a planing tool for the placement of solar panels on roofs. For this purpose we set up a machine learning model that accurately partitions those images into different types of roof parts and background. We learned that the UNet model with dice loss enforced with a pixel weighting strategy outperforms cross entropy based loss functions by a significant margin in semantic segmentation of satellite images. The following idealized pipeline illustrates the functionality of the planning tool: