We predict crop types from satellite data to support modern agriculture.
| Input: | Satellite images from different points in time |
| Output: | Crop type for each individual parcel |
| Goal: | Provide timely information on which crop is grown where |
The main objective of this research project is to predict which types of crops (such as wheat, maize, etc.) are growing and where. To this end, we're working with our project partner GFZ (German Research Centre for Geosciences).
The increasing availability of remote sensing data, driven by organisations such as the European Space Agency ESA (e. g. data provided through the satellite missions Sentinel 1 and Sentinel 2), allows the application of machine learning methods in fields like agriculture.
The specific task is to employ these methods for the prediction of crop types that are being grown on a certain parcel. An additional requirement is that these predictions are to be returned as early as possible during the crop’s growth cycle, so that stakeholders can base their planning on these predictions.
As with any solution that makes use of optical satellite data, cloud coverage preventing a clear picture of fields is one of the main challenges to be overcome in this field. In addition, early crop classification also contributes to the challenge of creating a model which can distinguish crop types with as little information as possible.
Another challenge is the large amount of data to be processed for the prediction of a single field parcel: Sentinel 2 satellite data are not normal RGB images, but a time series of multispectral images, i. e. images containing more information than just visible colours.
Finally, because satellites can only capture a limited area at a time, they will fly over different areas at different time points. Therefore there are different sequences of image capturing dates for different parcels, which is a challenge in itself.
We use a deep neural network model that processes time series of images from Sentinel 2 (and Sentinel 1) to find patterns in time that identify specific crop types. A particular emphasis is given to the classification of wheat, rye, barley, rapeseed, potatoes, maize, and sugar beet as these are the most important crop types in Germany.
The network architecture employed enables the algorithm to learn which time points in the growing season can be used to distinguish between different crop types: It can for example automatically learn if a certain crop always turns green earlier than other crops.
We use a deep neural network model that processes time series of images from Sentinel 2 (and Sentinel 1). The approach to this problem is based on the fact that different species of plants reflect specific spectra at specific points during their development. A deep neural network model based on temporal attention is used to find these time points (attention being one of the mechanisms mainly responsible for the success of the famous Transformer architecture). So-called pixel set encodings provide a way to leverage the enormous amount of information contained in multi spectral image time series in a data-efficient way, similar to the work of Garnot et al.
This network architecture is capable of capturing changes of reflectance through time of individual crop types. It is further enhanced, for example by incorporating (cloud independent) radar data from Sentinel 1 in order to provide time series with stable intervals between images.
Backend: Python, PyTorch, NumPy, geopandas, rasterio, gdal, shapely, pyproj, mgrs
Infrastructure: GCloud (Training), Git, dvc, tensorboard
As with any solution that makes use of optical satellite data, cloud occlusion is one of the main challenges to be overcome. This entails the reduction of available data points in time for a given region and an imbalance of data between seasons: Warmer seasons tend to have clearer skies. Consequently, more data points are usable as the satellites are able to capture more images that are not obstructed by clouds.
Another challenge is the fact that the data sets are time series of images with multiple channels per time step, which makes the training process memory consuming and demands smart data aggregation strategies.
Finally, because satellites can only capture a limited area at a time, different locations will have different temporal availability of data, i. e. the sequences of data points in time for different areas will be of different sizes and different dates.
A selection of projects we have done
See how machine learning can be used to stop environmental destruction.
We automated the detection of certain cloud structures for Deutscher Wetterdienst (DWD).
Creative solutions enabled us to automate the process of planning solar systems.
An image segmentation algorithm that supports sustainable city planning.
