Crop Type Classification

The main objective of this project is predicting which types of crops (such as wheat, maize, etc.) are growing and where.

The European Union enforces a common agricultural policy EU CAP, where one of the main goals is to subsidize farmers. To do so, it is imperative that farmers and local authorities provide and verify, respectively, information about what crops are being grown and where.

Automatic crop type classification is thus beneficial for multiple reasons:

• Authorities responsible for subsidy payments (e.g. ministries of agriculture) are required to do supervision as part of the EU CAP. Automatic crop class prediction provides an easy and fast method for this.

• Farmers, food suppliers, and food producers can use the results to improve their strategic decisions, as they often rely on futures contracts where knowledge of the expected supply for specific crops is essential for economic planning.

• Dependent industries, such as fertilizer producers and agricultural trading companies, can tailor their offers to the individual needs of farmers, not only based on what crops are grown but also taking into account the interactions with other crops in their vicinity.

The main objective of this research project is to predict which types of crops (such as wheat, maize, etc.) are growing and where. To this end, we're working with our project partner GFZ (German Research Centre for Geosciences).

The increasing availability of remote sensing data, driven by organisations such as the European Space Agency ESA (e. g. data provided through the satellite missions Sentinel 1 and Sentinel 2), allows the application of machine learning methods in fields like agriculture.

The specific task is to employ these methods for the prediction of crop types that are being grown on a certain parcel. An additional requirement is that these predictions are to be returned as early as possible during the crop’s growth cycle, so that stakeholders can base their planning on these predictions.

As with any solution that makes use of optical satellite data, cloud coverage preventing a clear picture of fields is one of the main challenges to be overcome in this field. In addition, early crop classification also contributes to the challenge of creating a model which can distinguish crop types with as little information as possible.

Another challenge is the large amount of data to be processed for the prediction of a single field parcel: Sentinel 2 satellite data are not normal RGB images, but a time series of multispectral images, i. e. images containing more information than just visible colours.

Finally, because satellites can only capture a limited area at a time, they will fly over different areas at different time points. Therefore there are different sequences of image capturing dates for different parcels, which is a challenge in itself.

We use a deep neural network model that processes time series of images from Sentinel 2 (and Sentinel 1) to find patterns in time that identify specific crop types. A particular emphasis is given to the classification of wheat, rye, barley, rapeseed, potatoes, maize, and sugar beet as these are the most important crop types in Germany.

The network architecture employed enables the algorithm to learn which time points in the growing season can be used to distinguish between different crop types: It can for example automatically learn if a certain crop always turns green earlier than other crops.

Our Solution: Temporal Attention

We use a deep neural network model that processes time series of images from Sentinel 2 (and Sentinel 1). The approach to this problem is based on the fact that different species of plants reflect specific spectra at specific points during their development. A deep neural network model based on temporal attention is used to find these time points (attention being one of the mechanisms mainly responsible for the success of the famous Transformer architecture). So-called pixel set encodings provide a way to leverage the enormous amount of information contained in multi spectral image time series in a data-efficient way, similar to the work of Garnot et al.

This network architecture is capable of capturing changes of reflectance through time of individual crop types. It is further enhanced, for example by incorporating (cloud independent) radar data from Sentinel 1 in order to provide time series with stable intervals between images.

Technologies Used 

Backend: Python, PyTorch, NumPy, geopandas, rasterio, gdal, shapely, pyproj, mgrs

Infrastructure: GCloud (Training), Git, dvc, tensorboard

Challenges

As with any solution that makes use of optical satellite data, cloud occlusion is one of the main challenges to be overcome. This entails the reduction of available data points in time for a given region and an imbalance of data between seasons: Warmer seasons tend to have clearer skies. Consequently, more data points are usable as the satellites are able to capture more images that are not obstructed by clouds.

Another challenge is the fact that the data sets are time series of images with multiple channels per time step, which makes the training process memory consuming and demands smart data aggregation strategies.

Finally, because satellites can only capture a limited area at a time, different locations will have different temporal availability of data, i. e. the sequences of data points in time for different areas will be of different sizes and different dates.