Computer Vision Case Study

Automatic planning of solar systems

Hint: switch between general information and a more technical view on this project


Our client is a company from the solar sector, which offers the planning, consultation and installation of solar systems for private households. The aim of this project is to automate the planning process of solar systems. Specifically, this involves preparing an individual offer for customers so that they can estimate the costs and benefits of the investment and get an aesthetic impression of the final result.

Our client is a company from the solar sector, which offers the planning, consultation and installation of solar systems for private households. The aim of this project is to automate the planning process of solar systems. Specifically, this involves preparing an individual offer for customers so that they can estimate the costs and benefits of the investment and get an aesthetic impression of the final result.

Starting Point

In order to prepare a quotation, our client first needs various information from the customer, such as the length and width of the roof, roof orientation, etc. Usually parts of this data are unknown to the homeowner and an employee ('Chris') has to find the information elsewhere.

Description of the old process

First, Chris creates a new project in a desktop solar software and asks the homeowner for a photo of the roof. He then enters this photo and the customer's data into the solar software. Now the roof circumference and the contours of each obstacle on the roof must be drawn in manually. Dormers are treated separately, since these usually take up more roof surface on the photo than in reality.

Now Chris adds a reference length (e.g. the length or width of the roof). Therefore he opens the browser and enters the customer address e.g. at Google Maps. If the quality of the satellite image is good enough, Chris counts the roof tile on the roof and the reference length is roughly determined. Otherwise he has to switch to another service. After entering the reference length, the program estimates the roof angle and the resulting roof area. This often leads to inaccuracies. Since the underlying calculation method is not transparent, a subsequent improvement is hardly possible.

After a few further steps, Chris saves the desired solar panel visualization and transfers it (including further offer-relevant data) to the software so that a final offer can be created.

The whole process takes on average 120 minutes per planning, is error-prone (unclear calculation methods, frequent manual interfaces) and the visualization of the solar panels on the house roof does not correspond to current standards.

Until now, the solar systems were planned with the help of a desktop application, which is only available for Windows, has an outdated GUI as well as high maintenance and license costs. Furthermore there are no interfaces (both for automatic reading of customer information as well as for passing on new information) and the usability is not optimized for the use case (many unnecessary clicks/steps).

For cases where the roof angles and roof dimensions are unknown to the customer, the application has an approximative triangulation method, which estimates the roof angle using the roof circumference drawn on a photo and the input of a reference length (e.g. roof width, roof length). However, this method often turned out to be incorrect. Since the application is not open source and there is no transparent description of the calculation method, this error can not be sufficiently analyzed and the method can not be improved.


The goal of the project was clearly defined: Significantly accelerate the planning process.

But in detail, there were many challenges, such as

  • Should we aim for a solution for all roof shapes? Which data sources should be used?
  • Which human-machine interfaces are optimal in the conflicting areas of planning quality vs. time savings vs. user-friendliness?
  • How does the solution scale for several users? Does the system run in the cloud or on the user's computer?

Available input: address and roof photo of the customer.

Desired output: roof dimensions, roof angle, southern orientation, number of solar panels, panel visualisation on the customer's roof picture.

The biggest challenges we were able to identify before the project started, were the following:

  1. recognition of small obstacles (chimney, skylight, antenna etc.)
  2. different input quality (e.g. shading)
  3. estimating (3-dimensional) roof dimensions (length, width, roof angle)
  4. handling of special roof shapes
  5. separation of different roof sides
  6. interfaces (man-machine, machine-machine)

We developed our solution within an agile project framework (MVP after 6 months, continuous user feedback). The entire process was automated up to the visualization of the solar panels.

Together with the customer's IT department the solution was implemented as a web application, which has several advantages: e.g. fast roll-out of improvements, unlimited user addition, operating system independence, etc.

Currently 13 employees are working with the this software (June 2019). The use of the old solar software has been discontinued.

The planning time was reduced from 120 minutes to approx. 15 minutes.

Technologies used

Backend: Python, Tensorflow, PyTorch, SciPy, NumPy, scikit-learn, scikit-image, OpenCV, Flask

Frontend: JavaScript, Angular5, Three.js, Fabric.js, Sass

Infrastructure: GCloud (Training), AWS (Production), Docker, Git

In the following we present solutions of some selected problems.

1. Segmentation of the roof

We segment the roof and the obstacles on it by means of neural networks both on the satellite photo and on the customer photo. The architecture of these networks is mainly based on U-Nets, a specific form of a convolutional neural network, which currently provide the best performance for such problems.

For the training of the nets, we generated numerous labels ourselves, i.e. marked outlines of roofs and obstacles. It was important to adapt the labeling scheme to the problem in order to learn and finally recognize small obstacles and semantic structures. The training required extensive finetuning (e.g. identification of the best loss function, data augmentation, hyper parameter optimization etc.) - for details please refer to our Blog.

For the segmentation of customer photos we were able to achieve significant improvements by means of transfer learning through a neural network pre-trained on ImageNet. The two lower pictures exemplify the impressive abilities of the neural network trained by us to recognize semantic structures.

2. Postprocessing

The segmentation of the roofs first provides a pixel-by-pixel classification (i.e. each pixel is assigned to a class = {roof, non-roof, obstacle, etc.}). The next step is the detection of geometric shapes and the recognition of semantic structures. In particular it is about the recognition of the roof corners, the separation of individual roof parts and the identification of the respective south side.

These subtasks turned out to be more difficult than initially assumed due to the not always perfect segmentation output. Through a creative combination of evolutionary algorithms and classical computer vision methods, we still achieve a very good quality and are able to correct segmentations of lower quality.

3. Inferring the roof dimensions

Photos are 2-dimensional projections and therefore not quite sufficient for estimating the actual roof area.

We solved this problem by a combination of satellite and customer photos. The former provides reference dimensions for horizontal distances and the latter allows us to calculate angles for rectangular roof surfaces using projective geometry and self-developed formulas.

With the help of simple trigonometric formulas, we then determine length and width and thus the area of the roof. The transformation matrix obtained from the formulas allows the visualization of the panels on the customer photo.

We would be happy to present more detailed information to you in a personal meeting.


To start the product demonstration, click the frame below.

Currently this product demonstration is optimized for desktop only.

Step 1: The user enters the address of the house.

Step 2: The neural network recognizes roof areas including obstacles, the user selects the desired roof.

Step 3: A post-processing algorithm determines the corners of the roof area, the user can adjust the suggested corners.

Step 4: Another neural network determines roof area and obstacles on the customer photo. The user can make adjustments if necessary.

Step 5: All information is merged, result is visualized. User can make last changes.

End of product demonstration

Case Studies in Computer Vision

Computer Vision

Convective clouds detection

We automated the detection of certain cloud structures for Deutscher Wetterdienst (DWD).
Our solution
© unsplash/@nicolasp
Computer Vision

Crop Type Classification

Predict crop types from satellite data to support modern agriculture
Image which visualizes the process of crop type prediction Our solution