EAGL-I (Embedded Autonomous Generator of Labelled Images) is TerraByte's first autonomous imaging system. The system started imaging 2020 and has since produced hundreds of thousands labelled images of weeds and crops. The system consists of a gantry-style cartesian robot that allows movement in the three spatial dimensions. On the gantry head a pan-tilt system is mounted that can pan and tilt an attached WiFi-controllable camera (a GoPro Hero 7). Controlled through our software the robot automatically repositions the camera every couple of seconds. From each position an image is taken of several plants positioned inside EAGL-I. The software calculates the position of the plants within the image taken to automatically label and crop out the individual plants. A large amount of such labelled images is key for the training of machine learning models. The so captured plants are automatically uploaded into our database and are available to researchers and eventually everyone.

• Further Information
• Date: January 2019 - Today

Beck et al. 2020
Back

There are many deep learning models and many related researches have been done to classify objects. Yet, plant classification has still been challenging because most plants are very similar to each other in nature. Our research is designing the best-fit CNN architecture to classify highly similar plants with high accuracy. With an architecture called generalist and specialists CNN model we can distinguish between highly identical weeds and crops, which are barely discernible by humans. The generalist and specialists network contains two different networks in terms of classification and application. The generalist determines and classifies similar photos and pass them to the related specialist network. The specialist networks then classify the plants and distinguish between crops and weeds that look incredibly similar.

• Date: April 2019 - Today

Back

Convolutional neural networks generally perform better on data that is similar to the data they had been trained on. To ensure that the models we train at TerraByte perform under field-conditions we have different strategies at our disposal. A very common one is transfer learning: Here a trained model is retrained on data from the new environment, while carrying over already trained filters for low- to high-level features. Another way is to modify our data such that it appears similar to field-data, although the images had been taken in the greenhouse. For transfer learning we collect a large amount of (initially unlabeled) field data. For the second approach we use image segmentation and background removal to digitally "plant" our lab-data in field-like environments.

• Date: January 2020

Back

The multispectral 3D scanning project establishes a framework of generating labeled 3D imaging datasets of real plants and developing 3D convolutional neural networks to solve detection, classification and regression problems related to agriculture. The imaging scanner used in the project is the Phenospex PlantEye, which is a unique plant sensor that produces real-time multispectral 3D point models of plants and crops. The output of the scanner contains detailed information of more than 14 parameters related to the morphological and spectral specifications of plants such as plant height, 3D leaf area, leaf inclination, digital biomass and light penetration depth. PlantEye also allows the visualization of the 3D point models with different spectral indices like NDVI or PSRI, where each index enables the calculation of multiple measures related to the plant health, performance and photosynthesis. We create fully automated systems to conduct data preprocessing and analysis by developing 3D convolutional neural networks to learn, interpret and analyze 3D point models of plants in order to monitor plant growth and to improve plant health, quality and yield.

• Date: November 2020 - Today

Close Project

Insects see the world differently from humans. While our eyes have only three types of colour sensitive cells, butterflies for example have four types of colour receptors, allowing them to see more colours than we do. These insects have evolved this trait to better find food, mate, and to avoid predators. A device that has a similar superhuman ability is the hyperspectral camera. It scans the scene at hundreds of different wavelengths in the visible and infrared spectrum. In other words, a hyperspectral camera can distinguish hundreds of colours of light, way more than the three colours – red, green, and blue – our cell phone camera captures. Hyperspectral imaging of plants enables scientists to discover plant properties that are invisible to humans. At the University of Winnipeg, we use a hyperspectral camera to find new ways to visually characterise plants.

• Date: November 2020 - Today

The following video shows the reflectance of a plant under different wavelengths. The video is colorized such that the color of the each image corresponds roughly to each wavelength's color in the visible part of the spectrum. Note that wavelengths beyond 800 nm (Infrared) are normally not visible to the human eye. We still gave it a red hue in this video.

Close Project

With the help of generative adversarial neural networks we can "translate" images of plants that we have grown in the lab to look like they had been growing in the field. This is a powerful method, as it allows us to combine any weed grown in the lab to be placed on any kind of soil recreating labelled field image data with each weed cirrectly identified! Such images form valuable training data for models to distinguish real weeds in real fields afterwards.

We have also trained generative neural networks on small weed data taken in the lab to construct images of weeds entirely from scratch. This procedure allows us to bootstrap weed image data in any volume desired from a small dataset of images of real plants.

• Date: 2023 - Today

Close Project

We investigate several methods of obtaining field data with different camera systems. From simply mounting several cameras on a tractor up to drones. We are developing a field rover with the goal of fully automating the collection of field data and starting first tests with it in the growing season of 2021. The imaging equipment ranges from normal RGB-cameras to multi- and hyperspectral cameras.

• Date: June 2019 - Today

Close Project

Using a rotary table and several low cost cameras we image plants from a variety of angles to obtain a 3-dimensional reconstruction of the plant. This reconstructed model of the plant can be used to take phenotypical measurements such as height, convex hull, or leaf area. At the same time the system remains affordable at around $1000.

• Date: May 2021 - Today

Close Project