Vision-Guided Robot Trims Tomato Plants with Chameleon3

Removing the leaves from tomato plants is a relatively simple procedure that traditionally has involved manually cutting the older leaves from the lower part of the stems. The process is performed on a weekly basis on commercially grown tomato crops to promote the ripening of the tomatoes. However, the unpredictability and high cost of the manual labor involved in the process and finding a motivated workforce for this monotonous task recently led a consortium of commercial tomato growers in the Netherlands to join forces with automation specialists at Priva to create an alternative solution. The culmination of their efforts has resulted in the development and optimization of a robot that provides tomato growers with an economically viable alternative to manually de-leafing tomato crops grown in greenhouses.

According to Dr. Tomas de Boer, the mechatronic design engineer within the Priva development team responsible for the design of the system, the Kompano Deleaf-Line robot travels on existing tube rails down lanes in the greenhouse which is populated with tomato plants spaced at intervals on either side of the tracks (Figure 1). The robot, which is fitted with a vision system and a telescopic cutter moves from plant to plant in a sequential fashion, identifying and then removing the leaves from each one.

Figure 1: The Kompano Deleaf-line robot travels on a tube rails in the greenhouse which are populated with tomato plants spaced at intervals on either side of the tracks. The robot, which is fitted with a vision system and a telescopic cutter moves from plant to plant in a sequential fashion, identifying and then removing the leaves from each one.

Design challenges

In the development of the system, engineers at Priva needed to ensure that the vision-based robotic system could perform continuously during day and night in the challenging environmental conditions found in commercial greenhouses. Hence the system needed to be capable of identifying small green leaf petioles - or the stalks that attachs tomato leaf blades to their stems - in a sea of other green leaves, and operate under a variety of lighting conditions, ranging from bright sunlight or artificial light to the complete absence of light during the night. 

To solve these challenges, the developers chose to build the vision system around a pair of stereoscopic cameras mounted on a moveable platform, together with a telescopic cutting arm. The stereo cameras - which were each custom built from two pairs of Chameleon3 cameras with OnSemi 1280 x 1024 resolution Python sensors - capture a wide field of view from the left and right side of each tomato plant. (Figure 2).

Figure 2: The vision system for the robot uses a pair of stereoscopic cameras mounted on a moveable platform together with the telescopic cutting arm. The stereo cameras - which were each custom built from two pairs of  Chameleon 3 cameras with OnSemi 1280 x 1024 resolution Python sensors - capture a wide field of view from both the left and right hand side of each of the tomato plants.

"To enable each pair of stereo cameras to capture reliable images of the tomato plants regardless of the lighting conditions in the greenhouse, the system employs a Xenon strobe light which illuminates the plant. As the strobe emits light every two seconds, the strobe triggers the stereo cameras to expose images at 30 microsecond intervals. This enables the system to capture a uniform set of images each time," said Dr. de Boer.

Once the two stereo cameras have captured a pair of images, they are transferred over a USB interface to a PC running the open source Ubuntu operating system and the Robot Operating System (ROS), an open-source software framework hosted by the Open Source Robotics Foundation, Inc. (OSRF) for creating robotic applications across a variety of platforms.

System Software

Bespoke image processing functions inside OpenCV are then used to process the images from both sets of cameras to identify the location of leaves in the tomato plants in a certain height range previously defined by the tomato growers. Having identified the location of the leaves, the image processing software then calculates the exact three dimensional co-ordinates of their petioles, which need to be cut by the effecter at the end of the robotic arm.

The co-ordinate data generated by the image processing software is passed to the Robot Operating System (ROS), which transfers the data to a set of intelligent servo drives that power motors that drive the telescopic robotic arm to the correct location on the plant. Once it reaches its destination, the end effecter then cuts the petioles, removing the leaves from the tomato plant (Figure 3).

Because the pair of stereo cameras are mounted on the same platform as the robotic arm, once the arm moves to cut one specific set of petioles, the cameras are also moved to a new location, where they capture a further set of images of the plant from a different perspective.

"These new images are processed in parallel during the cutting operation to identify any further leaves that need to be removed from the plant in any subsequent operation. The iterative procedure continues until the system cannot detect that any further leaves need to be removed from the plant. After performing a final check, the cutting operation ceases, and the robot moves along the tracks to the next plant in the greenhouse," said Dr. de Boer. 

Figure 3: Once the location of the petioles on the tomato plant has been determined by the image processing software, the data is used to drive the telescopic robotic arm on the robot to the correct location. Once it reaches its destination, the end effecter then cuts the petioles, removing the leaves.

Finishing Touches

At the present time, the developers of the Kompano Deleaf-Line robot are working with members of the tomato growing consortium in the Netherlands to finalize a pre-production prototype of the system. 
Those members who worked with Priva will be the first to take advantage of the system when it finally becomes available at the beginning of next year. Depending on the number of robots that are ordered by the consortium, it will then become available for other tomato growers from June 2017. As of June this year, it is also possible to pre-order the robot online.

In use, the de-leafing robot is expected to be capable of maintaining 0.75 to 1 hectares of tomato plants, depending on the density of the leaves on their stems. In larger facilities, growers can either choose to work with multiple robots or to combine the automated system with manual labor.

According to Dr. de Boer, the tomato de-leafing robot will be the first of many robots that the company plans to develop for the horticultural marketplace. These next generation systems will automate other labor intensive tasks, such as the harvesting of vegetables like cucumbers. 

About Priva Group

Priva creates a climate for growth. The company leads the way in the development and production of technology for the optimization of environmental conditions and process management. Priva provides solutions for horticulture, building automation and anything in between with a unique combination of software, hardware and services.
 www.privagroup.com