Advanced classification of volunteer potato in a sugar beet field

Volunteer potato is a major problem in sugar beet production in the Netherlands, and adequate control of volunteer potato is critical. This is stressed by a statutory obligation in the Netherlands under which farmers have to remove volunteer potato plants from their fields before the 1st of July in the growing season every year, to a maximum level of two remaining plants per square meter.In 2011, the EU SmartBot project, a cross-border collaboration project which involved 24 different partners from Germany and the Netherlands, was initiated to develop a robotic system for several applications including for agricultural use. In AgroBot, part of the SmartBot project, a small-sized and vision-based autonomous weed control system was to be developed for effective control of volunteer potato plants in a sugar beet field. As a robotic platform, the Clearpath Husky A200 UGV (Unmanned Ground Vehicle) was to be used in this project. Due to the reduced carrying capacity of the robotic platform (Husky), additional infrastructure like a hood was not a viable option. Moreover, artificial lighting was not considered feasible either because the mobile platform was battery operated. Thus, the system should be able to perform robustly in scenes that are fully exposed to ambient lighting conditions.Within the EU SmartBot project, the primary objective of this research was identified as:to develop a computer vision procedure that detects volunteer potato plants under ambient light conditions in a sugar beet fieldFor a complete weed control pipeline, including weed detection and weed removal, the following requirements were set. The automatic weeding system should:effectively control more than 95% of the volunteer potato;ensure less than 5% of undesired control of sugar beet plants;ensure a classification time of less than 1 second per field image for real-time operation in the field.It was indicated that due to the potential non-perfect performance of actual weed removal, classification accuracy should be considerably higher than 95%. The steps required to fulfil the above-mentioned objective form the main line of this thesis including vegetation segmentation (Chapter 2 and 3) and sugar beet/volunteer potato classification (Chapter 4 and 5).Chapter 2 addressed the research question: “Does a ground shadow detection and removal enhance the performance of vegetation segmentation under natural illumination conditions in the field?”In Chapter 2, an algorithm was described and evaluated for ground shadow detection and removal based on colour space conversion and a multilevel threshold. The advantage of using the proposed algorithm was assessed for vegetation segmentation with field images that were acquired by a High Dynamic Range (HDR) camera under natural illumination. Compared with no shadow removal, applying shadow removal enhanced the performance of vegetation segmentation under natural illumination conditions in the field with an average of 20%, 4.4% and 13.5% in precision, specificity and modified accuracy, respectively, and did not reduce segmentation performance when shadows were not present. The average processing time was 0.46 s, which is feasible when real-time application in the field is considered.Chapter 3 addressed the research question: “Do different combinations of colour index and threshold technique result in different segmentation performance when evaluated on field images? Given the varying conditions in the field, is it better to use one specific combination at all times or the combination should be adapted to the field conditions at hand for best segmentation performance?”In Chapter 3, the performance of 40 combinations of eight colour indices and five threshold techniques for vegetation segmentation were evaluated. A clear difference in performance, represented in terms of MA (Modified Accuracy), was observed among various combinations under the given conditions of this research. CIVE+Kapur showed the best performance, while VEG+Kapur showed the worst on the dataset. When adapting the combination to the given conditions yielded a slightly higher performance than when using a single combination for all (in this case CIVE+Kapur). Consistent results were obtained when validated on a different independent image dataset. The expected advantage of adapting the combination to the field condition is not large because it seems that for practical use, the slight improvement when adapting the combination to the field conditions does not outweigh the investment in sensor technology and software needed to accurately determine the different conditions in the field.Chapter 4 and 5 focussed on classification and addressed the following research questions: “Does an algorithm using a Bag-of-Visual-Words (BoVW) model and SIFT or SURF descriptors meet the requirements set for the classification of volunteer potato and sugar beet under natural and varying daylight conditions? If the BoVW model does not meet the requirements, does a deep learning approach, particularly transfer learning based on Convolutional Neural Network (ConvNet, or CNN) provide an effective and better performance to meet the requirements with limited amount of dataset? Are the processing times (or calculation times) fast enough for real-time application?”For the classification of sugar beet and volunteer potato under ambient varying daylight conditions, Chapter 4 proposed a classification algorithm using a Bag-of-Visual-Words (BoVW) model based on SIFT or SURF features as well as crop row information in the form of the Out-of-Row Regional Index (ORRI). The highest classification accuracy of 96.5% with false-negative of 0% obtained using SIFT and ORRI with SVM is considerably better than previously reported approaches for weed classification; however, the false-positive rate of 7% deviates from the requirements since misclassification should be less than 5%. The average classification time of 0.10-0.11 s met the real-time requirements. Adding location information (ORRI) improved overall classification accuracy significantly. The proposed approach proved its potential under varying natural light conditions.Since the required classification accuracy was not obtained in Chapter 4, further research was carried out for the classification of sugar beet and volunteer potato under ambient varying daylight conditions. Chapter 5 evaluated a transfer learning procedure with three different implementations of AlexNet (Part I), and then assessed the performance amongst different ConvNet architectures (Part II): AlexNet, VGG-19, GoogLeNet, ResNet-50, ResNet-101 and Inception-v3. In Part I, the highest classification accuracy (98.0%) was obtained with AlexNet in Scenario 2. In scenario 1 and 3, the highest classification accuracy of 97.0% and 97.3% were obtained, respectively. In Part II, the highest classification accuracy of 98.7% was obtained. This result, to the best of our knowledge, was considerably better than any other approaches mentioned in the literature for crop and weed classification. Transfer learning provided very promising performance for the classification of sugar beet and volunteer potato images under ambient varying light conditions. A deep learning approach based on ConvNet provided better performance than the one in Chapter 4, and satisfied the requirements. All procedures were feasible for real-time field applications (the classification time < 0.1 s).The full pipeline for weed detection consists of three steps: 1) vegetation segmentation, i.e. separating pixels in an image into plant pixels and non-plant pixels, 2) individual object identification, i.e. identification of individual plants (objects) in the set of plant pixels obtained after segmentation, and 3) classification of the plants into two classes, sugar beet (crop) and volunteer potato (weed).In this thesis, steps 1 and 3, i.e. image segmentation and classification of sugar beet/volunteer potato were successfully addressed. Step 2, the identification of individual plants in the images was not addressed. Despite this limitation, it can be concluded that significant progress has been made in this area of study, given the fact that reported algorithms were developed using images captured in full daylight with significant variations in light colour and intensity; a distinct challenge that so far has been circumvented by using hoods and artificial lighting. Yet, the question remains unanswered whether a full pipeline, including all three steps, would be able to meet the requirements identified at the onset of the research.With current hardware and suitable implementation of software, it seems that the requirement of 1 s per image for real-time operation of a weed control system can be attained. The highest classification accuracy of 98.7% obtained in Chapter 5 is supportive in meeting the required 95% control of volunteer potatoes, but when the ConvNet classification would be implemented in a full pipeline also containing vegetation segmentation and individual plant identification, a degraded performance can be expected. Given the fact that the images in this research were obtained under varying daylight conditions, the results showed potential of the proposed approach and compared favourably with classification results in the range of 85-90% that were obtained in various previous researches using hoods and artificial lighting. Therefore, it is safe to say that this research has laid the foundation for a small-sized robotic platform to come into action for weed control in the field