Developing a phone-based imaging tool to inform on fruit volume and potential optimal harvest time

Abstract

The objective of this project was to investigate the feasibility of developing a smartphone-based imaging tool for berry quality assessment that can be conveniently used within vineyards to provide real time and in-field evaluation of grape berry size and colour. After taking a photograph of the grape bunch on the vine, the smartphone application analyses the image, detects and samples the visible berries, and determines the volume and hue distributions of the sampled berries. Results are reported for the analysis of grape bunch images from Australian vineyards.

The resulting app (called WineOz SmartGrape) is programmed for use on an Android smartphone platform and can be downloaded from the app store using the following link:

https://play.google.com/store/apps/details?id=com.nwgic.grapeyield

The app is intended to be used to monitor the spatial and temporal data collected for vineyard blocks and is best used when interfaced to a dashboard which can be downloaded from the NWGIC web site along with instructions:

http://www.csu.edu.au/nwgic/research/wineoz-smartgrape-mobile-application

Challenges and recommendations for smartphone imaging in the field are also discussed.

Summary

This project arises from two developments: a deepening understanding of the chemistry associated with grape maturation; and rapid development of mobile imaging techniques, especially associated with smartphones.

The mature development of mobile phones with high-quality cameras, well-developed software environments, image processing, and statistical analysis (including machine learning) for data and image classification could offer a less expensive means to measure berry skin colour, fruit volume and distribution, grape berry homogeneity versus heterogeneity. That forms the justification for the present project.

White grape colour transition (from green to yellow, and yellow to orange) during the ripening is linked to chlorophylls and carotenoids degradation, and the formation of C13 norisoprenoids. Deloire 2013, showed that using a Dyostem™ instrument (Vivelys and Montpellier SupAgro, France) to measure berry volume and skin colour in white cultivars could help the harvest decision process to achieve desirable wine styles. However, this technique is expensive, slow, destructive, and cannot be deployed in the vineyard. Other techniques such as optical fibre evanescent field absorption (FEFA) have been used to measure grape colour but they also rely on specialised equipment and the destruction of grapes.

The link between sugar accumulation and flavour development in the berry is often non-existent, compound and environment dependent, and highly influenced by harvest time. Berry size can be manipulated most efficiently with irrigation strategies and is one of the key determinants of berry shrinkage. Heterogeneity in berry volume significantly alters grape composition and can only be rectified by expensive sorting techniques in the winery. Thus, the development of in-field decision support tools will assist vineyard managers to more effectively operate commercial vineyards for irrigation management and harvest decisions for specific wine styles.

This project commenced by determining the feasibility of using desktop computing systems for berry detection and colour measurements of images of bunches of grapes. A colour reference card located within the image allowed for colour correction algorithms to be applied and served as a depth of field calibration for accurate berry size estimation. Initial image analysis showed good correlations between berry measures from the image and laboratory measures for berry size and colour in white varieties, and berry size for red varieties. Several reference items were trialled for depth of field calibrations and for the purposes of this project a probe that could be inserted into the target bunches was chosen. Inserting a probe into the bunch of grapes facilitates image capture as the camera operator is not required to hold or place an object in the image area; the probe is not obscured by fingers or the hands of the operator and the probe can be easily removed and reused. Berry illumination is an important consideration to image capture prior to analysis as colour casting arises from differentially shaded portions of the vine canopy, which in turn is influenced by the degree of cloud cover, light angle and time of day. Several image databases have been created for grape bunches at varying levels of maturity and for several cultivars, along with traditional measures of grape composition. These databases will provide a valuable resource for the development of machine learning algorithms for future projects.
To determine the spatial and temporal variability of fruit composition within a single bunch, an investigation to characterise bunch heterogeneity at two developmental stages (veraison and full maturity immediately prior to harvest) was undertaken. Berry position within bunches and branches did not influence berry composition, however, berry seed number was strongly correlated with sugar concentration fresh and dry mass of the pericarp and skin colour at veraison.

Smartphone app development was undertaken to incorporate image capture, berry detection, colour and size measures within the Android Studio with OpenCV. This App can be operated using Android version 4.0.3 (Ice Cream Sandwich). To assist in the development of the user interface, feedback from industry representatives was sought prior to final GUI development. User data incorporating vineyard, block and grape cultivar can be entered prior to image capture and running berry detection algorithms. A considerable investment of time to overcome the programming challenge of incorporating image detection and machine learning algorithms from a desktop to a smartphone computing environment was required for this project.

The resulting Android platform App (WineOZ SmartGrape) can be downloaded from the app store and interfaced with an Excel™ based dashboard to monitor the temporal and spatial data created with the image analysis features of the software. The app can be downloaded from:

https://play.google.com/store/apps/details?id=com.nwgic.grapeyield

The dashboard, along with instructions and guidelines for imaging, is available from the NWGIC web site:

http://www.csu.edu.au/nwgic/research/wineoz-smartgrape-mobile-application

The berry detection pipeline was developed within this project using a sequential process that makes use of five stages of detection:

1. Colour image capture with reference probe detection.
2. Greyscale conversion.
3. Equalisation of the grayscale image to a standardised distribution of colour.
4. Berry detection using a circle detection algorithm.
5. Berry detection filtering using a support vector machine learning algorithm.

While the image undergoes a greyscale transformation, pixel colour information within each detected berry is retained and this is used to determine the medium berry colour within the HSV colour scale. Berry size estimation is determined using a ratio of pixels across the reference probe and detected circle following image analysis. Once berry diameter has been determined an estimate of berry volume is calculated using a standard sphere formula. Metadata collected during image capture allows spatial and temporal data curation within three separate data files which are easily linked to an Excel™ dashboard to display results within a desktop environment.

Ground truthing of the app results to the industry standard measures for berry size and colour was undertaken in the 2018 growing season for Chardonnay grapes growing in the Riverina. A suite of grape bunch images was collected in full sunlight and with shading to assess the impact of illumination on berry detection and colour measures. Good comparison for berry hue angle obtained from the smartphone app and a Dyostem™ instrument were obtained for bunches in full sunlight, however, such correlations were not evident for partially shaded or fully shaded bunches. These results were obtained without any colour correction applied to the images and demonstrated a requirement for this feature to be incorporated into future versions of the app. Berry volume estimates were poorly correlated with Dyostem™ values and a number of factors are important when comparing these methods. The Dyostem™ instrument is a proprietary system and the methods used for calculation of berry volume could not be determined. Attempts to access the source code for the instrument and requests to the instrument manufacturer to obtain this information were not successful. Some berry water loss and deformation may have arisen from the berries as these were measured up to 24 hours after their collection. The number of pixels that represented the reference probe diameter (0.8 cm) may vary from image to image depending on camera position and proximity to the probe. This variance associated with the pixel diameter of the probe translates to a significant ratio difference used for berry volume calculations. This arises due to the small size of the probe and can be corrected by enlarging the probe diameter.

Small lot winemaking of Chardonnay and Sauvignon Blanc at early and advanced levels of grape maturity were undertaken to explore the link between berry colour assessment and wine style. Clear and demonstrably different sensory profiles of the wines were noted by a trained sensory panel at the early and mature stages of grape development. Importantly, changes in the wine sensory profiles were correlated to a change in berry hue angle, demonstrating the opportunity to use grape berry colour as a predictor of wine style and establishing a sufficient body of evidence to support future research to move the outcomes of this project from a proof of concept to a mature decision support tool for practical vineyard management.