Abstract
Insects are the most important global pollinator of crops and play a key role in maintaining the sustainability of natural ecosystems. Insect pollination monitoring and management are therefore essential for improving crop production and food security. Computer vision facilitated pollinator monitoring can intensify data collection over what is feasible using manual approaches. The new data it generates may provide a detailed understanding of insect distributions and facilitate fine-grained analysis sufficient to predict their pollination efficacy and underpin precision pollination. Current computer vision facilitated insect tracking in complex outdoor environments is restricted in spatial coverage and often constrained to a single insect species. This limits its relevance to agriculture. Therefore, in this article we introduce a novel system to facilitate markerless data capture for insect counting, insect motion tracking, behaviour analysis and pollination prediction across large agricultural areas. Our system is comprised of edge computing multi-point video recording, offline automated multi-species insect counting, tracking and behavioural analysis. We implement and test our system on a commercial berry farm to demonstrate its capabilities. Our system successfully tracked four insect varieties, at nine monitoring stations within polytunnels, obtaining an F-score above 0.8 for each variety. The system enabled calculation of key metrics to assess the relative pollination impact of each insect variety. With this technological advancement, detailed, ongoing data collection for precision pollination becomes achievable. This is important to inform growers and apiarists managing crop pollination, as it allows data-driven decisions to be made to improve food production and food security.
Similar content being viewed by others
Data Availability
The datasets generated during and analysed during the current study are available at https://doi.org/10.26180/21533760
Code Availability
Code is available through https://github.com/malikaratnayake/Polytrack2.0.
Change history
11 January 2023
A Correction to this paper has been published: https://doi.org/10.1007/s11263-022-01741-2
References
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, & J., Zheng, X. (2016) TensorFlow: A system for large-scale machine learning. Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016 (pp. 265—283).
Abdel-Raziq, H. M., Palmer, D. M., Koenig, P. A., Molnar, A. C., & Petersen, K. H. (2021). System design for inferring colony-level pollination activity through miniature bee-mounted sensors. Scientific Reports, 11(1), 1–12.
Afonso, M., Fonteijn, H., Fiorentin, F. S., Lensink, D., Mooij, M., Faber, N., & Wehrens, R. (2020). Tomato fruit detection and counting in greenhouses using deep learning. Frontiers in Plant Science, 11, 1759.
Aizen, M. A., Garibaldi, L. A., Cunningham, S. A., & Klein, A. M. (2009). How much does agriculture depend on pollinators? Lessons from long-term trends in crop production. Annals of Botany, 103(9), 1579–1588.
Amarathunga, D. C. K., Grundy, J., Parry, H., & Dorin, A. (2021). Methods of insect image capture and classification: A systematic literature review. Smart Agricultural Technology, 1, 100023.
Aslanpour, M. S., Toosi, A. N., Cicconetti, C., Javadi, B., Sbarski, P., Taibi, D., & Dustdar, S. (2021). Serverless edge computing vision and challenges. Australasian Computer Science Week Multiconference (pp. 1–10).
Barreiros, M. O., Dantas, D. O., Silva, L. C. O., Ribeiro, S., & Barros, A. K. (2021). Zebrafish tracking using YOLOv2 and Kalman filter. Scientific Reports, 11(1), 1–14.
Batsleer, F., Bonte, D., Dekeukeleire, D., Goossens, S., Poelmans, W., Van der Cruyssen, E., & Vandegehuchte, M. L. (2020). The neglected impact of tracking devices on terrestrial arthropods. Methods in Ecology and Evolution, 11(3), 350–361.
Bjerge, K., Mann, H. M., & Høye, T. T. (2021). Real-time insect tracking and monitoring with computer vision and deep learning. Remote Sensing in Ecology and Conservation, 8(3), 315–327.
Bjerge, K., Nielsen, J. B., Sepstrup, M. V., Helsing-Nielsen, F., & Høye, T. T. (2021). An automated light trap to monitor moths (lepidoptera) using computer vision-based tracking and deep learning. Sensors, 21(2), 343.
Bochkovskiy, A., Wang, C -Y., & Liao, H -Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv preprintarXiv:2004.10934.
Branson, K., Robie, A. A., Bender, J., Perona, P., & Dickinson, M. H. (2009). High-throughput ethomics in large groups of drosophila. Nature Methods, 6(6), 451–457.
Breeze, T. D., Bailey, A. P., Balcombe, K. G., Brereton, T., Comont, R., & Edwards, M. (2021). Pollinator monitoring more than pays for itself. Journal of Applied Ecology, 58(1), 44–57.
Campbell, J., Mummert, L., & Sukthankar, R. (2008). Video monitoring of honey bee colonies at the hive entrance. Visual Observation and Analysis of Animal and Insect Behavior, ICPR, 8, 1–4.
Chagnon, M., Gingras, J. . De., & Oliveira, D. (1989). Effect of honey bee (hymenoptera: Apidae) visits on the pollination rate of strawberries. Journal of Economic Entomology, 82(5), 1350–1353.
Dennis, R., Shreeve, T., Isaac, N., Roy, D., Hardy, P., Fox, R., & Asher, J. (2006). The effects of visual apparency on bias in butterfly recording and monitoring. Biological Conservation, 128(4), 486–492.
FAO (2018). Why bees matter; the importance of bees and other pollinators for food and agriculture. https://www.fao.org/documents/card/en/c/i9527en/
Fijen, T. P., Scheper, J. A., Boom, T. M., Janssen, N., Raemakers, I., & Kleijn, D. (2018). Insect pollination is at least as important for marketable crop yield as plant quality in a seed crop. Ecology Letters, 21(11), 1704–1713.
Food & Agriculture Organization of the United Nation (2019). Global action on pollination services for sustainable agriculture https://www.fao.org/pollination/en/.
Garibaldi, L. A., Carvalheiro, L. G., Vaissière, B. E., Gemmill-Herren, B., Hipólito, J., Freitas, B. M., et al. (2016). Mutually beneficial pollinator diversity and crop yield outcomes in small and large farms. Science, 351(6271), 388–391.
Garibaldi, L. A., Requier, F., Rollin, O., & Andersson, G. K. S. (2017). Towards an integrated species and habitat management of crop pollination. Current Opinion in Insect Science, 21, 105–114.
Garibaldi, L. A., Sáez, A., Aizen, M. A., Fijen, T., & Bartomeus, I. (2020). Crop pollination management needs flower-visitor monitoring and target values. Journal of Applied Ecology, 57(4), 664–670.
Goscinski, W. J., McIntosh, P., Felzmann, U. C., Maksimenko, A., Hall, C. J., Gureyev, T., et al. (2014). The multi-modal Australian sciences imaging and visualization environment (massive) high performance computing infrastructure: applications in neuroscience and neuroinformatics research. Frontiers in Neuroinformatics, 8, 30.
Haalck, L., Mangan, M., Webb, B., & Risse, B. (2020). Towards image-based animal tracking in natural environments using a freely moving camera. Journal of Neuroscience Methods, 330, 108455.
Hall, M. A., Jones, J., Rocchetti, M., Wright, D., & Rader, R. (2020). Bee visitation and fruit quality in berries under protected cropping vary along the length of polytunnels. Journal of Economic Entomology, 113(3), 1337–1346.
Hallmann, C. A., Sorg, M., Jongejans, E., Siepel, H., Hofland, N., & Schwan, H. (2017). More than 75 percent decline over 27 years in total flying insect biomass in protected areas. PLoS One, 12(10), e0185809.
Howard, S. R., Nisal Ratnayake, M., Dyer, A. G., Garcia, J. E., & Dorin, A. (2021). Towards precision apiculture: Traditional and technological insect monitoring methods in strawberry and raspberry crop polytunnels tell different pollination stories. PLoS One, 16(5), e0251572.
Høye, T. T., Ärje, J., Bjerge, K., Hansen, O. L., Iosifidis, A., Leese, F., Raitoharju, J., et al. (2021). Deep learning and computer vision will transform entomology. Proceedings of the National Academy of Sciences, 118(2), e2002545117.
Jolles, J. W. (2021). Broad-scale applications of the raspberry pi: A review and guide for biologists. Methods in Ecology and Evolution, 12(9), 1562–1579.
Kamilaris, A., & Prenafeta-Boldú, F. X. (2018). Deep learning in agriculture: A survey. Computers and Electronics in Agriculture, 147, 70–90.
Kevan, P. G. (1975). Sun-tracking solar furnaces in high arctic flowers: Significance for pollination and insects. Science, 189(4204), 723–726.
Kirkeby, C., Rydhmer, K., Cook, S. M., Strand, A., Torrance, M. T., & Swain, J. L. (2021). Advances in automatic identification of flying insects using optical sensors and machine learning. Scientific Reports, 11(1), 1–8.
Koirala, A., Walsh, K. B., Wang, Z., & McCarthy, C. (2019). Deep learning-method overview and review of use for fruit detection and yield estimation. Computers and Electronics in Agriculture, 162, 219–234.
Kuhn, H. W. (1955). The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1–2), 83–97.
Lu, H., Fu, X., Liu, C., & Li, L. (2017). Cultivated land information extraction in uav imagery based on deep convolutional neural network and transfer learning. Journal of Mountain Science, 14(4), 731–741.
MacInnis, G., & Forrest, J. R. (2019). Pollination by wild bees yields larger strawberries than pollination by honey bees. Journal of Applied Ecology, 56(4), 824–832.
Magnier, B., Gabbay, E., Bougamale, F., Moradi, B., Pfister, F., & Slangen, P. (2019). Multiple honey bees tracking and trajectory modeling. Multimodal Sensing: Technologies and Applications, 11059, 110590Z.
Odemer, R. (2022). Approaches, challenges and recent advances in automated bee counting devices: A review. Annals of Applied Biology, 180(1), 73–89.
O’Grady, M., Langton, D., & O’Hare, G. (2019). Edge computing: A tractable model for smart agriculture? Artificial Intelligence in Agriculture, 3, 42–51.
Outhwaite, C., McCann, P., & Newbold, T. (2022). Agriculture and climate change reshape insect biodiversity worldwide. Nature, 605(7908), 97–102.
Pérez-Escudero, A., Vicente-Page, J., Hinz, R. C., De Arganda, S., & Polavieja, G. G. (2014). idtracker: Tracking individuals in a group by automatic identification of unmarked animals. Nature Methods, 11(7), 743–748.
Potts, S. G., Imperatriz-Fonseca, V., Ngo, H. T., Aizen, M. A., Biesmeijer, J. C., & Breeze, T. D. (2016). Safeguarding pollinators and their values to human well-being. Nature, 540(7632), 220–229.
Rader, R., Bartomeus, I., Garibaldi, L. A., Garratt, M. P., Howlett, B. G., & Winfree, R. (2016). Non-bee insects are important contributors to global crop pollination. Proceedings of the National Academy of Sciences, 113(1), 146–151.
Ratnayake, M.N., Dyer, A.G., Dorin, A. (2021a). Towards computer vision and deep learning facilitated pollination monitoring for agriculture. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2921–2930).
Ratnayake, M. N., Dyer, A. G., & Dorin, A. (2021). Tracking individual honeybees among wildflower clusters with computer vision-facilitated pollinator monitoring. PLoS One, 16(2), e0239504.
Real, L. (2012). Pollination Biology. Elsevier.
Redmon, J., & Farhadi, A. (2017). Yolo9000: better, faster, stronger. Proceedings of the IEEE conference on computer vision and pattern recognition Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7263–7271).
Rollin, O., & Garibaldi, L. A. (2019). Impacts of honeybee density on crop yield: A meta-analysis. Journal of Applied Ecology, 56(5), 1152–1163.
Schweiger, O., Biesmeijer, J. C., Bommarco, R., Hickler, T., Hulme, P. E., & Klotz, S. (2010). Multiple stressors on biotic interactions: How climate change and alien species interact to affect pollination. Biological Reviews, 85(4), 777–795.
Sekachev, B., Manovich, N., & Zhavoronkov, A. (2019). Computer vision annotation tool. Zenodo. GitHub: https://github.com/opencv/cvathttps://doi.org/10.5281/zenodo.3497106.
Settele, J., Bishop, J., & Potts, S. G. (2016). Climate change impacts on pollination. Nature Plants, 2(7), 1–3.
Simons, D. J., & Chabris, C. F. (1999). Gorillas in our midst sustained inattentional blindness for dynamic events. Perception, 28(9), 1059–1074.
Spaethe, J., Tautz, J., & Chittka, L. (2001). Visual constraints in foraging bumblebees: Flower size and color affect search time and flight behavior. Proceedings of the National Academy of Sciences, 98(7), 3898–3903.
Spencer, E. E., Barton, P. S., Ripple, W. J., & Newsome, T. M. (2020). Invasive European wasps alter scavenging dynamics around carrion. Food Webs, 24, e00144.
Stojnić, V., Risojević, V., Muštra, M., Jovanović, V., Filipi, J., Kezić, N., & Babić, Z. (2021). A method for detection of small moving objects in UAV videos. Remote Sensing, 13(4), 653.
Su, D., Kong, H., Qiao, Y., & Sukkarieh, S. (2021). Data augmentation for deep learning based semantic segmentation and crop-weed classification in agricultural robotics. Computers and Electronics in Agriculture, 190, 106418.
Vanbergen, A. J., Initiative, I. P., et al. (2013). Threats to an ecosystem service: Pressures on pollinators. Frontiers in Ecology and the Environment, 11(5), 251–259.
van der Kooi, C. J., Kevan, P. G., & Koski, M. H. (2019). The thermal ecology of flowers. Annals of Botany, 124(3), 343–353.
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., & Belongie, S. (2018). The inaturalist species classification and detection dataset. Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8769–8778).
Walter, T., & Couzin, I. D. (2021). Trex, a fast multi-animal tracking system with markerless identification, and 2D estimation of posture and visual fields. eLife, 10, e64000.
Wang, S., Liu, W., Wu, J., Cao, L., Meng, Q., Kennedy, P.J. (2016). Training deep neural networks on imbalanced data sets. 2016 international joint conference on neural networks (ijcnn) (pp. 4368–4374).
Wood, T. J., Michez, D., Paxton, R. J., Drossart, M., Neumann, P., & Gerard, M. (2020). Managed honey bees as a radar for wild bee decline? Apidologie, 51(6), 1100–1116.
Yang, C., Collins, J., & Beckerleg, M. (2018). A model for pollen measurement using video monitoring of honey bees. Sensing and Imaging, 19(1), 1–29.
Zivkovic, Z., & Van Der Heijden, F. (2006). Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recognition Letters, 27(7), 773–780.
Acknowledgements
The authors would like to thank Sunny Ridge Australia for the opportunity to conduct research at their farm.
Funding
Authors were supported by the Australian Research Council Discovery Projects grant DP160100161 and Monash-Bosch AgTech Launchpad primer Grant. This study was funded by AgriFutures grant PRJ-012993. Amarathunga is supported by ARC Research Hub IH180100002.
Author information
Authors and Affiliations
Contributions
MNR, AGD, AD: conceptualization; MNR: data curation; MNR: formal analysis; AGD, AD: funding acquisition; MNR, DCA, AZ: investigation; MNR, AGD, AD: Methodology; AGD, AD: project administration; AGD, AD: resources; MNR: software; AGD, AD: supervision; MNR, DCA: validation; MNR: writing—original draft; MNR, DCA, AZ, AGD, AD: writing—review & editing.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no competing interests to declare that are relevant to the content of this article.
Consent to participate
Not applicable.
Consent for publication
Not applicable.
Ethics approval
Not applicable.
Additional information
Communicated by Angjoo Kanazawa.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ratnayake, M.N., Amarathunga, D.C., Zaman, A. et al. Spatial Monitoring and Insect Behavioural Analysis Using Computer Vision for Precision Pollination. Int J Comput Vis 131, 591–606 (2023). https://doi.org/10.1007/s11263-022-01715-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-022-01715-4