Skip to main content

Advertisement

Log in

Spatial Monitoring and Insect Behavioural Analysis Using Computer Vision for Precision Pollination

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

A Correction to this article was published on 11 January 2023

This article has been updated

Abstract

Insects are the most important global pollinator of crops and play a key role in maintaining the sustainability of natural ecosystems. Insect pollination monitoring and management are therefore essential for improving crop production and food security. Computer vision facilitated pollinator monitoring can intensify data collection over what is feasible using manual approaches. The new data it generates may provide a detailed understanding of insect distributions and facilitate fine-grained analysis sufficient to predict their pollination efficacy and underpin precision pollination. Current computer vision facilitated insect tracking in complex outdoor environments is restricted in spatial coverage and often constrained to a single insect species. This limits its relevance to agriculture. Therefore, in this article we introduce a novel system to facilitate markerless data capture for insect counting, insect motion tracking, behaviour analysis and pollination prediction across large agricultural areas. Our system is comprised of edge computing multi-point video recording, offline automated multi-species insect counting, tracking and behavioural analysis. We implement and test our system on a commercial berry farm to demonstrate its capabilities. Our system successfully tracked four insect varieties, at nine monitoring stations within polytunnels, obtaining an F-score above 0.8 for each variety. The system enabled calculation of key metrics to assess the relative pollination impact of each insect variety. With this technological advancement, detailed, ongoing data collection for precision pollination becomes achievable. This is important to inform growers and apiarists managing crop pollination, as it allows data-driven decisions to be made to improve food production and food security.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

The datasets generated during and analysed during the current study are available at https://doi.org/10.26180/21533760

Code Availability

Code is available through https://github.com/malikaratnayake/Polytrack2.0.

Change history

References

  • Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, & J., Zheng, X. (2016) TensorFlow: A system for large-scale machine learning. Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016 (pp. 265—283).

  • Abdel-Raziq, H. M., Palmer, D. M., Koenig, P. A., Molnar, A. C., & Petersen, K. H. (2021). System design for inferring colony-level pollination activity through miniature bee-mounted sensors. Scientific Reports, 11(1), 1–12.

  • Afonso, M., Fonteijn, H., Fiorentin, F. S., Lensink, D., Mooij, M., Faber, N., & Wehrens, R. (2020). Tomato fruit detection and counting in greenhouses using deep learning. Frontiers in Plant Science, 11, 1759.

    Article  Google Scholar 

  • Aizen, M. A., Garibaldi, L. A., Cunningham, S. A., & Klein, A. M. (2009). How much does agriculture depend on pollinators? Lessons from long-term trends in crop production. Annals of Botany, 103(9), 1579–1588.

    Article  Google Scholar 

  • Amarathunga, D. C. K., Grundy, J., Parry, H., & Dorin, A. (2021). Methods of insect image capture and classification: A systematic literature review. Smart Agricultural Technology, 1, 100023.

    Article  Google Scholar 

  • Aslanpour, M. S., Toosi, A. N., Cicconetti, C., Javadi, B., Sbarski, P., Taibi, D., & Dustdar, S. (2021). Serverless edge computing vision and challenges. Australasian Computer Science Week Multiconference (pp. 1–10).

  • Barreiros, M. O., Dantas, D. O., Silva, L. C. O., Ribeiro, S., & Barros, A. K. (2021). Zebrafish tracking using YOLOv2 and Kalman filter. Scientific Reports, 11(1), 1–14.

    Article  Google Scholar 

  • Batsleer, F., Bonte, D., Dekeukeleire, D., Goossens, S., Poelmans, W., Van der Cruyssen, E., & Vandegehuchte, M. L. (2020). The neglected impact of tracking devices on terrestrial arthropods. Methods in Ecology and Evolution, 11(3), 350–361.

    Article  Google Scholar 

  • Bjerge, K., Mann, H. M., & Høye, T. T. (2021). Real-time insect tracking and monitoring with computer vision and deep learning. Remote Sensing in Ecology and Conservation, 8(3), 315–327.

    Article  Google Scholar 

  • Bjerge, K., Nielsen, J. B., Sepstrup, M. V., Helsing-Nielsen, F., & Høye, T. T. (2021). An automated light trap to monitor moths (lepidoptera) using computer vision-based tracking and deep learning. Sensors, 21(2), 343.

    Article  Google Scholar 

  • Bochkovskiy, A., Wang, C -Y., & Liao, H -Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv preprintarXiv:2004.10934.

  • Branson, K., Robie, A. A., Bender, J., Perona, P., & Dickinson, M. H. (2009). High-throughput ethomics in large groups of drosophila. Nature Methods, 6(6), 451–457.

    Article  Google Scholar 

  • Breeze, T. D., Bailey, A. P., Balcombe, K. G., Brereton, T., Comont, R., & Edwards, M. (2021). Pollinator monitoring more than pays for itself. Journal of Applied Ecology, 58(1), 44–57.

    Article  Google Scholar 

  • Campbell, J., Mummert, L., & Sukthankar, R. (2008). Video monitoring of honey bee colonies at the hive entrance. Visual Observation and Analysis of Animal and Insect Behavior, ICPR, 8, 1–4.

    Google Scholar 

  • Chagnon, M., Gingras, J. . De., & Oliveira, D. (1989). Effect of honey bee (hymenoptera: Apidae) visits on the pollination rate of strawberries. Journal of Economic Entomology, 82(5), 1350–1353.

    Article  Google Scholar 

  • Dennis, R., Shreeve, T., Isaac, N., Roy, D., Hardy, P., Fox, R., & Asher, J. (2006). The effects of visual apparency on bias in butterfly recording and monitoring. Biological Conservation, 128(4), 486–492.

    Article  Google Scholar 

  • FAO (2018). Why bees matter; the importance of bees and other pollinators for food and agriculture. https://www.fao.org/documents/card/en/c/i9527en/

  • Fijen, T. P., Scheper, J. A., Boom, T. M., Janssen, N., Raemakers, I., & Kleijn, D. (2018). Insect pollination is at least as important for marketable crop yield as plant quality in a seed crop. Ecology Letters, 21(11), 1704–1713.

    Article  Google Scholar 

  • Food & Agriculture Organization of the United Nation (2019). Global action on pollination services for sustainable agriculture https://www.fao.org/pollination/en/.

  • Garibaldi, L. A., Carvalheiro, L. G., Vaissière, B. E., Gemmill-Herren, B., Hipólito, J., Freitas, B. M., et al. (2016). Mutually beneficial pollinator diversity and crop yield outcomes in small and large farms. Science, 351(6271), 388–391.

    Article  Google Scholar 

  • Garibaldi, L. A., Requier, F., Rollin, O., & Andersson, G. K. S. (2017). Towards an integrated species and habitat management of crop pollination. Current Opinion in Insect Science, 21, 105–114.

    Article  Google Scholar 

  • Garibaldi, L. A., Sáez, A., Aizen, M. A., Fijen, T., & Bartomeus, I. (2020). Crop pollination management needs flower-visitor monitoring and target values. Journal of Applied Ecology, 57(4), 664–670.

    Article  Google Scholar 

  • Goscinski, W. J., McIntosh, P., Felzmann, U. C., Maksimenko, A., Hall, C. J., Gureyev, T., et al. (2014). The multi-modal Australian sciences imaging and visualization environment (massive) high performance computing infrastructure: applications in neuroscience and neuroinformatics research. Frontiers in Neuroinformatics, 8, 30.

    Article  Google Scholar 

  • Haalck, L., Mangan, M., Webb, B., & Risse, B. (2020). Towards image-based animal tracking in natural environments using a freely moving camera. Journal of Neuroscience Methods, 330, 108455.

    Article  Google Scholar 

  • Hall, M. A., Jones, J., Rocchetti, M., Wright, D., & Rader, R. (2020). Bee visitation and fruit quality in berries under protected cropping vary along the length of polytunnels. Journal of Economic Entomology, 113(3), 1337–1346.

    Article  Google Scholar 

  • Hallmann, C. A., Sorg, M., Jongejans, E., Siepel, H., Hofland, N., & Schwan, H. (2017). More than 75 percent decline over 27 years in total flying insect biomass in protected areas. PLoS One, 12(10), e0185809.

    Article  Google Scholar 

  • Howard, S. R., Nisal Ratnayake, M., Dyer, A. G., Garcia, J. E., & Dorin, A. (2021). Towards precision apiculture: Traditional and technological insect monitoring methods in strawberry and raspberry crop polytunnels tell different pollination stories. PLoS One, 16(5), e0251572.

    Article  Google Scholar 

  • Høye, T. T., Ärje, J., Bjerge, K., Hansen, O. L., Iosifidis, A., Leese, F., Raitoharju, J., et al. (2021). Deep learning and computer vision will transform entomology. Proceedings of the National Academy of Sciences, 118(2), e2002545117.

    Article  Google Scholar 

  • Jolles, J. W. (2021). Broad-scale applications of the raspberry pi: A review and guide for biologists. Methods in Ecology and Evolution, 12(9), 1562–1579.

    Article  Google Scholar 

  • Kamilaris, A., & Prenafeta-Boldú, F. X. (2018). Deep learning in agriculture: A survey. Computers and Electronics in Agriculture, 147, 70–90.

    Article  Google Scholar 

  • Kevan, P. G. (1975). Sun-tracking solar furnaces in high arctic flowers: Significance for pollination and insects. Science, 189(4204), 723–726.

    Article  Google Scholar 

  • Kirkeby, C., Rydhmer, K., Cook, S. M., Strand, A., Torrance, M. T., & Swain, J. L. (2021). Advances in automatic identification of flying insects using optical sensors and machine learning. Scientific Reports, 11(1), 1–8.

    Article  Google Scholar 

  • Koirala, A., Walsh, K. B., Wang, Z., & McCarthy, C. (2019). Deep learning-method overview and review of use for fruit detection and yield estimation. Computers and Electronics in Agriculture, 162, 219–234.

    Article  Google Scholar 

  • Kuhn, H. W. (1955). The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1–2), 83–97.

    Article  MathSciNet  MATH  Google Scholar 

  • Lu, H., Fu, X., Liu, C., & Li, L. (2017). Cultivated land information extraction in uav imagery based on deep convolutional neural network and transfer learning. Journal of Mountain Science, 14(4), 731–741.

    Article  Google Scholar 

  • MacInnis, G., & Forrest, J. R. (2019). Pollination by wild bees yields larger strawberries than pollination by honey bees. Journal of Applied Ecology, 56(4), 824–832.

    Article  Google Scholar 

  • Magnier, B., Gabbay, E., Bougamale, F., Moradi, B., Pfister, F., & Slangen, P. (2019). Multiple honey bees tracking and trajectory modeling. Multimodal Sensing: Technologies and Applications, 11059, 110590Z.

    Google Scholar 

  • Odemer, R. (2022). Approaches, challenges and recent advances in automated bee counting devices: A review. Annals of Applied Biology, 180(1), 73–89.

    Article  Google Scholar 

  • O’Grady, M., Langton, D., & O’Hare, G. (2019). Edge computing: A tractable model for smart agriculture? Artificial Intelligence in Agriculture, 3, 42–51.

    Article  Google Scholar 

  • Outhwaite, C., McCann, P., & Newbold, T. (2022). Agriculture and climate change reshape insect biodiversity worldwide. Nature, 605(7908), 97–102.

    Article  Google Scholar 

  • Pérez-Escudero, A., Vicente-Page, J., Hinz, R. C., De Arganda, S., & Polavieja, G. G. (2014). idtracker: Tracking individuals in a group by automatic identification of unmarked animals. Nature Methods, 11(7), 743–748.

    Article  Google Scholar 

  • Potts, S. G., Imperatriz-Fonseca, V., Ngo, H. T., Aizen, M. A., Biesmeijer, J. C., & Breeze, T. D. (2016). Safeguarding pollinators and their values to human well-being. Nature, 540(7632), 220–229.

    Article  Google Scholar 

  • Rader, R., Bartomeus, I., Garibaldi, L. A., Garratt, M. P., Howlett, B. G., & Winfree, R. (2016). Non-bee insects are important contributors to global crop pollination. Proceedings of the National Academy of Sciences, 113(1), 146–151.

    Article  Google Scholar 

  • Ratnayake, M.N., Dyer, A.G., Dorin, A. (2021a). Towards computer vision and deep learning facilitated pollination monitoring for agriculture. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2921–2930).

  • Ratnayake, M. N., Dyer, A. G., & Dorin, A. (2021). Tracking individual honeybees among wildflower clusters with computer vision-facilitated pollinator monitoring. PLoS One, 16(2), e0239504.

    Article  Google Scholar 

  • Real, L. (2012). Pollination Biology. Elsevier.

    Google Scholar 

  • Redmon, J., & Farhadi, A. (2017). Yolo9000: better, faster, stronger. Proceedings of the IEEE conference on computer vision and pattern recognition Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7263–7271).

  • Rollin, O., & Garibaldi, L. A. (2019). Impacts of honeybee density on crop yield: A meta-analysis. Journal of Applied Ecology, 56(5), 1152–1163.

    Article  Google Scholar 

  • Schweiger, O., Biesmeijer, J. C., Bommarco, R., Hickler, T., Hulme, P. E., & Klotz, S. (2010). Multiple stressors on biotic interactions: How climate change and alien species interact to affect pollination. Biological Reviews, 85(4), 777–795.

    Google Scholar 

  • Sekachev, B., Manovich, N., & Zhavoronkov, A. (2019). Computer vision annotation tool. Zenodo. GitHub: https://github.com/opencv/cvathttps://doi.org/10.5281/zenodo.3497106.

  • Settele, J., Bishop, J., & Potts, S. G. (2016). Climate change impacts on pollination. Nature Plants, 2(7), 1–3.

    Article  Google Scholar 

  • Simons, D. J., & Chabris, C. F. (1999). Gorillas in our midst sustained inattentional blindness for dynamic events. Perception, 28(9), 1059–1074.

    Article  Google Scholar 

  • Spaethe, J., Tautz, J., & Chittka, L. (2001). Visual constraints in foraging bumblebees: Flower size and color affect search time and flight behavior. Proceedings of the National Academy of Sciences, 98(7), 3898–3903.

    Article  Google Scholar 

  • Spencer, E. E., Barton, P. S., Ripple, W. J., & Newsome, T. M. (2020). Invasive European wasps alter scavenging dynamics around carrion. Food Webs, 24, e00144.

    Article  Google Scholar 

  • Stojnić, V., Risojević, V., Muštra, M., Jovanović, V., Filipi, J., Kezić, N., & Babić, Z. (2021). A method for detection of small moving objects in UAV videos. Remote Sensing, 13(4), 653.

    Article  Google Scholar 

  • Su, D., Kong, H., Qiao, Y., & Sukkarieh, S. (2021). Data augmentation for deep learning based semantic segmentation and crop-weed classification in agricultural robotics. Computers and Electronics in Agriculture, 190, 106418.

    Article  Google Scholar 

  • Vanbergen, A. J., Initiative, I. P., et al. (2013). Threats to an ecosystem service: Pressures on pollinators. Frontiers in Ecology and the Environment, 11(5), 251–259.

    Article  Google Scholar 

  • van der Kooi, C. J., Kevan, P. G., & Koski, M. H. (2019). The thermal ecology of flowers. Annals of Botany, 124(3), 343–353.

    Article  Google Scholar 

  • Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., & Belongie, S. (2018). The inaturalist species classification and detection dataset. Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8769–8778).

  • Walter, T., & Couzin, I. D. (2021). Trex, a fast multi-animal tracking system with markerless identification, and 2D estimation of posture and visual fields. eLife, 10, e64000.

    Article  Google Scholar 

  • Wang, S., Liu, W., Wu, J., Cao, L., Meng, Q., Kennedy, P.J. (2016). Training deep neural networks on imbalanced data sets. 2016 international joint conference on neural networks (ijcnn) (pp. 4368–4374).

  • Wood, T. J., Michez, D., Paxton, R. J., Drossart, M., Neumann, P., & Gerard, M. (2020). Managed honey bees as a radar for wild bee decline? Apidologie, 51(6), 1100–1116.

  • Yang, C., Collins, J., & Beckerleg, M. (2018). A model for pollen measurement using video monitoring of honey bees. Sensing and Imaging, 19(1), 1–29.

  • Zivkovic, Z., & Van Der Heijden, F. (2006). Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recognition Letters, 27(7), 773–780.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Sunny Ridge Australia for the opportunity to conduct research at their farm.

Funding

Authors were supported by the Australian Research Council Discovery Projects grant DP160100161 and Monash-Bosch AgTech Launchpad primer Grant. This study was funded by AgriFutures grant PRJ-012993. Amarathunga is supported by ARC Research Hub IH180100002.

Author information

Authors and Affiliations

Authors

Contributions

MNR, AGD, AD: conceptualization; MNR: data curation; MNR: formal analysis; AGD, AD: funding acquisition; MNR, DCA, AZ: investigation; MNR, AGD, AD: Methodology; AGD, AD: project administration; AGD, AD: resources; MNR: software; AGD, AD: supervision; MNR, DCA: validation; MNR: writing—original draft; MNR, DCA, AZ, AGD, AD: writing—review & editing.

Corresponding author

Correspondence to Malika Nisal Ratnayake.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Ethics approval

Not applicable.

Additional information

Communicated by Angjoo Kanazawa.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 6206 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ratnayake, M.N., Amarathunga, D.C., Zaman, A. et al. Spatial Monitoring and Insect Behavioural Analysis Using Computer Vision for Precision Pollination. Int J Comput Vis 131, 591–606 (2023). https://doi.org/10.1007/s11263-022-01715-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-022-01715-4

Keywords

Navigation