science & animal breeding
Citation for published version (APA):
Roma Barge, L. (2021). The design of the Insight pipeline for behavioral animal science & animal breeding.
Technische Universiteit Eindhoven.
Document status and date:
Published: 07/10/2021
Document Version:
Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers)
Please check the document version of this publication:
• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website.
• The final author version and the galley proof are versions of the publication after peer review.
• The final published version features the final layout of the paper including the volume, issue and page numbers.
Link to publication
General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.
• Users may download and print one copy of any publication from the public portal for the purpose of private study or research.
• You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal.
If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement:
www.tue.nl/taverne Take down policy
If you believe that this document breaches copyright please contact us at:
openaccess@tue.nl
providing details and we will investigate your claim.
Download date: 19. Sep. 2022
PDEng SOFTWARE TECHNOLOGY
The design of the Insight pipeline
for behavioral animal science & animal breeding
A subset of the Discovery Informatics Platform Luis Roma Barge
October 2021
Department of Mathematics & Computer Science
The design of the Insight pipeline
for behavioral animal science & animal breeding
A subset of the Discovery Informatics Platform
Luis Roma Barge October 2021
Eindhoven University of Technology Stan Ackermans Institiute – Software Technology
PDEng Report: PDEng report; 2021/078
Confidentiality Status:
Public
Partners
IMAGEN program: “Integrating be- havioral dynamics and social genetic effects to improve the health, welfare, and ecological footprint of livestock”.
Eindhoven University of Technology
Steering Group
Prof. Jakob de Vlieg Dr. Yanja Dajsuren, PDEng Dr. Rogier Brussee
Date October 2021
Composition of the Thesis Evaluation Committee:
Chair: Prof. dr. Jakob de Vlieg Members: Dr. Yanja Dajsuren, PDEng
Dr. Rogier Brussee Paul van Zoggel, MA Dr. Eleni Constantinou
The design that is described in this report has been carried out in accordance with the rules of the TU/e Code of Scientific Conduct.
Contact Address
Eindhoven University of Technology
Department of Mathematics and Computer Science
MF 5.072, P.O. Box 513, NL-5600 MB, Eindhoven, The Netherlands +31 402743908
Partnership This project was supported by Eindhoven University of Technology and NWO.
Published by Eindhoven University of Technology Stan Ackermans Institiute
PDEng-report PDEng report 2021/078 Preferred
reference
The design of the Insight pipeline for behavioral animal science & animal breed- ing: A subset of the Discovery Informatics Platform. Eindhoven University of Technology, PDEng Report 2021/078, October 2021
Abstract Aggressive animal social behavior costs millions to the livestock industry and af- fects the welfare of the farm animals. Understanding animal behavior is a time- consuming task difficult to perform in large groups.
IMAGEN project uses computer vision technology to automatically detect and track animals in a video. Insight provides a data pipeline to further analyze this AI-generated data in order to obtain useful knowledge.
Thanks to Insight behavioral researchers can obtain useful results such as social networks or activity diagrams on demand, improving the quality of their analysis.
Keywords Livestock, Welfare, Computer Vision, Software Architecture, Behavior analysis Disclaimer
Endorsement
Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the Eindhoven University of Technology or NWO. The views and opinions of authors expressed herein do not necessarily state or reflect those of the Eindhoven University of Technology or NWO and shall not be used for advertising or product endorsement purposes.
Disclaimer Liability
While every effort will be made to ensure that the information contained within this report is accurate and up to date, Eindhoven University of Technology makes no warranty, representation or undertaking whether expressed or implied, nor does it assume any legal liability, whether direct or indirect, or responsibility for the accuracy, completeness, or usefulness of any information.
Trademarks Product and company names mentioned herein may be trademarks and/or service marks of their respective owners. We use these names without any particular en- dorsement or with the intent to infringe the copyright of the respective owners.
Copyright Copyright © 2021. Eindhoven University of Technology. All rights reserved.
No part of the material protected by this copyright notice may be reproduced, mod- ified, or redistributed in any form or by any means, electronic or mechanical, in- cluding photocopying, recording, or by any information storage or retrieval sys- tem, without the prior written permission of the Eindhoven University of Technol- ogy and NWO.
i
Foreword
Meeting the growing global demand for high-quality protein while reducing the ecological footprint of protein production is undoubtedly one of the biggest challenges in the world. Novel protein sources must be identified that can meet ambitious sustainability goals. One of the opportunities and directions is the transition to more vegetable and/or insect-based proteins. However, demand for high-quality an- imal-based protein will continue to exist and is likely to grow as a larger share of the world population can afford meat. Animal agriculture and farm animal breeders have an essential contribution to a healthy and sustainable food supply chain. To keep the sector future-proof, it must work with more precision thereby improving its efficiency and reducing its ecological footprint. Fortunately, this is possible by benefitting more from several disruptive technology developments, e.g., in the field of genomics, com- puter-vision, multi-sensing technologies, and artificial intelligence (A.I.). However, applying genomics, multi-sensing, and computer vision technologies means creating huge amounts of complex and hetero- geneous data. Data that needs to be stored, connected, and analyzed to create new scientific Insights and innovations to improve sustainable food production, animal welfare and ultimately create new busi- ness models.
PDEng project is part of IMAGEN program
This data science PDEng project is part of the NWO-TTW funded IMAGEN program: “Integrating behavioral dynamics and social genetic effects to improve the health, welfare and ecological footprint of livestock”. IMAGEN brings research into animal behavior together with computer science to im- prove the health and welfare of pigs and laying hens and to reduce the ecological footprint of the food production chain. The interdisciplinary IMAGEN program is a collaboration of Wageningen University and Research, Utrecht University, and Eindhoven University of Technology, together with Dutch breed- ing and technology companies, organizations of farmers and veterinarians, and the Dutch organization for the protection of animals.
A.I. to automatically detect the behavior of individual animals in large groups
The ultimate goal of Imagen is an A.I.-based system to automatically detect individual behaviors in large groups of pigs and laying hens. For example, the final system should be able to automatically detect actors and victims of anti-social behaviors such as feather pecking and biting. Data from this A.I.
system will be integrated with genomics data to identify animals that are genetically superior with re- spect to social behavior. By using these individuals as the parents of the next generation, natural genetic variation in social behaviors can be exploited for sustainable genetic improvement of populations of pigs and laying hens.
Data science and computer vision for super social animals
Powerful discovery informatics platforms will be critical to identify new reliable relationships between the observable characteristics of an individual animal (its phenotype, e.g., obtained from livestock video streams) on the one hand, and on the other hand, the collection of its genes (genotype). In this PDEng project, Luis successfully builds further on the digital platform designed by Manu Agarwal in 2020 in the PDEng project: “SmartTurkeys - A Digital Platform for Behavioral Phenotyping”. The original software engineering and data design was based on FAIR (Findable, Accessible, Interoperable, and Reusable) computer science principles and layered microservice architecture. This is to manage and (semi)automatically connect and integrate heterogeneous data sources from several sensing experiments and/or live video-stream / computer vision observations. The underlying data fusion architecture allows both, future use of automated data analysis based on machine learning, or statistical models, and the use of visualizations or dashboards for human-based detection of individual health and behavior pheno- types. An important design requirement of the integrated data fusion and data analytics approach is the
possibility to manage and analyze live video streams of individual animals in barns to identify real-time behavior and to identify subsequent actions or a chain of actions.
Data pipelines building bridges across disciplines
In this project, Luis made important progress by developing powerful and creative data pipelines and analysis plots to support data-driven animal breeding research and animal behavioral science & welfare studies. Data pipelines to analyze livestock video recordings, e.g., by visualizing the movement trajec- tory of the animals and identifying specific social contacts between animals. The data pipelines and analysis plots are playing an important role to better utilize and analyze the complex data sets and video streams but are also important to bring different disciplines & scientists together. In fact, playing an essential role to “link data and minds” and formulating new follow-up interdisciplinary research ques- tions together. The current PDEng project was focused on developing the foundations for data manage- ment and processing huge amounts of video stream data. However, to realize the ultimate goal of IMAGEN (A.I. algorithms able to automatically detect the behavior of individual animals in large groups) novel data science and A.I. will be required.
Advancing to the next data science project phase: self-supervised learning for computer vision
The current analysis and data-driven predictions in Imagen are dependent on obtaining high-quality labeled data. Obtaining the labeled data set is an expensive and time-consuming task requiring human inspection and extensive animal behavior knowledge. A future research topic in this data science project is to explore the use of self-supervised learning methods for computer vision. This is based on the fast developments in A.I. algorithms and high-performance computing to allow the use of also non-labeled data sets. For example, developments of new evaluation methods in reinforcement learning that don’t require guidance and/or previous human expertise. Self-supervised Learning is an unsupervised learn- ing method where the supervised learning task is created by combining two important steps: supervised learning based on high quality labeled data (step 1) and the use of output obtained in step 1 to bootstrap self/unsupervised learning techniques to use unlabeled data (step 2). The digital architecture designed by Manu Agarwal and the data pipeline approach implemented by Luis is important first stages to real- ize step 1. Combining complex data sets, high-quality experimental design to produce sufficient labeled data based on relevant research questions, and cleansing and correcting data errors in systematic ways are crucial pre-processes for step 1. It opens the way for contrastive predictive coding and the develop- ment of reliable loss functions to train the current models automatically in step 2. Developing and using self-supervised learning techniques and/or other advanced A.I. breakthrough technologies will clearly be essential to achieve the final goal of IMAGEN.
Excellent team science & team worker
Luis, you have done a great job, especially considering the difficult circumstances you had to work in due to the corona measures. Despite these measurements, you build bridges between disciplines and colleagues. You are an independent and problem-solving PDEng student. A good presenter and able to explain your data science tools and methods in the context of complex interdisciplinary research ques- tions. You regularly visited the animal behavior scientists in Utrecht and Wageningen to better under- stand the demand and the scientific field, and you had several conversations with colleagues from TopigsNorvin and other companies on ICT systems and data-driven animal breeding, all of which was greatly appreciated.
In conclusion: Supervising you is a pleasure; thank you for all your work and achievements!
Prof. dr. Jakob de Vlieg
Chair of the Applied Data Science (ADS) research group Lead AgrifoodTech at TU/e and JADS
iii
v
Preface
This document is the main deliverable of The design of the Insight pipeline for behavioral animal sci- ence & animal breeding project. It describes the process of designing and implementing Insight, one of the processing pipelines of the Discovery informatics Platform (DIP).
DIP is a platform to store, share, and analyze animal data to be used in behavioral and phenotypic research. Insight helps the user to analyze AI-generated animal data. The main source of animal data during this project was in form of livestock video recordings.
When analyzing a scene of the animal recordings, some use cases for Insight include:
• Measuring the distance traveled by the animals
• Analyzing the movement trajectory of the animals
• Studying social contacts of pairs of animals
• Analyzing the movement and velocity of the animals, among other significative parameters
This project was carried out by the trainee Luis Roma Barge as part of his ten-month Software Tech- nology Professional Doctorate in Engineering (PDEng) graduation project. The project was carried out within the Applied Data Science (ADS) group of the Mathematics and Computer Science department (TU/e) and the Dutch Research Council (Nederlandse Organisatie voor Weternschappelijk NWO [1]
in Dutch).
The target audience of this document includes people with a technical background in computer science and animal behavior.
Eindhoven, October 7th, 2021 Luis Roma Barge
vii
Acknowledgments
I would like to express my deep gratitude to everyone who supported me during this project. The guid- ance, help, ideas, and support that I received were essential for completing this project.
I want to thank my supervisor Prof. Jakob de Vlieg. Your enthusiasm and energy were contagious throughout the project. Thanks to you, I was able to work on such an important topic, meet interesting people, and learn about the animal sciences. Your feedback and ideas were essential for successfully completing this PDEng project. Your ability to connect people and topics is a core skill for the progress of science, and I was lucky to learn it from you on this multidisciplinary project.
I want to express my most deep gratitude to my academic supervisor and the program director of the PDEng Software Technology program Dr. Yanja Dajsuren. You deposited your trust in me by accepting me on this program (that completely changed my life), encouraging me to give the best every day. You are an inspiration in management, trust, support, and professionalism. We all were lucky to have such an amazing director leading us. I enjoyed our weekly meetings and your guidance was key to find my path.
Manu Agarwal, I was enormously lucky to find an advisor, supporter, and most importantly, a friend. I was amazed by every one of your presentations. The stories you tell are crystal clear and the meetings you preside always conclude with the key topics discussed. Your communication skills inspired me.
You were always there for a chat or some guidance and I truly appreciate it.
I am grateful to Dr. Rogier Brussee. Your brainstorm ideas and suggestions shaped some of the deci- sions of this project. I enjoyed our conversations where you advised me on how to understand others and be understood by others better. Your feedback on key moments of the project was superb. Thanks, Rogier.
I want to thank the IMAGEN team for being such a charming company during this project. I met so many interesting people from diverse backgrounds working together for a better future for the animals.
Even though I could not meet you in real life until the last weeks of the project, I felt part of the team from the totality of it. I want to say my special thanks to the people with whom I had the pleasure to work closely. Joy, you always were there for me, answering all my questions and providing me your friendship and support. You are an amazing researcher, and your future is bright. Arjen, who taught me the secrets of holding and understanding a chicken.
There are some people not involved in the project that had a great impact on my development throughout these two years. I am talking about my PDEng family. Starting with Desiree, who made the Netherlands feel like home from day one. Our PD coaches, Kyril, Linda, Andre, and Peter, helped me to develop in ways I could never imagine. Judith helping us to become better professional writers. And, of course, my colleagues, who made these two years some of the best of my life.
I want to thank my parents, Jose and Isabel, my brother Gabriel, my godmother Elena, and my grand- mother Clara, who always believe in and support me. Thanks for teaching me the ways of love, hard work, and compassion. It is thanks to you that I am the person I am today.
And my most heartfelt gratitude goes to Cristina. You are the reason everything makes sense. I am a better person thanks to you. You inspire me, understand me, support me, and love me. You are all I need. I would follow you to the Netherlands and the end of the world.
Luis Roma Barge October 2021
ix In the loving memory of Marila
xi
Executive Summary
Even in the best conditions, livestock animals live in crowded farms and suffer stress. In most cases, this stress causes the animals to hurt fellow animals as an instinctive response. Additionally, this nega- tive behavior is contagious; if one animal starts doing it, others will follow. These stress-caused reac- tions have a negative impact not only on the welfare of the animals but also on the economy of the farm.
Until now, the only way of monitoring behavior was manual (i.e., actively observing the animals). This task results in difficult and tedious work considering that the average farm counts hundreds (or thou- sands) of animals. Reacting on time to stop negative behavior and minimize losses has proven challeng- ing to animal breeders.
This report describes the development of Insight, a subset of the umbrella project IMAGEN [2], funded by NWO and carried out by a consortium of Dutch universities and companies.
The IMAGEN program aims to automatically study, analyze, and recognize animal behavior by using Computer Vision technology and, at a later stage, link this behavioral information to specific genes of individual animals. Cameras have been set up in multiple livestock farms. These cameras are used to collect animal data (videos). These videos contain animals living on farms and how they behave in a social group. These videos are used to produce AI models that automatically detect and track individual animals.
Insight is a data pipeline that analyzes the output of AI algorithms to help researchers conduct animal behavior research. By analyzing a specific time period of a video of animals, these pipelines produce the following data plots:
•
Trajectory analysis•
Distance traveled by each animal (used to calculate feeding efficiency)•
Proximity encounters within a specific radius•
Basic animal state identification (Passive, Active, and Very Active)•
Social network based on proximity and animal state•
Recorded states of an individual animal over timeThese plots have been validated by animal researchers from different backgrounds and experts from the industry, all of them part of IMAGEN. These plots have been proven to be a helpful tool to quickly obtain measures of key information such as distance traveled and proximity between animals. The pipe- lines were developed considering adaptability and extensibility so including new types of data (e.g., RFID data [3], feeding station data) or new kinds of plots is a simple task.
We recommend that IMAGEN integrate Insight in their future data platform and include new animal and farm data (e.g., meteorological data).
Glossary
AI Artificial Inteligence
DL Deep Learning
CV Computer vision
RFID Radio-frequency identification
NWO Nederlandse Organisatie voor Wetenschappelijk
Onderzoek
MOT Multi-Object Tracking
xiii
Table of Contents
Foreword ... i
Preface ... v
Acknowledgments ... vii
Executive Summary ... xi
Glossary ... xii
Table of Contents ... xiii
List of Figures ... xv
List of Tables ... xvi
1. Introduction ... 1
1.1 The Need for Studying Animal Group Behaviour and Genetics ... 1
1.2 The NWO IMAGEN program ... 1
1.3 Insight ... 2
1.4 Goal and Objectives of the Project ... 2
1.5 Report Outline ... 2
2. Stakeholder Analysis ... 5
2.1 Stakeholders by Organization ... 5
2.1.1. Eindhoven University of Technology (TU/e) ... 5
2.1.2. Wageningen University (WUR) ... 6
2.1.3. Utrecht University (UU) ... 6
2.1.4. Topigs Norsvin (TN) ... 7
2.1.5. Other organizations ... 7
3. Problem Analysis ... 9
3.1 Animal behavior and genetics ... 9
3.2 Reference Architecture and Insight Pipeline ... 9
3.3 Data sources ... 10
3.4 Computer vision ... 12
3.5 Problem definition ... 15
3.6 Assumptions ... 16
3.7 Constraints ... 16
4. Requirements Analysis ... 19
4.1 Use cases ... 19
4.2 Requirements ... 19
5. Insight Architecture ... 21
5.1 Design Challenges ... 21 5.1.1. Functional challenges ... 21 5.1.2. Non-functional challenges ... 21 5.2 System Design ... 22 5.3 Prototype Design ... 23 5.3.1. Insight components ... 24 5.3.2. Insight implementation ... 24 5.3.3. Insight evaluation... 26 6. Case Study ... 28 7. Conclusions and future work ... 33
8. Project Management ... 34 8.1 Way of working ... 34 8.2 Planning ... 34 8.3 Risk management ... 35 8.4 Retrospective ... 36 Bibliography ... 37 A AdditionalData sources ... 39 B Project Management ... 40 C Using Insight on a subset of animals ... 41 D Platform Requirements (PR) ... 43 About the Author ... 44
xv
List of Figures
Figure 1. SmartTurkeys DPBP platform. Source: SmartTurkeys – A Digital Platform for Behavioral Phenotyping (DPBP) ... 10 Figure 2 Pen with pigs from the Volmer farm ... 11 Figure 3 Chicken pen from Utrecht farm (top view) ... 11 Figure 4 How the bounding boxes are measured ... 13 Figure 5 Example of YOLO output. Headers have been added to improve readability ... 14 Figure 6 Example of MOT output. Headers have been added to improve readability ... 14 Figure 7 Combined results of YOLO and MOT algorithms ... 15 Figure 8. Overview of the system. Insight pipelines and output are marked in blue ... 22 Figure 9 High-level overview of Insight pipeline ... 23 Figure 10. Class diagram of Insight pipeline ... 23 Figure 11. Configuration file. The user has chosen to obtain a Graph plot with all the animals. The animals to be analyzed are pigs and the searching radius is 100 pixels ... 25 Figure 12 The distance between two animals is calculated as Ecludien distance ... 26 Figure 13. Example of a frame... 28 Figure 14. Trajectory plot of pigs in a pen. Red dots are the starting points and black stars are the ending points. ... 29 Figure 15. Distance traveled per animal. ... 29 Figure 16. Heatmap plot representing the proximity between pair of pigs ... 30 Figure 17. Social network plot on pig data. The proximity radius is 80 pixels ... 31 Figure 18 Multiple State Timeline plots. These plots represent how the state of a pig changes during the scene. The available states are eating, stting, and moving ... 32 Figure 19. Planning of the project ... 35 Figure 20 Example of feeding station data ... 39 Figure 21 Configuration file with a subset of animals ... 41 Figure 22 Trajectories for pigs 1,3,5, and 7 ... 42 Figure 23 Distance traveled for pigs 1,3, 5, and 7 ... 42
List of Tables
Table 1: List of TU/e stakeholders ... 5 Table 2: List of WUR stakeholders... 6 Table 3 List of UU stakeholders ... 6 Table 4 List of TN stakeholders ... 7 Table 5 An example of a risk identified during the project. ... 35 Table 6 Complete table of risk mitigation plans ... 40
xvii
1
1.Introduction
The project The design of the Insight pipeline for behavioral animal science & animal breeding was offered to Luis Roma Barge as his graduation project, as a part of the Software Technology Professional Doctorate in Engineering program. This program is offered by the Department of Mathematics and Computer Science of the Eindhoven University of Technology in the context of the 4TU.School for Technological Design, Stan Ackerman’s Institute as part of the IMAGEN project.
1.1 The Need for Studying Animal Group Behaviour and Genetics
In the past decades, the human population has grown exponentially. The number of people living on Earth doubled from roughly 2 billion in 1930 to 4 billion in 1974 and doubled again with 7.8 billion in 2020 [4] [5]. The projected human population in 2050 is 10 to 12 billion [6] . With so many mouths to feed, food production must equally grow to satisfy human demand.
Animal livestock farms pose a challenge in keeping up with this increasing demand (horizontal growth).
These facilities are already densely populated, with the average farm having thousands of animals. In addition to the ever-increasing amount of food and water needed to feed the animals, space is also a problem. Even in the farms with the best conditions, these animals live in crowded spaces with several animals per square meter. This lack of space negatively affects the animals; they cannot establish the social hierarchies found in the wild [7], which leads to animals suffering stress and harmful behavior appearing.
In the case of chickens, they may be pecking at each other [8], causing injuries and suffering on the victim. In rare situations, chickens may opt to group against a wall or a corner, stacking on top of other chickens causing suffocation and even death. In the case of the pigs, they will bite each others’ tails [9], causing injuries that can produce infections. All these cases impact the health of the animals, the quality of the products, and the economy of the farm. In the past, these problems were fixed with harmful solutions such as removing chickens’ beaks or cutting off pigs’ tails. Current efforts aim to improve animals’ health in a less harmful way.
If the number of animals on a farm cannot be increased by increasing their living space, the solution is to improve the quality of the animals (vertical growth). For this, multiple worldwide known companies such as Topigs Norsvin [10] or Hendrix Genetics [11] research the way of producing (breeding) animals that respond better to living in large groups by studying their genes. One of the main objectives of these companies is to link phenotype information (the genetics of the animal) with the behavior of the animal.
Unfortunately, analyzing the behavior of a single animal within a group of thousands is difficult and tedious.
1.2 The NWO IMAGEN program
The IMAGEN program is a collaboration of Wageningen University and Research [12], Eindhoven University of Technology [13], and Utrecht University [14], together with Hendrix Genetics, Topigs Norsvin, and several other stakeholders. The program is funded by NWO-TTW.
The goal of IMAGEN is to analyze animal behavior in large groups. IMAGEN usesdata obtained from multiple types of sensors to better understand animal social behavior and link it to individual genes. In particular, IMAGEN uses Computer Vision (CV) [15] techniques to automatically identify and track individual animals in large groups. This allows to build up enough data for doing large scale to do statistics and analysis.
The program has three working packages. These are the following:
• WP1 – Generic sensing, AI, and data science technologies. This package focuses on devel- oping the generic elements of sensing, AI, and data technologies for the automated detection of traits in groups of livestock. WP1 will last for four years.
• WP2 – Behavioral interaction in pigs. This package focuses on the behavioral interactions in pigs, including tail-biting, mounting, aggression, and social-support behaviors. WP2 will last for five years, exceeding WP1 by one year.
• WP3 – Behavioral interaction in laying hens. This package focuses on the behavioral inter- actions in laying hens, including pecking behavior, smothering, and collective dustbathing.
Similar to WP3, it will last for five years.
The animals are provided by Topigs Norsvin (pigs) and Hendrix Genetics (chickens), for their interest in improving the genetics of their animals (see Section 3.3 ). These animals are recorded and analyzed by researchers from Topigs Norvin, Utrecht University, and Wageningen University. Additionally, Eindhoven University of Technology uses the data to produce AI algorithms that aim to identify and track animals and detect specific behavior.
This PDEng project belongs to WP1 but it had a tight collaboration with members of WP2 and WP3.
1.3 Insight
This PDEng project’s contribution to IMAGEN is the design and creation of the “Insight” data pipeline.
This pipeline will be used by all of the members of IMAGEN for behavioral research. Additionally, this pipeline will be integrated into a platform that stores the raw and annotated data and produces analytics from the AI-generated data.
Insight is the data pipeline part of the processing layer of the Discovery Informatics Platform. Insight takes the raw AI-generated data (i.e., files containing bounding boxes of detected and tracked ani- mals) and produces understandable data. This data is generated in the form of reports, analytics, plots, and graphs; It can be further analyzed by behavioral experts. The main goal is to aid animal research- ers in conducting their behavioral and genetic research.
1.4 Goal and Objectives of the Project
The goal of this project is to design a software pipeline capable of transforming complex animal data into useful Insight in order to improve animal welfare. To achieve this goal, we must meet the follow- ing objectives:
•
Design layers that facilitate the storing and sharing of raw data and results (e.g., cataloging and archiving a video stream.)•
Design layers that allow automated AI services to analyze the raw data to a stream of low-level events (e.g identifying an animal and detecting its position or movement.)•
Design pipeline(s) that allow the user to obtain high-level behavioral knowledge from AI-ana- lyzed data (e.g identifying which animals are sociable or aggressive towards which other ani- mals.)1.5 Report Outline
The remaining of the report is organized in the following way. Chapter 2 identifies and describes the project stakeholders. Chapter 3 analyzes the domain and describes the problems to be solved in this
3 project. Chapter 4 is the requirements elicitation. Chapters 5 describes Insight design and implementa- tion. Chapter 6 contains the expected results of applying Insight. Chapter 7 contains the conclusions of this project. Finally, Chapter 8 describes the way this project was managed.
5
2.Stakeholder Analysis
The purpose of this section is to analyze and identify the people involved in the project. Their roles, stakes, and concerns are stated here. They are grouped based on their participation, influence, and in- terest in the project.
As explained in Section 1, this PDEng project was part of the IMAGEN project, which is divided into three working packages (Technology project, Pig project, and Chicken project). Since the goal of the project is to extend a platform that will be used by all the members of IMAGEN, all of them can be considered direct or indirect stakeholders. In this chapter, only the ones that were direct stakeholders are listed.
2.1 Stakeholders by Organization
Even though several organizations are involved in the NWO IMAGEN project, not all of them were directly connected to this project. Four organizations took a primary stakeholder role in this project.
These organizations were Eindhoven University of Technology (TU/e), Wageningen University (WUR), Topigs Norsvin (TN), and Utrecht University (UU.)
2.1.1. Eindhoven University of Technology (TU/e)
TU/e collaborates with the IMAGEN project with two groups. The first group, the Applied Data Science group, is led by Professor Jakob de Vlieg who is also the project owner of this project. Dr. Rogier Brusse is also part of this group. In the context of the PDEng project, they act as company supervisors.
The second group, the Electrical Engineering group, is lead by Professor Peter de With and carries all the activities related to Computer Vision. The Post-Doctoral student, Joy Sue, is one of the main stake- holders of this project.
Table 1: List of TU/e stakeholders Yanja Dajsuren
Project Role TU/e Supervisor and Software Technology Pdeng Program Direction Interests • Project success
• Project deliverables (prototype(s), final report) comply with the PDEng’s quality standards
• Provide technical (design, architecting) and non-technical (communica- tion, planning) feedback
Jakob de Vlieg
Project Role Project Owner, Company Supervisor, and Chair of Applied Data Science Group Interests • ADS group providing useful guidance to NWO IMAGEN project with
regards to data solutions
• Insight pipeline helping animal researchers
• Provide feedback throughout the project Rogier Brusse
Project Role Company Supervisor
Interest • Provide feedback throughout the project Manu Agarwal
Project Role PhD Student
Interests • Architecture realization (i.e., the continuation of his work as former PDEng)
• Integrating Insight as a general-use pipeline for other projects (e.g., SmartTurkeys)
Yue (Joy) Sun
Project Role Post-Doctoral Student
Interests • Animal data available to be used for generating DL models
• Obtaining interesting research output for IMAGEN and her research group.
• Helping her Ph.D. students 2.1.2. Wageningen University (WUR)
WUR is the leader of Animal Research in the Netherlands. Their contribution to the IMAGEN project is expertise in animal behavior and they are focused on the Pig Project. In the context of this PDEng project, WUR is one of the major external stakeholders. Their opinion and interest were considered at all stages of the project. They provided feedback during the prototype, implementation, and validation phases.
They provided animal behavior expertise used to increase the value of the Insight tool.
The leader of the IMAGEN project, Peter Bijma, belongs to this organization. The researchers belong- ing to this organization conduct the Pig Project.
Table 2: List of WUR stakeholders Peter Bijma
Project Role Program Leader of IMAGEN project Interests • IMAGEN project success
• Research Output valuable for WUR
• Behavior linked to genetics in animals Liesbeth Bolhuis
Project Role Project Leader of the Pig Project Interests • IMAGEN project success
• Research Output valuable for WUR
• Behavior linked to genetics in animals
2.1.3. Utrecht University (UU)
The researchers belonging to this organization conduct the Chicken Project. These researchers take care of the chickens provided by Hendrix Genetics. They record the chicken data and conduct behavioral research on the chickens.
Table 3 List of UU stakeholders Arjen van Putten
Project Role Ph.D. student
Interests • Chicken data available for everyone in IMAGEN
• Conducting animal behavior research Maeva Manet
Project Role Ph.D. student
Interests • Conducting animal behavior research Saskia Kliphuis
Project Role Ph.D. student
7 Interests • Conducting animal behavior research
2.1.4. Topigs Norsvin (TN)
Topigs Norsvin is a company that specializes in pig genetics and breeding. They also have agreements with farms in Germany, the USA, and Canada. These farms provide the data information for the Pig Project.
Table 4 List of TN stakeholders Egbert Knol
Project Role Research Director of Topigs Norsvin Interests • Research Output valuable for TN
• TN economic profit
2.1.5. Other organizations
In addition to the aforementioned organizations, more organizations are part of IMAGEN. These or- ganizations are the following: Hendrix Genetics, Noldus Information Technology, Vencomatic, So- rama, Farm Result, Dutch Society for the Protection of Animals, LTO, KNMvD.
Regarding this PDEng project, they had minor influence but some of them were involved in several User Committee meetings where they provided feedback on Insight.
9
3. Problem Analysis
This chapter analyzes the application domain where this project was carried out, i.e., animal behavior and computer vision. I start by describing the problem that IMAGEN wants to solve. I continue by describing the animal data involved in this project and a brief explanation of the computer vision algo- rithms used in this project. Finally, I explain the assumptions and the constraints.
3.1 Animal behavior and genetics
The goal of the IMAGEN project is to automatically detect, analyze, and understand animal social be- havior using computer vision. Before any AI algorithm can be used, animal behavior needs to be de- scribed in a detectable way. Animal behavior can be classified as positive or negative behavior.
Positive behavior is commonly found in nature and has a positive effect on animals’ health. For exam- ple, chickens tend to groom their feathers when they are relaxed or happy [12]. This action further improves the animal's mental state. Sometimes, one chicken grooming will motivate other chickens in the flock to groom, causing the full group to groom.
Negative behavior is linked to stress or a lack of the natural social hierarchies found in the wild. One of the negative behaviors that is interesting for researchers and breeders is the aggression between animals.
Some parameters to consider regarding aggressions are the duration, the intensity, whether there were previous attacks, the attacker, and the victim.
Breeding companies study and produce animal breeding lines as their main product. Their current way of facing animal aggressions is reacting when the aggression is done. The aggressor is (normally) sep- arated from the group and the victim, in case of surviving with serious damage, is culled. Instead of operating with these losses, breeding companies are interested in linking behavior to specific genes so they can breed individuals with better genes. These individuals would have better resilience to external stimuli and produce less negative behavior.
3.2 Reference Architecture and Insight Pipeline
This project is a continuation of SmartTurkeys - A Digital Platform for Behavioral Phenotyping (DPBP) [13] project, carried out by the PDEng Manu Agarwal for his final project in the PDEng in Software Technology. The main goal of the project was to define the architecture of a digital platform that would be used for the behavioral phenotyping of turkeys. This architecture is based on the layered architectural pattern and it is technology agnostic. A depiction of this architecture can be found in Figure 1.
Figure 1. SmartTurkeys DPBP platform. Source: SmartTurkeys – A Digital Platform for Behav- ioral Phenotyping (DPBP)
In this figure, a Multilayer Service Architecture [14] is shown. The ingestion layer is used to accept the data from different data sources. This layer is connected to storage layers, which can be fast or slow, and it is also connected to a metadata layer. The fast storage layer is conceived for immediate usage (e.g., processing the data contained on it) while the slow storage layer is conceived for permanent re- tention (e.g., long-term storage). The metadata layer provides statistics and ingestion status. The pro- cessing layer is where all the analytics, data quality checks, and data manipulation
jobs are performed.
Due to the Applied Data Science group collaboration with both SmartTurkeys and IMAGEN, this plat- form was proposed to also be used in IMAGEN where Manu Agarwal continues his research as a Ph.D.
student. Additionally, the Insight pipeline will be also integrated into SmartTurkeys.
3.3 Data sources
The main sources of the data are video recordings of the animals. In addition, there are data sources such as RFID data, feeding station data, and climatic data.
For the pig project, the data is produced on a farm in Volmer1, Germany. A total of nine pens for this farm are used for this project. For each pig pen, there are 10 to 11 animals and two cameras. An example of a pig pen can be found in Figure 2. In this figure, there are 11 pigs and one feeding station on the top left. The cameras record 24 hours per day on a 720p quality.
1 This farm collaborates with Topigs Norvin, but it is not owned by them. TN is allowed to place cameras and record live footage of the pens in this farm.
11 Figure 2 Pen with pigs from the Volmer farm
For the chicken project, the data is produced in the Faculty Farm of Animal Health in Utrecht, The Netherlands. At the beginning of the project, there were four chicken pens. Each pen has ten chickens and two cameras, one for the top view and the other for the side view. In May, the number of pens was increased to 24, with the final count being 48 cameras and more than 200 chickens. These cameras record 24 hours per day. An example of a chicken pen can be found in Figure 3. In this figure, there are 9 chickens, a feeding station (yellow circle in the middle), a drinking station (bucket top left), two poles where the chickens sleep (right side on top of the shelf), and a covered area where the chickens lay eggs (underneath the shelf on the right).
Figure 3 Chicken pen from Utrecht farm (top view)
Besides the video data, there are other types of data. On the pig farm, there are sensors in the feeding stations. These sensors are connected to the electronic tag of the pigs and they can measure the differ- ence in weight of the pig, before and after the food ingestion. An example of this data can be found in Appendix A. IMAGEN plans to integrate these measurements into its platform. In the pig farm, there are other sensors such as humidity and temperature sensors. The recording of these sensing data is being discussed but they are not being recorded in IMAGEN at the time of writing.
On the chicken farm, each chicken pen has an RFID sensor This sensor is a cable situated in the middle of the rectangular pen, dividing it into two halves, on the long side. This sensor is triggered every time a chicken steps over it and identifies the chicken with an electronic tag on its body. RFID data will be incorporated in the platform to be used jointly the computer vision algorithms to identify individual animals.
3.4 Computer vision
In order to obtain detection and tracking algorithms, a set of steps must be followed. These steps nor- mally imply dividing the dataset into training and testing sets, annotating the training set, training the model, and finally testing it. This is an iterative process that concludes once the model is accurate enough. This section includes a more detailed explanation of the steps and their implication for this project
Annotation
Before training any algorithm, the data must be annotated [15]. A deep learning model needs annotated data with the features that it will learn. This is a time-consuming process that consists of manually drawing boxes around the animals on most of the frames of the video. For doing this more efficiently, Computer Vision Annotation Tool (CVAT) was provided by ADS. For each video, frames are obtained with a frame step of 20 (one out of every 20 frames is considered for the annotation.) In a one-minute video over 100 frames are obtained. Each of these frames contain animals that need to be annotated. On average, a five-minute video takes five hours to annotate2.
Once the annotation is done, the CVAT produces one file for each annotated frame of the video. On each file, there is the number of objects (i.e., animals) and also the positions of the drawn bounding boxes. These positions are defined as the distance from the top left corner of the frame to the top left corner of the box, in pixels. The width and the height of the box are also measured in pixels. A graphical representation of this is shown in Figure 4.
2 Based on the work of IMAGEN’s computer vision team.
13 Figure 4 How the bounding boxes are measured
In Figure 4, bb_top represents the vertical distance between corners and bb_left the horizontal distance.
The variables bb_h and bb_w represent height and width respectively.
Training
With the annotated scenes ready, the model can be trained. In this case, the deep learning algorithm used is YOLOv3 [16]. The resulting model can detect the kind of objects (i.e., pigs or chickens) that it was trained with. This model is then fed with new animal data (i.e., data not used to train the model) and it detects the animals within this data.
AI-generated output
The results are structured data files (.csv style) that contain the coordinates of the bounding box that encapsulates the animal in the frame. An example of this file can be seen in Figure 5. In this figure, the first column represents the ID of the object, and the next four columns are the same quantities explained in Figure 4. In this case, the IDs of the animals are set as zero. This is because YOLO only detects objects, but it cannot track or identify them (i.e., assign to them a unique ID).
Figure 5 Example of YOLO output. Headers have been added to improve readability
Due to this, an additional algorithm is needed for tracking the animals along with the scene. A Multi- Object Tracking (MOT) [17] algorithm is used to keep temporal information between frames. This algorithm takes a group of files (such as the ones shown in Figure 5) and assigns an individual identifi- cation number to each animal (row). These IDs are consistent along the sequence of frames. The output of this algorithm is a structured data file as shown in Figure 6. In this figure, the first column represents the frame, the second column represents the ID of the animals and the next four elements represent the bounding boxes. The rest of the numbers are for 3D cases and are not used for this project.
Figure 6 Example of MOT output. Headers have been added to improve readability
If the bounding boxes are plotted on top of a frame (an image) of the video, the graphical results of the detection are appreciated. Each of the detected objects (i.e., the pigs) is surrounded by bounding boxes at the positions and with the dimensions set by the file. The results image is shown in Figure 7.
15 Figure 7 Combined results of YOLO and MOT algorithms
One of the main problems with detection algorithms is that they can not perfectly keep temporal infor- mation of the identified animals, resulting in ID switches (e.g., an animal identified with ID 1 may be identified with ID 7 in the next frame). This is the reason why training these models is an iterative process. Tuning the hyperparameters and the structure of the model may help to improve this situation.
3.5 Problem definition
With the increasing demand for food, livestock farms can not keep up with the pace if they want to maintain minimal space conditions that satisfy the well-being of the animals. Even if the surface of the farm is increased there is a density limit (number of animals placed in the same pen) before the sanity and health of the animals is compromised. These animals negatively affected by its crowded environ- ment incur an economic loss for the farmers. Due to this, researchers and farmers are interested in understanding what causes specific animal behavior and prevent it if it is negative.
Most of the time animal social behavior is hard to observe, even for the expert eye. The analysis of this behavior comes from manual work. The situation gets worse with a bigger number of animals in the group. To add to this, analyzing animal behavior is a complex and qualitative task. Behavior needs to be observed and also put into context. A simple situation such as a pig approaching his nose to the tail of another pig could imply biting (aggression, negative) or sniffing (social, positive). This is why con- text, and more important, temporal information, needs to be considered for behavioral analysis.
Breeding companies such as Topigs Norvin have as their product specific breeding lines of animals.
These breeding lines are characterized by the genes of their individuals. For these companies, it is key to understand and link the social behavior of individuals with the genes of their line. Individuals with negative behavior can incur losses of up to 4% of the animals of the same line3. These breeding com- panies would be interested in obtaining a breed of animal that is less negatively affected by crowded space and more inclined to have positive social interactions on these conditions.
The added value of the IMAGEN program is the automatic detection of this social behavior and later on the connection to specific genes.
3 Statistic obtained from Egbert Knol (Topigs Norsvin)
Since the development of AI algorithms that detect behavior is a long process, it is useful for animal researchers to have mechanisms of processing data and generating usable information. In order to gen- erate this information, the following questions must be answered:
• What features (detectable by an IA model) determine social behavior?
• What information is obtainable from AI-generated data?
• Which of this information is relevant for behavioral research?
• How to separate relevant information from non-relevant information?
Insight aims to answer these questions by introducing a pipeline between AI algorithms and researchers.
AI-generated files are processed by Insight and transformed into analytics. Even though Insight cannot detect complex behavior (e.g., tail biting, feather pecking) it can detect simple behavior and patterns (e.g., crowding, animals spending time together). This information is graphically presented to the re- searchers so that they can focus on the details. Insight acts as a filter (a large amount of data) to narrow the search of behavior (only focus on scenes prefetched by Insight).
3.6 Assumptions
During the initial analysis of the problem, several assumptions were made. These assumptions were discussed and verified with the relevant stakeholders. The list of assumptions was updated iteratively during the design and prototype phase. The curated final list of assumptions is the following:
1. The main users of the platform will be researchers (AI researchers and animal behavior re- searchers) and breeders (genetic companies, farmers).
2. AI researchers will develop detection and tracking algorithm. This algorithm will produce a data file (.txt, .csv,…) from each video.
3. These data files have information related to the animals such as identification tags and coordi- nates. These data files contain the trackings of the animals.
4. These data files will aid in the behavioral research (e.g., finding if aggression between animals happens, identifying aggressor and victim.)
5. Behavioral researchers will use these files to study the behavior of the animals.
6. The AI modules will be trained in local machines of the universities. Each research group has different dependencies and they believe it would be too difficult to have a general model for different groups.
7. The researchers are interested in sharing only the recent data on the cloud.
8. The researchers are interested in archiving the rest of the data for the whole project, in partic- ular all the videos.
9. All the produced data must be shareable, identifiable, and (to some extent) searchable.
10. A microservice layered architecture (such as the one proposed by Manu Agarwal) is the best kind of architecture for this type of platform.
3.7 Constraints
This project was executed under several constraints. These constraints are the following:
1. The duration of the PDEng project was ten months including analysis, designing, implementa- tion, and documentation. PDEng-related activities (such as PDEng defense, thesis writing, comeback days) were also included in this period.
2. The Pig Project generates ~32 Gb of video data per day.
3. The Chicken Project generates ~1TB of video data per day.
4. Storing all the generated data would incur a massive price. Therefore, only the relevant data will be stored.
5. The definition of relevant data was discussed by the relevant stakeholders.
17 6. The prototype of the Insight pipeline must be demonstrated to WUR researchers by the end of
April.
7. The prototype of the Insight pipeline must be demonstrated to UU researchers by the end of June.
8. The final version of the Insight pipeline must be demonstrated to the whole IMAGEN group by the end of July.
19
4.Requirements Analysis
This chapter shows the Uses Cases and the Requirements for the data platform and the Insight pipeline.
In the case of the platform, a qualitative study was carried out as a continuation of the SmartTurkeys project adapted to the necessities of IMAGEN. In the case of Insight, the Requirements have been cu- rated from brainstorming sessions, discussions, and agreements with the stakeholders.
4.1 Use cases
The following actors of the system have been identified:
• Data provider is a person or system that produces and stores animal data.
• AI researcher is a person who uses raw data to generate AI models that predict and track ani- mals.
• Behavioral researcher is a person who uses the predictions generated by the AI model to con- duct animal behavior research.
The following use cases of the platform were identified:
• UC 1 – Store data. This use case describes any type of data stored in the platform. These storage types can be raw data generated by the specific sensor, annotated data by AI researchers, or AI- generated data.
• UC 2 – Data manipulation. This use case describes the aspects of data preprocessing. This preprocessing can refer to the movement, renaming, and copying of data.
• UC 3 – Provide data. This use case describes the aspects of data retrieval. This data retrieval can be raw data generated by the specific sensor, annotated data by AI researchers, or AI-gen- erated data.
• UC 4 – Analyse AI data. This use case describes how a behavioral researcher can use the Insight pipeline to obtain analytics from previously generated AI data.
The following use cases of the Insight pipeline were identified:
• UC 5 – Obtain animal trajectory. This use case describes how a behavioral researcher can use the Insight pipeline to obtain the trajectory of the animals. This trajectory can be obtained for a single animal or the whole pack.
• UC 6 – Obtain traveled distance plot. This use case describes how a behavioral researcher can use the Insight pipeline to obtain the total distance traveled by the animals in the selected scene.
• UC 7 – Obtain proximity heatmap. This use case describes how a behavioral researcher can use the Insight pipeline to obtain a heatmap plot that describes one-on-one interactions between animals. This plot describes during how many frames two animals are closer to a certain thresh- old to each other.
• UC 8 – Obtain a social network graph. This use case describes how a behavioral researcher can use the Insight pipeline to obtain a social network graph. This plot describes the overall state of the animals and the interaction between them.
4.2 Requirements
These are requirements based on the MoSCoW method (Must, Should, Could, Won’t). This list of re- quirements was generated from conversations between the main stakeholders, the Project Manager, and the Project Owner.
During the project, two lists of requirements were elicited. The first list is related to the platform for IMAGEN. This list of Platform Requirements (PR) contains high-level requirements results of the
analysis of the first weeks of the project. IMAGEN members can use this list as a future reference for the further development of their platform. This list can be found in Appendix D. The second list contains the requirements related to Insight. These are the Insight Requirements (IR) that Insight needs to comply with to guarantee an end product that satisfies the demands of the key stakeholders.
Insight Requirements (IR#)
Req.# Priority Description
IR1 Must The Insight pipeline shall accept AI-generated data as input.
IR1.1 Could The Insight pipeline shall accept heterogeneous data as input.
IR2 Must The Insight pipeline shall analyze the input data.
IR3 Must The Insight pipeline shall produce reports or plots of processed ani- mal data.
IR3.1 Must The Insight pipeline shall produce trajectory plots.
IR3.2 Must The Insight pipeline shall produce proximity plots.
IR3.3 Could The Insight pipeline shall produce relevant statistical reports.
IR3.4 Must The Insight pipeline shall produce basic animal state (passive, ac- tive, very active) detection and definition.
IR4 Must The Insight pipeline shall be usable. Its usage shall be intuitive for non-technical people.
IR5 Must The Insight pipeline shall be scalable.
IR5.1 Must The Insight pipeline shall be usable for at least ten researchers at the same time.
IR5.2 Must The Insight pipeline shall be able to analyze at least 50 animals in a scene.
IR6 Must The Insight pipeline shall be extendable.
IR6.1 Must The Insight pipeline shall be extendable with new types of plots.
IR6.2 Must The Insight pipeline shall be extendable with new types of data.
IR6.3 Must The Insight pipeline shall be extendable with new types of animals.
IR7 Must The Insight pipeline shall be portable.
IR8 Must The Insight pipeline shall be reliable.
IR8.1 Must The Insight pipeline shall obtain the expected results at least 95% of the time.
21
5.Insight Architecture
System architecture can be defined as the designing of a set of guidelines for a product in order to comply with a set of requirements. System (or Software) architecture and requirements complement and justify each other with the architecture explaining how a feature is achieved and requirements set- ting what feature needs to exist.
As explained in Section 4.2 , the requirements elicitation was an iterative task during this project. In a project with so many moving parts and multiple teams working on it, a new update could require the revisit and review of the requirements. Due to the research nature of the project, an iterative prototype- based design was chosen. Since the results needed to be shared and validated with multiple teams, a Rapid Prototyping Approach [18] was followed.
I start this chapter by describing the design challenges that motivated the design. After that, I briefly describe the architecture of the system that contains Insight. Lastly, I explain Insight design, implemen- tation, and evaluation.
5.1 Design Challenges
During the design phase of Insight, a set of challenges were encountered. These challenges are related to the requirements of the project and they influenced the design choices. These were either the most important features or the hardest-to-tackle problems.
The design challenges are divided into functional or non-functional challenges, depending on if they are related to functional or non-functional requirements respectively.
5.1.1. Functional challenges
These challenges are related to the functional requirements. These requirements define basic system behavior; they are features that allow the system to function as intended.
In the case of Insight, the main functional design challenge was the sequential processing of the data.
From the first three Insight Requirements (IR1, IR2, and IR3) it is stated that the system must accept data, analyze it, and produce reports and/or plots. These actions must be strictly executed in order. First, the data must be read from the source. After that, the data must be analyzed. This analysis can change depending on the situation and the demands of the user. Lastly, the output will be produced depending on the analysis.
This challenge presented the perfect opportunity to use the software pipeline design pattern [19]. This design pattern allows to build and execute a sequence of operations.
5.1.2. Non-functional challenges
While the functional requirements defined what the system should do, the non-functional requirements define how the system should do it. Even if these requirements were not met, the system would perform its basic purpose. These requirements are important because they improve usability. In the case of In- sight, they are related to extensibility and scalability.
Extensibility
This design challenge is based on the Insight Requirement related to extensibility (IR6, IR6.1, IR6.2, IR6.3). Being a research project, some decisions such as the type of data, the type of animals, or the
plots generated may need to be changed or extended in the future. Insight was designed considering this to make it simple to extend any new functionality needed.
Scalability
This design challenge is based on the Insight Requirement related to extensibility (IR5, IR5.1, IR5.2).
Since the number of animals can change based on the pen, the system must be able to handle a variable number of animals. The performance of the system and the quality of the outputs should not be greatly influenced by a large number of individuals in the input. If the pipeline produces an output that is more demanding in resources of memory (e.g., a video as output) it should ensure that the creation and de- struction of objects are properly managed so that no memory leaks happen. Additionally, computations that only require individual animal data could be executed in parallel, improving the performance of the system (if that would be required).
Lastly, Insight is conceived to be run in the processing layer of a platform used by multiple team mem- bers at the same time (PR1, IR5.1). The pipeline must allow multiple independent executions coming from different users with variable configurations.
5.2 System Design
As mentioned in Section 1.3, Insight is integrated into the Discovery Informatics Platform of the IMAGEN project. In this platform, the animal data obtained by different sources (i.e. the farms) is processed. This data (generally videos) is used to train AI models as well as to validate said models.
The results of this validation process are files containing bounding boxes of the animals. These files are used as input for Insight to produce information such as the trajectories of the animals or a summary of their states.
An overview of this process is shown in Figure 8. In this Figure, the Input Data is divided into Training and Testing. The training data is annotated using the CVAT to produce Annotated videos. These An- notated videos are used for training a DL model. This model and the testing data are used to detect and track the animals. This tracking information is used as input for Insight (in blue) to produce plots.
Figure 8. Overview of the system. Insight pipelines and output are marked in blue
