The First DeepLabCut AI Residency Program — 2022 Recap 🤩

7 min readSep 15, 2022

Written by Timoklia Kousi, DLC Community Manager & Research Scientist

Are you 🔥passionate about open source software? Do you want to empower and inspire 💪groups underrepresented in computer science? Do you want to be part of the next-gen leaders in behavioral analysis? We do too❗

🎥This is our story…

DeepLabCut AI Residency Program was established in 2022 in EPFL, Campus Biotech by the lead developers of DeepLabCut 🐭 Mackenzie Mathis and Alexander Mathis, with the generous support of the CZI DEI Program, aiming to democratize AI-based behavioral analysis in life sciences.

We selected a diverse team of scientists who flew to Switzerland in July 2022 to take part in an intense, hacking mission to modify code, develop new add-ons, and become scientists within the DeepLabCut consortium.

✨Meet the 2022 AI Residents:

🎯 What drew you to this field? What do you want to achieve?

Neslihan: Working with birds was kind of destiny to me. I was constantly feeding birds, primarily crows and sparrows, every morning in my window. From feeding them, I started now to understand why they behave the way they behave. I was not planning on working specifically with pigeons, but I got interested in the work of my Ph.D. supervisor, and I applied to work in his lab.

My current research goal is to develop an open-source system for automated behavioral analysis of pigeons that will be accessible to everyone. I hope that such a tool will decrease the working hours needed to label the experimental data and reduce the bias introduced by annotations produced by different people. As a result, the work will be reproducible. In addition, precise and efficient data analysis will decrease the necessary amount of data to achieve solid results and, as a result, will reduce the number of animals needed in experiments.

I received a lot of support from the open source community, especially during the confinement, and this project will be my contribution and an expression of gratitude.

I have come to realize that many researchers don’t know how to use open source tools for their work, and also, many software engineers struggle to reach the right audience to put their work to good use. So I aim to become the bridge between these two communities.

Nirel: During my bachelor’s studies, I participated in many hackathons developing tools for protecting nature, more specifically, coral reef biodiversity. Now, I aim to find my voice in science, be happy and engage in meaningful work. I also plan to begin my start-up adventures.

I want to continue supporting under-resourced communities in Kenya, bring tools that can operate in this context, empower women and help to bridge the gap between women and men.

Jonas: Bats are mysterious, majestic, and goth, so if you are drawn to those things then, well, every now and then, someone falls in love with bats.

When I was contemplating whether or not to do a PhD on bats, I was reading Batman Year One (1986), and there is a scene where batman is also reflecting on how to move forward when he remembers being frightened by bats and says to himself “yes, father, I shall become a bat”, and I also thought that yes, I will become a bat. Probably, I would have done it anyway, but this felt like the tipping point.

In the future, I want to continue working with bats and maybe go into conservation. Ultimately, I want to work in a museum and combine research with a more public facing role.

Sofia: I have just finished my PhD and will soon publish my results on hawks’ bird gaze strategies. My topic stems from my passion for conservation and behavior, and I enjoy the computational aspects and work on software for research.

I want to specialize in computer vision for motion tracking and pose estimation. I plan to shift my interest into developing software and open-source tools for research to support scientists in improving their practices and making their results more reproducible. There is not enough guidance currently on how to do this. I hope it will help the research community share their results better and reach wider audiences.

Sabrina: I am investigating how memory works through recording neural activity in mice while observing and analyzing their behavior. Through my work, I hope to understand the role of hippocampus better and apply this knowledge in treating people with memory deficit pathologies like dementia.

I am fresh in the research world, building my research identity and developing my research goals. I am sure that I want to contribute to the open source community and develop tools accessible to everyone.

Vic Shao-Chih: The neuroscience of sexual Diversity is a topic neglected in neuroscience research. It has been an existential question for me as a gay, cisgender male. What does it mean to be sexually diverse? Also, I wanted to empower marginalized minorities. So when I applied for graduate school, I was thinking ‘How can I help?’.

Currently, I am working with hybrid mice to show the independence between sexual behavior and the hormonal cycle and with fish specie to explore sexual conduct in a collective behavior context. Also, I am looking at the neuroscience of sexual behavior in wildlife, a novel concept as it has not been studied before.

🚧 What projects did you work on during the DLC residency program 2022?

🔝Data Augmentations

The team applied and experimented with different augmentation methods to increase the models’ accuracy.

Knowing that fieldwork can be challenging, and the footage is not always high quality (for example, blurry images, dark images, and low-resolution images), it is essential to identify clever ways to improve models’ performance under these circumstances. The traditional way dictates increasing the amount of labeled data used to train the model, which is time-consuming. Instead, the team identified several augmentations that can achieve the same result without increasing the training dataset.

🔝Improving the DLC Napari Plug-in

The residents added a new feature in the DLC Napari plug-in, which can scan the predictions produced by the DLC models and detect outliers (possible mistakes) that the user can identify, correct and retrain the model to improve future results.

🔝Combining MegaDetector and DLC live

The team combined MegaDetector and DLC in one application powered by Hugging Face.

MegaDetector is an open-source model that detects animals, humans, and vehicles in camera trap images. Combining it with DLC, it produces a powerful tool that can automatically identify images with animals in large datasets and apply tracking points for behavioral analysis and pose estimation.

Example: test out the new App!

🔝Tutorials

The team developed guidance, documentation, and tutorials to help current and future users understand and use the tools in their research.

🆒 What is the coolest thing about DeepLabCut and DeepLabCut Residency Program 2022?

✨Accessibility. It is a free, open-source tool that decreases economic barriers and can be learned and used by anyone.

✨Trans-disciplinary approach and teamwork. In academia, people often work by themselves and feel isolated. Here we worked collaboratively. Apart from our personal goals, we had team goals. Our team consisted of people with different backgrounds and expertise, which helped to bring a diverse perspective. If you had questions, someone would be there to help.

✨ Diversity in the field of AI. It tries to have more representation in AI and helps underrepresented people become more familiar with AI.

✨Efficiency. DLC is an automated, markerless pose estimation tool that has radically decreased the amount of annotated data needed for an experiment, thus reducing the working hours spent on this task.

✨Reduction of the risk of bias. Labeling behavioral data by many researchers could introduce various types of bias, which can be avoided when the annotation happens automatically by a computer vision system.

✨Applicability. It can easily be applied in analyzing large amounts of data, making it ideal for researching wildlife, conservation, and fieldwork.