DeepLabCut AI Residency 2024 Recap: working with the SuperAnimal-Bird Model and DLC-3.0🔥 Live 🦜
Curious about what we have been working on this summer? 🌞 Our 2024 DLC AI Residency team, supported by Shaokai Ye, PhD student🦄, in the Mathis Lab of Adaptive Intelligence, and core-dlc dev. software engineer wizard Niels Poulsen🪄, took on an exciting challenge with the residents: developing & testing the SuperAnimal-Bird model (aka “SuperBird🐦”), a pre-trained model for 2D bird pose estimation that aims to enhance how we study avian behavior.
The Journey of SuperBird 🐦
Shaokai Ye rightly noticed that people love birds, and for a bit of a fun extension to his collaborative work on SuperAnimal models (just published in Nature Communications), he started collecting open-source data of birds! But, there is always a ton of checks and prep work in training such SuperModels… in come the 2024 Residents!
The residency kicked off by building a data analyzer that is able to visualizing COCO-format poses and bounding boxes and plotting annotation statistics. All the code they developed is open-sourced here! Then the team analyzed and cleaned multiple bird datasets. The analysis step is critical as it helps estimate the strength and weakness of the produced model, paving the path to training the SuperBird model.
The team then moved on to training this model with Shaokai and Niels using a top-down ResNet50 architecture and a SSDLite detector. The result? A promising new model designed to handle the complexities of bird pose estimation with pretty good precision and reliability!! 🐣🐧🦆 The model achieves 87.9 mAP on the IID test set (yay)!
But it wasn’t just about building the model in DLC3.0🔥 — residents also tackled the challenge of learning to merge diverse datasets 🗂️. From single-bird images to groups of birds in cages, the team worked to create a unified dataset that could support robust model training “The Bird60K Dataset”. The project also highlighted the importance of key point standardization across datasets, ensuring that the SuperBird model would be versatile enough to handle various types of bird images.
The Bird60K Dataset: A Feathered Collection 🐦
To build the SuperBird model, we combined data from four diverse, open-source bird datasets:
- Penn Aviary Dataset: 864 images Penn Aviary
- Nabirds: 48,000 images across 400 species Nabirds
- CUB-200–2011: 11,788 images of 200 species CUB-200–2011
- Pigeons: 1,151 labeled frames Pigeon Supermodel
Altogether, these datasets total around 60,000 images with 42 keypoints standardized across them.
The Science Behind the SuperBird 🔬
With over 11,000 bird species worldwide, developing a universal bird model poses unique challenges. Birds vary greatly in appearance and behavior, and no single dataset is comprehensive enough for training modern neural networks. By merging multiple datasets and following the SuperAnimal approach, we unified keypoints into a superset, creating a comprehensive bird dataset. The gradient masking algorithm was used to train the model and video adaptation techniques further enhanced it, reducing jitter in video predictions and paving the way for a versatile tool in avian behavior research.
Benchmarking in DLC 3.0 🖥️
The residents didn’t stop at just training the model. They conducted a thorough benchmarking process within the new DLC 3.0 framework, comparing TensorFlow and PyTorch architectures💪. They made a new demo video to guide you through the DeepLabCut 3.0 GUI🌟:
Real-Time Bird Tracking with DLC Live 3.0 🎥
One of the major highlights of the residency was adapting DLC Live to the new PyTorch backend. The updated DLC Live now supports the SuperBird model, enabling researchers to perform real-time pose estimation on bird videos. Whether working with static images or live video streams, this tool is set to be a game-changer for anyone studying avian behavior.
The team even experimented🔬 with various species. These efforts not only demonstrated the versatility of DLC Live 3.0 but also set the stage for future developments in real-time animal tracking.
Looking Ahead 🌍
As we wrapped up this residency for 2024, the SuperBird model will soon take its place in the DLC Model Zoo 🏞️, providing powerful tools🛠 for researchers worldwide. The DLC Live 3.0 adaptation is also set to enable more options for real-time tracking in animal behavior research. The work doesn’t stop here — stay tuned for the official release of SuperBird, and get ready to explore new frontiers in animal behavior research with DeepLabCut🐭! 🌟
