Wildbook AI: Using Computer Vision and Machine Learning to Discover Whale Shark Encounters on YouTube

The proliferation of mobile phones and GoPros has created a wealth of video and photographic material ripe for mining scientific data. At Wild Me, we have extended our open-source wildlife conservation software, Wildbook, to... [ view full abstract ]

The proliferation of mobile phones and GoPros has created a wealth of video and photographic material ripe for mining scientific data. At Wild Me, we have extended our open-source wildlife conservation software, Wildbook, to take advantage of this by autonomously searching social platforms for animal sightings.

Our pilot project runs on Wildbook for Whale Sharks (whaleshark.org), using YouTube as a source.  Nightly, our automated process, “Wildbook AI”, uses YouTube’s API to search for videos, then uses custom computer vision software to detect the presence of whale sharks in the video frames.  Next, we use natural language processing to derive a date and location of the sighting.  We then use our pattern matching algorithm to match the individual against our database of nearly 9000 whale sharks. Finally, the software follows up by notifying the creator of the video of these results by leaving a comment on the source YouTube video.

This initial trial has already surpassed the next most prolific (human) researcher in number of sightings collected by a factor of 3-to-1, discovering upward of 150 encounters each month.  We are testing extending this method to other social platforms, as well as other species.