Google Cloud Video Intelligence API - Object Tracking in Python Example [Single Video]

One of the reasons why I’ve started learning Python is due to API integrations with Google Cloud Platform.

As part of GCP, Video Intelligence API is quite interesting as it lets, among other features, track objects within a video and annotates the timestamp for these objects. This can be really helpful as it can help the process of shot selection quicker as you would know where to look for.

Note: This blog post uses GCP Video Intelligence API documentation, Jie Jenn’s helpful YT channel in framing the initial part and help from StackOverflow.

This page provides a summary of Video Intelligence, in general. While this page explains the difference between AutoML and Video Intelligence API. When you use the former, you can train a custom model for labels, objects + use the UI in GCP. With the API mode, you get to access the pre-trained model with access to 20,000 objects and you access it via Python [among other platforms].

The below script can be summarized as:

Import libraries for google cloud and video-intelligence.

Use assigned service credentials JSON key for the API call.

Use OBJECT_TRACKING method in the features to detect objects, timestamps and bounding boxes.

Note: Video Intelligence is only supported (so far) for videos uploaded to a Google Cloud Storage bucket [folder] from where the API can read the video and do the object tracking. AutoML also requires videos to be in GCS. Hence, the input_uri and output_uri locations are a GCP object id in an existing bucket.

For example, I downloaded this adidas video from YouTube and uploaded it to GCS bucket: https://www.youtube.com/watch?v=V_ozu_7Xweg

Once the job is completed, you should be able to see the response in PyCharm and the response.json file is also saved in your GCS bucket.

This object: Helicopter - shows in segment 0 sec to 4.8 sec. Let’s check!

Pretty cool for a standard API, without any custom model training.

You can now download the JSON from GCS bucket and decide if you want to convert the JSON to CSV/XLSX OR open the file as is in Data Studio. This can be done using the SuperMetrics Custom JSON connector.

Blog post covering Object Tracking Multiple Videos Within A Bucket - Python.