Reverse Video Search

You may have used some kind of reverse image search before. Put simply, instead of searching using text: australian shepherds running, you can use an image: australian_shepherd_running.png. The search engine will then find all similar images based on that input.

But have you used reverse video search? The approach is the same: use your video as a query to find other videos.

💻 Embedding Models Used: https://docs.mixpeek.com/overview/models/available-models

📓 Features Extracted: https://docs.mixpeek.com/ingestion/extractors

🧑🏻‍🏫 Multimodal University: https://mixpeek.com/learn

Lets first explore reverse image search

Try it on Google Images: https://images.google.com/

In the example below, I'll upload a picture of an Australian Shepherd dog, and Google's reverse image search will find all similar pictures of Australian Shepherds.

Use cases for reverse image search

There's tons of awesome use cases for reverse image like:

• E-commerce: Helps customers find products by uploading images, increasing sales by simplifying the shopping experience.
• Intellectual Property: Identifies unauthorized use of images, aiding in copyright enforcement and protecting creators' rights.
• Content Verification: Verifies the authenticity of images in news and social media, combating misinformation.
• Real Estate: Allows users to find properties by uploading photos, enhancing user experience and engagement.

Image Feature Extraction

To perform a search we need to extract features from the image. Below, we're just leaving the default options, but you can go crazy with how many features you can pull out

import requests

url = "https://api.mixpeek.com/ingest/images/url"

payload = {
    "url": "https://www.akc.org/wp-content/uploads/2017/11/Australian-Shepherd.1.jpg",
    "collection": "sample_dogs",
    "feature_extractors": {
        "embed": [
            {
                "type": "url",
                "embedding_model": "image"
            }
        ]
    }
}
headers = {  
  'Authorization': 'Bearer API_KEY', # removed after for brevity
  "Content-Type": "application/json"
}

response = requests.request("POST", url, json=payload, headers=headers)

print(response.text)

Ingest Image Url - Mixpeek

Mixpeek

Reverse Image Search

import requests

url = "https://api.mixpeek.com/features/search"

payload = {
    "queries": [
        {
            "type": "url",
            "value": "https://www.akc.org/wp-content/uploads/2017/11/Australian-Shepherd.1.jpg",
            "vector_index": "image"
        },
    ],
    "collections": ["sample_dogs"]
}

Search Features - Mixpeek

This endpoint allows you to search features.

Mixpeek

But what about video?

Reverse video search works the same way. We first embed a couple videos, then provide a sample video as a search.

For our index, we'll use a movie trailer from the 1940s classic, The Third Man:

Prepare the video(s)

We'll split the video up by 5 secton intervals, then embed each interval using the multimodal embedding model. We'll also pull out a description from each interval.

import requests
import json

url = "https://api.mixpeek.com/ingest/videos/url"

payload = json.dumps({
  "url": "https://mixpeek-public-demo.s3.us-east-2.amazonaws.com/media-analysis/The+Third+Man++Official+Trailer.mp4",
  "collection": "my_video_collection",
  "feature_extractors": [
    {
      "interval_sec": 5,
      "describe": {
        "enabled": True
      },
      "embed": [
        {
          "type": "url",
          "embedding_model": "multimodal"
        }
      ]
    }
  ]
})

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Embed the video to search and run!

Now we have a grainy video clip from some CCTV that we'll use for our reverse video search:

We'll do the same thing, only difference is we'll want the embedding from the video we want to search across the already indexed and embedded videos:

import requests

url = "https://api.mixpeek.com/features/search"

payload = {
    "queries": [
        {
            "type": "url",
            "value": "https://mixpeek-public-demo.s3.us-east-2.amazonaws.com/media-analysis/video_queries/exiting_sewer.mp4",
            "vector_index": "multimodal",
        },
    ],
    "collections": ["my_video_collection"],
}

response = requests.request("POST", url, json=payload, headers=headers)

print(response.text)

This will return an object that contains the key: embedding

Compare results

Now that we have our embeddings we can run a KNN search:

This will return an array of objects that we can use to render in our application indicating what the most similar video timestamps are based on the video embedding as a query

results = [
    {"start_time": 25.0, "end_time": 30.0, "score": 0.6265061},
    {"start_time": 5.0, "end_time": 10.0, "score": 0.6025797},
    {"start_time": 30.0, "end_time": 35.0, "score": 0.59880114},
]

Now if we look at the original video @ 25 seconds in:

Amazing, we found a challenging scene to describe using a video query as an input. Now imagine doing that across billions of videos 🤯

Using this template, we set it so that whenever a new object is added to our S3 bucket it's automatically processed and inserted into our database (connection established prior). Additionally, if a video is ever deleted from our S3 bucket its' embeddings are deleted from our database as well.

Use cases for video search

1. Content Creation: Enables creators to find specific video clips quickly, streamlining the editing process.
2. Media Monitoring: Identifies reused video content across platforms, aiding in tracking content spread and copyright enforcement.
3. E-commerce: Helps customers find products by uploading video snippets, enhancing the shopping experience.
4. Security and Surveillance: Analyzes footage to detect specific events or objects, improving security measures.