feat: revamp multimodal search notebook

multimodal-search/Multimodal_Search_with_Cohere_and_Cloud_Inference.ipynb

@@ -0,0 +1,289 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "YIVBTkiDAlfy"
+   },
+   "source": [
+    "### Tutorial\n",
+    "\n",
+    "We will use [**Cohere Embed 4.0**](https://cohere.com/blog/embed-4) through [Qdrant Cloud Inference](https://qdrant.tech/documentation/inference/inference-api/) for generating multimodal embeddings and a [**Qdrant Collection**](qdrant.tech/documentation/manage-data/collections/) for storing and retrieving them.\n",
+    "\n",
+    "> _To follow along with this example, you need a Cohere API key. Create a free one [here](https://dashboard.cohere.com/api-keys)_"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "id": "TrkMVQD0H6Qu"
+   },
+   "outputs": [],
+   "source": [
+    "! pip install -q qdrant-client"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "source": "We will be using a [Qdrant Cloud Free Tier Cluster](/documentation/cloud/create-cluster/#free-clusters).\n\n[Create a free cluster](https://cloud.qdrant.io/), save the associated API key and endpoint URL, and instantiate the Qdrant Client (make sure to set `cloud_inference=True` to enable Cloud Inference):",
+   "metadata": {
+    "id": "-wTBQHkedYIp"
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "Jn6xx48IJf7D"
+   },
+   "outputs": [],
+   "source": [
+    "from qdrant_client import QdrantClient, models\n",
+    "from getpass import getpass\n",
+    "\n",
+    "client = QdrantClient(url=getpass(\"Qdrant URL: \"), api_key=getpass(\"Qdrant API key: \"), cloud_inference=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "vfkXbWRhBbPT"
+   },
+   "source": [
+    "\n",
+    "Let's embed a very short selection of images and their captions in the **shared embedding space**."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "id": "7ZZYV2aqKD5b"
+   },
+   "outputs": [],
+   "source": [
+    "import base64\n",
+    "\n",
+    "def image_to_base64_url(image_path: str) -> str:\n",
+    "  prefix = \"data:image/png;base64\"\n",
+    "  with open(image_path, \"rb\") as image_file:\n",
+    "    return prefix + \",\" + base64.b64encode(image_file.read()).decode(\"utf-8\")\n",
+    "\n",
+    "documents = [\n",
+    "    {\"caption\": \"An image about plane emergency safety.\", \"image\": \"images/image-1.png\"},\n",
+    "    {\"caption\": \"An image about airplane components.\", \"image\": \"images/image-2.png\"},\n",
+    "    {\"caption\": \"An image about COVID safety restrictions.\", \"image\": \"images/image-3.png\"},\n",
+    "    {\"caption\": \"A confidential image about UFO sightings.\", \"image\": \"images/image-4.png\"},\n",
+    "    {\"caption\": \"An image about unusual footprints on Aralar 2011.\", \"image\": \"images/image-5.png\"},\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "yEn3HWTiYl_u"
+   },
+   "source": [
+    "Create a **Collection**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "id": "98oNUlicaiXK"
+   },
+   "outputs": [],
+   "source": [
+    "COLLECTION_NAME = \"multimodal-embeddings\"\n",
+    "\n",
+    "if not client.collection_exists(COLLECTION_NAME):\n",
+    "    client.create_collection(\n",
+    "        collection_name=COLLECTION_NAME,\n",
+    "        vectors_config={\n",
+    "            \"image\": models.VectorParams(size=512, distance=models.Distance.COSINE),\n",
+    "            \"text\": models.VectorParams(size=512, distance=models.Distance.COSINE),\n",
+    "        }\n",
+    "    )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "4vwr7UbRFV5M"
+   },
+   "source": [
+    "Now let's upload our images with captions to the **Collection**. Each image with its caption will be embedded by the Cohere model, [through Cloud Inference](https://qdrant.tech/documentation/inference/external-inference-providers/#cohere), and uploaded, as a [Point](https://qdrant.tech/documentation/concepts/points/), to the collection."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "mpqu-qzbP8Eh"
+   },
+   "outputs": [],
+   "source": [
+    "from qdrant_client.context_headers import headers\n",
+    "\n",
+    "cohere_api_key = getpass(\"Cohere API key: \")\n",
+    "\n",
+    "with headers({\"cohere-api-key\": cohere_api_key}):\n",
+    "  client.upsert(\n",
+    "      collection_name=COLLECTION_NAME,\n",
+    "      points=[\n",
+    "          models.PointStruct(\n",
+    "              id=idx,\n",
+    "              vector={\n",
+    "                  \"text\": models.Document(\n",
+    "                      text=doc[\"caption\"],\n",
+    "                      model=\"cohere/embed-v4.0\",\n",
+    "                      options={\n",
+    "                          \"output_dimension\": 512\n",
+    "                      }\n",
+    "                  ),\n",
+    "                  \"image\": models.Image(\n",
+    "                      image=image_to_base64_url(doc[\"image\"]),\n",
+    "                      model=\"cohere/embed-v4.0\",\n",
+    "                      options={\n",
+    "                          \"output_dimension\": 512\n",
+    "                      }\n",
+    "                  ),\n",
+    "              },\n",
+    "              payload=doc\n",
+    "          )\n",
+    "          for idx, doc in enumerate(documents)\n",
+    "      ]\n",
+    "  )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "pAkj-XFVGm9N"
+   },
+   "source": "Let's see what image we get for the query \"*Plane components*\""
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "DY2PvcZVnZwN"
+   },
+   "outputs": [],
+   "source": [
+    "from PIL import Image\n",
+    "\n",
+    "with headers({\"cohere-api-key\": cohere_api_key}):\n",
+    "  image_path = client.query_points(\n",
+    "      collection_name=COLLECTION_NAME,\n",
+    "      query=models.Document(\n",
+    "          text=\"Plane components\",\n",
+    "          model=\"cohere/embed-v4.0\",\n",
+    "          options={\n",
+    "            \"output_dimension\": 512\n",
+    "          }\n",
+    "      ),\n",
+    "      using=\"image\",\n",
+    "      with_payload=[\"image\"],\n",
+    "      limit=1\n",
+    "  ).points[0].payload['image']\n",
+    "\n",
+    "Image.open(image_path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "mg77HNQv7T3s"
+   },
+   "source": [
+    "Let's also run the same query in Italian (one of the 30+ languages supported by the model) and compare the results."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "C7m5bOOM7T3s"
+   },
+   "outputs": [],
+   "source": [
+    "with headers({\"cohere-api-key\": cohere_api_key}):\n",
+    "  image_path = client.query_points(\n",
+    "      collection_name=COLLECTION_NAME,\n",
+    "      query=models.Document(\n",
+    "          text=\"Componenti di un aereo\",\n",
+    "          model=\"cohere/embed-v4.0\",\n",
+    "          options={\n",
+    "            \"output_dimension\": 512\n",
+    "          }\n",
+    "      ),\n",
+    "      using=\"image\",\n",
+    "      with_payload=[\"image\"],\n",
+    "      limit=1\n",
+    "  ).points[0].payload['image']\n",
+    "\n",
+    "Image.open(image_path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "IihSWFsnHfwx"
+   },
+   "source": "Now let's do a reverse search for the following image:"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "2ScFbT38I2rO"
+   },
+   "outputs": [],
+   "source": [
+    "Image.open(\"images/image-2.png\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "HLmPdjktJic7"
+   },
+   "outputs": [],
+   "source": [
+    "with headers({\"cohere-api-key\": cohere_api_key}):\n",
+    "  client.query_points(\n",
+    "      collection_name=COLLECTION_NAME,\n",
+    "      query=models.Image(\n",
+    "          image=image_to_base64_url(\"images/image-2.png\"),\n",
+    "          model=\"cohere/embed-v4.0\",\n",
+    "          options={\n",
+    "            \"output_dimension\": 512\n",
+    "          }\n",
+    "      ),\n",
+    "      # Now we are searching only among text vectors with our image query\n",
+    "      using=\"text\",\n",
+    "      with_payload=[\"caption\"],\n",
+    "      limit=1\n",
+    "  ).points[0].payload['caption']"
+   ]
+  }
+ ],
+ "metadata": {
+  "colab": {
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}