diff --git a/docs/config/yaml.md b/docs/config/yaml.md index c3cf870097..453eac831e 100644 --- a/docs/config/yaml.md +++ b/docs/config/yaml.md @@ -74,7 +74,7 @@ embedding_models: ### input -Our pipeline can ingest .csv, .txt, or .json data from an input location. See the [inputs page](../index/inputs.md) for more details and examples. +Our pipeline can ingest `.csv`, `.txt`, `.json`, `.jsonl`, `.parquet`, or MarkItDown-supported files from an input location. See the [inputs page](../index/inputs.md) for more details and examples. #### Fields @@ -86,16 +86,16 @@ Our pipeline can ingest .csv, .txt, or .json data from an input location. See th - `container_name` **str** - (blob/cosmosdb only) The Azure Storage container name. - `account_url` **str** - (blob only) The storage account blob URL to use. - `database_name` **str** - (cosmosdb only) The database name to use. -- `type` **text|csv|json** - The type of input data to load. Default is `text` +- `type` **text|csv|json|jsonl|parquet|markitdown** - The type of input data to load. Default is `text` - `encoding` **str** - The encoding of the input file. Default is `utf-8` -- `file_pattern` **str** - A regex to match input files. Default is `.*\.csv$`, `.*\.txt$`, or `.*\.json$` depending on the specified `type`, but you can customize it if needed. +- `file_pattern` **str** - A regex to match input files. Defaults are derived from `type` (for example `.*\.csv$`, `.*\.txt$`, `.*\.json$`, `.*\.jsonl$`, `.*\.parquet$`), but you can customize it if needed. - `id_column` **str** - The input ID column to use. - `title_column` **str** - The input title column to use. - `text_column` **str** - The input text column to use. ### chunking -These settings configure how we parse documents into text chunks. This is necessary because very large documents may not fit into a single context window, and graph extraction accuracy can be modulated. Also note the `metadata` setting in the input document config, which will replicate document metadata into each chunk. +These settings configure how we parse documents into text chunks. This is necessary because very large documents may not fit into a single context window, and graph extraction accuracy can be modulated. You can also prepend selected document fields (standard fields like `title` or values from `raw_data`) into each chunk. #### Fields @@ -103,7 +103,7 @@ These settings configure how we parse documents into text chunks. This is necess - `encoding_model` **str** - The text encoding model to use for splitting on token boundaries. - `size` **int** - The max chunk size in tokens. - `overlap` **int** - The chunk overlap in tokens. -- `prepend_metadata` **list[str]** - Metadata fields from the source document to prepend on each chunk. +- `prepend_metadata` **list[str]** - Document fields to prepend on each chunk. These can be standard document fields (`id`, `title`, `text`, `creation_date`) or fields from the source row/object stored in `raw_data`. For structured inputs, the `raw_data` object contains any other fields present in the source file (for CSV, this is all other column data). ## Outputs and Storage @@ -120,8 +120,6 @@ This section controls the storage mechanism used by the pipeline used for export - `container_name` **str** - (blob/cosmosdb only) The Azure Storage container name. - `account_url` **str** - (blob only) The storage account blob URL to use. - `database_name` **str** - (cosmosdb only) The database name to use. -- `type` **text|csv|json** - The type of input data to load. Default is `text` -- `encoding` **str** - The encoding of the input file. Default is `utf-8` ### update_output_storage @@ -136,8 +134,6 @@ The section defines a secondary storage location for running incremental indexin - `container_name` **str** - (blob/cosmosdb only) The Azure Storage container name. - `account_url` **str** - (blob only) The storage account blob URL to use. - `database_name` **str** - (cosmosdb only) The database name to use. -- `type` **text|csv|json** - The type of input data to load. Default is `text` -- `encoding` **str** - The encoding of the input file. Default is `utf-8` ### cache diff --git a/docs/examples_notebooks/drift_search.ipynb b/docs/examples_notebooks/drift_search.ipynb index 8d53c7d9cc..871b65120f 100644 --- a/docs/examples_notebooks/drift_search.ipynb +++ b/docs/examples_notebooks/drift_search.ipynb @@ -19,14 +19,10 @@ "import os\n", "\n", "import pandas as pd\n", - "from graphrag.config.enums import ModelType\n", "from graphrag.config.models.drift_search_config import DRIFTSearchConfig\n", - "from graphrag.config.models.language_model_config import LanguageModelConfig\n", - "from graphrag.language_model.manager import ModelManager\n", "from graphrag.query.indexer_adapters import (\n", " read_indexer_entities,\n", " read_indexer_relationships,\n", - " read_indexer_report_embeddings,\n", " read_indexer_reports,\n", " read_indexer_text_units,\n", ")\n", @@ -35,60 +31,10 @@ ")\n", "from graphrag.query.structured_search.drift_search.search import DRIFTSearch\n", "from graphrag.tokenizer.get_tokenizer import get_tokenizer\n", - "from graphrag_vectors.lancedb import LanceDBVectorStore\n", - "\n", - "INPUT_DIR = \"./inputs/operation dulce\"\n", - "LANCEDB_URI = f\"{INPUT_DIR}/lancedb\"\n", - "\n", - "COMMUNITY_REPORT_TABLE = \"community_reports\"\n", - "COMMUNITY_TABLE = \"communities\"\n", - "ENTITY_TABLE = \"entities\"\n", - "RELATIONSHIP_TABLE = \"relationships\"\n", - "COVARIATE_TABLE = \"covariates\"\n", - "TEXT_UNIT_TABLE = \"text_units\"\n", - "COMMUNITY_LEVEL = 2\n", - "\n", - "\n", - "# read nodes table to get community and degree data\n", - "entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n", - "community_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet\")\n", - "\n", - "print(f\"Entity df columns: {entity_df.columns}\")\n", - "\n", - "entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL)\n", - "\n", - "# load description embeddings to an in-memory lancedb vectorstore\n", - "# to connect to a remote db, specify url and port values.\n", - "description_embedding_store = LanceDBVectorStore(\n", - " db_uri=LANCEDB_URI,\n", - " index_name=\"entity_description\",\n", - ")\n", - "description_embedding_store.connect()\n", - "\n", - "full_content_embedding_store = LanceDBVectorStore(\n", - " db_uri=LANCEDB_URI,\n", - " index_name=\"community_full_content\",\n", - ")\n", - "full_content_embedding_store.connect()\n", - "\n", - "print(f\"Entity count: {len(entity_df)}\")\n", - "entity_df.head()\n", - "\n", - "relationship_df = pd.read_parquet(f\"{INPUT_DIR}/{RELATIONSHIP_TABLE}.parquet\")\n", - "relationships = read_indexer_relationships(relationship_df)\n", - "\n", - "print(f\"Relationship count: {len(relationship_df)}\")\n", - "relationship_df.head()\n", - "\n", - "text_unit_df = pd.read_parquet(f\"{INPUT_DIR}/{TEXT_UNIT_TABLE}.parquet\")\n", - "text_units = read_indexer_text_units(text_unit_df)\n", - "\n", - "print(f\"Text unit records: {len(text_unit_df)}\")\n", - "text_unit_df.head()\n", - "\n", - "report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n", - "reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL)\n", - "read_indexer_report_embeddings(reports, full_content_embedding_store)" + "from graphrag_llm.completion import create_completion\n", + "from graphrag_llm.config import ModelConfig\n", + "from graphrag_llm.embedding import create_embedding\n", + "from graphrag_vectors import IndexSchema, LanceDBVectorStore" ] }, { @@ -99,33 +45,61 @@ "source": [ "api_key = os.environ[\"GRAPHRAG_API_KEY\"]\n", "\n", - "chat_config = LanguageModelConfig(\n", - " api_key=api_key,\n", - " type=ModelType.Chat,\n", + "chat_config = ModelConfig(\n", + " type=\"litellm\",\n", " model_provider=\"openai\",\n", " model=\"gpt-4.1\",\n", - " max_retries=20,\n", - ")\n", - "chat_model = ModelManager().get_or_create_chat_model(\n", - " name=\"local_search\",\n", - " model_type=ModelType.Chat,\n", - " config=chat_config,\n", + " api_key=api_key,\n", ")\n", + "chat_model = create_completion(chat_config)\n", "\n", "tokenizer = get_tokenizer(chat_config)\n", "\n", - "embedding_config = LanguageModelConfig(\n", - " api_key=api_key,\n", - " type=ModelType.Embedding,\n", + "embedding_config = ModelConfig(\n", + " type=\"litellm\",\n", " model_provider=\"openai\",\n", " model=\"text-embedding-3-large\",\n", - " max_retries=20,\n", + " api_key=api_key,\n", + ")\n", + "\n", + "text_embedder = create_embedding(embedding_config)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# parquet files generated from indexing pipeline\n", + "INPUT_DIR = \"./inputs/operation dulce\"\n", + "LANCEDB_URI = \"./lancedb\"\n", + "COMMUNITY_TABLE = \"communities\"\n", + "COMMUNITY_REPORT_TABLE = \"community_reports\"\n", + "ENTITY_TABLE = \"entities\"\n", + "RELATIONSHIP_TABLE = \"relationships\"\n", + "TEXT_UNIT_TABLE = \"text_units\"\n", + "COMMUNITY_LEVEL = 2\n", + "\n", + "community_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet\")\n", + "report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n", + "entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n", + "relationship_df = pd.read_parquet(f\"{INPUT_DIR}/{RELATIONSHIP_TABLE}.parquet\")\n", + "text_unit_df = pd.read_parquet(f\"{INPUT_DIR}/{TEXT_UNIT_TABLE}.parquet\")\n", + "\n", + "reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL)\n", + "entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL)\n", + "relationships = read_indexer_relationships(relationship_df)\n", + "text_units = read_indexer_text_units(text_unit_df)\n", + "\n", + "# Connect to the existing GraphRAG vector index for entity-description embeddings.\n", + "description_embedding_store = LanceDBVectorStore(\n", + " index_schema=IndexSchema(index_name=\"default-entity-description\")\n", ")\n", + "description_embedding_store.connect(db_uri=LANCEDB_URI)\n", "\n", - "text_embedder = ModelManager().get_or_create_embedding_model(\n", - " name=\"local_search_embedding\",\n", - " model_type=ModelType.Embedding,\n", - " config=embedding_config,\n", + "print(\n", + " f\"Loaded reports={len(reports)}, entities={len(entities)}, relationships={len(relationships)}, text_units={len(text_units)}\"\n", ")" ] }, @@ -188,7 +162,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "graphrag-monorepo (3.12.10)", "language": "python", "name": "python3" }, diff --git a/docs/examples_notebooks/global_search.ipynb b/docs/examples_notebooks/global_search.ipynb index 605f704bd2..0585cfdb50 100644 --- a/docs/examples_notebooks/global_search.ipynb +++ b/docs/examples_notebooks/global_search.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "code", - "execution_count": null, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ @@ -12,16 +12,13 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "import pandas as pd\n", - "from graphrag.config.enums import ModelType\n", - "from graphrag.config.models.language_model_config import LanguageModelConfig\n", - "from graphrag.language_model.manager import ModelManager\n", "from graphrag.query.indexer_adapters import (\n", " read_indexer_communities,\n", " read_indexer_entities,\n", @@ -31,7 +28,9 @@ " GlobalCommunityContext,\n", ")\n", "from graphrag.query.structured_search.global_search.search import GlobalSearch\n", - "from graphrag.tokenizer.get_tokenizer import get_tokenizer" + "from graphrag.tokenizer.get_tokenizer import get_tokenizer\n", + "from graphrag_llm.completion import create_completion\n", + "from graphrag_llm.config import ModelConfig" ] }, { @@ -58,18 +57,13 @@ "source": [ "api_key = os.environ[\"GRAPHRAG_API_KEY\"]\n", "\n", - "config = LanguageModelConfig(\n", - " api_key=api_key,\n", - " type=ModelType.Chat,\n", + "config = ModelConfig(\n", + " type=\"litellm\",\n", " model_provider=\"openai\",\n", " model=\"gpt-4.1\",\n", - " max_retries=20,\n", - ")\n", - "model = ModelManager().get_or_create_chat_model(\n", - " name=\"global_search\",\n", - " model_type=ModelType.Chat,\n", - " config=config,\n", + " api_key=api_key,\n", ")\n", + "model = create_completion(config)\n", "\n", "tokenizer = get_tokenizer(config)" ] @@ -87,7 +81,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 7, "metadata": {}, "outputs": [], "source": [ @@ -104,9 +98,213 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Total report count: 10\n", + "Report count after filtering by community level 2: 10\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idhuman_readable_idcommunitylevelparentchildrentitlesummaryfull_contentrankrating_explanationfindingsfull_content_jsonperiodsize
09785e64b429dacd70cce8238051f3438b387ebcd6920ef...7710[]Paranormal Military Squad and Operation DulceThis community centers on the Paranormal Milit...# Paranormal Military Squad and Operation Dulc...8.5The impact severity rating is high due to the ...[{'explanation': 'The Paranormal Military Squa...{\\n \"title\": \"Paranormal Military Squad and...2026-01-134
1212c8ff1796bff731abe3d54f3216511ab15d5fb0d97eb...8810[]Paranormal Military Squad: Leadership and Scie...This community centers on the Paranormal Milit...# Paranormal Military Squad: Leadership and Sc...8.0The community poses a high impact due to its i...[{'explanation': 'Taylor Cruz, often referred ...{\\n \"title\": \"Paranormal Military Squad: Le...2026-01-133
2dbff99eaacade8ac4337a9bde0f533313445f0643ab3d0...9910[]Paranormal Military Squad and Operation: Dulce...This community centers around the Paranormal M...# Paranormal Military Squad and Operation: Dul...8.5The community poses a high impact severity due...[{'explanation': 'The military complex is the ...{\\n \"title\": \"Paranormal Military Squad and...2026-01-133
3b6e8ab1ac8ecc605bbec09ad78819b95ac08d62c5a9991...000-1[7, 8, 9]Paranormal Military Squad and Operation: DulceThis community centers on the Paranormal Milit...# Paranormal Military Squad and Operation: Dul...8.5The community poses a high impact due to its e...[{'explanation': 'Taylor Cruz is the central a...{\\n \"title\": \"Paranormal Military Squad and...2026-01-1310
497038bee58306d13b709b615fb2b816cae0c11273a9e73...110-1[]Team of Agents Investigating Dulce BaseThis community centers on a specialized team o...# Team of Agents Investigating Dulce Base\\n\\nT...7.5The impact severity rating is high due to the ...[{'explanation': 'The core of this community i...{\\n \"title\": \"Team of Agents Investigating ...2026-01-135
\n", + "
" + ], + "text/plain": [ + " id human_readable_id \\\n", + "0 9785e64b429dacd70cce8238051f3438b387ebcd6920ef... 7 \n", + "1 212c8ff1796bff731abe3d54f3216511ab15d5fb0d97eb... 8 \n", + "2 dbff99eaacade8ac4337a9bde0f533313445f0643ab3d0... 9 \n", + "3 b6e8ab1ac8ecc605bbec09ad78819b95ac08d62c5a9991... 0 \n", + "4 97038bee58306d13b709b615fb2b816cae0c11273a9e73... 1 \n", + "\n", + " community level parent children \\\n", + "0 7 1 0 [] \n", + "1 8 1 0 [] \n", + "2 9 1 0 [] \n", + "3 0 0 -1 [7, 8, 9] \n", + "4 1 0 -1 [] \n", + "\n", + " title \\\n", + "0 Paranormal Military Squad and Operation Dulce \n", + "1 Paranormal Military Squad: Leadership and Scie... \n", + "2 Paranormal Military Squad and Operation: Dulce... \n", + "3 Paranormal Military Squad and Operation: Dulce \n", + "4 Team of Agents Investigating Dulce Base \n", + "\n", + " summary \\\n", + "0 This community centers on the Paranormal Milit... \n", + "1 This community centers on the Paranormal Milit... \n", + "2 This community centers around the Paranormal M... \n", + "3 This community centers on the Paranormal Milit... \n", + "4 This community centers on a specialized team o... \n", + "\n", + " full_content rank \\\n", + "0 # Paranormal Military Squad and Operation Dulc... 8.5 \n", + "1 # Paranormal Military Squad: Leadership and Sc... 8.0 \n", + "2 # Paranormal Military Squad and Operation: Dul... 8.5 \n", + "3 # Paranormal Military Squad and Operation: Dul... 8.5 \n", + "4 # Team of Agents Investigating Dulce Base\\n\\nT... 7.5 \n", + "\n", + " rating_explanation \\\n", + "0 The impact severity rating is high due to the ... \n", + "1 The community poses a high impact due to its i... \n", + "2 The community poses a high impact severity due... \n", + "3 The community poses a high impact due to its e... \n", + "4 The impact severity rating is high due to the ... \n", + "\n", + " findings \\\n", + "0 [{'explanation': 'The Paranormal Military Squa... \n", + "1 [{'explanation': 'Taylor Cruz, often referred ... \n", + "2 [{'explanation': 'The military complex is the ... \n", + "3 [{'explanation': 'Taylor Cruz is the central a... \n", + "4 [{'explanation': 'The core of this community i... \n", + "\n", + " full_content_json period size \n", + "0 {\\n \"title\": \"Paranormal Military Squad and... 2026-01-13 4 \n", + "1 {\\n \"title\": \"Paranormal Military Squad: Le... 2026-01-13 3 \n", + "2 {\\n \"title\": \"Paranormal Military Squad and... 2026-01-13 3 \n", + "3 {\\n \"title\": \"Paranormal Military Squad and... 2026-01-13 10 \n", + "4 {\\n \"title\": \"Team of Agents Investigating ... 2026-01-13 5 " + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "community_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet\")\n", "entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n", @@ -133,7 +331,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 9, "metadata": {}, "outputs": [], "source": [ @@ -174,7 +372,6 @@ "map_llm_params = {\n", " \"max_tokens\": 1000,\n", " \"temperature\": 0.0,\n", - " \"response_format\": {\"type\": \"json_object\"},\n", "}\n", "\n", "reduce_llm_params = {\n", @@ -194,10 +391,10 @@ " context_builder=context_builder,\n", " tokenizer=tokenizer,\n", " max_data_tokens=12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)\n", - " map_llm_params=map_llm_params,\n", - " reduce_llm_params=reduce_llm_params,\n", + " map_llm_params=dict(map_llm_params),\n", + " reduce_llm_params=dict(reduce_llm_params),\n", " allow_general_knowledge=False, # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases.\n", - " json_mode=True, # set this to False if your LLM model does not support JSON mode.\n", + " json_mode=False,\n", " context_builder_params=context_builder_params,\n", " concurrent_coroutines=32,\n", " response_type=\"multiple paragraphs\", # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report\n", @@ -206,9 +403,39 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 22, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "## Overview of Operation Dulce\n", + "\n", + "Operation Dulce is a classified, high-stakes mission led by the Paranormal Military Squad, an elite unit specializing in the investigation and management of anomalous phenomena. The operation centers on the enigmatic Dulce Base, a secretive and technologically advanced underground facility. The primary objective of Operation Dulce is to uncover hidden technologies, existential threats, and anomalous phenomena within the base—matters considered to have profound implications for security, intelligence, and potentially the future of humanity [Data: Reports (4, 0, 3, 8, 9, +more)].\n", + "\n", + "## Mission Objectives and Scope\n", + "\n", + "The operation is specifically tasked with investigating and managing unconventional threats, particularly those associated with the Dulce Base. This includes handling advanced alien technology and responding to existential risks posed by the secrets and anomalies discovered within the facility. The mission is regarded as a convergence point for key personnel, including Sam Rivera, Alex Mercer, Taylor Cruz, and Dr. Jordan Hayes, and is considered a career-defining moment for those involved [Data: Reports (4, 0, 1, 3, 8)].\n", + "\n", + "Safeguarding national security is a central driver of the operation, as the threats emerging from Dulce Base are unconventional and potentially far-reaching. The Paranormal Military Squad operates from a secure military complex, relying on specialized agents and robust technical infrastructure to carry out its mission [Data: Reports (7, 0, 4, 8)].\n", + "\n", + "## Operational Environment and Infrastructure\n", + "\n", + "Operation Dulce is supported by a high-security operational environment at the Dulce Base. The base itself serves as the secure foundation and logistical center for the mission, providing the necessary infrastructure and a compartmentalized environment to support advanced military operations. Facilities include briefing rooms equipped with advanced technological devices such as projectors and monitors, which facilitate mission planning, review, and data analysis. This environment ensures operational readiness and security for the Paranormal Military Squad as they execute the operation [Data: Reports (5)].\n", + "\n", + "Additionally, the operation is underpinned by advanced communication infrastructure, secure military facilities, and specialized equipment such as encrypted radio transmitters. These measures are designed to ensure operational discipline and security during the investigation of the Dulce Base [Data: Reports (0, 9, 1)].\n", + "\n", + "## Complexity, Protocols, and Implications\n", + "\n", + "The mission is marked by significant complexity and uncertainty, requiring strict adherence to protocols and adaptability among squad members. The level of preparation and security measures reflects the gravity of the operation and the potential for uncovering cosmic secrets or existential threats within the Dulce Base. The success or failure of Operation Dulce may have far-reaching consequences, not only for national security but also for the broader understanding of anomalous phenomena and advanced technologies [Data: Reports (4, 0, 9)].\n", + "\n", + "## Summary\n", + "\n", + "In summary, Operation Dulce is a classified mission of critical importance, focused on investigating and managing the unknown and potentially dangerous phenomena associated with the Dulce Base. It is characterized by its high level of security, advanced technological support, and the involvement of specialized personnel. The operation’s outcomes may significantly impact national security and humanity’s understanding of existential threats and advanced technologies [Data: Reports (4, 0, 3, 8, 9, 7, 1, 5, +more)].\n" + ] + } + ], "source": [ "result = await search_engine.search(\"What is operation dulce?\")\n", "\n", @@ -217,9 +444,153 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 13, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idtitleoccurrence weightcontentrank
04Dulce Base and Operation: Dulce Community1.0# Dulce Base and Operation: Dulce Community\\n\\...9.0
10Paranormal Military Squad and Operation: Dulce1.0# Paranormal Military Squad and Operation: Dul...8.5
23Paranormal Military Squad and Operation: Dulce1.0# Paranormal Military Squad and Operation: Dul...8.5
38Paranormal Military Squad: Leadership and Scie...1.0# Paranormal Military Squad: Leadership and Sc...8.0
49Paranormal Military Squad and Operation: Dulce...0.2# Paranormal Military Squad and Operation: Dul...8.5
51Team of Agents Investigating Dulce Base0.2# Team of Agents Investigating Dulce Base\\n\\nT...7.5
66Dulce Base Mainframe Room Technology Network0.2# Dulce Base Mainframe Room Technology Network...7.5
77Paranormal Military Squad and Operation Dulce0.4# Paranormal Military Squad and Operation Dulc...8.5
85Dulce Base Operational Command: Briefing Room ...0.4# Dulce Base Operational Command: Briefing Roo...7.5
92Alien Technology Retrieval and Analysis at Dul...0.2# Alien Technology Retrieval and Analysis at D...9.0
\n", + "
" + ], + "text/plain": [ + " id title occurrence weight \\\n", + "0 4 Dulce Base and Operation: Dulce Community 1.0 \n", + "1 0 Paranormal Military Squad and Operation: Dulce 1.0 \n", + "2 3 Paranormal Military Squad and Operation: Dulce 1.0 \n", + "3 8 Paranormal Military Squad: Leadership and Scie... 1.0 \n", + "4 9 Paranormal Military Squad and Operation: Dulce... 0.2 \n", + "5 1 Team of Agents Investigating Dulce Base 0.2 \n", + "6 6 Dulce Base Mainframe Room Technology Network 0.2 \n", + "7 7 Paranormal Military Squad and Operation Dulce 0.4 \n", + "8 5 Dulce Base Operational Command: Briefing Room ... 0.4 \n", + "9 2 Alien Technology Retrieval and Analysis at Dul... 0.2 \n", + "\n", + " content rank \n", + "0 # Dulce Base and Operation: Dulce Community\\n\\... 9.0 \n", + "1 # Paranormal Military Squad and Operation: Dul... 8.5 \n", + "2 # Paranormal Military Squad and Operation: Dul... 8.5 \n", + "3 # Paranormal Military Squad: Leadership and Sc... 8.0 \n", + "4 # Paranormal Military Squad and Operation: Dul... 8.5 \n", + "5 # Team of Agents Investigating Dulce Base\\n\\nT... 7.5 \n", + "6 # Dulce Base Mainframe Room Technology Network... 7.5 \n", + "7 # Paranormal Military Squad and Operation Dulc... 8.5 \n", + "8 # Dulce Base Operational Command: Briefing Roo... 7.5 \n", + "9 # Alien Technology Retrieval and Analysis at D... 9.0 " + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "# inspect the data used to build the context for the LLM responses\n", "result.context_data[\"reports\"]" @@ -227,9 +598,17 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 14, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "LLM calls: 2. Prompt tokens: 11395. Output tokens: 0.\n" + ] + } + ], "source": [ "# inspect number of LLM calls and tokens\n", "print(\n", @@ -240,7 +619,7 @@ ], "metadata": { "kernelspec": { - "display_name": "graphrag", + "display_name": "graphrag-monorepo (3.12.10)", "language": "python", "name": "python3" }, diff --git a/docs/examples_notebooks/global_search_with_dynamic_community_selection.ipynb b/docs/examples_notebooks/global_search_with_dynamic_community_selection.ipynb index 6b3763d73b..13464438c3 100644 --- a/docs/examples_notebooks/global_search_with_dynamic_community_selection.ipynb +++ b/docs/examples_notebooks/global_search_with_dynamic_community_selection.ipynb @@ -19,9 +19,6 @@ "import os\n", "\n", "import pandas as pd\n", - "from graphrag.config.enums import ModelType\n", - "from graphrag.config.models.language_model_config import LanguageModelConfig\n", - "from graphrag.language_model.manager import ModelManager\n", "from graphrag.query.indexer_adapters import (\n", " read_indexer_communities,\n", " read_indexer_entities,\n", @@ -30,7 +27,9 @@ "from graphrag.query.structured_search.global_search.community_context import (\n", " GlobalCommunityContext,\n", ")\n", - "from graphrag.query.structured_search.global_search.search import GlobalSearch" + "from graphrag.query.structured_search.global_search.search import GlobalSearch\n", + "from graphrag_llm.completion import create_completion\n", + "from graphrag_llm.config import ModelConfig" ] }, { @@ -59,18 +58,13 @@ "\n", "api_key = os.environ[\"GRAPHRAG_API_KEY\"]\n", "\n", - "config = LanguageModelConfig(\n", - " api_key=api_key,\n", - " type=ModelType.Chat,\n", + "config = ModelConfig(\n", + " type=\"litellm\",\n", " model_provider=\"openai\",\n", " model=\"gpt-4.1\",\n", - " max_retries=20,\n", - ")\n", - "model = ModelManager().get_or_create_chat_model(\n", - " name=\"global_search\",\n", - " model_type=ModelType.Chat,\n", - " config=config,\n", + " api_key=api_key,\n", ")\n", + "model = create_completion(config)\n", "\n", "tokenizer = get_tokenizer(config)" ] @@ -193,7 +187,6 @@ "map_llm_params = {\n", " \"max_tokens\": 1000,\n", " \"temperature\": 0.0,\n", - " \"response_format\": {\"type\": \"json_object\"},\n", "}\n", "\n", "reduce_llm_params = {\n", @@ -213,10 +206,10 @@ " context_builder=context_builder,\n", " tokenizer=tokenizer,\n", " max_data_tokens=12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)\n", - " map_llm_params=map_llm_params,\n", - " reduce_llm_params=reduce_llm_params,\n", + " map_llm_params=dict(map_llm_params),\n", + " reduce_llm_params=dict(reduce_llm_params),\n", " allow_general_knowledge=False, # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases.\n", - " json_mode=True, # set this to False if your LLM model does not support JSON mode.\n", + " json_mode=False,\n", " context_builder_params=context_builder_params,\n", " concurrent_coroutines=32,\n", " response_type=\"multiple paragraphs\", # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report\n", diff --git a/docs/examples_notebooks/local_search.ipynb b/docs/examples_notebooks/local_search.ipynb index f7f0c5a54b..bba7a7b38c 100644 --- a/docs/examples_notebooks/local_search.ipynb +++ b/docs/examples_notebooks/local_search.ipynb @@ -190,39 +190,29 @@ "metadata": {}, "outputs": [], "source": [ - "from graphrag.config.enums import ModelType\n", - "from graphrag.config.models.language_model_config import LanguageModelConfig\n", - "from graphrag.language_model.manager import ModelManager\n", "from graphrag.tokenizer.get_tokenizer import get_tokenizer\n", + "from graphrag_llm.completion import create_completion\n", + "from graphrag_llm.config import ModelConfig\n", + "from graphrag_llm.embedding import create_embedding\n", "\n", "api_key = os.environ[\"GRAPHRAG_API_KEY\"]\n", "\n", - "chat_config = LanguageModelConfig(\n", - " api_key=api_key,\n", - " type=ModelType.Chat,\n", + "chat_config = ModelConfig(\n", + " type=\"litellm\",\n", " model_provider=\"openai\",\n", " model=\"gpt-4.1\",\n", - " max_retries=20,\n", - ")\n", - "chat_model = ModelManager().get_or_create_chat_model(\n", - " name=\"local_search\",\n", - " model_type=ModelType.Chat,\n", - " config=chat_config,\n", + " api_key=api_key,\n", ")\n", + "chat_model = create_completion(chat_config)\n", "\n", - "embedding_config = LanguageModelConfig(\n", - " api_key=api_key,\n", - " type=ModelType.Embedding,\n", + "embedding_config = ModelConfig(\n", + " type=\"litellm\",\n", " model_provider=\"openai\",\n", " model=\"text-embedding-3-small\",\n", - " max_retries=20,\n", + " api_key=api_key,\n", ")\n", "\n", - "text_embedder = ModelManager().get_or_create_embedding_model(\n", - " name=\"local_search_embedding\",\n", - " model_type=ModelType.Embedding,\n", - " config=embedding_config,\n", - ")\n", + "text_embedder = create_embedding(embedding_config)\n", "\n", "tokenizer = get_tokenizer(chat_config)" ] diff --git a/docs/index/architecture.md b/docs/index/architecture.md index c1c9e05f98..c5d86938fe 100644 --- a/docs/index/architecture.md +++ b/docs/index/architecture.md @@ -41,7 +41,7 @@ Several subsystems within GraphRAG use a factory pattern to register and retriev The following subsystems use a factory pattern that allows you to register your own implementations: - [language model](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-llm/graphrag_llm/completion/completion_factory.py) - implement your own `chat` and `embed` methods to use a model provider of choice beyond the built-in LiteLLM wrapper -- [input reader](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-input/graphrag_input/input_reader.py) - implement your own input document reader to support file types other than text, CSV, and JSON +- [input reader](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-input/graphrag_input/input_reader.py) - implement your own input document reader to support file types other than the built-in text, CSV, JSON, JSONL, Parquet, and MarkItDown readers - [cache](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-cache/graphrag_cache/cache_factory.py) - create your own cache storage location in addition to the file, blob, and CosmosDB ones we provide - [logger](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/logger/factory.py) - create your own log writing location in addition to the built-in file and blob storage - [storage](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-storage/graphrag_storage/tables/table_provider_factory.py) - create your own storage provider (database, etc.) beyond the file, blob, and CosmosDB ones built in diff --git a/docs/index/default_dataflow.md b/docs/index/default_dataflow.md index 9a810accb0..051c06e6eb 100644 --- a/docs/index/default_dataflow.md +++ b/docs/index/default_dataflow.md @@ -2,7 +2,7 @@ ## The GraphRAG Knowledge Model -The knowledge model is a specification for data outputs that conform to our data-model definition. You can find these definitions in the python/graphrag/graphrag/model folder within the GraphRAG repository. The following entity types are provided. The fields here represent the fields that are text-embedded by default. +The knowledge model is a specification for data outputs that conform to our data-model definition. You can find these definitions in the `packages/graphrag/graphrag/data_model` folder within the GraphRAG repository. The following entity types are provided. The fields here represent the fields that are text-embedded by default. - `Document` - An input document into the system. These either represent individual rows in a CSV or individual .txt files. - `TextUnit` - A chunk of text to analyze. The size of these chunks, their overlap, and whether they adhere to any data boundaries may be configured below. diff --git a/docs/index/inputs.md b/docs/index/inputs.md index a15f1418c1..6aeeb2fec5 100644 --- a/docs/index/inputs.md +++ b/docs/index/inputs.md @@ -12,7 +12,7 @@ All input formats are loaded within GraphRAG and passed to the indexing pipeline | text | str | The full text of the document. | | title | str | Name of the document. Some formats allow this to be configured. | | creation_date | str | The creation date of the document, represented as an ISO8601 string. This is harvested from the source file system. | -| metadata | dict | Optional additional document metadata. More details below. | +| raw_data | dict | Optional source row/object for structured inputs. This can be used for metadata prepending during chunking. | Also see the [outputs](outputs.md) documentation for the final documents table schema saved to parquet after pipeline completion. @@ -26,7 +26,7 @@ We use an injectable InputReader provider class. This means you can implement an ## Formats -We support three file formats out-of-the-box. This covers the overwhelming majority of use cases we have encountered. If you have a different format, we recommend either implementing your own InputReader or writing a script to convert to one of these, which are widely used and supported by many tools and libraries. +We support multiple file formats out-of-the-box. If you have a different format, we recommend either implementing your own InputReader or writing a script to convert to one of these. ### Plain Text @@ -40,11 +40,23 @@ With the CSV format you can configure the `text_column`, and `title_column` if y ### JSON -JSON files (typically ending in a .json extension) contain [structured objects](https://www.json.org/). These are loaded using python's [`json.loads` method](https://docs.python.org/3/library/json.html), so your files must be properly compliant. JSON files may contain a single object in the file *or* the file may contain an array of objects at the root. We will check for and handle either of these cases. As with CSV, multiple files will be concatenated into a final table, and the `text_column` and `title_column` config options will be applied to the properties of each loaded object. Note that the specialized jsonl format produced by some libraries (one full JSON object on each line, not in an array) is not currently supported. +JSON files (typically ending in a .json extension) contain [structured objects](https://www.json.org/). These are loaded using python's [`json.loads` method](https://docs.python.org/3/library/json.html), so your files must be properly compliant. JSON files may contain a single object in the file *or* the file may contain an array of objects at the root. We will check for and handle either of these cases. As with CSV, multiple files will be concatenated into a final table, and the `text_column` and `title_column` config options will be applied to the properties of each loaded object. -## Metadata +### JSON Lines -With the structured file formats (CSV and JSON) you can configure any number of columns to be added to a persisted `metadata` field in the DataFrame. This is configured by supplying a list of column names to collect. If this is configured, the output `metadata` column will have a dict containing a key for each column, and the value of the column for that document. This metadata can optionally be used later in the GraphRAG pipeline. +JSONL files (typically ending in a .jsonl extension) are loaded one JSON object per line. Blank/whitespace-only lines are skipped. + +### Parquet + +Parquet files (typically ending in a .parquet extension) are loaded as structured rows, similar to CSV/JSON behavior for `text_column`, `title_column`, and `id_column`. + +### MarkItDown + +MarkItDown input supports document conversion via the MarkItDown reader. + +## Field Prepending + +For structured file formats, source row/object content is preserved in `raw_data`. During chunking, you can use `chunking.prepend_metadata` to prepend selected fields into each chunk. Fields can reference standard document fields (`id`, `title`, `text`, `creation_date`) or keys from `raw_data`. ### Example @@ -59,16 +71,16 @@ An early space shooter game,Space Invaders,arcade settings.yaml ```yaml -input: - metadata: [title,tag] +chunking: + prepend_metadata: [title,tag] ``` Documents DataFrame -| id | title | text | creation_date | metadata | +| id | title | text | creation_date | raw_data | | --------------------- | -------------- | --------------------------- | ----------------------------- | ---------------------------------------------- | -| (generated from text) | Hello World | My first program | (create date of software.csv) | { "title": "Hello World", "tag": "tutorial" } | -| (generated from text) | Space Invaders | An early space shooter game | (create date of software.csv) | { "title": "Space Invaders", "tag": "arcade" } | +| (generated from text) | Hello World | My first program | (create date of software.csv) | { "text": "My first program", "title": "Hello World", "tag": "tutorial" } | +| (generated from text) | Space Invaders | An early space shooter game | (create date of software.csv) | { "text": "An early space shooter game", "title": "Space Invaders", "tag": "arcade" } | ## Chunking and Metadata @@ -78,13 +90,13 @@ Imagine the following scenario: you are indexing a collection of news articles. ### Input Config -As described above, when documents are imported you can specify a list of `metadata` columns to include with each row. This must be configured for the per-chunk copying to work. +As described above, structured source fields are available via `raw_data` and may be referenced directly by field name during chunk prepending. ### Chunking Config Next, the `chunks` block needs to instruct the chunker how to handle this metadata when creating text units. By default, it is ignored. We have the following setting to include it: -- `prepend_metadata`. This instructs the importer to copy the contents of the `metadata` column for each row into the start of every single text chunk. This metadata is copied as key: value pairs on new lines. +- `prepend_metadata`. This instructs the importer to copy the selected document fields into the start of every text chunk. Values are copied as key: value pairs on new lines. For structured inputs, the `raw_data` object contains any other fields present in the source file (for CSV, this is all other column data). ### Examples diff --git a/docs/index/outputs.md b/docs/index/outputs.md index 89a48e8526..e76ea89d59 100644 --- a/docs/index/outputs.md +++ b/docs/index/outputs.md @@ -40,7 +40,7 @@ This is the list of summarized reports for each community. | full_content | str | LM-generated full report. | | rank | float | LM-derived relevance ranking of the report based on member entity salience | rating_explanation | str | LM-derived explanation of the rank. | -| findings | dict | LM-derived list of the top 5-10 insights from the community. Contains `summary` and `explanation` values. | +| findings | dict[] | LM-derived list of the top 5-10 insights from the community. Each item contains `summary` and `explanation` values. | | full_content_json | json | Full JSON output as returned by the LM. Most fields are extracted into columns, but this JSON is sent for query summarization so we leave it to allow for prompt tuning to add fields/content by end users. | | period | str | Date of ingest, used for incremental update merges. ISO8601 | | size | int | Size of the community (entity count), used for incremental update merges. | @@ -69,7 +69,8 @@ List of document content after import. | title | str | Filename, unless otherwise configured during CSV import. | | text | str | Full text of the document. | | text_unit_ids | str[] | List of text units (chunks) that were parsed from the document. | -| metadata | dict | If specified during CSV import, this is a dict of metadata for the document. | +| creation_date | str | Creation timestamp of the source document, represented as an ISO8601 string. | +| raw_data | dict | Source row/object data loaded from structured inputs (or `null` for plain text). | ## entities List of all entities found in the data by the LM. @@ -104,5 +105,5 @@ List of all text chunks parsed from the input documents. | n_tokens | int | Number of tokens in the chunk. This should normally match the `chunk_size` config parameter, except for the last chunk which is often shorter. | | document_id | str | ID of the document the chunk came from. | | entity_ids | str[] | List of entities found in the text unit. | -| relationships_ids | str[] | List of relationships found in the text unit. | +| relationship_ids | str[] | List of relationships found in the text unit. | | covariate_ids | str[] | Optional list of covariates found in the text unit. | \ No newline at end of file