Skip to content

testing-guide.md

Latest commit

 

History

History
244 lines (169 loc) · 12.5 KB

testing-guide.md

File metadata and controls

244 lines (169 loc) · 12.5 KB

Testing the Pinecone SDK

We have a lot of different types of tests in this repository. At a high level, they are structured like this:

tests
├── dependency
├── integration
├── integration_manual
├── perf
├── unit
├── unit_grpc
└── upgrade

  • dependency: These tests are a set of very minimal end-to-end integration tests that ensure basic functionality works to upsert and query vectors from an index. These are rarely run locally; we use them in CI to confirm the client can be used when installed with a large matrix of different python versions and versions of key dependencies. See .github/workflows/testing-dependency.yaml for more details on how these are run.

  • integration: These are a large suite of end-to-end integration tests exercising most of the core functions of the product. They are slow and expensive to run, but they give the greatest confidence the SDK actually works end-to-end. See notes below on how to setup the required configuration and run individual tests if you are iterating on a bug or feature and want to get more rapid feedback than running the entire suite in CI will give you. In CI, these are run using .github/workflows/testing-integration.yaml.

  • integration_manual: These are integration tests that are not run automatically in CI but can be run manually when needed. These typically include tests for features that are expensive to run (like backups and restores), tests that require special setup (like proxy configuration), or tests that exercise edge cases that don't need to be validated on every PR. To run these manually, use: uv run pytest tests/integration_manual

  • perf: These tests are still being developed. But eventually, they will play an important roll in making sure we don't regress on client performance when building new features.

  • unit and unit_grpc. These are what you would probably expect. Unit-testing makes up a relatively small portion of our testing because there's not that much business logic that makes sense to test in isolation. But it is ocassionally useful when doing some sort of data conversions with many edge cases (e.g. VectorFactory) or merging results (e.g. QueryResultsAggregator) to write some unit tests. If you have a situation where unit testing is appropriate, they are really great to work with because they are fast and don't have any external dependencies. In CI, these are run with the .github/workflows/testing-unit.yaml workflow.

  • upgrade: These are also still being developed and if you are reading this guide you probably don't need to worry about them. The goal of these is to ensure we're not introducing breaking changes without realizing it.

Running the ruff linter / formatter

These should automatically trigger if you have enabled pre-commit hooks with uv run pre-commit install. But in case you want to trigger these yourself, you can run them like this:

uv run ruff check --fix # lint rules
uv run ruff format      # formatting

If you want to adjust the behavior of ruff, configurations are in pyproject.toml.

Running the type checker

If you are adding new code, you should make an effort to annotate it with type hints.

You can run the type-checker to check for issues with:

uv run mypy pinecone

Automated tests

Running unit tests

Unit-testing makes up a relatively small portion of our testing because there's not that much business logic that makes sense to test in isolation. But it is ocassionally useful when doing some sort of data conversions with many edge cases (e.g. VectorFactory) or merging results (e.g. QueryResultsAggregator) to write some unit tests.

Unit tests do not automatically read environment variables in the .env file because some of the tests relate to parsing values from environment variables and we don't want values in our .env file to impact how these tests execute.

To run them:

  • For REST: uv run pytest tests/unit
  • For GRPC: uv run pytest tests/unit_grpc

If you want to set an environment variable anyway, you can do it be prefacing the test command inline. E.g. FOO='bar' uv run pytest tests/unit

Running integration tests

Integration tests make real calls to Pinecone. They are slow but give the highest level of confidence the client is actually working end to end. In general, only Pinecone employees should run these because the cost of the creating underlying resources can be quite significant, particularly if errors occur and some resources are not cleaned up properly.

For these tests, you need to make sure you've set values inside of an .env file (see .env.example for more information). These will be read using dotenv when tests are run.

I never run all of these locally in one shot because it would take too long and is generally unnecessary; in CI, the tests are broken up across many different jobs so they can run in parallel and minimize the amount of retesting when a failure results in the entire build being re-run.

If I see one or a few tests broken in CI, I will run just those tests locally while iterating on the fix:

  • Run the tests for a specific part of the SDK (example: index): uv run pytest tests/integration/db/control/sync/resources/index
  • Run the tests in a single file: uv run pytest tests/integration/db/control/sync/resources/index/test_create.py
  • Run a single test uv run pytest tests/integration/db/control/sync/resources/index/test_list.py::TestListIndexes::test_list_indexes_includes_ready_indexes

Test Sharding

To speed up CI runs, we use a custom pytest plugin to shard (split) tests across multiple parallel jobs. This allows us to run tests in parallel across multiple CI workers, reducing overall test execution time.

The sharding plugin is automatically available when running pytest (registered in tests/conftest.py). To use it:

Command-line options:

# Run shard 1 of 3
uv run pytest tests/integration/rest_sync --splits=3 --group=1

# Run shard 2 of 3
uv run pytest tests/integration/rest_sync --splits=3 --group=2

# Run shard 3 of 3
uv run pytest tests/integration/rest_sync --splits=3 --group=3

Environment variables (alternative to command-line options):

# Set environment variables instead of using --splits and --group
export PYTEST_SPLITS=3
export PYTEST_GROUP=1
uv run pytest tests/integration/rest_sync

How it works:

  • Tests are distributed across shards using a hash-based algorithm, ensuring deterministic assignment (the same test will always be in the same shard)
  • Tests are distributed evenly across all shards
  • The --group parameter is 1-indexed (first shard is 1, not 0)
  • All shards must be run to execute the complete test suite

In CI: The CI workflows (.github/workflows/testing-integration.yaml) automatically use sharding to split tests across multiple parallel jobs. Each job runs a different shard, allowing tests to execute in parallel and complete faster. Different test suites use different shard counts based on their size:

  • rest_sync tests: 8 shards
  • rest_asyncio tests: 5 shards
  • grpc tests: No sharding (runs all tests in a single job, including tests/integration/rest_sync/db/data with USE_GRPC='true')

Local development: When running tests locally, you typically don't need to use sharding unless you want to simulate the CI environment or test the sharding functionality itself.

Fixtures and other test configuration

Many values are read from environment variables (from .env) or set in CI workflows such as .github/workflows/testing-integration.yaml.

At the level of the testing framework, a lot of test fixtures as well as setup & cleanup tasks take place in special files called conftest.py. This file name has special significance to pytest and your fixtures won't be loaded if you mispell the name of the file, so be careful if you are setting up a new group of tests that need a conftest.py file.

Within a conftest file, a fixture can be defined like this with the @pytest.fixture decorator:

@pytest.fixture()
def foo(request):
    return "FOO"

Then in the test file, you can refer to the fixture by name in the parameters to your test function:

class MyExampleTest:
    def test_foo(self, foo):
        assert foo == "FOO"

This is a highly contrived example, but we use this technique to access test configuration controlled with environment variables and resources that have heavy setup & cleanup cost (e.g. spinning up indexes) that we want to manage in one place rather than duplicating those steps in many places throughout a codebase.

Testing data plane: REST vs GRPC vs Asyncio

Integration tests for the data plane (i.e. uv run pytest tests/integration/db/data/sync) are reused for both the REST and GRPC client variants since the interfaces of these different client implementations are nearly identical (other than async_req=True responses). To toggle how they are run, set USE_GRPC='true' in your .env before running.

There are a relatively small number of tests which are not shared, usually related to futures when using GRPC with async_req=True. We use @pytest.mark.skipif to control whether these are run or not.

class TestDeleteFuture:
    @pytest.mark.skipif(
        os.getenv("USE_GRPC") != "true", reason="PineconeGrpcFutures only returned from grpc client"
    )
    def test_delete_future(self, idx):
        # ... test implementation

Asyncio tests of the data plane are unfortunately separate because there are quite a few differences in how you interact with the asyncio client. So those tests are found in a different directory, tests/integration/db/data/asyncio

Manual testing

With an interactive REPL

You can access a python REPL that is preloaded with the virtualenv maintained by uv (including all dependencies declared in pyproject.toml), any changes you've made to the code in pinecone/, the environment variables set in your .env file, and a few useful variables and functions defined in scripts/repl.py :

$ uv run repl

    Welcome to the custom Python REPL!
    Your initialization steps have been completed.

    Two Pinecone objects are available:
    - pc: Built using the PINECONE_API_KEY env var, if set
    - pcci: Built using the PINECONE_API_KEY_CI_TESTING env var, if set

    You can use the following functions to clean up the environment:
    - delete_all_indexes(pc)
    - delete_all_pod_indexes(pc)
    - delete_all_collections(pc)
    - delete_all_backups(pc)
    - cleanup_all(pc)

>>> pc.describe_index(name='jen')
{
    "name": "jen",
    "metric": "cosine",
    "host": "jen-dojoi3u.svc.preprod-aws-0.pinecone.io",
    "spec": {
        "serverless": {
            "cloud": "aws",
            "region": "us-east-1"
        }
    },
    "status": {
        "ready": true,
        "state": "Ready"
    },
    "vector_type": "dense",
    "dimension": 2,
    "deletion_protection": "disabled",
    "tags": null
}

Investigating module import performance

We don't have automated tests for this, but if you want to do some one-off testing to check on how efficiently the package can be imported and initialized, you can run code like this:

uv run python3 -X importtime -c 'from pinecone import Pinecone; pc = Pinecone(api_key="foo")' 2> import_time.log

And then inspect the results with a visualization tool called tuna.

uv run tuna import_time.log

This is a useful thing to do when you are introducing new classes or plugins to ensure you're not causing a performance regression on imports.

Installing SDK WIP in another project on your machine

pip, uv, and similar tools know how to install from local files. This can sometimes be useful to validate a change or bugfix.

If your local files look like this:

workspace
├── pinecone-python-client/
└── repro_project/

You should be able to test changes in your repro project by doing something like

cd repro_project

# With uv
uv add ../pinecone-python-client

# With pip3
pip3 install ../pinecone-python-client