add chroma to benchmark

[LukasWestholt]

No description provided.

[LukasWestholt]

Hello dear Chroma maintainers,

Are there existing optimisation options and how can they be used (for example CHROMA_WORKERS)?
Thank you.
@tazarov

[LukasWestholt]

Waiting for chroma-core/chroma#2581

[tazarov]

A few minor nits and caveats, but generally looks good.

[LukasWestholt]

Is something like this helpful?

@classmethod
def delete_client(cls):
    if cls.client is not None:
        cls.client._admin_client._server._session.close()
        cls.client._server._session.close()

for the problem of chroma-core/chroma#2581.

This is because in the benchmark, search queries are sent in several parallel processes (on the client side) with parallel=100 (multiprocessing). And a new HttpClient instance is created in each of these processes. Does this cause problems? Since Chroma then works serially on the server side, this benchmark probably just doesn't make sense.

This also results in worse times with parallel > 1 than with parallel = 1, I think.

Let me test this right now. Update: Yes, parallel > 1 is slower then parallel = 1, even with this snippet above, so i deleted this part of the benchmark.

[tazarov]

Is something like this helpful?

@classmethod
def delete_client(cls):
    if cls.client is not None:
        cls.client._admin_client._server._session.close()
        cls.client._server._session.close()

for the problem of chroma-core/chroma#2581.

This is because in the benchmark, search queries are sent in several parallel processes (on the client side) with parallel=100 (multiprocessing). And a new HttpClient instance is created in each of these processes. Does this cause problems? Since Chroma then works serially on the server side, this benchmark probably just doesn't make sense.

This also results in worse times with parallel > 1 than with parallel = 1, I think.

Let me test this right now. Update: Yes, parallel > 1 is slower then parallel = 1, even with this snippet above, so i deleted this part of the benchmark.

Due to how the WAL is currently implemented, mixing reads and writes will result in lots of blocking, even for reads. Regarding the client.close(), lots of people expressed various degrees of frustration with the lack of it, as depending on your use case, it can exhibit differently—sockets left open, open file handles, failures during shutdown, etc.

I'll try to get the close done shortly.