feat: Evaluation Module Prototype

[frayle-ons]

✨ Summary

These changes add a new evaluation module that can quantitatively evaluate the performance of VectorStores against a user-provided ground truth dataset using package-provided classification metrics.

The current implementation views the evaluation as a 'multi-class-single-label' problem, with each evaluation metric assessing only the top candidate result output from the VectorStore search method. Additional metrics could be considered such as 'hit@k' where we assess if the correct label is in the top K results of a VectorStore search, to give users more options on how to evaluate VectorStores.

The main entry point for the new code; the evaluation function, importable from the Evaluation module, has several parameters:

• one or more VectorStore objects or callable functions that would instantiate a vectorstore (allowing users to efficiently instantiate each VectorStore only when its evaluation run starts, saving memory),
• a list of string names to keep track of each corresponding vectorstore,
• a ground truth dataframe of ['qid', 'text', 'label'], where text is the query, and label is the ground truth correct label,
• a list of metrics to calculate (currenctly `['acccuracy', 'macro_precision', 'macro_recall', and 'macro_f1'] are available,
• (optionally) an output file name to save the results to.

The evaluation function performs several actions:

1. checks the validity of the input arguments
2. searches each vectorstore with the provided eval data queries.
3. computes specified metrics for each vectorstore search results using provided ground truth labels
4. collects all metric results for all vectorstores
5. returns the dataframe of results with metric names as columns and row index as the provided name of the vectorstore. Saves dataframe to file if output_file argument is provided with a valid .csv string.

Final thought:

• Currently the ground truth data assumes there is 1 label. We could make this module accept different kinds of ground truth data: one gold standard label per sample, or multiple ground truths per sample. This could mean we have different evaluation modes where the internal logic is different based on which kind of dataframe is passed (and possibly we have a separate argument in the evaluate function that specifies what mode of evaluation we're doing.

📜 Changes Introduced

• (feat:) introduces new evaluation function
• (feat:) new error classes and pandera schemas to ensure data
• (feat:) metrics.py to collect metrics to evaluate.
• (feat:) init file for new package module
• (feat:) smart vectorstore loading for memory efficiency

✅ Checklist

Passes all pre-commit checks.

🔍 How to Test

The tester should obtain some data that has queries and gold standard labels for a give vectorstore, loading the queries into a pandas dataframe with ['qid', 'text', 'label'] columns. All columns should be loaded with string types.

Next they should instantiate the corresponding VectorStore(s) they wish to evaluate against the ground truth dataset.

Passing these items are arguments to the new evaluate function as in the code below, and specifying a list of metric names, the user can run the script to see the evaluation function,

from classifai.evaluation import evaluate 


results = evaluate(
    vectorstores = [vectorstore1, vectorstore2],
    vectorstore_names = ["vectorstore_1", "vectorstore_2"],
    metrics = ["macro_precision", "accuracy"],
    ground_truth = eval_data_df,
    output_file="./classifai_eval_results.csv"
)

Additionally test the functionality that allows users to pass callable to the vectorstores argument of the eval function. To do this the user could write a zero-arg function that returns an instantiated vectorstore object and pass the function pointer as an argument (not the call the to function). See below for the normal process of instantiation versus the same vectorstore wrapped as a callable.

from classifai.vectorisers import HuggingFaceVectoriser
from classify.indexers import VectorStore


# normal instantiation 
vectoriser = HuggginFaceVectoriser('model_name'='sentence-transformers/all-MiniLM-L6-v2')

# normal vectorstore
vectorstore = VectorStore(
    file_name="./testdata.csv",
    data_type="csv",
    vectoriser=vectoriser,
)


# building a function that returns a vectorstore only when its time for its evaluation run.
def second_vectorstore():

    built_vectoriser = HuggingFaceVectoriser(model_name="sentence-transformers/all-MiniLM-L6-v2")

    built_vectorstore = VectorStore(
        file_name="./testdata.csv",
        data_type="csv",
        vectoriser=built_vectoriser,
        )
    
    return built_vectorstore


# this time passing the pointer to the callable to the second item in the list
results = evaluate(
    vectorstores = [vectorstore, second_vectorstore],
    vectorstore_names = ["in_memory_vectorstore", "memory_efficient_vectorstore"],
    metrics = ["macro_precision", "accuracy", "macro_recall", "macro_f1"],
    ground_truth = eval_data_df,
)

Finally try out some edge cases such as:

• Passing items that aren't a vectorstore or callable, or a callable that does not return a vectorstore object.
• make vectorstore_names and vectorstores different lengths.
• try passing a metric that does not exist
• pass the classification_suite string to the metrics, which triggers all current metrics to run
• pass a ground_truth dataframe with missing columns