You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* After installation, the following commands are available:
21
19
*`ragability_info` : show the versions of all relevant Python packages installed
22
-
" `ragability-cc-wc1` : convert the wiki-contradict based corpus to ragability format
20
+
*`ragability_cc_wc1` : convert the wiki-contradict based corpus to ragability format
23
21
*`ragability_query` : given an input file with facts and queries, a list of candidate LLMs and a prompt template, produce and output file that contains
24
22
LLM answers to the queries
25
23
*`ragability_check` : given a query result file and a judge LLM, evaluate the answers received against the pre-defined answers and create an output file that contains the evaluation scores and meta-information for each example
26
24
*`ragability_eval` : given the data created with `ragability_check`, calculate detailled performance statistics
27
-
*`ragability_hjson_info` : show some infor about the number of entries and keys present in a hjson, json, or jsonl file
25
+
*`ragability_hjson_info` : provides an overview of the major information present in a hjson, json, or jsonl file
28
26
*`ragability_hjson_cat` : concatenate several hjson, json or jsonl files into one hjson or json file
29
27
*`llms_wrapper_test`: for the configured/specified LLMs, test if they are working and returning an answer. This is useful to check a config file and
30
28
test if all the API keys are correctly set and working
31
29
* All commands take the `--help` option to get usage information
32
30
33
31
LLMs currently supported: support is based on the [LiteLLM](https://github.com/BerriAI/litellm) backend via the [llms_wrapper](https://github.com/OFAI/python-llms-wrapper/) package. The supported LLMs are listed [here](https://docs.litellm.ai/docs/providers/)
34
32
35
-
### Current usage
33
+
### Example usage
36
34
37
-
Example usage with the converted wiki-contradict dataset:
35
+
with the converted wiki-contradict dataset:
38
36
39
37
* (optional conversion step, the current converted dataset is already part of the repo): convert the dataset tsv file to ragability format
0 commit comments