pypi i pytrec-eval-git


pytrec_eval is an Information Retrieval evaluation tool for Python, based on the popular trec_eval.

by Christophe Van Gysel

0.5 (see all)
pypi i pytrec-eval-git


pytrec_eval is a Python interface to TREC's evaluation tool, trec_eval. It is an attempt to stop the cultivation of custom implementations of Information Retrieval evaluation measures for the Python programming language.


The module was developed using Python 3.5. You need a Python distribution that comes with development headers. In addition to the default Python modules, numpy and scipy are required.


Installation is simple and should be relatively painless if your Python environment is functioning correctly (see below for FAQs).

pip install pytrec_eval


Check out the examples that simulate the standard trec_eval front-end and that compute statistical significance between two runs.

To get a grasp of how simple the module is to use, check this out:

import pytrec_eval
import json

qrel = {
    'q1': {
        'd1': 0,
        'd2': 1,
        'd3': 0,
    'q2': {
        'd2': 1,
        'd3': 1,

run = {
    'q1': {
        'd1': 1.0,
        'd2': 0.0,
        'd3': 1.5,
    'q2': {
        'd1': 1.5,
        'd2': 0.2,
        'd3': 0.5,

evaluator = pytrec_eval.RelevanceEvaluator(
    qrel, {'map', 'ndcg'})

print(json.dumps(evaluator.evaluate(run), indent=1))

The above snippet will return a data structure that contains the requested evaluation measures for queries q1 and q2:

    'q1': {
        'ndcg': 0.5,
        'map': 0.3333333333333333
    'q2': {
        'ndcg': 0.6934264036172708,
        'map': 0.5833333333333333

For more like this, see the example that uses parametrized evaluation measures.

Frequently Asked Questions

Since the module's initial release, no questions have been asked so frequently that they deserve a spot in this section.


If you use pytrec_eval to produce results for your scientific publication, please refer to our SIGIR paper:

  title={Pytrec\_eval: An Extremely Fast Python Interface to trec\_eval},
  author={Van Gysel, Christophe and de Rijke, Maarten},


pytrec_eval is licensed under the MIT license. Please note that trec_eval is licensed separately. If you modify pytrec_eval in any way, please link back to this repository.

No alternatives found
No tutorials found
Add a tutorial
No dependencies found

Rate & Review

No reviews found
Be the first to rate