Fig. 1

An overview of the A2A benchmarking system. We present a layered view with key interactions between the elements. Nore that the role of the user in an IR benchmarking scenario is to define the experiment (a combination of topics, document collection, human judgements, and a retrieval model with its parameters), which is accomplished through A2A front-end (bottom layer). The processing path leading from a topic to the evaluation report is presented in the Artifacts an Conceptual steps layers; the former layer presents the intermediate entities (artifacts) obtained at respective steps of the latter layer. The A2A back-end layer lists physical elements of the system’s pipeline involved at respective conceptual steps. The front end application starts an instance of the A2A back-end pipeline, parameterised per user request; a process queue is used for asynchronous scheduling and execution of the pipeline process. The back-end pipeline is implemented in Python 3.7, with Apache Solr search engine. We use NIST-supplied TRECeval script to calculate metrics. The process queue is implemented with Celery and Redis. The front-end application is implemented in Flask and deployed on a gunicorn web server