Rules

Registration
Data confidentiality
Submitting results
Viewing results
Publishing results

Team registration

  • Once you have successfully registered a team and emailed the signed agreement of data confidentiality to coronarystenoses@bigr.nl, you can download data.
  • Anonymous registration is not allowed; you must use your real name, affiliation and e-mail address to register.
  • The provided information will only be listed on the site if you choose to publish the results (see below)

Data confidentiality

  • CTA datasets and associated reference standard provided by the organizers may not be given or distributed under any circumstances to persons other than the person registered for the team and the people mentioned in the data confidentiality agreement.
  • The data of the evaluation framework may only be used for the evaluation of coronary artery stenoses detection/quantification and lumen segmentation methods, through the Rotterdam Coronary Artery Algorithm Evaluation framework.
  • When publishing results obtained using one of the provided set of coronary artery centerlines, you have to cite the corresponding paper describing the centerline extraction method. Citation details can be found in the publishing results section.
  • When publishing obtained results, you have to cite the paper describing the evaluation framework. Citation details can be found in the publishing results section.

Submitting results

  • Result archives to be uploaded on the submit page are limited to 100Mo.

Viewing results

  • Results on the training data can be viewed any time after the uploaded results have been processed.
  • Testing results can only be inspected after disclosure by the organizers.
    To prevent training on the testing data, results will be disclosed only once per algorithm.
    An algorithm is considered to be new as compared to another algorithm if any other changes apart from 1) tuning parameters (better training), 2) using different set of provided centerline, and 3) fixing bugs are performed.
    Disclosure for a submission can be requested once a week by sending an e-mail to the organizers.

Publishing results

  • Results of your submission will only be published on the website when a document (e.g. a journal paper) describing the method is provided. Please e-mail this document (or a link to it) to the organizers if you want to publish your results.
  • If a commercial system is evaluated no method description is necessary, but the system has to be publicly available and the exact name and version number has to be provided.
  • The organizers of the evaluation framework will check the method description before your results will be published on the website.
  • When publishing results obtained using one of the provided set of coronary artery centerlines, you have to cite the corresponding paper describing the centerline extraction method:
  • When publishing the obtained results, you have to cite the following journal paper describing the evaluation framework:

    H.A. Kirişli, M. Schaap, C. Metz, A.S. Dharampal, W.B. Meijboom, S.L. Papadopoulou, A. Dedic, K. Nieman, M.A. de Graaf, M.F.L. Meijs, M.J. Cramer, A. Broersen, S. Cetin, A. Eslami, L. Flórez-Valencia, K.L. Lor, B. Matuszewski, I. Melki, B. Mohr, I. Öksüz, R. Shahzad, C. Wang, P.H. Kitslaar, G. Unal, A. Katouzian, M. Orkisz, C.M. Chen, F. Precioso, L. Najman, S. Masood, D. Ünay, L. van Vliet, R. Moreno, R. Goldenberg, E. Vuçini, G.P. Krestin, W.J. Niessen, T. van Walsum,
    "Standardized evaluation framework for evaluating coronary artery stenosis detection, stenosis quantification and lumen segmentation algorithms in computed tomography angiography", Medical Image Analysis, 17(8), 859–876 (2013)