What should be sent in the solution?

Zip-archive with the solution should contain jsonl files named according to the tasks titles, for example, RuMedDaNet.jsonl or RuMedTest.jsonl. An example solution can be found here.

After downloading the solution, I get an error, what should I do?

Please make sure once again that the format of the solution is correct (files naming and format, exact match of identifiers, etc.). If the error persists, create an issue in the project repository.

Is it possible to upload an incomplete archive?

Yes, you can submit solutions to specific tasks for review.

How many times a day can solutions be downloaded?

We trust the scientific community. We leave the question of the permissible number of solution checks on the conscience of each of the participants.

Is there a prize for solving tasks?


How can I refer to this work?

Please use the article:

@misc{blinov2022rumedbench, title={RuMedBench: A Russian Medical Language Understanding Benchmark}, author={Pavel Blinov and Arina Reshetnikova and Aleksandr Nesterov and Galina Zubkova and Vladimir Kokh}, year={2022}, eprint={2201.06499}, archivePrefix={arXiv}, primaryClass={cs.CL} }

I don't know where to start, is there any tutorial?

You can see the existing solutions in the project repository.

Do I need to register on the site to download the solution?

No, you can check your solutions without registering, just upload the archive with answers through the "Upload solution" button on the "Leaderboard" page.

I found a bug / I have a suggestion / questions about the benchmark, where can I report it?

Please open requests in the Issues section or write to MedBench@yandex.ru.

Does this project have a discussion group?

No, all discussions are through the tools in the GitHub-repository.

Under what license is the data distributed?

All data is available under license CC BY-NC 4.0.

Does the data contain personal information / how is it protected?

No, the data does not contain personal information.

Is there a deadline for submitting solutions?

No, we do not set a deadline for submitting solutions.

Can I check my solution without posting it on the leaderboard?

Yes, send the solution for verification, the result will be available only to you. In order for your solution to appear on the leaderboard, it must undergo additional verification and its code must be published in the repository.

How to upload a solution so that it gets on the leaderboard?

If you want your solution to appear on the leaderboard, you need to issue it as a pull-request or send the code for verification to MedBench@yandex.ru.

What organization is this benchmark affiliated with?

None, this benchmark is an open initiative of interested researchers.

Are there any rules or restrictions for developing systems?

"Manual" markup of data testing is prohibited. Beyond that, the choice of methods and details of the automatic intelligent systems implementation is at the discretion of the participants.

Can additional data/markup be used?

Yes, participants can use additional training data, but when publishing a solution such data must also be made publicly available.