Hands-on Tutorial for Model Owners¶
In this guide, you will learn how a Model Owner can use MedPerf to take part in a benchmark. It's highly recommend that you follow this or this guide first to implement your own model MLCube and use it throughout this tutorial. However, this guide provides an already implemented MLCube if you want to directly proceed to learn how to interact with MedPerf.
The main tasks of this guide are:
- Testing MLCube compatibility with the benchmark.
- Submitting the MLCube.
- Requesting participation in a benchmark.
It's assumed that you have already set up the general testing environment as explained in the setup guide.
Before You Start¶
Prepare the Local MedPerf Server¶
For the purpose of the tutorial, you have to initialize a local MedPerf server with a fresh database and then create the necessary entities that you will be interacting with. To do so, run the following: (make sure you are in MedPerf's root folder)
Download the Necessary files¶
A script is provided to download all the necessary files so that you follow the tutorial smoothly. Run the following: (make sure you are in MedPerf's root folder)
This will create a workspace folder
medperf_tutorial where all necessary files are downloaded.
Login to the Local MedPerf Server¶
The local MedPerf server is pre-configured with a dummy local authentication system. Remember that when you are communicating with the real MedPerf server, you should follow the steps in this guide to login. For the tutorials, you should not do anything.
You are now ready to start!
1. Test your MLCube Compatibility¶
Before submitting your MLCube, it is highly recommended that you test your MLCube compatibility with the benchmarks of interest to avoid later edits and multiple submissions. Your MLCube should be compatible with the benchmark workflow in two main ways:
- It should expect a specific data input structure
- Its outputs should follow a particular structure expected by the benchmark's metrics evaluator MLCube
These details should usually be acquired by contacting the Benchmark Committee and following their instructions.
To test your MLCube validity with the benchmark, first run
medperf benchmark ls to identify the benchmark's server UID. In this case, it is going to be
Next, locate the MLCube. Unless you implemented your own MLCube, the MLCube provided for this tutorial is located in your workspace:
After that, run the compatibility test:
Assuming the test passes successfuly, you are ready to submit the MLCube to the MedPerf server.
2. Submit the MLCube¶
How does MedPerf Recognize an MLCube?¶
The MedPerf server registers an MLCube as metadata comprised of a set of files that can be retrieved from the internet. This means that before submitting an MLCube you have to host its files on the internet. The MedPerf client provides a utility to prepare the files of an MLCube that need to be hosted. You can refer to this page if you want to understand what the files are, but using the utility script is enough.
To prepare the files of the MLCube, run the following command ensuring you are in MedPerf's root folder:
This script will create a new folder in the MLCube directory, named
assets, containing all the files that should be hosted separately.
Host the Files¶
For the tutorial to run smoothly, the files are already hosted. If you wish to host them by yourself, you can find the list of supported options and details about hosting files in this page.
Submit the MLCube¶
The submission should include the URLs of all the hosted files. For the MLCube provided for the tutorial:
- The URL to the hosted mlcube manifest file is
- The URL to the hosted mlcube parameters file is
- The URL to the hosted additional files tarball file is
Use the following command to submit:
medperf mlcube submit \ --name my-model-cube \ --mlcube-file "https://raw.githubusercontent.com/hasan7n/medperf/99b0d84bc107415d9fc6f69c4ea3fcdfbf22315d/examples/chestxray_tutorial/model_mobilenetv2/mlcube/mlcube.yaml" \ --parameters-file "https://raw.githubusercontent.com/hasan7n/medperf/99b0d84bc107415d9fc6f69c4ea3fcdfbf22315d/examples/chestxray_tutorial/model_mobilenetv2/mlcube/workspace/parameters.yaml" \ --additional-file "https://storage.googleapis.com/medperf-storage/chestxray_tutorial/mobilenetv2_weights.tar.gz"
The MLCube will be assigned by a server UID. You can check it by running:
3. Request Participation¶
Benchmark workflows are run by Data Owners, who will get notified when a new model is added to a benchmark. You must request the association for your model to be part of the benchmark.
To initiate an association request, you need to collect the following information:
- The target benchmark ID, which is
- The server UID of your MLCube, which is
Run the following command to request associating your MLCube with the benchmark:
This command will first run the benchmark's workflow on your model to ensure your model is compatible with the benchmark workflow. Then, the association request information is printed on the screen, which includes an executive summary of the test mentioned. You will be prompted to confirm sending this information and initiating this association request.
What Happens After Requesting the Association?¶
When participating with a real benchmark, you must wait for the Benchmark Committee to approve the association request. You can check the status of your association requests by running
medperf association ls. The association is identified by the server UIDs of your MLCube and the benchmark with which you are requesting association.
You have reached the end of the tutorial! If you are planning to rerun any of the tutorials, don't forget to cleanup:
To shut down the local MedPerf server: press
Cin the terminal where the server is running.
To cleanup the downloaded files workspace (make sure you are in the MedPerf's root directory):
- To cleanup the local MedPerf server database: (make sure you are in the MedPerf's root directory)
- To cleanup the test storage: