Runbox Grading Script

This document explains how to write a grading script for Runbox.

Runbox Grading Structure

Runbox grading is based on file uploads. Students submit their files, and a Python script validates these submissions. The total size of all files for each submission is limited to 50MB after compression. Scores and submissions are stored on the server for later review.

Recommended Submitable File Types

There are no specific limitations on the file types that can be submitted, but the following types are recommended. For Python-based practices, using pickle for object serialization is common. However, due to security reasons (such as executing arbitrary code), its use is not recommended (see official documentation).

JSON

Recommended as the file format for submissions in most cases, except competitions.

CSV

Recommended as the submission file format for competitions. For other cases, it is advised not to use CSV for both the submission file and the answer file but rather to use JSON.

PNG

Used for grading plot results from libraries like Matplotlib and seaborn. Instead of checking for exact image matches, it is preferable to calculate similarity.

NumPy array (.npy)

Used when grading functions that return NumPy arrays directly.

Writing the Grading Script

When a grading request is received, Runbox executes the grader.py file included in the uploaded zip file on the Runbox creation/management page. The Grader script can access the submissions submitted by the student at /mnt/elice/userfile. It validates these submissions using Python scripts and returns scores and messages to the student.

The zip file can include not only grader.py but also other resources needed during grading. For example, if you want to grade by comparing with a reference file, you can include this file in the zip. Files included in the zip can only be accessed by the grading script at /mnt/elice/grader, while the student cannot access them.

• To record the score, you simply need to print the score encoded in UTF-8 to the /var/run/elice/grade_score file. Both integer and floating-point scores are supported.
• For the message, you can print the feedback encoded in UTF-8 to the /var/run/elice/grade_message file to relay information to the student.

Here is a simple example of a grading script:

import json
from scipy.spatial.distance import cosine

score = None
message = ""

try:
    with open("/mnt/elice/grader/reference.json", "r") as f:
        reference = json.load(f)
    with open("/mnt/elice/userfile/result.json", "r") as f:
        user_result = json.load(f)

    similarity = cosine(
        reference,
        user_result['submission']
    )

    score = similarity
    if similarity < 0.1:
        score = "100"
        message = "good job!"
    elif user_result < 0.5:
        score = score = 100 * (1 - (similarity - 0.1) / 0.4)
        score = f"{score:.3f}"
        message = "well done"
    else:
        score = "0"
        message = "try again"

except FileNotFoundError:
    score = "0"
    message = "missing required files"

except Exception:
    score = "0"
    message = "something goes wrong :("

finally:
    if score is not None:
        with open("/var/run/elice/grade_score", "w") as f:
            f.write(str(score))

    with open("/var/run/elice/grade_message", "w") as f:
        f.write(message)

Executing the Grading Script

After composing the grading script, you can run the following command within Runbox to perform grading:

elice_grade result.json code.ipynb

For practices based on JupyterLab, it is advisable to provide the commands in advance within the learning materials to facilitate easy execution for learners. For practices based on Orange3 and VSCode, a user interface for executing the grading script is provided.

Example of grading code inserted within Jupyter Notebook learning materials

Complex Grading Example

• You can receive multiple files at once and grade based on weights assigned for each problem.
• For example, in tasks that generate graphs or images, you can grade by receiving the images or binaries.
• You can also grade based on the inference results of a simple machine learning model submitted.
  • Note that grading uses only CPU resources regardless of the student's runtime, so caution is needed.