Case 1. Input and Output-Based Grading

Add the following folders & files in the 기본 세팅된 (default setup) file structure.

• Input folder / Output folder: Defines the test cases for grading.

• testcases.py file: Sets the partial scores and correct/incorrect messages for each test case.

• Input Folder

  • Save test case input values as .txt files.
  • Each case should contain one input value.

• Output Folder

  • Save test case output values as .txt files.
  • Save corresponding output values for each input value created in the input folder.

2. Writing Grading Messages

Define the correct and incorrect messages for each test case set in the test cases.
Also, set partial scores for each test case.

# Correct messages for each test case
correct_messages = [
    "testcase 1. Correct!",
    "testcase 2. Correct!"
]

# Incorrect messages for each test case
wrong_message = [
        "testcase 1. Incorrect!",
        "testcase 2. Incorrect!"
     ]

# Partial scores for each test case
scores = [50, 50]

3. Writing the Grading Code in grader.py

Run the student's code for each test case and check the output. Then, compare the output result of the student's code with the expected output value (from the output file) to determine whether the exercise is correct or incorrect.

Partial scores can be provided for each test case, and the final grading result is sent to the platform (LXP) via elice_utils.secure_send_score(total_score).

Here is an example code:

import os
import subprocess
import sys
from testcases import *
sys.path.append(os.getcwd())
from grader_elice_utils import EliceUtils  # isort:skip

elice_utils = EliceUtils()
elice_utils.secure_init()

SUM_TESTCASE_SCORES = 100
INPUT_DIR = '.elice/input/'
OUTPUT_DIR = '.elice/output/'

try:
    total_score = 0

    # 0. Load input and output files
    input_data = [x for x in os.listdir(INPUT_DIR) if x.endswith('.txt')]
    output_data = [x for x in os.listdir(OUTPUT_DIR) if x.endswith('.txt')]

    # 1. Check the number of test cases
    if len(input_data) != len(output_data):
        sys.exit(1)
    NUM_TESTCASES = len(input_data)

    # 2. Check file names
    matching = True
    for i in range(1, NUM_TESTCASES + 1):
        input_file = '%d.txt' % i
        output_file = '%d.txt' % i

        if input_file not in input_data:
            matching = False
        if output_file not in output_data:
            matching = False
    if not matching:
        sys.exit(1)

    # 3. Grade each test case
    for i in range(0, NUM_TESTCASES):
        testcase_score = scores[i]
        input_file = '%d.txt' % (i+1)

        input_text = subprocess.run(['cat', '%s%s' % (INPUT_DIR, input_file)],
                                    stdout=subprocess.PIPE).stdout
        result = subprocess.run(['/bin/bash', '.elice/runner.sh'],
                                input=input_text,
                                stdout=subprocess.PIPE)
        student_result = result.stdout.decode('utf-8')

        answer = ''.join(open('%s%s' % (OUTPUT_DIR, input_file)).readlines())

        student_result = student_result.strip()
        answer = answer.strip()

        # Determine correct or incorrect for each test case
        if answer == student_result:
            total_score += testcase_score # Add partial score for the test case
            elice_utils.secure_send_grader('✅ {} \n'.format(correct_messages[i]))

        else:
            elice_utils.secure_send_grader('❌ {} \n'.format(wrong_message[i]))

    # 5. Calculate final score
    total_score = int(total_score)
    elice_utils.secure_send_score(total_score)

except Exception as err:
    elice_utils.secure_send_grader('An error occurred during grading. Please check if your code is running correctly.\n')
    elice_utils.secure_send_score(0)
    sys.exit(1)