Python Script-Based Grading
grader.sh is executed when the learner presses the submit button, and it runs grader.py to start the grading process. Below, I will explain the commonly used functions from elice_utils.py that are helpful in writing grader.py along with their general formats.
secure_send_grader()
: Outputs messages to the user via the terminalsecure_send_score()
: Sets the final score for the practice
from grader_elice_utils import EliceUtils
elice_utils = EliceUtils()
elice_utils.secure_init() # Import and initialize elice_utils.
try: # Proceeding with grading.
elice_utils.secure_send_grader('String') # Sends a string to the learner's terminal.
elice_utils.secure_send_score(100) # Sends the final score of the practice. When scoring 100, a correct answer bunny pops up.
except Exception: # In case an error occurs during the grading process
elice_utils.secure_send_score(0) # Sets the practice score to 0
sys.exit(1)
You can modify grader.py to grade the student's code in various ways.
Here are some key points to keep in mind while writing grader.py.
Note 1
Sometimes, comparing values in an if statement with mismatched formats or types can lead to errors like TypeError or RangeError. In such cases, the Grader halts immediately, and grading cannot be correctly performed. Therefore, it is recommended to follow this format when using an if statement:
try:
if (submission.drink_relfreq == drink[drink['Attend'] == 1]['Name'].value_counts(normalize=True)).all():
total_score += 100
elice_utils.secure_send_grader('You have correctly calculated the relative frequency!\n')
except Exception:
elice_utils.secure_send_grader('The relative frequency was not calculated correctly. Shall we try calculating it again?\n')
Note 2
In some cases, it may be necessary to specify the details of the error that occurred during grading. In this situation, please use internal exceptions to craft your messages. For more on Python's built-in exceptions, visit the following link: https://docs.python.org/ko/3/library/exceptions.html#concrete-exceptions
import student_codes
try:
if student_codes.answer == 12:
total_score += 100
elice_utils.secure_send_grader('Correct!\n')
# index range error
except IndexError:
elice_utils.secure_send_grader('The index is out of the specified range.\n')
# name error
except NameError:
elice_utils.secure_send_grader('You used a variable that hasn’t been declared.\n')
# zero division error
except ZeroDivisionError:
elice_utils.secure_send_grader('You cannot divide by 0.\n')
Note 3
If you provide the RuntimeError message that occurred during grading to the student, it could allow them to potentially leak test case details based on the message's content. Therefore, it is recommended to write error messages in the following manner:
# Not recommended
try:
# Grading code here
except Exception as err:
elice_utils.secure_send_grader('Internal Error:\n%s\n' % str(err))
elice_utils.secure_send_score(0)
sys.exit(1)
# Recommended method
try:
# Grading code here
except Exception:
elice_utils.secure_send_grader('An error occurred during grading. Please check if your code is running correctly.\n')
elice_utils.secure_send_score(0)
sys.exit(1)
"""
What if the grading code is written in the not recommended manner?
"""
# The full output of grader.py can be raised.
raise Exception(open('.elice/grader.py').read())