Add a Workshop

Workshop is a peer assessment activity in five phases controlled by the instructor.  Students submit their work via an online text tool and attachments, then they are allocated the work of another student or students. They assess their peer(s) work using a peer assessment form the instructor creates. Students are graded for their submission and their assessment of their peers' work; however, instructors can set grading out of zero and require only comments.  If grades are used, instructors can always override the grade average of student assessments, and they can adjust the mark that CourseSpaces gives for assessment.  Finally, grades and/or comments are released when the Workshop is closed.  Peer assessment can be set to blind review so students do not know whose work they are assessing and don't know who is assessing their own work.  Workshop is considered an "advanced activity," but don't let that put you off. Once you become familiar with how it works, you'll see it can be an effective and efficient way to automate peer review/ assessment.

Some features to know about

  • No group submissions are possible in Workshop.
  • Workshop activity creates two grade items in the course Gradebook and they can be aggregated there as needed.
  • The process of peer assessment and understanding the assessment form can be practised in advance on example submissions.
  • The instructor can select some submissions and publish them so they are available to the others at the end of Workshop activity.

Workshop phases

Workshop usually follows a straight path: Setup, Submission, Assessment, Grading/Evaluation, and ending with the Closed phase. However, an advanced recursive path is also possible.  Activity progress is visible in the Workshop planner tool that displays all Workshop phases and highlights the current one. The planner tool also lists all the tasks the user has in the current phase with an indication of whether the task is finished, not yet finished, or failed. In the illustration below, the activity is in the submission phase with four students out of nine having submitted their work.

workshop visual planner

Setup phase

In this initial phase, students cannot do anything (neither modify their submissions nor their assessments). Instructors use this phase to change workshop settings, modify the grading strategy or tweak assessment forms. You can switch to this phase any time you need to change the Workshop setting and prevent users from modifying their work.

Submission phase

In the submission phase, students submit their work. Access control dates can be set so that even if the Workshop is in this phase, submitting is restricted to the given time frame. Submission start date (and time), submission end date (and time), or both can be specified. It's difficult to accommodate late submissions, so we recommend that students be told they will not be reviewed if late. When submissions are complete, the instructor allocates submissions for assessment (auto or manually).

Assessment phase

In this phase, students assess the submissions allocated to them for the review. As in the submission phase, access can be controlled by date and time.

Grading evaluation phase

The major task during this phase is for the instructor to approve/adjust the final grades for submissions and assessments and provide feedback to authors and reviewers. Workshop participants cannot modify their submissions or their assessments in this phase. Instructors can manually override the calculated grades. Also, selected submissions can be set to publish so they become available to all Workshop participants in the next phase.

Closed

When the Workshop is switched to this phase, the final grades calculated in the previous phase are pushed into the course Gradebook.This will result in the Workshop grades appearing in the Gradebook and in the Workshop activity. In this phase, students can view their submissions, their submission assessments, and any published submissions.

Workshop grading

The grades for a Workshop activity are obtained at several stages and then finalized. The following scheme illustrates the process and also provides the information in what database tables the grade values are stored.

Every student gets two numerical grades that are pushed to the course Gradebook. During the Grading evaluation phase, cthe instructor can let the Workshop module calculate these final grades. Grades are stored in the Workshop module only until the activity is switched to the final (Closed) phase. Therefore it is safe to play with grades unless you are happy with them and then close the Workshop and push the grades into the Gradebook. You can even switch the phase back, recalculate or override the grades and close the Workshop again so the grades are updated in the Gradebook again. You can override the grades in the Gradebook, too.

During the grading evaluation, Workshop grades report provides you with a comprehensive overview of all individual grades. The report uses various symbols and syntax:

Value Meaning
- (-) < Alice There is an assessment allocated to be done by Alice, but it has been neither assessed nor evaluated yet
68 (-) < Alice Alice assessed the submission, giving the grade for submission 68. The grade for assessment (grading grade) has not been evaluated yet.
23 (-) > Bob Bob's submission was assessed by a peer, receiving the grade for submission 23. The grade for this assessment has not been evaluated yet.
76 (12) < Cindy Cindy assessed the submission, giving the grade 76. The grade for this assessment has been evaluated 12.
67 (8) @ 4 < David David assessed the submission, giving the grade for submission 67, receiving the grade for this assessment 8. His assessment has weight 4
80 (20 / 17) > Eve Eve's submission was assessed by a peer. Eve's submission received 80 and the grade for this assessment was calculated to 20. Teacher has overridden the grading grade to 17, probably with an explanation for the reviewer.

Grade for submission

The final grade for every submission is calculated as weighted mean of particular assessment grades given by all reviewers of this submission. The value is rounded to a number of decimal places set in the Workshop settings form.

Course facilitator can influence the grade for a given submission in two ways:

  • by providing their own assessment, possibly with a higher weight than usual peer reviewers have
  • by overriding the grade to a fixed value

Grade for assessment

Grade for assessment tries to estimate the quality of assessments that the participant gave to the peers. This grade (also known as grading grade) is calculated by the artificial intelligence hidden within the Workshop module as it tries to do typical teacher's job.

During the grading evaluation phase, you use a Workshop subplugin to calculate grades for assessment. At the moment, only one subplugin is available called Comparison with the best assessment. The following text describes the method used by this subplugin. Note that more grading evaluation subplugins can be developed as Workshop extensions.

Grades for assessment are displayed in the braces () in the Workshop grades report. The final grade for assessment is calculated as the average of particular grading grades.

There is not a single formula to describe the calculation. However the process is deterministic. Workshop picks one of the assessments as the best one - that is closest to the mean of all assessments - and gives it 100% grade. Then it measures a 'distance' of all other assessments from this best one and gives them the lower grade, the more different they are from the best (given that the best one represents a consensus of the majority of assessors). The parameter of the calculation is how strict we should be, that is how quickly the grades fall down if they differ from the best one.

If there are just two assessments per submission, Workshop can not decide which of them is 'correct'. Imagine you have two reviewers - Alice and Bob. They both assess Cindy's submission. Alice says it is a rubbish and Bob says it is excellent. There is no way how to decide who is right. So Workshop simply says - ok, you both are right and I will give you both 100% grade for this assessment. To prevent it, you have two options:

  • Either you have to provide an additional assessment so the number of assessors (reviewers) is odd and workshop will be able to pick the best one. Typically, the teacher comes and provide their own assessment of the submission to judge it
  • Or you may decide that you trust one of the reviewers more. For example you know that Alice is much better in assessing than Bob is. In that case, you can increase the weight of Alice's assessment, let us say to "2" (instead of default "1"). For the purposes of calculation, Alice's assessment will be considered as if there were two reviewers having the exactly same opinion and therefore it is likely to be picked as the best one.

Backward compatibility note

In Workshop 1.x this case of exactly two assessors with the same weight is not handled properly and leads to wrong results as only the one of them is lucky to get 100% and the second get lower grade.

It's not final grades what is compared

It is very important to know that the grading evaluation subplugin Comparison with the best assessment does not compare the final grades. Regardless the grading strategy used, every filled assessment form can be seen as n-dimensional vector of normalized values. So the subplugin compares responses to all assessment form dimensions (criteria, assertions, ...). Then it calculates the distance of two assessments, using the variance statistics.

To demonstrate it on example, let us say you use grading strategy Number of errors to peer-assess research essays. This strategy uses a simple list of assertions and the reviewer (assessor) just checks if the given assertion is passed or failed. Let us say you define the assessment form using three criteria:

  1. Does the author state the goal of the research clearly? (yes/no)
  2. Is the research methodology described? (yes/no)
  3. Are references properly cited? (yes/no)

Let us say the author gets 100% grade if all criteria are passed (that is answered "yes" by the assessor), 75% if only two criteria are passed, 25% if only one criterion is passed and 0% if the reviewer gives 'no' for all three statements.

Now imagine the work by Daniel is assessed by three colleagues - Alice, Bob and Cindy. They all give individual responses to the criteria in order:

  • Alice: yes / yes / no
  • Bob: yes / yes / no
  • Cindy: no / yes / yes

As you can see, they all gave 75% grade to the submission. But Alice and Bob agree in individual responses, too, while the responses in Cindy's assessment are different. The evaluation method Comparison with the best assessment tries to imagine, how a hypothetical absolutely fair assessment would look like. In the Development:Workshop 2.0 specification, David refers to it as "how would Zeus assess this submission?" and we estimate it would be something like this (we have no other way):

  • Zeus 66% yes / 100% yes / 33% yes

Then we try to find those assessments that are closest to this theoretically objective assessment. We realize that Alice and Bob are the best ones and give 100% grade for assessment to them. Then we calculate how much far Cindy's assessment is from the best one. As you can see, Cindy's response matches the best one in only one criterion of the three so Cindy's grade for assessment will not be as high.

The same logic applies to all other grading strategies, adequately. The conclusion is that the grade given by the best assessor does not need to be the one closest to the average as the assessment are compared at the level of individual responses, not the final grades.

Adding a Workshop

To add a new workshop to a course, the instructor has to follow the steps:

  1. With the editing turned on, in the section you wish to add your workshop, click the "Add an activity or resource" link (or, if not present, the "Add an activity" drop down menu )and choose Workshop.
  2. The "Adding a new workshop" form will be displayed
  3. Enter the Workshop Name and give a description of the workshop in the “Introduction” field.
  4. Choose the Workshop features, more details in Workshop Features
  5. Choose the Grading Settings, for more details on this refer to the sub-section Grading Settings
  6. Choose the Submission Settings, for more details on this refer to the sub-section Submission Settings
  7. Specify the Assessment Settings, for more details on this refer to the sub-section Assessment Settings
  8. Specify the Access Control, this deals with when the workshop is open for submission and assessment, for more details refer to the sub-section Access Control
  9. Choose the Common Module Settings, for more details on this refer to the sub-section Common Module Settings
  10. Finally set the Restrict Access settings as well as Activity Completion. After completing this the Instructor can also set any advanced settings they wish to do. For more details simply refer to the Restrict Access and Activity Completion sub-sections.

Updating a Workshop

To update a workshop simply make sure that you are in Editing Mode and then click on the update workshop button, represented by two arrows facing opposite directions. Most of the settings mentioned in "Adding a Workshop" will be available for change but please note some features may only be changed during the setup phase. This is mentioned in detail in each of the sub-sections

Workshop features

Students can assess example submissions for practice before assessing their peers' work if this feature is enabled. They can compare their assessments with reference assessments made by the Instructor. The grade will not be counted in the grade for assessment.

Instructors need to upload one or more example submissions and the corresponding reference assessment to support this function:

  1. Click the ‘Add example submission’ button in the first page of the workshop setup for the assignment.
  2. Click the ‘continue’ button and provide a corresponding reference assessment based on the assessment form. If you fail to provide corresponding reference assessment, you can assess the example submissions later by clicking ‘assess’ button in the first page of the workshop setup. Also, unassessed example submissions will be highlighted in pink by the CourseSpaces system.
  3. Instructors can also edit the reference assessment later by clicking the ‘re-assess’ button in the first page.

Peer assessment:

If this feature is enabled, a student will be allocated a certain amount of submissions from his peers to assess. He will receive a grade for each assessment, which will be added together with the grade for his own submission and this will be used as his final grade for this assignment.

This is the key feature of workshop: To encourage students to assess the work of their peers and learn from each other. Through this, they will see the strengths of their classmates’ submissions and have a better understanding about how to do a good job. In addition, the advices they get from their peers will give them a more comprehensive view of their own work: The comments from their peers will point out the weakness of their work, which is generally difficult to find out by themselves.

Self-assessment:

If this option is turned on, a student may be allocated his own work to assess. The grade he receives from assessment of his own work will be counted into the grade for assessment, which will be added together with the grade for submission and used to calculate his final grade for this assignment.

This setting enables Instructors to see whether students can find out the strength and weakness of their own submissions and judge them objectively. It is a good way to help students think more comprehensively.

Grading settings

There are three settings in this section in the basic mode but with the click of the “Show Advanced” button there is one additional option. This button is found in the top right hand corner of the Grading Settings section of the form.

  • Grade for Submission: This sets the maximum grade a student can attain from a Instructor for a given submission. It is scaled between 0-100
  • Grade for Assessment: Sets the maximum grade a student can receive for assessing other students’ work. It is also scaled between 0-100

Grading Strategies

The grading strategy chosen here determines the assessment form students will use and this method is also used for grading submissions. Please note that the descriptions of the grading strategies given are just summaries, for a detailed description refer to the page Grading Strategies.

  • Accumulative grading: Comments and a grade are given regarding the aspects of the workshop specified.
  • Comments: Comments are given but no grade can be given to the specified aspects
  • Number of Errors: A yes/no assessment is used and comments are given for specified assertions
  • Rubric: A level assessment is given regarding specified criteria

To select the grading strategy simply select one of the four options from the drop down menu.

The only advanced setting given for grading settings is the ability to set how many decimal places are allowed in the grade. By default this is 0; that is grades can only be integers. To set this simply make sure you have clicked the “Show Advanced” button and then select a number from 0-5 from the drop down menu labelled “Decimal Places in Grade” found below the menu for selecting the grading strategy.

It is important to take note of the fact that grading strategies can't be changed once we have entered the submission phase of a workshop

Submission settings

These settings are to outline to students what they must submit as well as the control what students are allowed to submit. This is regarding uploading files, the number of files they can upload and additionally the size of each file. In the advanced settings for this section Instructors can also decide if they would like to allow late submission.

  • Instructions for submission: Here the Instructor should fill in instructions on what students must submit for the workshop submission.
  • Maximum number of submission attachments: In this drop down box Instructors select the number of attachments a student is allowed to upload in this workshop submission. By default it is set to 0 and Instructors can allow as many as 7 attachments for a single workshop submission.
  • Maximum File Size: Here Instructors can control how big of a file students are allowed to upload. The range of sizes starts from 10KB to 2MB
  • Late Submissions: After clicking the “Show Advanced” button in the top right of the submission settings Instructors can check the box indicating the allowance of late submissions. If not checked then students must either submit before the deadline of the submission period or not at all.

Once the workshop has been made we can then set more settings relating to submissions. This is done through clicking on the menu highlighted below which is found when you click on the workshop’s link or after clicking “Save and Display” on completion of the workshop. To access the menu simply click on “Allocate Submissions”. It is highlighted in the picture by the red box.

Manual Allocation

In manual allocation menu once a student has submitted work the Instructor can then choose which other students get to access their work. If the student has not submitted any work the Instructor cannot assign other students to access the work. However without anything submitted the Instructor can choose which students’ work this student will access.

Note that if a student has not submitted any work they can only be assigned to review other students who have submitted work and likewise other students cannot be assigned to review any student who hasn’t submitted any work.

Random Allocation

In random allocation the Instructor is given 5 settings that determine how the random allocation will work.

  • Number of reviews: Here the Instructor picks between 0 and 30 reviews for either each submission or per reviewer. That is the Instructor may choose to either set the number of reviews each submission must have or the number of reviews each student has to carry out
  • Prevent Reviews: If the Instructor wishes for students of the same group to never review each other’s work, as most likely it is their work too in a group submission, then they can check this box and CourseSpaces will ensure that they are only allocated other students out of their group’s work to access
  • Remove current allocations: Checking this box means that any manual allocations that have been set in the Manual Allocation menu will be removed
  • Can access with no submission: Having this box checked allows students to assess other students’ work without having already submitted their own work.
  • Add self assessments: This options when checked make sure that as well as assessing other students’ work they must also assess their own. This is a good option to teach students how to be objective to their own work.

Assessment settings

Instructors can write instructions for assessments online. This is very useful to help students have a better understanding about the important points of a task before assessing their classmates’ submissions.

Mode of example assessment:

By clicking the 'Show advanced' button at the upper right corner of the Assessment settings field, you can see a drop down menu named Mode of examples assessment. This setting is available only after the 'use examples' check box in the Workshop Features section has been ticked.

There are three options in this drop down menu: The first option means that the assessment of the example submission is voluntary, while the second and the third ones make it mandatory, which either requires students to assess example submissions before submitting their own work or after their own submission but before peer-assessment.

After the workshop has been made, Instructors can set more settings related to assessments-Edit assessment form, Grading evaluation settings and Workshop toolbox.

Editing assessment form

In order to set the criterion for an assignment, Instructors need to fill out an assessment form during the setup phase. Students can view this assessment form in the submission phase and focus on what is important about the task when working on their assignment. In the next phase-the assessment phase, students will assess their peers’ work based on this assessment form.

According to the grading strategy chosen in the grading settings, Instructors will get corresponding original assessment form to edit by clicking ‘Edit assessment form’ button in the first page of the workshop setup for the assignment. The grading strategy can be one of Accumulative grading, Comments, Number of errors or Rubrics. Instructors can set each criterion in detail in the assessment form.

Grade evaluation settings

In the grade evaluation phase, Instructors can do specific settings for calculation of the grade for assessments.

Grade calculation method:

This setting determines how to calculate grade for assessments. Currently there is only one option- comparison with the best assessment. For more detail, please refer to 4.2 Grade for assessment subsection of Workshop module.

First, Comparison with the best assessment method will try to imagine what a hypothetical absolutely fair assessment would look like.

For example, a Instructor uses Number of errors as grading strategy to peer-assess one assignment. This strategy uses a couple of assertions and assessors just need to check if the given assertion is passed or failed. That is, they only need to choose ‘yes’ or ‘no’ for each criterion in the assessment form. In this case, there are three assessors, Alice, Bob and Cindy. And the assessment form contains three criteria. The author will get 100% grade if all the criteria are passed, 75% if two criteria are passed, 25% if only one criterion is passed and 0% if the assessor gives ‘no’ for all three assertions. Here are the assessments they give to one certain work:

Alice: yes/yes/no
Bob: yes/yes/no
Cindy: no/yes/yes

Then the best assessment will be: Yes/yes/no

Second, workshop will give the best assessment 100% grade. Next it will measure the ‘distance’ from other assessments to this best assessment. The farther the distance, the lower grade the assessment will receive. And Comparison of assessments setting, next to the Grade evaluation setting, will determine how quickly the grade falls down if the assessment differs from the best one.

Note: Comparison with the best assessment method will compare responses to each individual criterion instead of comparing the final grades. In the example above, all of the three assessors give 75% to the submission. However, only Alice and Bob will get 100% grade for their assessments, while Cindy will get a lower grade. Because Alice and Bob agree in individual responses too, while the responses in Cindy’s assessment are different.

Comparison of assessments:

This setting has 5 options: very lax, lax, fair, strict and very strict. It specifies how strict the comparison of assessment should be. By using comparison with the best assessment method, all assessments will be compared with the best assessment picked up by workshop. The more similar one assessment is with the best assessment, the higher grade this assessment will get, and vice versa. This setting determines how quickly the grades fall down when the assessments differ from the best assessment.

Workshop toolbox

Clear all aggregated grades:

Clicking this button will reset aggregated grades for submission and grades for assessment. Instructors can re-calculate these grades from scratch in Grade evaluation phase.

Clear assessments:

By clicking this button, grades for assessments along with grades for submission will be reset. The assessment form will remain the same but all the reviewers need to open the assessment form again and re-save it to get the given grades calculated again.

Miscellaneous settings

Conclusion

It is possible to add some custom text which the students will see once they reach the end of the workshop process. This might be a general summary or suggestions on what should be done next, such as writing a blog post to reflect on the experience.

Access control

The whole of this section falls under the category of “Advanced Settings”. Without having clicked the “Show Advanced” button there are no options under the access control section.

Access Control deals with setting submission times and assessment times for the workshop. That is the time when students can start submitting as well as the deadline for submitting, and similarly the time assessments for other students’ work start and when they must finish assessing other students work.

All access control dates - open for submissions from, submissions deadline, open for assessment from and assessment deadline - are displayed in the course calendar.

Access control lets Instructor decide if they want a workshop with a closed schedule or one that is open ended. Setting an opening time but leaving the deadline open makes it an ongoing activity. To set the opening time and deadline for either submissions or assessments Instructors must click the enable button next to the option they want to set. Once the enabled has been checked the 5 drop down boxes will be activated and available for change.

The first three boxes correspond to the date and we can either set this box by box or by selecting the date in the calendar that pops up when any of the date boxes are selected. The last two boxes correspond to the time in 24 hour time, the first of the two being hours and the second being minutes. Instructors simply set the desired time for each of the sections they wish to activate.

Common module settings

By clicking the 'Show advanced' button at the upper right corner of the common CourseSpaces settings field, you will be able to see two more settings, Grouping and Available for group members only, besides Group mode, Visible and ID number in the basic mode.

Group mode:

    • No groups - There are no groups and each student of the course is a member of the same big community. The whole workshop is available to all the students of the course. All students can submit their work in one single submission area.
    • Separate groups - Each group member can only submit his work within the separate group based submission area, while other groups are invisible. Instructors can sort submissions by ‘Group’ or ‘View all participants’.
    • Visible groups - Each group member works within his own group. All students submit their work within the same submission area, but they can choose their own group to associate their submissions with before uploading. Unlike separate group mode, all other groups are visible but submissions of other groups are read-only. Instructors can sort submissions by ‘Group’ or ‘View all participants’.

Note: If ‘Separate group’ mode or ‘Visible group’ mode is selected, students need to be part of at least one group to get peer-assessments allocated to them by this tool. Nevertheless, students who do not belong to any groups can still be given new self-assessments or have existing assessments removed.

Grouping:

Grouping is a cluster of groups within one course. Students assigned to groups of one grouping are able to access the workshop and work together if that grouping is selected.

This setting helps Instructors a lot when they have many different topics for projects paired with different activity types. For example, there are four groups in one course: groups A, B, C and D. Groups A and B are supposed to write in a forum while groups C and D are supposed to make a wiki. The Instructor can create a forum grouping containing groups A and B, and another wiki grouping containing groups C and D. Within each grouping, you have two sub-groups for an added layer of distinction.

Available for group members only:

If this setting is enabled, only students assigned to groups within the selected grouping can access the activity or resource.

Visible:

Instructor can choose whether to show or hide the activity or resource.

ID number:

Each unique ID number associates with an activity, which is easier to type than the name of the activity. This setting aims at providing a way to identify an activity for grade calculation purposes. If the activity is not included in any grade calculation, then the ID number field can be left blank.

The ID number can also be set on the edit grade calculation page in the Gradebook, but it can only be edited on the update activity page of the module in a course context.

Restrict access

Allow access from/until:

Access from/to date determines the exact period during which students can access an activity via a link on the course page. Unlike Availability Settings, outside the set dates students can not access this activity at all. However, in the case of Availability Settings, students can still view the activity description after the set period.

Grade conditions:

This setting requires students to meet certain grade conditions before accessing the activity. Instructors can choose one particular grade condition from the drop down menu and then fill in the blanks next to the menu with the acceptable range of grade. Multiple grade conditions can be set for one single activity by clicking ‘Add 2 grade conditions to form’ button. In that case, students need to meet all the grade conditions in order to access the activity.

Before activity can be accessed:

Instructors can decide whether they want their students to see the activity before it is available. They can choose from the drop down menu either to show the activity greyed-out with restriction information or just hide the activity entirely.

Activity completion

Completion tracking:

This setting allows Instructors to specify conditions that define when an activity is considered to be completed. If this setting is enabled, activity completion will be tracked either manually or automatically, based on those certain conditions. Multiple conditions can be set for the completion of one activity, like when a grade has been reached or a certain number of posts have been made. If so, only after all the conditions are met can the activity be considered completed

A tick next to the name of the activity on the course page indicates whether the activity is completed or not.

Require view:

If this setting is enabled, students need to view the activity before they can complete it. In most cases, clicking the link is enough to view the activity. Also, please do not turn this option on unless you have extra requirements, because it will make extra work for the server and students generally can not meet other requirements without viewing the activity.

Require grade:

If this option is turned on, an activity will be considered complete when a student receives a grade. It does not matter how high the grade is: Getting any grade will mark the activity completed. Pass and Fail icons can be displayed so that the activity becomes ‘completed, passed’ or ‘completed, not passed’ instead of just ‘completed' .

Expect completed on:

If this setting is enabled, it will specify the date when the activity is expected to be completed. The date will not be shown to students and will only be displayed in the activity completion report.