Moderation sessions: Establishing shared performance expectations among faculty at a distance

Author Information
Author(s): 
Jeff Grann
Institution(s) or Organization(s) Where EP Occurred: 
Capella University
Effective Practice Abstract/Summary
Abstract/Summary of Effective Practice: 

To promote quality online learning, administrators need effective practices to engage many faculty in establishing shared performance expectations. A moderation session is a synchronous meeting in which faculty discuss points of consensus and disagreement about their assessment of a student’s work. A facilitator works to identify the basis for faculty judgments and to articulate performance expectations shared by faculty.

Description of the Effective Practice
Description of the Effective Practice: 
Online courses are taught increasingly by instructors who did NOT create the content, discussions, or assignments of the course. This one course-many instructors model has several advantages (such as curricular alignment and consistency), but also introduces new challenges for instructors to deliver the courses. In particular, instructors may not hold the same performance expectations as the faculty that created the course. These differences in performance expectations can undermine the effectiveness of the program’s curriculum as learners receive inconsistent feedback on their performance.  Perceived differences in performance expectations may also undermine faculty sharing of effective teaching practices or discussion of instructional challenges. Unfortunately, few universities address the role of performance expectations explicitly, either through role responsibilities, committee charters, policy, or procedures.
Moderation sessions are a method for improving faculty-based learning outcome assessments by promoting shared performance expectations among faculty. Operationally, a moderation session involves faculty individually assessing a student’s work using a scoring guide and then as a group discussing points of consensus and disagreement. A facilitator asks questions to identify the basis for assessment judgments and to articulate shared performance expectations amongst the faculty. Facilitators may also prompt analysis of the scoring guide language and its concordance with faculty’s performance expectations. Based on findings from a moderation session, faculty may decide to revise course assignments or scoring guides to improve alignment amongst the program’s curriculum, instruction, and assessment practices. Inter-rater reliability statistics can also be calculated from these sessions as part of an evaluation of a program’s learning outcome assessment system.
Procedure
Setting up a moderation session: Clear and early communication is key to a successful moderation session. Initial communications about the moderation session should describe your goals, timeline, participation expectations, and the value of high-quality assessments for students' success. Make sure faculty know why they were selected to participate in the session and how their participation is key to the success of the session. Give faculty at least one week to complete the assessment before the scheduled moderation session. Also, ensure that the contact details for the conference call and any web conferencing environments are contained in the meeting invite.
Select an assignment and scoring guide for the moderation session that is relevant and appropriate for many program faculty to assess, such as a course early in the program or a final capstone course. Embedded assignments that directly measure program outcomes are excellent choices for moderation sessions.
You may intentionally select a student’s work to focus the moderation session on particular performance expectations or randomly select a student’s work. In either case, avoid revealing your selection criteria to minimize unintentionally biasing the faculty’s judgments. Make sure to remove all identifying information from the student’s work, including the student’s name, the document’s properties, comments, and any other identifying information.
If using a web conferencing environment, set up a poll question for each scoring guide criterion before the moderation session. The response options should be the performance levels. Make sure each poll is set to hide participant’s responses. The facilitator will reveal the poll results once the polling is complete.
Facilitating a moderation session: Begin the session by welcoming the faculty and reinforcing the goals for the session. Facilitators have found it helpful to describe at a high-level the agenda for the session. An effective practice is to have faculty answer each of the poll questions at the beginning of the meeting so the facilitator can review the poll results and decide which criterion to discuss first. To ensure the discussion is focused effectively on building shared performance expectations, it is recommended to discuss the most discordant criteria first as opposed to moving sequentially through the criteria as listed on the scoring guide. Once poll results are revealed, the facilitator uses observations and questions to identify the basis for faculty judgments, such as “What evidence is there in the student’s work suggesting a ‘distinguished’ level of performance on this criterion?” After working through a few criteria, the facilitator can often simply verbally state the results of the poll question to stimulate appropriate discussion. You may also consider collecting evaluation data at the end of the moderation session using poll questions.

After a moderation session: Thank faculty for participating in the moderation session and seek opportunities to recognize their service through inter-department and university communications. If the web conferencing environment saves poll results, inter-rater reliability statistics can be calculated and communicated to participants for monitoring the quality of outcome assessments over time.

Supporting Information for this Effective Practice
Evidence of Effectiveness: 

Since 2007, moderation sessions have been conducted periodically with program faculty in support of direct assessments of learning outcomes in bachelor’s and master’s degree capstone courses. Aggregate faculty survey data findings from these sessions are described below. Faculty participants report moderation sessions to be a productive use of their time (>90%) and enthusiasm to participate in future sessions(>95%). Participants also reported that the sessions are most effective when several roles participate, especially subject matter experts, faculty leadership, curriculum specialists, course developers, and assessment specialists. Most participants reported one hour to be sufficient (76%), with the remaining participants preferring more time and no participant preferring less time.

How does this practice relate to pillars?: 
Moderation sessions promote several of Sloan’s quality pillars, including learning effectiveness, student satisfaction, and faculty satisfaction. While most any assessment practice could claim similar relations, moderation sessions are unique in focusing on promoting faculty satisfaction by engaging faculty with peers to establish shared performance expectations. Moderation sessions give faculty a space to assess a student’ work according to their own performance expectations and evaluate the quality of those judgments quickly via concordance with their peer’s assessments. These sessions support a larger goal of enhancing student learning, which is an important purpose for most faculty. These features of moderation sessions address several of the factors described as being key to promoting internal motivation (Pink, 2009), which can be a challenge for administrators engaging faculty in learning outcome assessment initiatives.

Moderation sessions are also a key step toward developing explicit models of student cognition for learning outcomes that could anchor future assessments and aid in interpreting assessment results (NRC, 2001). While few examples of these models exist in higher education, they promise efficiencies in the development of curriculum and instruction. Moderation sessions support the development of these models by promoting shared performance expectations amongst faculty. Once faculty have explicit and shared performance expectations for learning outcomes, they are better positioned to study and describe a detailed model of learning outcome demonstration and acquisition. In this way, moderation sessions support the Sloan quality pillar of scale by engaging many faculty in building a basis for future curricular and instructional decisions.

Equipment necessary to implement Effective Practice: 
While a moderation session could be conducted in many different ways, a conference call in which all participants can synchronously discuss the student’s work and their assessment is recommended. A web conference environment (such as, Adobe Connect) is also helpful for facilitating the meeting, collecting assessment data with poll questions, and sharing screen images of the student’s submission.
Estimate the probable costs associated with this practice: 

Moderation sessions are relatively inexpensive to conduct depending on the communication technologies available at your institution. Most institutions provide access to web-based conferencing  tools that can be used to facilitate the collection of faculty assessment data, however such tools are not required. Assessment data could be collected from participants via email by a facilitator for aggregation and presentation. The moderation session itself could also be conducted using free web-based conference call technologies, such as Skype, and free online survey technologies. The indirect costs of having faculty and administrators participating in a moderation session (2-3 hours per participant) should also be considered.

References, supporting documents: 
National Research Council (2001). Knowing what students know: The science and design of educational assessment. Pellegrino, J., Chudowsky, N., and Glaser, R., editors. Board on Testing and Assessment, Center for Education, Division of Behavioral and Social Science and Education. Washington, DC: National Academy Press.

Pink, D. (2009). Drive: The surprising truth about what motivates us. Riverhead Press.

Useful Links: 
The LOT Design Process
Contact(s) for this Effective Practice
Effective Practice Contact: 
Jeff Grann
Email this contact: 
jgrann@capella.edu