Sponsor Videos

Adapt Courseware




Conference News

Twitter LinkedIn FaceBook YouTube GooglePlus www.instagram.com/onlinelearningconsortium


Missed the conference? Purchase a post conference virtual package!

Post Conference Individual Virtual Package

Post Conference Institutional Virtual Package

OLC Conference App Icon

Download (or Update) the OLC Conferences Mobile App!
- Apple iTunes store for iPhone and iPad
- Google Play store for Android

OLC Fellows Class of 2014 Announced

OLC Award Winners Announced

Best-in-Track Selections Announced

Watch the recording of our Preview Webinar

Workshop line-up is posted! Add a pre-conference workshop to your registration

New for 2014! Technology Test Kitchen - BYOD to learn, explore, and share knowledge within this lab environment

Technology Test Kitchen

Thurs Eve (Oct 30) Disney Event Ticket Purchase now available - 2 Options:



Exhibit or Sponsor at This Conference

Save the Dates

22st Annual OLC International Conference
November 16-18, 2016 | Orlando, Florida | Walt Disney World Swan/Dolphin Resort

OLC Innovate 2016 - Innovations in Blended and Online Learning
April 20-22, 2016 | New Orleans, LA | Sheraton New Orleans Hotel

Whenever a Thing is Done for the First Time, It Releases a Little Demon: Pilot Strategies for Blended Learning Programs

Roxanne Russell (Emory University, USA)
Additional Authors
Timothy Harfield (Emory University, USA)
Session Information
October 30, 2014 - 4:10pm
Learning Effectiveness
Areas of Special Interest: 
Blended Learning
Major Emphasis of Presentation: 
Blended Program/Degree
Institutional Level: 
Multiple Levels
Audience Level: 
Session Type: 
Information Session
Oceanic 3
Session Duration: 
35 Minutes
Concurrent Session 10

Pilot testing to mitigate risk and optimize student experience before launching a high profile, nationally accredited, online doctoral program

Extended Abstract

The Challenge
We are in a transitional time for online learning at traditional higher education institutions- passing from a period of majority resistance to online learning and into a period of either acquiescent or enthusiastic acceptance of online learning as an inevitable disruption to traditional practices (Selingo, 2013). Generational, social and economic drivers are prompting more higher education administrators and faculty to offer traditional programs using online and hybrid models. Just as the approach to online course development requires more front-end comprehensive planning and development than traditional face-to-face courses, program level design requires a more systematic approach to cross course consistency and reliability. Pilot testing for usability and feasibility is crucial to the design process for online programs and introduces a unique set of challenges and opportunities to improve student learning and experiences.

This session will share the design, experiences, interview and survey feedback, and learning analytics gathering process from a pilot conducted for a new online doctoral program, the first online doctoral program of any kind at Emory University and the first online version of this particular degree to be nationally accredited.

The Context
In 2013, Emory University's Candler School of Theology decided to revive the Doctor of Ministry (DMin) degree program that had run for over twenty years before it was discontinued in 1995 because of financial pressure, declining enrollment and demands on faculty time. In response to a steady stream of requests to re-institute the degree, the leadership decided to revamp the program as a 90% online degree.

By taking this innovative step, Candler stands out at Emory University because the DMin program will be the first 90%online doctoral degree program on campus. This DMin degree is also the first of its kind to be accredited by the Association for Theological Schools, which approves degree programs for its 270 member association of "post-baccalaureate professional and academic degree programs to educate persons for the practice of ministry and for teaching and research in the theological disciplines." Furthermore, before making the decision to offer a new online program, Candler's faculty had only just begun to experiment with online learning components and had never offered even one class fully online. After an active year of faculty and administrative collaboration and program design and development, we knew a pilot test of the program was essential to ensure a program launch geared towards faculty and student success.

The Approach
To gather usability and feasibility data from an administrative, faculty and student perspective, we took the following approach:

  • Designed a pilot course to include all activities and tools of a typical week of program courses in the pilot course
  • Recruited faculty and administrators who would be active in the program in the first year to help build and deliver the pilot course
  • Offered the pilot to a volunteer subset of newly admitted students to the program six months prior to the program's start
  • Developed learning analytics based on learning management system log data to systematically examine student experience patterns and optimize program and course level design
  • Administered a survey to and conducted follow-up individual and focus group interviews with all pilot participants, including administrators, students and faculty
  • Analyzed the data for design-related program implications and to inform programmatic student support infrastructure, student orientation, administrative training and faculty development

The Takeaways
In this session, we will share experiences and results from this pilot program and ask for audience input of past experiences, reactions and suggestions. Using a collaborative digital tool, we will invite audience members to actively engage in producing a session debrief towards identifying successful strategies for implementation and evaluation of online program pilot testing. We and the audience should leave this session with a grounded understanding of programmatic level pilot design, usability data gathering techniques and cross-referencing learning analytics to inform design.

Program Track: Learning Effectiveness

Target audience: All audiences

Institutional Level: multiple

Type of session: Information Session

Media: Digital projector, wireless access

Lead Presenter

Roxanne Russell is a teacher and instructional designer with over nineteen years of university-level teaching experience and ten years of professional experience with designing, developing, integrating and evaluating online learning environments in a global context. She now brings her teaching, instructional design and web development background to helping schools navigate the complexities of technology integration for learning.