A Quality Scorecard for the Administration of Online Education Programs

Award Winner: 
2010 Sloan-C Effective Practice Award
Author Information
Author(s): 
Kaye Shelton, Ph.D.
Institution(s) or Organization(s) Where EP Occurred: 
A research study that involved multiple institutions.
Effective Practice Abstract/Summary
Abstract/Summary of Effective Practice: 

As the demands for public accountability increase for the higher education industry, institutions are seeking methods for continuous improvement in which they may demonstrate the level of quality within educational programs. Because of the rapid growth of online programs, institutions are further called upon to demonstrate that quality education is being delivered to students at a distance. This study created such a method to provide institutions offering online education an instrument for assessing quality within their programs: a quality scorecard for the administration of online education programs.

 

A six round Delphi study was undertaken with 43 seasoned administrators of online education programs from a variety of institutions in higher education. This panel of experts agreed upon 70 quality indicators that administrators of online education programs should examine within their programs to assess quality. A method for scoring was also developed. The original set of 24 quality indicators from the Institute for Higher Education Policy study, Quality on the Line: Benchmarks for Success in Internet-Based Distance Education (2000) were used as a starting point and were determined to be still valid in 2010, but with modifications (wording, some were divided, and some were combined). Forty-five additional quality indicators were added which resulted in a quality scorecard which provides industry agreed upon standards for online education programs to use for quality assessment. Each quality indicator has a potential range of 0-3 points with a perfect score on the scorecard resulting in 210 points.

Description of the Effective Practice
Description of the Effective Practice: 

The purpose for this study was the development of a scorecard to measure and quantify elements of quality within online education programs in higher education. The scorecard is an easy-to-use tool for online administrators to use for program evaluation. By evaluating each of the respective quality indicators within the established categories, an online administrator can determine strengths and weaknesses of their program. The identification of the weaknesses can be used to support program improvement and strategic planning initiatives. The scorecard could also be used to demonstrate to accreditors elements of quality within the program as well as an overall quality level.

A scorecard of 70 quality indicators was developed by a panel of experts with 86% of the panel have 9 or more years of experience Each indicator is worth up to three points. The administrator will determine at what level their program meets the intent of the quality indicator:

  • 0 points = Not Observed. The administrator does not observe any indications of the quality standard in place.

  • 1 point = Insufficiently Observed. The administrator has found a slight existence of the quality standard in place. Much improvement is still needed in this area.

     
  • 2 points = Moderate Use. The administrator has found there to be moderate use of the quality standard. Some improvement is still needed in this area.

     
  • 3 points = Meets Criteria Completely. The administrator has found that the quality standard is being fully implemented and there is no need for improvement in this area.

A perfect score = 210 points.

90-99% = 189-209 - Exemplary (little improvement is needed)

80-89% = 168-188 - Acceptable (some improvement is recommended)

70-79% = 147-167 - Marginal (significant improvement is needed in multiple areas)

60-69% = 126-146 - Inadequate (many areas of improvement are needed throughout the program)

59% and below = 125 pts and below - Unacceptable.

 

Supporting Information for this Effective Practice
Evidence of Effectiveness: 

While there are rubrics being used to assess quality in online course materials, such as Quality Matters, until now, there was not an industry agreed upon instrument that was being used to evaluate online education programs. Many institutions prolifically advertise they offer quality online education but have not had a way to quantify or benchmark their programs. How do students know they are enrolling in a quality program? The scorecard developed as a result of this research study provides an instrument that could identify strengths and weaknesses of an online education program and be used as a benchmarking tool for evaluation against other like programs in the industry. The results of the scorecard may also provide valuable information for strategic planning and budgeting.

The quality scorecard was developed by a panel of online education administrators with all having more than 7 years of experience. The following table shows the distribution of the types of institutions represented.

Institutional Classification for Expert Panel Members

Institutional Classification

Type

Size

Total

Public (4 year)

Non-profit

Large

24

Public Community College (2 year)

Non-profit

Large

  2

Private (4 year)

Non-profit

Large

  4

Private (4 year)

For-profit

Large

  1

Private Faith-Based (4 year)

Non-profit

Large

  1

Public (4 year)

Non-profit

Medium

  2

Private (4 year)

Non-profit

Medium

  3

Private Faith-based (4 year)

Non-profit

Medium

  3

Public (4 year)

Non-profit

Small

  1

Private (4 year)

Non-profit

Small

  2

How does this practice relate to pillars?: 

Each of the Sloan-C pillars of quality for online education are focused on establishing minimum guidelines for quality effectiveness. While this effective practice actually supports all of the pillars of quality, it may be most closely aligned with the Scale pillar - The provider continuously improves services while reducing costs because the scorecard enables the institution or program director to determine if they meet minimum quality standards in all areas of the program.

Equipment necessary to implement Effective Practice: 

Nothing more than a pen or pencil is needed and the time necessary for all processes and procedures to be carefully examined and reviewed before filling out the quality scorecard. 

Estimate the probable costs associated with this practice: 

$0

References, supporting documents: 
Institute for Higher Education Policy. (2000). Quality on the line: Benchmarks for success in Internet-based distance education. Washington, DC: Author.
 
Pond, W. K. (2002). Distributed education in the 21st Century: Implications for quality assurance. Online Journal of Distance Learning Administrators, V(II). Retrieved from http://www.westga.edu/~distance/ojdla/summer52/pond52.pdf
 
 
 

 

Useful Links: 
View the scorecard
Other Comments: 

The identification of quality online education programs satisfies a great need in our field and has been requested by many online education administrators as a tool for program improvement. The assessment of quality online education has never been more important as fierce competition from for-profit programs as well as many non-profits programs continues to increase and students all over the world are clicking to find a respectable degree program. Quality in education really does matter as the ultimate impact is to our students.

The interactive scorecard on the Sloan-C website is open to institutional members, see http://sloanconsortium.org/node/add/scorecard-online-program.

Contact(s) for this Effective Practice
Effective Practice Contact: 
Kaye Shelton
Email this contact: 
kaye@dbu.edu