Massachusetts Institute of Technology's pedagogical effectiveness index provides a simple yet mutlidimensional tool to the learning effectiveness of the online courses based on media, learning models and interactivity elements.
The pedagogical effectiveness index has been now used in many institutions of higher learnings involved with the online education. The number of user for the given methodology is increasing. The instrument is also available online for those who would like to use it for institutional course evaluation.
These factors are evaluated with reference to the learning technology standards proposed by IMS, AICC, and SCORM.
Content Factors. The content is the basis for course delivery. The quality of the content has to be good to begin with. Mediocre content cannot be made better just by the infusion of the pedagogical styles or multimedia enhancements. It is important to note that the independent authority authenticates the accuracy and the quality of the content. The source and author of the content must be given proper attribution to avoid copyright and compensation issues and to hold the author responsible for the quality of the content.
Learning Factors. The effectiveness of an online course depends on the quality of pedagogically driven instructional design. The learning factors that are at the core of the educational quality of an online course include concept identification, pedagogical styles, media enhancements, interactivity with the educational content, testing and feedback, and collaboration. Often the objectives of the courses are not well defined and therefore do not provide a clear intent of the course. The learning styles define content sequencing and presentation. It is important that the instructional design has sensitivity to the functional learning style that allows accommodation of the individual content sequencing and aggregation preferences.
Delivery Support Factors. The success of an online course depends heavily on the delivery support function essential for the course instructors, administrators, and users. The user authentication, the portfolio information, and records of the users activities during the completion of the online course should be administered by a user management module. Management of the course content elements, including video streaming servers, audio servers and HTML server is managed in this module. Also note that it is now federal requirement to provide access to course Web site to the visually and hearing impaired students. The federal acts 255, 504 and 508 require that course Web sites are designed to allow screen readers have alt tag for the graphics and video with the sign language closed captions.
Usability Factors. The usability is an important element for human factor based man-machine interface design. Despite the high quality of content, pedagogical styles and media enhancement, an online course can be a complete failure if the usability of the course is poor. The user interacts with the online Web course through the GUI. The design of the graphical elements, color scheme, font type, and navigational elements all can affect how a course is organized and perceived by the student.
Information overload in a Web page with excessive scrolling within a window can be detrimental to the educational quality of the presentation. It is recommended by several information design experts that a small chunk of information on an 800x600 pixal window is optimum. The page layout for the information chunks and access from the page to various part of the course Web site through the navigation bars is very important in the success of an online course.
Technology factors. Online course Web sites run on the technological infrastructure. The issues that influence the technological success of online courses include bandwidth, target system configuration, server capacity, brower client, and database connectivity. The network bandwidth defines what is the lowest common denominator for the course Web page design. Design for 56Kbps modem network access has more constraints than a network connection of T1 line with 1Mbps bandwidth. The number of simultaneous users that a Web server can handle is an important constraint for the large-scale deployment of online courses.
It is important to emphasize that the intent of the methodology described here is to create an objective criteria for evaluation of the quality of the online course based on the existing elements that represent pedagogical content.
Summative Evaluation Instrument Most rating systems are summative and depend on the precise definition of the quantitative scale. The most widely used rating system is the Likert scale, which I have selected for the proposed summative evaluation instrument (Figure 5).
The summative evaluation results and the pedagogical effectiveness index can be combined to give a final result that provides a view of the overall effectiveness of the online course:
Overall Pedagogical Rating = PEI x Summative Rating Score
The advantage of using therating formula is to evaluate both the pedagogical and delivery system-based scores and provide a final rating that will be useful for the comparison of online course offerings and online course delivery systems.
Towards a Successful Online Education The pedagogical effectiveness index and the summative evaluation instrument used in combination can be powerful tools for the evaluation of large numbers of online offerings. These criteria have clear emphasis towards the pedagogically driven design. Widespread use of these tools could guide and motivate online education developers, universities, and training centers towards the creation of educational systems marked by measurable success.
No. Evaluation Factors Absent Poor Average Good Excellent
1 Content Factors 0 1 2 3 4 Quality Authenticity Validity Media Presentation Attribution 2 Learning Factors 0 1 2 3 4
Concept identification Pedagogical Styles Media enhancements Interactivity Testing andFeedback Collaboration
3 Delivery & Support 0 1 2 3 4 User management Course Content Accessibility Reporting
4 Usability Factors 0 1 2 3 4
GUI Interactive design Clarity Chunk size Page layouts
5 Technological Factors 0 1 2 3 4
Network bandwidth Target system configuration Server capacity Browser client Database connectivity Ratings key:
0 = Absent 1 = Poor 2 = Average 3 = Good 4 = Excellent
Dr. Nishikant Sonwalkar Principal Educational Architect, AMPS, MIT email@example.com Ph$ (617)642-1767 cell (617)258-8730