Saturday, April 25, 2009

Week 5 / 6_Evaluation Methods

Requirement for Week 5 & 6

Outline some information about the type of evaluation you believe will best suit your project with your reasoning.

Summarise the article(s) you have found (300 words) - full reference, evaluation design (paradigm, model, questions, methods), findings and how it aligns with the evaluation project you would like to conduct.

As mentioned in our reading material, to evaluate an e-Learning solution, the first important work to do is to create a practical evaluation plan. The key part of the planning is to decide using which methods of collecting data and evidence, and also to think about how to choose appropriate source of information, for example, collect the information from students, staff or documents.

As mentioned in my Week 4 post, I will evaluate how well students achieve their learning goals for a current online unit 2791-Intergrating data. The evaluation will be focused on effectiveness evaluation. Reeves and Hedberg ( 2003) states that the main focus of effectiveness evaluation is the difference between the KSAs (Knowledge, Skills and Attitudes) before and after the course.

The effectiveness evaluation in Six Facets of Instructional Product Evaluation listed several dimensions of effectiveness evaluation, which will help me in my plan greatly. I will stress on the following aspects of effectiveness evaluation: task-oriented, challenging, constructionist and formative in our learning environment.

Eclectic – Mixed Methods – Pragmatic Paradigm fits in my case. As stated in Project of Evaluation Toolkit :

It's best to use a number of data gathering techniques and/or sources of data to substantiate findings. This is known as a process of triangulation — the use of multiple investigative methods or information sources to home in on the question in focus.

Reeves and Hedberg (2003) emphasis that we should not rely on only one or two measurements while evaluate effectiveness of current learning system. We need to triangulate effectiveness by measuring it with as wide a variety of approaches as our resources allow. Therefore I will conduct a formative evaluation using multiple methods, e.g. questionnaire, interview, checklist, etc. And try to collect data from different groups such as students, facilitators and assessors.

Next challenge for me is implement the evaluation. Need to think about the advantages and disadvantages of different data collecting methods and carefully design the questions so appropriate data can be collected for further analysis. Acceptability, reliability and validity of measurement need to be considered thoroughly during the evaluation.



Project of Evaluation Toolkit. (n.d.). Retrieved April 18, 2009, from University of Tasmania:

Reeves Thomas C., John G. Hedberg. (2003). Chapter 8 Effectiveness Evaluation. In J. G. Reeves Thomas C., Learning Systems Evaluation (p. 173). Englewood Cliffs: Educational Technology Publications.

1 comment:

  1. Good work Joy. I am glad you have decided on an effectiveness evaluation. Good choice the Eclectic – Mixed Methods – Pragmatic Paradigm. The data collection methods you suggest will fit well with this.

    The elearning guidelines you chose previously now need some sub-questions under each to help you focus more tightly on what you are trying to find out.

    TD2 (Do students get clearly defined learning objectives that assist them in focusing on their learning activities?)
    For example: Are learning tasks relevant and do they motivate learning?

    To help with this I need to know more about the course and the aspects you wish to evaluate. Are there particular areas of concern - such as too much content, too little communication between students and lecturer, is the assessment working well? etc.

    ST4 (Does the course require students to engage in analysis, synthesis and evaluation as part of their course and program requirements?
    sub-question suggestion: How well are the current strategies for learning being used by students?
    Please email me a time for a chat about where to now with your plan.