Week 5 / 6_Evaluation Methods

Requirement for Week 5 & 6

Outline some information about the type of evaluation you believe will best suit your project with your reasoning.

Summarise the article(s) you have found (300 words) - full reference, evaluation design (paradigm, model, questions, methods), findings and how it aligns with the evaluation project you would like to conduct.
========================================

As mentioned in our reading material, to evaluate an e-Learning solution, the first important work to do is to create a practical evaluation plan. The key part of the planning is to decide using which methods of collecting data and evidence, and also to think about how to choose appropriate source of information, for example, collect the information from students, staff or documents.

As mentioned in my Week 4 post, I will evaluate how well students achieve their learning goals for a current online unit 2791-Intergrating data. The evaluation will be focused on effectiveness evaluation. Reeves and Hedberg ( 2003) states that the main focus of effectiveness evaluation is the difference between the KSAs (Knowledge, Skills and Attitudes) before and after the course.

The effectiveness evaluation in Six Facets of Instructional Product Evaluation listed several dimensions of effectiveness evaluation, which will help me in my plan greatly. I will stress on the following aspects of effectiveness evaluation: task-oriented, challenging, constructionist and formative in our learning environment.

Eclectic – Mixed Methods – Pragmatic Paradigm fits in my case. As stated in Project of Evaluation Toolkit :

It's best to use a number of data gathering techniques and/or sources of data to substantiate findings. This is known as a process of triangulation — the use of multiple investigative methods or information sources to home in on the question in focus.

Reeves and Hedberg (2003) emphasis that we should not rely on only one or two measurements while evaluate effectiveness of current learning system. We need to triangulate effectiveness by measuring it with as wide a variety of approaches as our resources allow. Therefore I will conduct a formative evaluation using multiple methods, e.g. questionnaire, interview, checklist, etc. And try to collect data from different groups such as students, facilitators and assessors.

Next challenge for me is implement the evaluation. Need to think about the advantages and disadvantages of different data collecting methods and carefully design the questions so appropriate data can be collected for further analysis. Acceptability, reliability and validity of measurement need to be considered thoroughly during the evaluation.

===========================================

Bibliography

Project of Evaluation Toolkit. (n.d.). Retrieved April 18, 2009, from University of Tasmania: http://www.utas.edu.au/pet/index.html

Reeves Thomas C., John G. Hedberg. (2003). Chapter 8 Effectiveness Evaluation. In J. G. Reeves Thomas C., Learning Systems Evaluation (p. 173). Englewood Cliffs: Educational Technology Publications.

Week 4 Evaluation Paradigms and Models

When reading the four paradigms I was really confused by the terminologies – they are totally above my head :-(

In my planned evaluation schedule, two issues I raised in Week 3 post are finding out how clear objectives will help students achieve their learning goals, and how effective the online learning encourage students to not only remember the content, but explorer their learning further by analysis, synthesis and apply their learning skills into practical activities.

Regarding experimental evaluation model and multiple methods evaluation model. As Bronwyn (2003) stated, experimental (quasiexperimental) evaluation are generally carried out “quantitatively through the study of experimental and control group”. I believe multiple methods evaluation model is a proper tool to be used in my plan. To apply guidelines of TD2 (Do students get clearly defined learning objectives that assist them in focusing on their learning activities?) and ST4 (Does the course require students to engage in analysis, synthesis and evaluation as part of their course and program requirements?), I have to say it’s hard to use quantitive tests to explain them straight away; also we can’t quantify every single learning outcomes to evaluate guideline TD2 and ST4 in my planned evaluation solution . The issues need to be measured from different perspectives and information should be collected in different methods. That’s the reason why I think multiple methods is proper to be used here.

Bronwyn (2003) mentioned in multiple methods evaluation model, “ two main approaches are used, triangulation and bracketing”. To use a range of methods gathering data in several ways, I may use diverse methods (just a rough thought) and collect data from different groups. Maybe for TD2, I will ask students to fill in questionnaire and checklist, or interview students to find how important role learning objectives play during the study journey. For guideline ST4, I will probably try to collect all relevant information as much as I can from students, facilitators, program manager and assessors by using questionnaire, rate ranking forms, interviews, etc. So the factors can be fully triangulated and bracketed.