Week7-8 Evaluation Draft Plan

Evaluation Draft Plan

Introduction

This evaluation plan will evaluate how well students achieve their learning goals of a current online course: unit 2791 - Integrating data. The evaluation will be focused on effectiveness evaluation.

I am the person design and conduct this evaluation plan.

Background

MIT offers National Certificate in Computer (NCC) Lever 2 & Level 3 courses to adult students through several classrooms within different communities across Auckland. Any adult students can enrol into the course. They learn in their own paces and can get face to face help if they attend any classroom sessions.

The paper I am going to evaluate is Unit 2791 – Integrating Data. Generally before students start this unit standard, they should have finished their courses on Word, Excel and Access. So this unit is about integrating data from Excel and Access into Word document. Learning material of Unit 2791 is totally delivered online. Content of 2791 is divided into different topics. At the end of each topic exercises info is presented with detailed instruction on how to complete the exercises.

While students doing this paper, I found they can’t assess themselves due to missing of learning objectives. Many students can follow the instructions finishing exercises independently. But once they are required to do a task all by themselves without provided instructions, many of them are struggling with it. The following are two main issues I found with students doing unit 2791.

Issue 1: There are no clear learning objectives provided at the beginning of the course, therefore students lost the chance to assess themselves whether they achieved their learning goals. When students were doing their courses online I observed some of them were too focus on following instructions of practical activities. Most of them believe they achieved the learning outcomes just because they were able to complete the activities by successfully following the step by step instructions.

Issue 2: Students read the topic information first, and then they do their consolidation exercises of each topic. Most of the exercises are displayed as PDF file links. In the PDF files step by step instructions are given so students can follow them finish their exercises. But I found students are following the instructions without any understanding, further thinking or analysis on the topic, they just focus on what they are told to do step by step, but lost the whole picture, e.g. why they have to do so. Quite a lot of our students can complete the practical activities but fail to understand basic concepts. Once there’s no guidance from any given instructions, students really struggled finding their way to complete required tasks.

Purposes

The purpose of this evaluation is to evaluate the effectiveness of current online course. Try to find how well students get the required skills and seek the possibilities to improve the online course.

Objectives

To complete the evaluation properly, the following steps will be undertaken:

• Create evaluation plan;
• Determine proper paradigm, model and methods that will be applied;
• Create appropriate sub-questions for each guideline applied;
• Implement the evaluation by collecting data using different methods from focus group;
• Collect information and analysis data;
• Complete the evaluation report.
  

Outcomes

Proper data is collected by using different methods and the evaluation will suggest managers the effectiveness of the online material so that they can decide if it’s necessary to take further action on optimise the online course.

Audiences

The result of the evaluation will be reported to our area managers and program leaders. So they can decide whether to improve current online course based on the evaluation of collected data.
During the period of conducting the evaluation plan, current students will be involved to provide feedback of the course, and facilitators working at classrooms will be interviewed too to give their observation of students’ learning. Assessors maybe asked to provide information on students’ final assessments.

The person create the evaluation plan, conduct it, design data, collect information and analyse all information to give an evaluation on the effectiveness of 2791 online course material is me.

Decisions

The evaluation will give an analysis on the effectiveness of Unit 2791 online course material. By collecting feedback from students, facilitators and program leaders, the evaluation will give detailed and convincing information to help managers determine further action about any adjustment on the following aspects:

• Clearly define learning objectives or learning outcome checklist for each module;
• Review current learning material and try to give more explanation on relevant knowledge to help students understand basic concepts and facts;
• Review detailed step by step instructions for exercises, try to encourage student analyse, synthesize and apply their knowledge into practical skills.
  

Questions

Regarding issues mentioned above, I choose guidelines TD2 and ST4 to evaluate the effectiveness of current online course – Unit 2791: Integrating Data.

TD2 Do students get clearly defined learning objectives that assist them in focusing on their learning activities?

Clear learning goals are very important in online learning. It gives students a guideline assess themselves with their learning. If there are no clear learning objectives students won’t be able to know whether they achieved the planned goals and be able to move further. I believe learning without clear goals will diminish the effectiveness of learning.

Learning objectives, concepts and ideas with clear documentation are mentioned in TD2 guideline. For this guideline, I would like to know what an important role that clear study objectives play in online learning, and how it affects students’ learning outcomes. Evaluate how helpful the designed learning activities helping students achieve learning outcomes.

The sub-questions about guideline TD2 are:

1. How do students know whether they achieved their learning outcomes?
2. Does the learning material provide enough and clear description of topics, concepts explanation?
3. What activities have been designed to help students understand what they have learned?
   

ST4 Does the course require students to engage in analysis, synthesis and evaluation as part of their course and program requirements?

I will use this guideline to check whether the course material is well organised for students to understand the whole idea of integrating data thoroughly. I am also interested to know how effective the existing course activities enable students analyse on what they have learned, and combine the knowledge they gained so they can apply their skills with fully comprehension.

Questions I will focus according to guideline ST4 are:

1. Does the course material focus on students’ comprehension and application of the skills rather than simply recall or just memorise course content?
2. Are students aware that they need to use analysis, synthesis and evaluation skills to successfully complete the unit standard?
3. What tasks have been well designed to provide opportunities for students engaging in the use of analysis, synthesis and evaluation?
   

Methods

Regarding experimental evaluation model and multiple methods evaluation model, as Bronwyn (2003) stated, experimental (quasi experimental) evaluation are generally carried out “quantitatively through the study of experimental and control group”. I believe multiple methods evaluation model is a proper tool to be used in my plan. To apply guidelines of TD2 (Do students get clearly defined learning objectives that assist them in focusing on their learning activities?) and ST4 (Does the course require students to engage in analysis, synthesis and evaluation as part of their course and program requirements?), it’s hard to use quantitative tests to explain them straight away; also we can’t quantify every single learning outcomes to evaluate guideline TD2 and ST4 in my planned evaluation solution . The issues need to be measured from different perspectives and information should be collected in different methods. That’s the reason why I think multiple methods is proper to be used here.

Reeves and Hedberg (2003) emphasis that we should not rely on only one or two measurements while evaluate effectiveness of current learning system. We need to triangulate effectiveness by measuring it with as wide a variety of approaches as our resources allow. Bracketing techniques will be employed while considering factors related to effectiveness of evaluated course. Therefore I will conduct a formative evaluation using multiple methods, e.g. questionnaire, interview and observation.

Details will be prepared after getting feedback.

Sample

The sample participant involved in this evaluation is listed below:

• Past students – Students have finished Unit 2791;
• Current student – Students are working on unit 2791;
• Facilitators – facilitators from different classrooms;
• Management – Area manager, program leader, student support advisor and course assessor.
  

Instrumentation

Current student will be interviewed and asked to answer questionnaire to reflect their comprehension of learning material and ability to complete practical tasks independently.

Former students will be asked to fill in a questionnaire / checklist to show their impression about the whole unit standard overall.

Facilitators and management staff will be interviewed and answer questionnaire to get their feedback on course material and students’ reactions towards the unit standard.

Observations will be applied on current students trying to get how they utilise the learning material.

Limitations

Students in our classroom are doing different papers and they can do their course inside and / or outside classroom. Especially for this unit stand that all course material is delivered online, so not sure if there will be enough students attend the classroom in the period of the evaluation process. This might affect the size of sample group.

For former students, same situation exists. The information collected from former students really depends on how long ago students finish the evaluated course. For example students just finish unit 2791, compared another student finished almost a year ago, will give more detailed and precise information due to their ‘fresh’ memory. So sample size and information collected is likely to be affected.

Students from class to class have different demography. For example, our classroom (Howick) has more Asian students than other classrooms while classrooms in south Manukau will have more Maori and Island students. Average age of students in classroom on MIT campus is lower than us. So due to the strain of working places, students from different background may not be picked up in a well balanced mixture.

Logistics

I am the only evaluator for this plan. I will create and describe the evaluation plan prior conduct it practically. After proper data has been collected, I will be the one analysis and complete the evaluation report.

Time line

The evaluation is planned to be finished by the end of June, 2009. The outline of activities is listed below.

Activities ...................................................................Completed Date
Draft evaluation plan..................................................... 03/05/09
Feedback and approval of draft evaluation plan.................10/05/09
Conduct evaluation....................................................... 24/05/09
Analysis data.............................................................. 07/06/09
Draft evaluation report...................................................13/06/09
Feedback and approval of draft report..............................20/06/09
Finalise evaluation report...............................................27/06/09

Budget

Estimated cost would be about $480. The details are listed below

Personnel Time $400
Document duplication: $40
Telephone calls $10
Travel to interviews $30

Reference

Reeves Thomas C., John G. Hedberg. (2003). Chapter 8 Effectiveness Evaluation. In J. G. Reeves Thomas C., Learning Systems Evaluation (p. 173). Englewood Cliffs: Educational Technology Publications.

eLearning guidelines for New Zealand. (2008). Retrieved March 21, 2009, from http://elg.massey.ac.nz/index.php?title=TD2