Wednesday, May 13, 2009

Draft Formative Evaluation Plan

Hi All

I am posting my Draft Plan(2a) in Google Docs for your Feedback/Comments. This is my second draft, on which I have already received feedback from Bronwyn, and I plan to incorporate that feedback along with any welcome feedback from fellow classmates. I will now spend some time looking at the fine examples aready presented. Please post feedback as comments on this blog.
regards|
Debra

Sunday, April 26, 2009

Week 7-8 Draft ideas for Sampling Methods/Tools:

As outlined in the last post- it looks like I should be concentrating on a 'formative evaluation', given the timing and the scale of the project (both for this assessment and for the pending release of the programme to students). Therefore these comments are based on Bronwyns' feedback, thanks, Bronwyn. A slight rewording of my original intention of evaluating this summative point: "Are the activities effective and matching the L.O ?" to this formative point: "Are the activities relevent for the L.O"....

E-Learning Guidelines to evaluate against:

Given we need to evaluate against two e-learning guidelines (and I have been previously discussing three): my draft plan now will include the following:

ST9: - reworded - for a user review - to investigate useability ( access, navigation, ease of use)and effectiveness (content, activities, L.O and assessment)-
"Do the technologies employed successfully help the students participate and learn"?

e.g Does the use of interactive tools and forums help the student complete (engage with) the course (successfully)?


SD3: - reworded - for a peer review - to investigate if the activites are relevent for the L.O.
"Are students provided with relevent information and current thinking in their field".
e.g Can the students 'successfully' engage with the activities provided with no f2F contact?


I see that the next step is to work on further sub-questions however I will make contact with Brownwyn first.


Paradigm/Model:
Defineatly Multiple-methods model/Ecletic-mixed methods-pragmatic paradigm for a Formative Evaluation- of the Interactive web-based programm as outlined in the previous Blog postings.


Sampling Methods: I have not completely thought through the specifics of the triangulation and bracketing yet.

But- I would like to discuss further and negotiate the following:

Peer review-discussion forum- qualititative-small groups
Questionaires- for the students while using the course-limitations on numbers-qualitative
Check lists-ditto, for the students
some paper-based, some email/ online questionaire/evaluation in Moodle.

Saturday, April 18, 2009

Weeks 5 and 6: Evaluation Methods

Progress on the new programme ( my evaluation project) is a bit slower than I thought it would be, but given that I am not working on the project full time- we are not too far behind. After having had the last week off work, it seems achievable now while I am still at home, but I might feel differently when I go back to work next week. The study/activity guide has been written- just formatting it onto Moodle, and my colleague and I have thought of some areas that we could introduce discussion forums-hopefully,as Bronwyn suggested, encouraging them to engage with each other (and therefore improve interaction). The study guide will be in the form of an 'activity guide'-"what to do and when to do it"- as they work through the programme. Topics for the Forums include theory topics such as thinking about their own working and learning styles-adding that to a thread-sharing the importance of these styles for job seeking purposes, with other students; communication issues to deal with when under stress- and difficulties this may cause in the workplace;-sharing their own experiences-these are a couple of examples of discussion topics. We are also in the process of developing some quizzes, so that we can monitor (or attempt to), understanding of specific topics.

In terms of the evaluation model/paradigms and methods that I am persuing:
It's become clear that the type of model best suited is the mixed-multiple-methods model, simply because of the types of data collection available. The programme will be up and running very soon, albeit in a pilot form (as it will still be possible to make running additions to the learning activities as needed, since they will be hosted on a UCOL Moodle site). I may not have made it clear in the previous posting, but one of the final assessments tasks on the programme is to complete a face-to-face interview with the student, based on the CV they will have completed, as well as using the details of a research project (job seek process) that they have to complete. So there is an opportunity to evaluate with 'interview' methods, as well as questionaire methods- as we will be in e-mail contact with the students, as well as contact through the Moodle LMS.

We have discussed paradigms at length now and it's been an ongoing learning journey for me to see and read so much about 'them'. Referring to Phillips' discussions of popular research paradigms, there are so many different 'worldwide views of inquiry', that in order to find a traditional paradigm that fits with e-learning, it was easier to explain away with another multiple approach-the fourth paradigm (aka Reeves et al), critical -realist-pragmatic, which fits in well with the multiple methods of data collection. Hence the term 'eclectic' (borrowed) as used by Reeves, et al. Pragmatic meaning a practical problem-solving method. I believe this fourth model/paradigm suits my project. It may be quite difficult to measure all the factors that contribute to improvements of 'learning' with the other more scientific methods, since I am looking at interaction as a quality issue, as well as the relevence of added course content. Feedback should be able to be incorporated quite easily since UCOL has deleveloped the extra learning material and the associated Moodle site (with activities etc).

Why this model?
I see my project as a Formative type of evaluation and some of my ideas from the eLearning guidelines also fit under Effectiveness (summative) evaluation. While the programme is essentially already developed, the 'online delivery' is being 'fine tuned', and the new content added by UCOL is, at present, untested. So in terms of the elearning guidelines already identified, I would be asking something along the lines of 'are the activities effective and matching the L.O ?' Key to this is evaluating 'useability' (tested by users) and 'effectiveness' (is it working?). So it is a bit like a 'user review'. While not a formal pilot launch of the programme, the initial release will be to existing students whom we already have a relationship with (well, we are planning for this approach at least).

Reading Chapter 8 of Reeves and Hedburg helped confirm why I have chosen an Effectiveness Evaluation, rather than Impact. The ultimate measure of 'success' would be to say "did the the student get a job after completing the programme and creating a CV?" But this is looking too far into the future to evaluate the programme- too many variables out of our control; perhaps a more appropriate short term effectiveness aspect would be " Did the Virtual Building Tour enable a more descriptive (and therefore effective) CV to be completed"? This sort of data would be easy to gather at the Interveiw I imagine.

Articles that I have been reading:
Mehelenbacher,B.,Bennett, L.,Bird, T., Ivey,M.,Lucas, J.,Morton, J.and Whitman,L. (2005). Useable E-Learning: A Conceptual Model for Evaluation and Design. Interaction, Vol 4-Theories, Models and Processes in HCI. Retreived 29 March, 2009 from http://www.pedagogy.ir/images/pdf/usable-e-learning05.pdf

This article arose from a search relating to "evaluation and useability". Was of interest to me because of the authors oversimplfied statement that 'useability is the study of intersection between tools, tasks, users and expectations in the context of use'. Since my evaluation project is on effectiveness based on the quality of interaction- in an e-learning environment, I perservered with reading it.
It's a conference paper (International Conference on Human-Computer Interaction). The aim was to outline the challenges faced by researchers merging the (numerous) theories of useability and evaluation with e-learning developments. The approach considered neccesary with evaluation of e-learning environments is that it is more task-orientated than other types of useability research, therefore a different model is suggested. I have interpreted the model to be more inline with the fourth paradigm described above- because of the task-orientated perspective on e-learning instruction. The group suggests that 'useability evaluation' is founded on theories stemming from early Human-Computer Interaction theories- the social science research side versus the design of the tools for human interaction. "Understanding audience is where useability begins". The paper describes 10 heuristsics from Jacob Neilsen (1994) (http://www.useit.com/papers/heuristic/heuristic_list.html) for useable design
(the ones that stand out for me: match between system and real world; and recognition rather than recall). The secret it seems, is to match the design heuristics with the development of the evaluation-testing theories of e-learning useability. So with the groups emphasis on users and tasks- the model they propose is outlined by a set of useability heuristic tools-for designers evaluating e-learning environments- the questions to ask are defined under the following headings:
Learner Background and Knowledge (e.g accessibility, suport and feedback)
Social Dynamics (e.g communication protocols)
Instructional Content (e.g examples and case studies)
Interaction Display (e.g appeal, consistencey and layout)
Instructor Activities (e.g authority and authenticity)Environment and Tools (e.g organisation and information relevence).

These headings (tools) have been drawn from the five dimensions of all Instructional Situations, modified for e-learning. The paper details these dimensions in great detail, so I won't repeat them. I guess they are saying that further development is needed, and their ideas step away from those of Reeves etc (who directly apply these heuristics to e-learning evaluation); towards a more 'synergistic collaboration between useability and e-learning research'. Thus the task-interaction orientated emphasis on the questioning.

Another paper I was reading:
I have not fully summarized it yet, sorry.
Developing a Usability Evaluation Method for E-learning Applications:From Functional Usability to Motivation to Learn (http://www.dmst.aueb.gr/en2/diafora2/Phd_thesis/Zaharias.pdf)

The paper describes the development of a questionnaire-based usability evaluation method (formative) for e-learning applications. As with the previous paper, the focus for the study is the 'poor usability of e-learning applications'. The development of the method was based upon a very well known methodology in HCI research and practice.
More to follow.

References:
Phillips, R. (2004). We can't evaluate e-learning if we don't know what we mean by evaluating e-learning. Retrieved 29 March 2009 from http://www.tlc.murdoch.edu.au/staff/phillips/PhillipsV2Final.doc

Reeves - Chapter 8 (already referenced).

Sunday, March 29, 2009

Week 4 musings: "We can't evaluate e-learning if we don't know what we mean by evaluating e-learning".

Since we were not really required to post some thing for our Week 4 readings (and yes, I am a week behind- but have started with the week 5 tasks, honestly), I'd thought I'd share a paper with you all that I found this morning. Early Sunday morning, and there I was reading about Educational Paridigms (groan) and I suddenly thought- I can't cope so early on in the day with these combinations of words such as Analytic-empirical-positivist-quantitative and Constructivist-hermeneutic-interpretivist-qualitative ......so I did some searching on GOOGLE and came up with the following article- from a Rob Phillips (Murdoch University, 2005)- a publication from the University of Bristol Called Interact- sorry, not sure which one but the link to the document is here. The title said it all for me, and may help those of you who are struggling with getting started on the paradigms and models. As an overveiw, I didn't think it was too bad, and made the mornings readings a bit easier.
"We can't evaluate e-learning if we don't know what we mean by evaluating e-learning". He makes reference to an earlier paper by Reeves and Hedberg (2002), and talks about the first three paradigms that Reeves discusses in what seemed to me this morning, slightly more simpler terms than the review paper (first attempt anyway).

Thanks
Debra

5.4.09 I have just noted that this paper is referenced in the WikiEdProfessional eLearning Guidebook. I still thought it was worth a read! :-)

Sunday, March 15, 2009

Week 3 Bits

Week 3: Discussion on Quality and eLearning Guidelines:


The project that I hope to use for the Evaluation is related to quality aspects of a new eLearning course that I am presently assisting in getting up and running here at UCOL. Briefly, the project is an eLearning Solution to a new Short Course in Preparing for and Enhancing your Career (Level 2).

With this course, the students use a 'virtual visit' to a large Company, as a platform for discovering the neccesary skills and experiences that are important as a job seeker. While they 'visit and observe' all the departments in the Company and 'talk' to relevent people, they are also completing an online electronic CV (which will eventually be hosted by the following company, (www.qjumpers.co.nz). The elearning component is called GETSET (the 'Virtual Tour'), and UCOL have created additional materials to make it into a 15 credit Level 2 programme, with material such as Stress Management, how this affects you in the workplace, along with the neccessary Interview Skills. Currently, the GETSET component is used in a f2f situation, with students being stepped through the programme in a traditional classroom environment. However, with the new UCOL programme, the students will be distance students, studying off campus and will be stepping themselves through the GETSET programme. Therefore all the
neccessary guidance and interaction/feedback will need to be included in the form of extra instructions, with tutor support and learning/feedback e-tivities built into the Elearning part of the course.

The end result of this programme, is that students will have a better understanding of what employers are looking for, understand more of their own traits and skills, how they can contribute to a workplace, resulting in the production of a relevent and
current CV, and to eventually have that CV verfied by UCOL (whilst completing the course). Embedded into the course are 3 Unit Standards, relating to Interview Skills, Problem solving and Stress Management.

One of my concerns is that the students may simply scroll too quickly through the virtual Company Tour, just so that they can get their CV up and running, and therefore possibly not engaging in the real learning aspect of the programme.
We also want to them add as many details as they can to their CV, and if they skim through the programme too quickly, some of the subtle learning that we would want occurring may be missed.

So, these are two of the issues that relate to quality in elearning:

1. Is their enough interaction with the students- how will the tutuors know when the students are struggling? How can we get
them to engage fully?


2. Is the extra course material relevent / will the students use it to add to their current experience?

Following on from the e-Guideline that I looked at last week (which I misquoted actually- it was SD 5 I referred to, not SD 4(http://elg.massey.ac.nz/index.phptitle=Students/Learning_design/Good_practice):

Listed under (Students/Learning Design/Good practice) are some guidelines that can be related to the (quality) issues, the two I hope to explore further for this course are as follows:

SD 3:Do students gain knowledge relevant to employment and/or current thinking in their field?

SD 5: "Do students aquire the learning skills for successfully completing the course?"

In the example of my course, if we were evaluating aspects of the course in terms of quality, I'd still ask "Did the use of interactive tools and forums help the student complete the course"?

By asking such questions, we can evaluate the technology-assess the interaction and check that the knowledge gained is relevent by the CV verifcation process (verify the content). If the CV is verifiable- we could say that the intended outcome is acheived...

There is another quideline as well, that is relevent-
http://elg.massey.ac.nz/index.php?title=Students/Teaching_relationships/Good_practice
ST 9: Do the technologies employed help students successfully meet the learning outcomes?

I would ask- can the students successfully engage and complete the course in an eLearning format with no f2f contact (apart from the "Interveiw" at the end of the course (to verify the CV)?

Saturday, March 7, 2009

Week 2 Discussions

1.Why is evaluation important to you and how do you define it?


For me, evaluation is important as it is a reminder that what we are doing is of benefit and allows us the opportunity to consider different options.

My first thoughts lead to the mainly summative evaluation processes that are in place, then I realised, that even though I was already familiar with the ADDIE model, I considered that ongoing formative evaluation can also occur amongst peers in an informal way- perhaps by attending conferences, belonging to online Discussion forums, talking to staff from other Insitutions. I am refering to evaluation of the teaching process/system as well as the course content.

For example, within an online course, an e-activity may be presented in a new way to students as a result of some learning opportunity gained though a shared experience with another lecturer. Evaluation (formal and informal) is also important personally- it can be used to 'pat yourself' on the back, as well as 'allowing room for improvement'.I also was thinking of the efects 'tick box' situation whereby you present the end of course evaluations to the online students, gather the results for the Annual Programme Reporting process, list any outcomes and then add them to the 'Action List' and find, that a year later, no further action was taken because it got
lost in the 'redtape'.

So, during this course, I hope to learn more about evaluation in a 'user friendly' environment.


While listening to the presentation I jotted down the following in response the question "What is evaluation?" ...a review of a process, how well did it suit the 'intended outcome' (purpose), are there areas which can be improved?, did the process result in
'buy in'? will it be reproducable? can it be transferred to another format?



2.What sort of evaluations mentioned on the presentation are familiar to you already and why?

Evaluation processes that were mentioned in the presentation that are familair to me are as follows:

  • paper based questionaires- either handed out in class or posted to students- (this is for ongoing Community Computing classes that are facilitated rather than 'taught';

  • end of course evalutions available to students participating in papers or units online;

  • Annual Student Satification Surveys, -sent out to students who may or may not still be studying online-at the end of the year. Usually sent via email with a web link to follow;

  • focus groups- participation as a staff member on elearning initiatives;

  • discussion forums (as a student on this eLearning Course-in another paper)-evaluating fellow students work e.g webpages;

  • checklists to follow before a Unit is relased to students on an online course.



3.Why is quality important in eLearning?

Quality: For me this is much the same as in a retail environment - would you buy a suit from a Designer who you had heard via word of mouth from friends (who
had previously brought a similar suit) -that he used inferior materials? If you were looking for a product that will last, you probably wouldn't buy it. But if you were
looking for a product that is cheaper and doesn't have to last the distance, then possibly yes, you would consider it.

The analogy for the student wanting a quality course is something like this. They will want a 'quality course' if they plan to do further study, especially at other insitutions. WORD OF MOUTH is very important - a student who has a bad experience will probably do more damage that we realise- students talk and may remember the poor quality aspects longer than we want them to!!!
Quality is perception of value, perhaps something like "am I getting the same treament/value for money in a $300 elearning course as I would in a $1500 f2f class".

In the early days of blended and 'online courses', it could be said that many courses were delivered 'online' as result of the perception that they are more cost
effective. I don't have a specific reference for this, but I think that today- students expect a lot more than just reading their resources on line- they want all the bells and whistles that Web 2.0 technology offers- and that means that the developers and tutors alike have to come to grips with quality tools and good instructional design. I recall looking at some readings from Frank Rennie in the Educational Design for eLearning paper, but can't track them at present.


Quality is also about 'how good is the course'?-'how well does it meet the intended outcome(s)'? (taking into account the context of the measure of quality).What one student calls "good quality"- another may disagree or shun. e.g with elearning, we assume plenty of opportunity for feedback through (for example) forums, etc. A less vocal, or less interactive student, may not perceive the inclusion of this feedback method as a sign of good quality.

I had a look at the eLearning Guidelines (http://elg.massey.ac.nz/index.php?title=Students/Learning_design/Good_practice).

Listed under guideline SD4 (Students/Learning Design/Good practice) is the following question which I think can be related to quality:

"Do students aquire the learning skills for successfully completing the course?"

In the example above, if we were evaluating this aspect of the course in terms of quality, we'd ask "Did the use of interactive tools and forums help the student complete the course"? So, the first student would say it was 'good'! the second might not think so.

Lots to think about obviously....

Thanks

Debra

Tuesday, March 3, 2009

Hi All

Apologies for the delay. I have been having trouble getting this account (and google mail) working at home. This blog is one I set up Sept 2007 for some PD here at UCOL ( all things related to Web 2.0 technologies), so please ignore any 'posts' prior to this course.....
I work at UCOL along with Heather and Kay, in the School of Business and Computing. I have been at UCOL for ten years now, about 5 years involved on Online Programmes, both as tutor and some development. I am in the middle of a transfer of 3 courses over to Moodle- and on a fast learning curve for Moodle, the behind the scenes things are quite interesting!!! This is my third paper of this course, and it's especially timely for me, with this transfer of courses, I am hoping get a chance to reflect on "best practice' a bit more, because in reality it seems that sometimes the "best practice" gets in the way of " we must have the course up and running by such and such a date". I wonder what the students think of changing learning platforms? I am thinking that my project might be based around this new course platform and the acceptance of the students, especially the things that we can do differently in Moodle.
What do I know about evaluation already ? I'll work on that- however I do know that just because we ask the students to complete an online evaluation at the end of each paper- doesn;t mean they do it. And when it comes to the end of the course evaluations- we have to follow them up. Another point- getting the mix of questions "just right" so that we get meaningful information (which allows for change if appropriate).

Looking forward to getting more 'active' on the course- feels like a slow start so far :-(

Thanks