Constructing effective online assessment

I have worked alongside my academic colleagues Isabelle Marcoul and Svenja Erich of the Centre for Language Studies at City University London for the last two years to help develop effective online assessment. This project has now been written up for the recently published Learning at City Journal Vol 2 (2). You can download a full copy of our article here for free.

I’m providing a summary of the article here, focusing on the way the technology was used and how we measured the effectiveness of a multiple choice Moodle quiz.

Background

City University London runs a programme of language modules, some for course credit, some are extra curriculum. The languages taught are French, Spanish, Mandarin, Arabic and German. Before they can join a class the students need to be assessed and assigned to the language course appropriate to their level of linguistic competence, ranging from beginner to advanced levels. In 2011 more than 1000 students took a diagnostic test.

Prior to 2011, the language tests were handed out in a printed format and marked by language lecturers. The administrative burden for this was heavy with very tight marking deadlines, a lot of administrative work to assign students to the correct course, communicate this to students etc. It was concluded that an online system would help automate this, ensure the students received immediate feedback about which level and class was appropriate to them and would speed up the administrative process.

Practicalities

Each year the university runs a Language Fair during Freshers week. Traditionally this was when students took the written test and completed questionnaire (to gather basic information e.g. degree course etc). In September 2011 this assessment was done via multiple choice quiz on Moodle, the questionnaire was also online in a googleform. This meant that

  • a computer room was needed for the language fair
  • an audio/visual component was deemed to be difficult to manage as a large number of headphones would be required so listening was not part of the test

Design of the test

The languages team wanted to assess different types of language ability while being restricted to using a multiple choice online system. Each language had a test comprising of 100 questions. Please see the article for a full description of the choice of question type and what was assessed.

As a learning technologist I was very interested in how the languages department wrote their multiple choice questions in order to assess different types of language ability. For example, students were asked to read a generic text in the source language and were given comprehension questions to see how much they had understood. Some of the questions also asked that the students not only understand the words but also the cultural context and concept in order to get the answer right.

e.g.

What would you like as a main course?
A sorbet with strawberries
Six oysters
Steak and kidney pie with chips

To answer this question students needed to demonstrate understanding of it and the choices and to pick the correct answer from their own knowledge.

In the article Isabelle writes about how we construct language and how we can assess higher order thinking skills using online assessment methods so please do access the article if you are interested in this.

Use of Moodle and googleforms

City University London uses Moodle as it’s virtual learning environment. This was seen to be the perfect platform for the language testing. I met with the lecturers that would be preparing the questions for the test and explained how the Moodle quiz tool worked. This was to help them understand the types of question that would and would not be appropriate.

Once the questions had been written we had a two hour hands-on training session where the staff were trained in using Moodle quiz and then used it to add their questions with my support. I would recommend this approach. It meant that I could immediately troubleshoot any problems and the staff involved have been successfully using Moodle quiz ever since.

We also needed to collect some personal data from the students e.g. name, degree course etc. We used a googleform for this as they are very easy to set up and the data can be exported in excel format which the administrator requested.

Effectiveness of the language diagnostic multiple choice test

Effectiveness of the test was measured by the number of students that stayed in the group/level they were identified as during testing i.e. the language level of the course matched the language level that the student tested at. We were very pleased to see that the test proved very accurate in determining level for French, German and Spanish (small numbers of students took Mandarin and Arabic so the data was not conclusive).

This shows that an online test can effectively measure language ability in the majority of cases with very little movement of students between levels.

You can download a copy of the full article here

Case study – creating presentations using prezi instead of powerpoint

Professor Suzanne Franks presenting at the symposium

Professor Suzanne Franks, Head of Undergraduate Journalism at City University London, wanted a more innovative way to present at the International Symposium on “China as a Development Aid Actor: Rethinking Development Assistance and its Implications for Africa and the West”  hosted by the Department of Politics and International Relations, University of Oxford. We discussed the options and she decided to try to use prezi.com. Prezi is an online, cloud-based tool for creating presentations, exploring ideas and storytelling. Prezis can be played on a computer with internet access, downloaded for playing offline or played on an ipad by downloading a free app.

Professor Franks’ presentation was about the influence of China in Africa, particular in the media and media training. She decided she wanted to use images with no words and wanted to position the images/photos in the correct part of the country. She also used a map of Africa overlaid with the flag of the People’s Republic of China. You can see an image of this to the left (see the full prezi here).

Prezi is a really innovative presentation tool that can be used in many different ways. Here are some reasons to try prezi

1. Prezi moves slickly between images, graphics, text, video and audio so is very flexible

2. A prezi can be linear but is best utilised to introduce concepts or projects

3. Prezi allows you to zoom in and out of a bigger picture

4. Prezi is free for educators to use, just sign up with your university email account

I recommend that you take a look at their popular and award winning prezi presentations to give you an idea of how versatile the tool is and this blog post for tutorial videos

Presentations from the Learning @ City Conference

As Anna mentioned in a previous post, the entire team recently presented at the Learning Development Centre‘s Learning @ City Conference. Anna’s presentation with Isabelle Marcoul and Svenja Erich from the Centre for Language Studies won the prize for best paper.

My presentation, which was appropriately about lecture capture, was recorded using our Echo360 Personal Capture system. Click on the link below to watch it:

.

Kate co-presented with Gunter Saunders from the University of Westminster on the FeedForward project, which we have piloted in Social Sciences as part of the JISC Making Assessment Count project. They gave a similar presentation at the Cass Teaching and Learning Showcase in May, which you can watch here:

MPG session from City University on Vimeo

Creating interactive conference posters

I was pleased to have a poster accepted at the EDEN (European Distance and E-Learning Network) conference in Porto, Portugal this year. I have written a blog post about the content of the poster which you can find here.

Being in Education Technology I did feel the need to add a bit extra to the poster! So, alongside a QR code that links to the blog post I also used layar, an augmented reality app. On the poster there was a screenshot from a youtube video of my colleague and co-author Evelyn Reisinger speaking about setting up the course on Moodle (seen at the bottom left of the above image). To view it you needed the layar app on an apple or android device. My colleague Farzana Latif has been doing some work with augmented reality and set this up for me, so thanks Farzana!

By scanning the screenshot using the layar app the video of Evelyn started to play. It went down very well at the conference so I would recommend it. It’s great to be able to make a one dimensional poster more interactive.

Learning at City Conference – still time to book

Learning at City 2012 banner

Each member of the Education Support Team is speaking at the  Learning at City Conference on Wednesday 13th June. With a keynote from Professor Graham Gibbs it looks to be a worthwhile event to attend.

Our sessions

1.20-2.05 Making assessment count project with Kate Reader and Professor Gunther Saunders

2.15-3 Constructing online assessment with Anna Campbell, Isabelle Marcoul and Svenja Erich

2.15 -3 Students use of lecture capture for revision with Mo Pamplin and Kate Reader

Click here for the full programme for the day

There is still time to register for this event if you are City University London staff. Click here to register

A Guide to Writing Feedback for Undergraduates: By Academics and Students

Top ten tips

I ran a session at the LDC conference last week called ‘an undergraduate guide to feedback’.  I showed a few of the different ways in which we deliver feedback to students in the School of  Social Sciences.  This included using Wiki’s for continuous feedback, using the quiz tool in moodle to deliver operational feedback to students, and paying PhD students to write a guide to feedback for undergraduates with the purpose of providing students with a clearer idea of what we expect them to do with their feedback, and PhD students a more comprehensive idea of the types of feedback they should be giving undergraduates when they are marking.

I then opened the topic up to the floor for debate, first we discussed the different types of feedback that others were using in their Schools, and this ranged from using video and audio to providing group feedback for assignments.

We also had some students in the audience so I asked them to think of the most useful and productive feedback they had received as students and they came up with the following list of feedback preferences:

  • General points delivered to the entire class before they received their grade
  • Feedback that is delivered electronically as they often have difficulties deciphering handwriting
  • Lessons learned from previous years students
  • The option to discuss feedback with a tutor or module leader specifically in the first year
  • Voice recording (Audio Feedback)

We then moved the session on to try and write a series of tips for academic staff writing feedback for undergraduates.  The audience was made up of Academics, Support Staff and Students who all helped input into the following list of tips/ideas:

  • Consider giving general feedback across the class before grades are released so students engage.
  • Stay away from Jargon and ensure students understand the language you are using i.e. critical discussion
  • Criteria specific – link your feedback to your assessment criteria
  • Don’t hand write – consider electronic feedback (moodle?) or typed cover sheets, students have problems with handwriting.
  • All feedback needs to be appropriate to the type of assessment – one size does not fit all!
  • Suggest solutions, don’t just point out errors!
  • Make sure you let the students know when you are giving feedback – flag it as feedback!
  • Be creative consider audio, peer assessment and other innovative forms of  feedback.

iTunes U in Arts and Social Sciences

Last week I ran a workshop on iTunes U in the schools of Arts and Social Sciences at City. I wanted to give people an update on how far we have got with introducing podcasting in the Schools, as well as going over the basics of what iTunes U is and how to access the content we have on there. As it happened, every member of the audience owned an iPod, iPad or iPhone,which made my job a little easier.

This is the first year in which City has been recording lectures, seminars and conferences and hosting them on iTunes U. In Arts and Social Sciences, we’ve published guest lectures, research seminars and conferences in Cultural Policy and Management, Economics, International Politics, Music, Psychology, Publishing and Sociology.

Besides recording lectures – which can be a valuable tool to aid students’ revision, boost the research profile of the university, and expand the audience for our events – the big challenge for our iTunes U and podcasting project is to start creating more tailor-made content, which takes advantage of the podcast’s key characteristics – mobility, brevity and ‘linkability’. Research and best practice (references below)  into educational podcasting suggests a number of ways in which downloadable, subscribable audio/video files can help students, and we’re planning to get underway with some of these over the summer and in the coming year.

In particular, we’re hoping to interview academic staff in different departments about their careers, their research interests, and talk in detail about the issues that are central to the courses they teach on. We are also working on a series called ‘Introduction to…’ which discusses what a potential applicant should expect when they choose a subject to study at university. Further, we need to explore the more pedagogic applications of podcasting, such as creating revision presentations for students before exams, giving tasters or teasers ahead of new modules, and as a way of giving supplementary explanation of difficult concepts and topics beyond what can be given in lectures.

A number of questions were raised during the presentation, including the issue of how we can monitor usage and downloads from our iTunes U site. Now that we have these stats, we can use them to inform the content, format and pitch of our future material. Another question was around the ability to link to podcast episodes from and to iTunes. iTunes allows you to link directly to a podcast series, so you can email or tweet the URL of a podcast’s location in the iTunes store. Here’s a link to one of our most recently updated podcasts.

If you are a member of staff in Arts and Social Sciences at City University and you are interested in experimenting with podcasting, or if you already have some experience and want to get more involved, please contact Mo Pamplin or a member of the Education Support Team.

References

Cheetham, J., S. Ackerman and K. Christoph (2009). “Podcasting: a Stepping Stone to Pedagogical Innovation”. [Online] http://engage.doit.wisc.edu/edu_podcasting/. Accessed 18/05/11.

Division of Information Technology, University of Wisconsin (ND). “Teaching & Learning with Podcasting”.[Online] http://engage.wisc.edu/podcasting/teaching/index.html. Accessed 18/05/11.

Guertin, L. et al (2007). “Questioning the Student Use of and Desire for Lecture Podcasts”. Journal of Online Learning and Teaching, 3 (2), 2007. [Online] http://jolt.merlot.org/vol3no2/guertin.htm. Accessed 18/05/11.