In my last post, I outlined a process for analysing the content of the Learner Reports generated by the TEC Literacy and Numeracy for Adults Assessment Tool.
The reason for this is that I think that tutors should be using the data that they have access to to help them make better decisions about what to teach. This is a kind of data-driven decision making for educators.
It’s also something that is going to become a big deal due to the fact that the TEC is going to make funding decisions about your organisation based on the results that your learners get in the assessment tool, combined with the fact that tutors will come under pressure to demonstrate that learners have progressed by showing gains in subsequent reports.
What I wanted to do in this post was extend my discussion from last time and address a common issue faced by many tutors and educators when they come to deal with these Learner Reports.
It goes something like this: As a tutor you’re using the Assessment Tool with your learners and dutifully reporting the results back to management. You even print out or access the Learner Reports so you can see what step your learners are, but that’s typically where it stops.
The reason for this is simple.
- The language used in the reports is the technical jargon of literacy and numeracy and often makes no sense to a “non expert”.
This means that as a tutor, you cannot easily make a connection from what the report details are telling you to your own (or any) content.
Here’s how to fix this problem… at least partially anyway. The specialised language of literacy and numeracy is not going away any time soon. But then neither is the specialised language and jargon of your trade.
Here you just need to suck it up and deal with the fact that you have to learn some new words.
But there is something you can do that makes it easier to get your head around what’s going on.
As an educator who is using the Assessment Tool you should have access to the Learner Reports online. If you don’t then talk to management. You should.
When you look at the Learner Reports online the question numbers are hyperlinked. See the screenshot at the top. This means that once you are looking at the report you can click through to get some more information including:
- A further breakdown of this question item.
- What the correct answer should have been.
- What the incorrect answers could indicate
Here’s an example:
And then you can click the words “View Learner Response” and you’ll see the following:
- What your learner saw on the screen (or page) when they answered. In other words, you get to see the test item. This is important if you’re using the computer adaptive version.
- What their original and incorrect answer actually was
Here’s the rest of the example:
This helps you in a number of ways. One of these is it gives you some kind of insight into what was going through your learner’s head when they answered incorrectly.
And more importantly, it puts the literacy and numeracy jargon of the report, in particular the language used in the Question Intent in to some kind of context. In this case, you get an idea of the kind of ratio the question intent is talking about and you see the context from the tool.
And this should help you decide on how you could design, teach, or assess something similar in your own particular context.
Does that make sense? Let me know…
Yes this does make sense 🙂
And yes I know I should be doing this more often…
The thing is…
With 50 Learners all making a minimum of say 10 mistakes, x3 assessments = 1,500 questions to look at and think where was the learner at, what were they thinking and how can I help them?
I was sitting with the Learners after their Assessments and going over each question they got wrong with them, that was useful, but very time consuming.
Graeme, I was wondering if there is a way that we can print out the analysis of the questions they get wrong?
What I have started doing is highlighting the highest occurring mistakes and getting the learner to start there.
For example a lady today had about 6 mistakes with measurement. So I directed her to the corresponding Step in Pathways and found the measurement module.
It would be fantastic, if in the Adult Assessment Tool Printout it included which Module in Pathways the Learner could go to.
Great Idea!!
Graeme can you organise that? 🙂
Thanks for your time writing your blog and encouraging us tutors to be better.
Hi Karen:
You’re right… you can’t deal with every mistake that every learner makes.
What you can do though is continue to develop your pattern recognition skills over time. What I’m suggesting here is for tutors who are new to this process. You’ve been at this a few years now, so I think your process is probably sound.
I especially like the idea of directing them to Pathways. The only thing you can do at present is go off the overall scale score in Pathways, but I’ll ask this question to the Assessment Tool people. You probably know your way around Pathways well enough to point people to different parts, but it would be great to automate this somehow.
There may be some way to aggregate the data from various learner reports by using the export features. I think you can export as a CVS file, but this is beyond me at the moment. I’ll keep thinking about it. Perhaps if you exported each report as a CVS and kept combining the reports to get a larger data set you could then sort the columns for frequency or something.
The other thing is that if you can get a “best guess don’t stress” sense of what direction your interventions need to be with a learner or group of learners, that’s actually probably enough. You want to avoid being buried in the data at one extreme – and simply ignoring it at the other.
Buddha says take the middle road
Thanks and regards, G