Technology-Enhanced Items: Leveraging Technology to Enhance Assessments

Over the last decade, technology has improved exponentially. Technology continues to impact our lives positively in many ways. From shopping to ordering food, to banking to making payments, technology has made everything simpler and easier. It has also changed the face of education by offering platforms for teachers to connect with their students more easily and rapidly. While many video-conferencing applications have allowed us to use technology to enable learning, technology can be further harnessed to improve teaching-learning and assessments.

Young students are at ease using a mobile.
If learning has come to them through mobile phones or tablets, why not assessments?

 

 

 

 

 

 

With the ongoing pandemic, most of the formative assessments in schools as well as some of the high-stakes national assessments like the Joint Entrance Examination (JEE) are conducted online. There are a few advantages to using online assessments, like the ease of evaluation, high scalability, and the ability to conduct deeper analysis using response data, which can help teachers and educators make informed decisions, etc. One of the advantages of online assessments is the option of using technology-enhanced items, which is not possible on a traditional paper-based test.

What are Technology-Enhanced Items (TEIs)?

TEIs are assessment questions/items[1] or tasks that require users to use technology to interact with the question (A traditional multiple-choice question appearing on the screen of a mobile phone/tablet or a computer is not a technology-enhanced item). Unlike the traditional MCQ or a free-response item where the test-taker just answers the question by selecting one option or by filling in the blank, here he can interact with the item multiple times. The process data captured by the system gives details of the user’s interactions with the question, which helps in understanding the cognitive or meta-cognitive processes the user applied in solving the question. Due to the interactive nature of TEIs, they improve the test-taker’s engagement and make assessments more interesting.

Large scale international assessments like the National Assessment of Educational Progress (NAEP) and the Programme for International Student Assessment (PISA) have started using TEI’s.

Here are two examples of a TEI from one of the PISA assessments for 15-year olds on tiling to create patterns.

Item 1:

(Source: pisa-2021-mathematics-framework-draft)

The item requires a test-taker to recreate the pattern by understanding the logic behind it using the drag-and-drop functionality. When a similar question is included in a paper-based assessment, the trial and error needed to get the pattern right will be too time-consuming and not very engaging for the user due to the amount of work involved in drawing the pattern. If assessing without technology, it can be in the form of an activity where students complete the pattern using the given tiles. But that requires assessors to create such tiles using cardboard and paper. It can be effort-intensive and time-consuming not just for devising an activity but also for having all the students do an activity and get evaluated.

Item 2:

(Source: pisa-2021-mathematics-framework-draf)

The question tests the user’s problem-solving skills by directing him to write an algorithm to recreate the pattern. By using TEI, multiple test takers can easily be evaluated simultaneously, with each test taker taking the test on his/her device. In a classroom setting, an assessor would have to ask each test taker to narrate the algorithm and evaluate one test-taker at a time.

The two examples thus demonstrate how technology enhances assessors’ capabilities to assess learning unlike traditional pen-and-paper assessments.

With more schools and students moving to online education, it is important for the content creators and teachers to understand the scope of TEI of it in their assessments and leverage it to improve learning.

What are the advantages and limitations of TEI’s?

Advantages:

TEIs:

• Increase students’ engagement and reduces the anxiety of assessment
• Offer for a variety of contexts and scenarios
• Reduce the effects of guessing
• Test higher-order cognitive skills
• Capture more process data that helps in understanding what students think while solving the question
• Are easy to evaluate

Limitations:

TEIs:

• Are expensive and take more time to create
• Dependent on technology
• Are biased to the test-takers who are more familiar with technology

Types of TEI’s with examples from Mindspark^[2]

In this section, we discuss some TEIs used in Mindspark.

1. Drag-and-drop items: Such items require a test-taker to drag-n-drop objects on screen to answer.

Here is one of the drag-and-drop items from Mindspark.

(Q1: Drag-and-Drop TEI)

In the example (Q1), students construct a pictograph to represent the given data using the drag and drop functionality primarily. The TEI makes it easier for students to construct a pictograph in less time than on a paper-based test. It also reduces unnecessary cognitive load on students by not requiring them to draw the pictograph. It also allows an assessor to focus on the objective of the question, which is to test if students can scale the graph correctly and identify the right index.

The system also stores the process data, such as the time taken for each drag-and-drop action and to solve the question. So, teachers can not only see the final answer given by the student, but they can also see their different actions, which help in understanding the students’ thought processes in solving the item. This cannot be captured in a traditional assessment.

2. Multi-select multiple-choice questions: This question type contains one or more options as the correct answer. All the correct options need to be selected here instead of just one answer. This question type is mostly useful in identifying misconceptions and the learning level of a student.

Let us understand this with an example of such a TEI from Mindspark below.

(Q2: Multi-Select TEI)

In this question (Q2), a student who is at level 0 as per Van Hiele levels may choose any shape that looks like a triangle, even if it is not a 3-sided closed shape. Students at level 1 may choose only the triangles that are commonly shown in textbooks. Some students may think that the base of a triangle should always be a horizontal line and may choose only a specific group of triangles as the correct answer. So, the multi-select item not only differentiates the students at different geometric levels but also helps in identifying specific misconceptions with just one question in the assessment.

In a paper-based large-scale test, analysing students’ answers and grouping them into different levels based on their responses is very tedious and hence practically impossible to do. In such cases, TEIs make the analysis easier through automation.

Study another TEI of the same type below.

(Q3: Multi-Select TEI)

The above multi-select item (Q3) on identifying the vertically opposite angles helps in identifying students who see only the angles POR and SOQ as vertically opposite instead of TOP and QOU as also vertically opposite angles.

Students with inadequate understanding or a misconception that one (or more) pair of arms of vertically opposite angles may not lie on the same line(s) may also select the second option, angles POR and QOV, as their answer. Here, arms OR and OV do not lie on the same line. The same question also helps to identify students who have a misconception that angles forming a linear pair are vertically opposite angles. Such students will select the first option.

When such TEIs are used in an Intelligent Tutoring System (ITS) like Mindspark, differentiated learning can be practiced. ITS can take students to different learning paths based on their responses to the TEIs and thereby enable learning as per their learning needs.

3. Interactive questions using measurement or construction tools: In the question below (Q4), students can solve the question graphically by plotting the equations using the given tool. To do the same solution on paper is so time consuming and difficult to evaluate because, due to less precision with the hand drawing, one can end up with the closest points but not exactly the actual solution.

(Q4: Interactive based TEI using construction tool)

4. Step-by-step solver (Equation editor): In a traditional test, teachers prefer a free-response question to a multiple-choice question, as they can see the students’ solutions in detail and check for any errors they make. But it is very time-consuming to check and evaluate each student’s response. In such instances too, TEIs can be of great help.

Here is a TEI (Q5) that allows students to enter each and every step while factorising the given algebraic expression. In the figure, a student has entered the two steps.

(Q5: TEI using the step-by-step solver)

Feedback on the response:

The system can evaluate each step and give feedback to the student about any errors. Teachers can also use the system data to identify the common errors that students make, and design remedial activities accordingly.

5. Interactive games or activities: In this question type, technology can be used to simulate a situation, a game or an activity.

Refer to one of the TEIs (Q6) of such a type below.

(Q6: Interactive based TEI)

In this question on scale factor, students can actually observe how the image changes with the scale factor and then answer the items that test application of the concept.

Here is another TEI (Q7) of such a type.

(Q7: Interactive based TEI in the form of a game)

The above activity is a game on the application of comparison of decimal fractions. A learner uncovers digits and identifies the larger of the two decimal numbers given. (Initially all the digits are shown covered but in this figure 4 digits have already been uncovered). There is an incentive to uncover as few digits as possible and identify the larger number. In this way, a learner applies the understanding of the place value of digits to order numbers. A student with a desired level of understanding should answer as option A, uncovering the digits as shown in the figure, revealing the digits before the decimal point first. Such an item also helps to diagnose misconceptions. For example, if a student has a misconception, the decimal with the greater number of digits is larger than the decimal with few digits. Such a student with misconception will answer as option A without uncovering any digits or by uncovering just the digit 2 before the decimal point in the given two decimals.

To sum up, such questions are desirable in an assessment although they may not be possible on a traditional paper-based assessment.

Do these items capture students’ interests and encourage them to solve a problem? As a teacher/educator, don’t you think it is easier to analyse this data and understand students’ errors or misconceptions than checking each student’s answer scripts? If your answer to these questions is ‘Yes’, then start using TEI’s in your assessments and show students that math is fun and interesting.

___

 

[1] Questions in an assessment are also called as items.

[2] Ei Mindspark is a personalised learning software that allows children to effectively advance at their own pace.