In this article
In this article you find the following information:
- About item analysis
- Psychometric data in item analysis
- Opening the item analysis of an exam
- Assignment overview
- Item overview
- Item detail page
- Excluding a question
- Filtering the item analysis of an exam by schedule
- Exporting the item analysis of an exam to Excel
- Opening an assignment report via Item analysis
- Previewing or opening an assignment via Item analysis
- Getting access to the Item analysis module
- Restrictions in the Item analysis module
About item analysis
In the module Item analysis, you find actionable insights on the quality and effectiveness of the items in your assignment and the assignment as a whole.
Actionable insights are given through psychometric values. Psychometric values include statistical measures such as the P-value, RIT- and RIR-values, the A-value, Cronbach's alpha, and the standard deviation.
These psychometrics will allow you to take data-driven decisions to:
- exclude items from your exam
- improve the quality of your items in the item bank for future use.
As such, the Item analysis module provides you with a powerful and valuable tool for easily checking the health parameters of your items and exams directly within assessmentQ. For more in-depth analysis, you can export the data for use in your own analysis tools.
Note: If the psychometric data are not included in the Item analysis module in your assessmentQ environment, please contact Televic Education via support@televic-education.com.
Psychometric data in assessmentQ
In the Item analysis module you will find the following psychometric data:
-
P-value (measure of item difficulty):
The P-value represents the proportion of participants who answered a particular item correctly. It indicates the difficulty level of the item. Lower P-values suggest more difficult questions and higher P-values indicate easier ones. This value helps to understand how well each question performs in distinguishing between different performance levels of participants. Typically, a P-value between 0.3 and 0.7 is considered ideal for most exams.
-
RIT- and RIR values (measure the correlation between an item and the overall exam performance): these values allow you to assess the contribution of each item to the overall exam score, ensuring that each item is aligned with the exam objectives.
The RIT-value (Item-Total Correlation) measures how well a particular item correlates with the overall exam score. It reflects the extent to which correctly answering the item relates to higher performance on the exam as a whole. A higher RIT-value indicates that the item is a good discriminator, meaning participants who do well on the exam are more likely to answer this item correctly.
The RIR-value (Item-Rest Correlation) is similar to the RIT-value but with a slight difference in what it measures. Both values are used to assess how well an item correlates with the overall exam performance, but the RIR-value specifically measures the correlation between an individual item and the total score of the exam, excluding the score for the item itself.
A RIT or RIR value of 0.2 or higher is generally considered acceptable, with values closer to 0.4 or above being ideal for better item discrimination. A low RIR-value (below 0.2) suggests that the item doesn't correlate well with the overall exam score, meaning it fails to distinguish between high and low performers. This could indicate that the item is either poorly written or doesn't align well with the overall objectives of the exam.
On the other hand, a high RIR-value (above 0.4) signals that the item is an effective discriminator, meaning that those who perform well on the exam are more likely to answer this item correctly, while those who perform poorly are less likely to do so. -
A-value:
The A-value indicates the attractiveness of a distractor (i.e. an incorrect choice in a multiple choice or multiple response item). The A-value ranges from 0 to 1. An A-value of 0 means no participant chooses the distractor, while a value of 1 means all participants choose it.
In assessmentQ, you will find the A-value in the item detail pages of multiple choice and multiple response items. More information.
-
Standard deviation and normal distribution curve:
The standard deviation measures the spread of the scores around the mean. A low standard deviation means that most participants scored similarly, resulting in a narrow distribution curve. A high standard deviation indicates a wide range of scores, resulting in a wide (flat) normal distribution curve.
-
Cronbach's alpha
Cronbach's alpha measures the internal consistency of an exam, indicating how well the items (questions) are related to each other. A high Cronbach’s alpha (above 0.7) suggests that the exam is reliable, with items consistently measuring the same construct. On a bell curve, this means that most students' scores should cluster predictably around the mean. A low alpha (below 0.6) implies poor internal consistency, leading to more scattered scores across the curve.
Together, Cronbach’s alpha and standard deviation provide insights into both the reliability of the exam and the variability of student performance.
Opening the item analysis of an exam
To open the item analysis of an exam, proceed as follows:
- Go to Analysis > Item analysis.
Result: The list of your exams is displayed.Tip: If you want to customise the table columns on the overview page, click on the table icon
in the upper right corner and (de)select the columns of your choice.
- Search for and/or filter the exam for which you want to open the item analysis.
Notes:
- Item analysis is available for assignments of the type exam only. Exercises are not included, see Restrictions below.
- No item analysis is available for assignments preceded by a pending icon. The pending icon indicates that the item analysis needs to be updated to include newly finished sessions. Updates are done once a day. When hovering over the icon, you will see when the item analysis will become available.
- Click on the exam of your choice.
Result: The Assignment tab is opened, see Assignment overview below. You can switch to the Items tab, see Item overview below.Note: If no participants have finished the assignment yet or if participants finished the assignment on the same day that you want to consult the analysis, you will see a message indicating when the item analysis will be available.
Assignment overview
The item analysis of an exam by default opens the Assignment tab. Under the Assignment tab, you will find a detailed overview aggregating all data at the level of your assignment.
Example:
- At the top left, you find general data on your exam (number of items, number of participants, minimum and maximum score), the pass rate, the time spent on the assignment and the scores that were obtained.
- At the bottom left, you find the standard deviation and the normal distribution curve of the scores. Here you see the variation or dispersion of the item scores in your exam.
- At the top right of the Statistics section, you see a diagram displaying the P value of all items in your exam on the X-axis and the RIT/RIR-value of all items on the Y-axis. You can toggle between viewing the P-value and RIT-value or the P-value and RIR-value.
Each item in your exam is represented by a dot on the diagram, corresponding to its P value and RIT/RIR-value. This diagram allows you to quickly identify the items that may need to be reviewed again.
Items that are located in the lighter square combine a moderate to high P-value (around 0,3 to 0,8) with a moderate to high RIT/RIR-value (around 0,2 or higher). These items are neither too easy nor too hard and distinguish well between high and low performing participants. Items in the greyed-out areas have a less balanced combination of P value and RIT/RIR value and may need to be examined more closely.
Note: You can click through on a dot to open the item detail page, see Item detail page below.
- At the bottom right of the Statistics section, you see the average P-value, the average RIT- and RIR- value and the Cronbach's Alpha value of your exam.
The average P-value indicates the overall difficulty of the exam by showing the proportion of participants answering each question correctly, helping to ensure the exam is neither too easy nor too difficult.
The average RIR-value measures how well the exam discriminates between participants with varying levels of ability, ensuring that the questions effectively differentiate high performers from low performers.
Note: The Assignment overview by default includes the data of all sessions for the selected exam, but it is possible to filter by any schedule. For more information, see Filtering the item analysis of an exam per schedule.
Items overview
At the top of the item analysis screen, you can switch between the Assignment and Items tab. Under the Items tab, you will find detailed information related to each item in your exam.
For example, for each item you can see how many participants answered it in total, and how many answers were correct, partially correct, incorrect, or unanswered. You see the item exposure and statistic values such as P-value, RIR- and RIT-value,...
Example:
Notes:
- You will see a warning icon next to the P-value, RIT- or RIR-value when the value is under the 'acceptable' threshold (0,3 for the P-value and 0,0 for the RIT- or RIR-value). This allows you to quickly identify items that are outside the 'acceptable' thresholds.
- You can click through on an item to open the item detail page, see Item detail page below.
Customising the item overview
You can use the column selector to customise the items overview to your needs. That way, you can view only the information that is important to you.
Filtering the item overview
You can filter the items overview to view only the items you want to see. You can use (a combination of) the following filters:
- Item type
- P-value
- RIT-value
- RIR-value
In the example below, a filter is set to view only the items that have a P-value between 0 and 0,3. A low P-value indicates that the item was answered correctly only by a small proportion of the participants and thus suggests that the item has a high difficulty level.
Item detail page
To open an item detail page, proceed as follows:
- Under the Assignment tab, click on a dot in the statistics diagram.
- Under the Items tab, click on an item in the items overview.
Result: The item detail page is opened.
Example:
- At the left, you find general data on your item (item name, item type, assignment part in which it occurs, minimum and maximum score), the results (correct, incorrect, partially correct, unanswered), the time spent on the item, the item score, the item exposure and the item statistics (P-value, RIT-value and RIR-value).
- At the right, you see detailed information on the answers of the item: the specific item answers, the number of times each answer was given, whether or not the answer was correct.
Note: For multiple choice and multiple response questions, the A-value of the question distractors is added. An A-value of 0 means no participant chose the distractor, while a value of 1 means all participants have chosen it.
The information on the item detail page will help you to improve the quality of your items. For example:
- It can be useful to know that a certain distractor of a multiple-choice question is never selected by the participants. You may want to replace this distractor with a better one.
- Or it can be useful to know which answers are often given in fill gaps items, so you can update the item and let it take other possible answers into account.
Tip: You can navigate through the item detail pages of an assignment using the navigation bar at the top of the screen.
Excluding questions from your exam
When checking the item analysis of your exam, it is possible that you detect an incorrect or incomplete question. Based on this information, you can decide to exclude questions from your exam.
You have two options to exclude questions: you can either eliminate or neutralise questions:
- Neutralising means that all participants get the maximum score for the question. The maximum score of the exam does not change.
-
Eliminating means that you will not take the scores of the question into account because you remove the question from the exam. The maximum score of the exam will then decrease.
Notes:
- Neutralising or eliminating an item applies to all sessions (even future sessions) for your exam.
- Neutralising or eliminating items will impact the pass rate and score for your exam.
- Neutralising or eliminating items will change the statistical values for other items in your exam (RIT or RIR, for example). However, the statistics and calculations that are available on the item itself remain unchanged. For example, the P-value of a neutralised item will not change to 1.
- Answers to neutralised or eliminated questions are still displayed in the Grading module, but their score cannot be changed anymore and a message explains why.
- When a question is eliminated or neutralised, this will immediately be visible in the Results modules, so the reporting is always up to date.
- When a question has been eliminated or neutralised, the participants will no longer see it when they review their exam. They will see a message that the item was neutralised or eliminated.
Neutralising a question
To neutralise an item in an exam, proceed as follows:
- Open the item analysis of your choice. More information.
- Open the Items tab.
- Click on the three dots next to the item of your choice and select Neutralise.
-
Optionally, add a comment why you want to neutralise this question and click Yes.
Result: All participants get the maximum score for the question. The maximum score of the assignment does not change. When the participants review their exam, the item is no longer shown. The participants will see a message that the item was neutralised.
Eliminating a question
To eliminate a question, proceed as follows:
- Open the item analysis of your choice. More information.
- Open the Items tab.
- Click on the three dots next to the item you want to eliminate and select Eliminate.
-
Optionally, add a comment why you want to eliminate this question and click Yes.
Result: The maximum score of the assignment is decreased by the maximum score of the eliminated item. The scores for this item are no longer taken into account. When the participants review their exam, the item is no longer shown. The participants will see a message that the item was eliminated.
Filtering the item analysis of an exam by a schedule
By default, the data you see in the Assignment overview or Item overview is calculated based on all sessions for the selected exam.
You can, however, filter by schedule to recalculate the data based on the sessions in a specific schedule. For example, if you want to compare how participants performed on an exam in two different schedules, you can activate the schedule filter.
To filter the item analysis on a schedule, proceed as follows:
- Open the item analysis of your choice. More information.
- In the top right corner, select the schedule you want to filter on.
Result: The color of the dropdown button changes to green, clearly indicating that a filter is in use. The data columns on your screen are calculated based on the sessions in the schedule.Note: The schedule filter is applied to all tabs in the item analysis (Assignment and Items tab) and in the item detail pages.
- De-activate the schedule filter to recalculate the data columns again for all sessions linked to the assignment.
Note: You can only filter by one schedule at a time. Multi-selection of schedule filters is not supported.
Exporting the item analysis of an exam to Excel
As an analyst, you may want to perform advanced analytics through data exports.
To export the item analysis of an assignment to Excel, proceed as follows:
- In the overview screen of all assignments, click on the three dots at the right side of your assignment and select Export to Excel.
- In the Assignment or Items tab, click on the three dots in the top right corner and select Export to Excel.
Result: You'll be able to download the file from the notification center whenever the export is ready.
Note: All data that is shown on your screen will be exported. If a schedule filter is applied, only the filtered data will be exported. If you want to add or remove columns for the items overview, see Customising the item overview.
Opening an assignment report via Item Analysis
When you consult the item analysis for an assignment, you might want to click through to the report for that specific assignment.
To open the assignment report from the Item analysis module, proceed as follows:
- In the overview screen of all assignments, click on the three dots at the right side of your assignment and select Open report.
- In the Assignment or Items tab, click on the three dots in the top right corner and select Open report.
Previewing or opening an assignment via Item analysis
When you consult the item analysis for an assignment, you might want to preview or open the assignment.
To preview or open the assignment from the Item analysis module, proceed as follows:
- In the overview screen of all assignments, click on the three dots at the right side of your assignment and select Preview or Open Assignment.
- In the Assignment or Items tab, click on the three dots in the top right corner and select Preview or Open assignment.
Getting access to the item analysis module
To get access to the Item analysis module, you need to have the Analyst role. Administrators also have access out of the box, as they can see everything in the backoffice.
Tip: If you are an author and you would like to have access to the item analysis of your exams, please contact the administrator of your assessmentQ environment.
Restrictions in item analysis
When working in the Item analysis module, please note the following limitations:
- Item analysis is available for assignments of the type exam only. Exercises are not included.
Exercises are generally taken multiple times and the participants may become familiar with the content or questions, leading to a bias in the difficulty and discrimination of the items. - Assignments with on-the-fly item selection or adaptive paths are not included in the Item analysis module. In adaptive or on-the-fly exams, the participants receive a variable set of questions.
The calculation of the psychometric values in assessmentQ requires all participants to answer the same set of questions in an exam. - The data in the Item analysis module is updated once a day (nightly batch). If participants finish the assignment on the same day that you want to consult the analysis, the item analysis will not be available until the next update.
-
Minimum number of sessions
Some data (for example, the normal distribution) may become unreliable when only a few participants have taken the exam. Therefore, a minimum threshold of 10 sessions is required before the data is used in calculations. - The information in the item detail page is available for a limited number of item types. A message is shown in case the item type is not supported.
- The A-value is supported for multiple choice and multiple response questions only.