|Year : 2016 | Volume
| Issue : 1 | Page : 58-65
Graphical modeling for item difficulty in medical faculty exams
L Tomak1, Y Bek1, MA Cengiz2
1 Department of Biostatistics and Medical Informatics, Faculty of Medicine, Ondokuz Mayis University, Atakum, Samsun, Turkey
2 Department of Statistics, Faculty of Arts-Science, Ondokuz Mayis University, Atakum, Samsun, Turkey
|Date of Acceptance||26-Jun-2015|
|Date of Web Publication||12-Jan-2016|
Department of Biostatistics and Medical Informatics, Faculty of Medicine, Ondokuz Mayis University, Atakum, Samsun
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Background: There are different indexes used in the evaluation of exam results. One important index is the difficulty level of the item that is also used in this study to obtain control charts. This article offers some suggestions for the improvement of multiple-choice tests using item analysis statistics.
Materials and Methods: The graphical modeling is important for the rapid and comparative evaluation of test results. The control chart is a tool that can be used to sharpen our teaching and testing skills by inspecting the weaknesses of measurements and producing reliable items. The research data for the application of control charts were obtained using the results of the fourth and fifth-grade student's exams at Ondokuz Mayis University, Faculty of Medicine. I-chart or moving range chart (MR) is preferred for whole variable data.
Results: It is seen that all observations are within control limits for I-chart, but three points on MR-chart are settled on the LCL. Using X–-chart with subgroups, it was determined that control measurements were within the upper and lower limits in both charts. The difficulty levels of items were examined by obtaining different variable control charts. The difficulty level of the two items exceeded the upper control limit in R- and S-charts.
Conclusion: The control charts have the advantage for classifying items as acceptable or unacceptable based on item difficulty criteria.
Keywords: Item difficulty, quality control, statistical process control, variable control charts
|How to cite this article:|
Tomak L, Bek Y, Cengiz M A. Graphical modeling for item difficulty in medical faculty exams. Niger J Clin Pract 2016;19:58-65
| Introduction|| |
Statistical process control (SPC) is an important technique to assess a production process through control charts. Control charts are developed to monitor SPC and to determine variations in the production process.
Shewhart developed the first type of chart in 1920. Initially, control charts were used in order to solve industrial problems. After Shewhart, Levey and Jennings used control chart and SPC in clinical laboratory in 1950. The chart of Levey Jennings is widely preferred in clinical laboratory nowadays. This chart uses individual values, and it is plotted according to the reference value.,
Control charts are classified according to whether they monitor attribute (discrete) data or variable (continuous) data. Attribute characteristics are used as binary data that are “conforming-not conforming,” “good-bad” or “defective-nondefective” in quality control.,
Some control charts monitor variability of characteristics. The characteristics are measured as variable data for variable control charts. Control charts for variables include charts for individual measurements; charts for variability or dispersion of several measures, such as the range and the standard deviation, and charts for central tendency of several measures, such as averages, medians, and midranges. Many applications include the companion use of a chart for variability and a chart for central tendency.
The principles of creating both variable control charts and attribute control charts are same. The aim is to determine the mean, the standard deviation, and the distance between the mean and the control limits based on the standard deviation. Control charts for variable characteristics are often used, and they are very effective in providing feedback about the performance of the process. These graphs can be classified in different ways for the individual measurement, for subgroups and for different combinations of subgroups.
The value of each measurement in control charts for individual observations is shown by a single point on the chart. There are three different charts for individual observations. They are individual observations (I) chart, moving range (MR) chart, and I-MR chart that are obtained using both of them.
Other control charts for variable characteristics are preferred if there are existing subgroups. These charts are respectively average (X̄ chart, range (R) chart, standard deviation (S) chart, X̄-R chart that consists of using a combination of X̄ and R and X̄ -S Chart that consists of using a combination of X̄ and S. In addition, there is I-MR-R/S-chart that is obtained with the combination of I, MR and R, or S.
The exam results were evaluated using the following control charts in this study. They are individual observations (I) chart, MR chart, and I-MR chart, average (X̄ chart, R-chart, standard deviation (S) chart.
The production process to improve individual test items and to increase the quality of the test as a whole is evaluated through these control charts. When statistical process is an “in control” process, the test results are explained as a “good” item, but if statistical process is not an “in control” process, the test item identified need to be revised or thrown out; we may want to find out why that particular item had problems producing reliable measurements.
The purpose of this study is to offer some suggestions for the improvement of multiple-choice tests using item analysis statistics and control charts that are mainly used to control a manufacturing process. The control charts will allow us to identify outliers among parts of items. Inspecting that particular part may give us some insights into the weaknesses in our measurements. As a result, the statistical processes of these examinations were evaluated by plotting different variable control charts.
| Materials and Methods|| |
The data for the application of control charts were obtained using the results of the fourth and fifth-grade student's exams at Ondokuz Mayis University, Faculty of Medicine.
There are different indexes used in the evaluation of exams. The difficulty level was used to obtain control charts for this study. Item difficulty is a measure of the percentage of students answering a question correctly. Values for the difficulty index range from 0% (very difficult) to 100% (very easy). The difficulty index can be used to alert the instructor about potential problems. The level of difficulty is considered as between 0.2 and 0.8 for the acceptability of a problem.
We chose all of fourth-grade blocks which consist of 12 examination results; we also chose randomly one block among all blocks of fifth-grade class which consists of 20 questions results. Sensory block was recruited for the whole fifth class. The difficulty levels of 20 questions were evaluated for five different training groups. Because the number of training groups is five, the number of subgroups is accepted as 5.
We used control charts for individual observations of fourth-graders examination results of whole class, we illustrated the individual observation graphs for endocrine, gastroenterology, pediatrics, and pulmonary circulation blocks on the same graphics that belong to fourth grade class. We also used all of X̄
-chart, R-chart and S-chart for examination of sensory block of fifth grade class to see control charts of subgroups.
The analysis of examination was evaluated by control charts. MINITAB Statistical Software, Release 15 for Windows was used to get variable control charts.
Control charts for individual observations
Individual observation chart
The individual observation chart is the most basic variable control chart used for individual measurements of a variable type characteristic. For this chart, the measurable characteristics are selected. The frequencies of obtained measurements are determined.
The control limits of this chart are calculated according to the formula1, where “k” indicates the coefficient of the desired standard deviation. The arithmetic mean (X̄) is generally used to estimate the mean of the process characteristic. The equality for k = 3 is as follows:,
The MR is used to estimate the standard deviation (σ). The MR is defined as the absolute value of the difference between the largest and smallest observations in consecutive subsets of “n” observations. So the MRs and the average are computed as:,
When MR is used to estimate the standard deviation, derivation of control limits for the individual observations chart makes use of the relationship between standard deviation and ranges. The constant of proportionality is referred to as d2. The standard deviation is obtained as . Control limits are derived as follows:
Where d2, as a function of the subset size “n,” is tabulated in special table (Montgomery, 1996). For n = 2, d2 is usually equal 1.128, and the coefficient is 3/1.128 = 2.66. So control limits are simplified as follows:,
If both µ and σ are known, they can be used directly. If only σ is known, X̄ is calculated but σ can be used directly. Control limits are determined accordingly:,
Moving range chart
The MR is also used to estimate the standard deviation (s) for MR-chart.,
Once all of MR values are obtained, the average of MR shows center line. Control limits are as follows:
The following equations (D3, D4) are used to simplify the formulas of the control limits:,
Where both D3 and D4, as a function of the subset size for “n,” are tabulated in specific table. So control limits are shown as follows:,
I-MR-chart is created using both I-chart and MR-chart.
Control charts for subgroups
Average (X̄) chart
General principle for the creation of control limits of X̄-chart is µ ± 3σ for cases given standard and for cases with no standard or
When standard value is unknown, the mean average value is calculated. In order to determine , should be calculated for each subgroup (k). creates value of CL:,
The next step is to calculate UCL and LCL:,
To obtain control limits, the value of standard deviation must be determined for the population. This can be determined in several ways. The first option is to use the standard error estimate σ/√n, and another option is to use the mean range.
Standard error-based X̄-chart
Based on the central limit theorem, to obtain control limits, standard deviation of the process is divided by the square root of the sample's size:
When the value of σ is estimated by S/c4 for example, control limits are created as follows:
Control limits can be simplified by the equation of A3:,
Mean range-based X̄-chart
For the data to have normal distribution, there is a special relationship between the range and the standard deviation as follows:
R is called the relative range. The mean range is:
'k' is the number of subgroups. The estimator of σ is
The estimator of σ/√n is
Control limits are:
When the abbreviation of A2 is used, control limits are:,
If both m0 veya σ0 are known, control limits will be as follows:
The value of A is found using a special table that shows the different values of A for different subgroup sizes.
Primarily, a measurable feature of the process is selected for the creation of R-chart. The range value for each subgroup is calculated. The mean range is detected using the range values for subgroups.,
The center line of R-chart is the mean range that is shown by R̄.,
The equation of sR = d3.σ is used to calculate the standard deviation of the mean range. Because, the equation of σR= d3.σ can be written as follows:
Both of the equations D3 and D4 are used to obtain control limits:,
When both of the equations D3 and D4 are used, the control limits will be as follows:
If the value of σ is known, the control limits are as follows:
Both of the equations D1 and D2 are used according to the size of subgroups.
A measurable feature of the process is used for S-chart, too. The value of standard deviation of every group is detected.
When neither μnorσ is known, standard deviation (S) is calculated using variance (S 2) of sample:
S̄ is the average standard deviation for “k” groups. When S is used the estimate of σ, the biased results appear. To prevent this situation, it is recommended to use c4.σ. Here, c4 is a constant value associated with sample size. If the equation of is considered, s will be equal to The expected value of S [E(S)] will be equal to c4.σ, which is center line. The standard deviation of S is ,
When is used as the estimate ofs, the control limits are:
The equations of both B3 and B4 are given below:
The control limits can be shown to use both B3 and B4:
If both μ and σ are known, the control limits are:
The equations of both B5 and B6 are given as follows:
When these equations are used, control limits will be as follows:
Control charts for different combinations of subgroups
X̄-R-chart consists of a combination of X̄ and R. X̄-S-chart consists of a combination of X̄ and S. I-MR-R/S-chart is the combination of I, MR and R or S.,
| Results|| |
The variable control charts were created to evaluate item difficulty level derived from exam analysis.
The control charts for individual observations are as follows: I-chart is given in [Figure 1] and MR-chart is given in [Figure 2]. There were four separate item groups and exam blocks. There are 12 points on these charts. The first four points refer to difficulty level of endocrine block items for every group (four groups). The second four points refer to difficulty level of gastrology block items for every group (four groups). The third-four points refer to difficulty level of pediatrics block items for every group (four groups). The last four points refer to difficulty level of pulmonary circulation block items for each group (four groups).
All of the data are within control limits on I-chart [Figure 1].
All the observations are within the control limits on MR-chart. However, the points 8, 12, and 13 are settled on the LCL [Figure 2].
Standard error-based X̄-chart is given in [Figure 3]. All of the observations are within control limits and the variations exhibit a random pattern, so the process is stable and under control.
Mean range-based X̄-chart is shown in [Figure 4]. Because all of the data are settled within control limits, we can conclude that the process is under control.
S-chart is given in [Figure 5]. As two points, 6 and 20 are outside UCL, statistical process is out of control.
R-chart is shown in [Figure 6]. Two points, 6 and 20, are outside UCL on this chart, so the process is unstable.
| Discussion|| |
Six different variable control charts are used for continuous data, and the quality of item analysis process is evaluated. Here, difficulty levels of items were examined by obtaining different variable control charts.
The graphical modeling is important for the rapid and comparative evaluation of test results. The control chart is just one more tool we can use to sharpen our teaching and testing skills. The control charts will allow us to identify outliers among parts of items. Inspecting that particular unstable part may give us some insights into the weaknesses in our measurements.
The control charts have the advantage for quick summaries of various aspects of the quality of the items of exams. Simply classify items as acceptable or unacceptable based on item difficulty criteria. Also, this type of charts tend to be more easily understood by lecturer or other related people unfamiliar with quality control procedure or other statistical quality control index expectations.
The item difficulty data has normal distribution. So it is assumed that 68% of the values fall in the interval ± 1.S; 95% of the values fall in the interval ± 2.S; 99.7% of the values fall in the interval ± 3.S., If a value is distant ± 3.S from the average value, this is an unexpected observation (0.3% randomly). So this usually indicates a problem with the process. A measurement value that is distant ± 2.S from the average value may occur at least 5% randomly.,,
When there are not any subgroups, I-chart or MR-chart is preferred for variable data. Both I-chart and MR-chart were created to analyze item difficulty of fourth-grade exam. When these are evaluated, it is seen that all observations are within control limits, but three points on MR-chart are settled on the LCL. However, the processes are assumed to be in control for both charts.
In I-chart, the mean difficulty index was found to be 0.7738. This indicates that mean difficulty level was high, and it is a proof that the questions were easy. However, it was determined that values of all the blocks were within the desired limits. While the distribution for most of the observations was close to the mean difficulty level, it was determined that the questions of gastrology items block of the second group were quite easy (0.90) and the questions of pediatric items block of the first group were more difficult (0.63).
MR-chart is a chart that demonstrates the variability. Although it is within the control limits, there is a variation on this chart originating from the first items group. This difference is specifically remarkable in pediatric and pulmonary circulation training groups (internships).
I-chart is the simplest chart among all of control charts. This chart is preferred to get a result in a short time. It is particularly preferred when using X̄, R and S control charts is not practical. It is much more sensitive in detecting significant variations (variances) in cases where the sample size to be determined by the binormal distribution is very large., MR is generally used to evaluate the mean variability of the difficulty of item. If we do not take the subgroups into consideration, one or both of these charts can be preferred.,,,,,,,,,,,,,,,,,,,
Other variable charts used in the presence of subgroups were obtained in this study. First of them is X̄-chart. In cases with unknown standard values, for the estimation of σ, this chart was generated in two different ways, using standard error and mean range, and it was determined that control measurements were within the upper and lower limits in both charts. So we can say that the process is “in-control.” The mean value was detected to be 0.86 in both charts. This indicates that the questions were easy.
X̄-chart is the most commonly used control chart for the central tendency since it is highly sensitive in detecting the variations related to the distribution of the mean., Standard Error-based X̄-chart is not very practical. Instead of standard error, using the mean range as the measurement of variation, an alternative mean range-based X̄-chart can be created more easily. When sample sizes are relatively small (n ≤ 10), the mean range-based X̄-chart can replace standard error-based X̄
-chart. Since the number of the subgroups in this study is five, it would be better to use the mean range-based X̄-chart.,
In the charts obtained in this study, it was found out that difficulty level values of the two questions exceeded the upper control limit in R- and S-charts. Variability in the process was outside the determined limit at two points. R-chart is one of the most widely used control charts in variation measurement. The chart is popular because it is easy to calculate the mean range. Its effectiveness in estimating the variation of the process is better when the number of subgroups is smaller. S-chart is used to determine if the level of variation in the process is significant or not. Thus, control limits of the standard deviations related to the samples taken at regular intervals are created in this chart. When there is a strong variation within the established limits, the process is considered to be unstable.,, Since the number of subgroups in this study were five, R-chart was preferred to S-chart.
| Conclusion|| |
Control charts are effective instruments in detecting the specific causes of the variability of the difficulty of items in exams. If some points are outside the control limits, it indicates that the item is out of control; it could be either easy or difficult. The balance of the construction of items of exam has been deteriorated, and corrective operations should be performed. As soon as deterioration occurs in the process, the factor leaving the present observation outside the limits (the factor pushing the present observation outside the limits) should be detected. It does help identify questions that need to be revised or thrown out. The tests can be improved by maintaining and developing a pool of “good” items from which future tests will be drawn in part or whole.
| References|| |
Bass I. Six Sigma Statistics with Excel and Minitab. USA: McGraw-Hill Company; 2007.
Winkel P, Zhang NF. Statistical Development of Quality in Medicine. England: John Wiley and Sons Press; 2007.
Westgard JO. Basic QC Practices. 2nd
ed. Madison: Westgard QC Inc.; 2002.
Westgard JO, Groth T. Design and evaluation of statistical control procedures: Applications of a computer “quality control simulator” program. Clin Chem 1981;27:1536-45.
Montgomery DC. Introduction to the Statistical Quality Control. 3rd
ed. New York: John Wiley and Sons Inc.; 1996.
Oakland JS. Statistical Process Control. 5th
ed. London: MPG Books Limited; 2003.
Chandra MJ. Statistical Quality Control. USA: CRC Press; 2001.
Wadsworth HM, Stephens K, Godfrey AB. Modern Methods for Quality Control and Improvement. New York: John Wiley and Sons Inc.; 1986.
Ryan TP. Statictical Methods for Quality Improvement. New York: John Wiley and Sons Inc.; 1986.
Backhoff E, Larrazolo N, Rosas M. The level of difficulty and discrimination power of the basic knowledge and skills examination (EXHCOBA). Revista Electrónica de Investigación Educativa 2000;2:1-16.
Minitab Inc. MINITAB Statistical Software, Release 15 for Windows. State College, Pennsylvania; 2009.
Kume H. Statistical Methods for Quality Improvement. 10th
ed. Tokyo: 3A Corporation; 1992.
Ishikawa K. Guide to Quality Control. Tokyo: Asian Productivity Organization; 1986.
Westgard JO, Barry PL, Hunt MR, Groth T. A multi-rule Shewhart chart for quality control in clinical chemistry. Clin Chem 1981;27:493-501.
Gray JJ, Wreghitt TG, McKee TA, McIntyre P, Roth CE, Smith DJ, et al.
Internal quality assurance in a clinical virology laboratory. I. Internal quality assessment. J Clin Pathol 1995;48:168-73.
Westgard JO, Klee GG. Qualitymanagement. In: Burtis CA, Ashwood ER, Bruns DE, editors. Tietz Textbook of Clinical Chemistry and Moleculer Diagnostics. 4th
ed. New York: Elsevier Press; 2006. p. 485-523.
Cembrowski GS, Martindale RA. Qualitycontrolandstatistics. In: Bishop ML, Fody EP, Schoeff LE, editors. Clinical Chemistry: Principles, Procedures, Correlations. 4th
ed. Philadelphia: Lippincott Williams and Wilkins; 2004. p. 48-89.
Goris N, De Clercq K. Quality assurance/quality control of foot and mouth disease solid phase competition enzyme-linked immunosorbent assay – Part II. Quality control: Comparison of two charting methods to monitor assay performance. Rev Sci Tech 2005;24:1005-16.
Taghizadegan S. Essentials of Lean Six Sigma. USA: Elsevier Press; 2006.
Lenz HC, Wilrich TH. Frontiers in Statistical Quality Control. Germany: German Copyright Law; 2006.
Koch DD, Oryall JJ, Quam EF, Feldbruegge DH, Dowd DE, Barry PL, et al.
Selection of medically useful quality-control procedures for individual tests done in a multitest analytical system. Clin Chem 1990;36:230-3.
Thompson JR, Koronacki J. Statistical Process Control. 2nd
ed. USA: Chapman and Hall Press; 2002.
Juran JM, Godfrey AB. Juran's Quality Handbook. 5th
ed. New York, USA: McGraw-Hill; 1999.
Revere L, Black K. Integrating six sigma with total quality management: A case example for measuring medication errors. J Healthc Manag 2003;48:377-91.
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5], [Figure 6]