Ñî¹óåú´«Ã½

Test Security

A New Approach to Detecting Cluster Aberrancy (RR 16-05)

by Dmitry I. Belov

This report addresses a general type of cluster aberrancy in which a subgroup of test takers has an unfair advantage on some subset of administered items. Examples of cluster aberrancy include item preknowledge and test collusion. In general, cluster aberrancy is hard to detect due to the multiple unknowns involved: Unknown subgroups of test takers have an unfair advantage on unknown subsets of items. The issue of multiple unknowns makes the detection of cluster aberrancy a challenging problem from the standpoint of applied mathematics. This report presents a novel algorithm to detect cluster aberrancy. The algorithm is general and applicable to all types of testing programs: paper-and-pencil testing, computer-based testing, multistage testing, and computerized adaptive testing; it can also be applied in areas outside of psychometrics, such as finance (e.g., detecting financial fraud). Both simulated and real data were used to study the performance of this algorithm.

Back to report gallery

Request the full report

Additional reports in this collection

Detecting Groups of Test Takers Involved in Test...

Test collusion (TC) is the sharing of test materials or answers to test questions (items) before or during a test. Because of the potentially large advantages for the test takers involved, TC poses a serious threat to the validity of score interpretations. The proposed approach applies graph theory methodology to response similarity analyses to identify groups involved in TC while minimizing the false-positive detection rate. The new approach is illustrated and compared with a recently published method using real and simulated data.

Read More

Comparison Study of Item Preknowledge Detectors (RR 14-01)

When a test taker has prior knowledge about an administered test question (item), then this event is called item preknowledge, the test taker is called aberrant, and the item is called compromised. Item preknowledge negatively affects the corresponding testing program and its test score users (universities, companies, government organizations) because the scores produced for aberrant test takers will be invalid. The performance of eight statistics for detection of item preknowledge (five existing, two modified, and one new) was studied via computer simulations.

Read More