Can automated item generation be used to develop high quality MCQs that assess application of knowledge?

Research and Practice in Technology Enhanced Learning

Table 6 Mean agreement rate inter-judge agreement rates for each of the six quality etrics and the overall cognitive domain judgment item by modality

Item	Mean (SD)
	AIG	Traditional
1^a	0.78 (0.17)	0.68 (0.19)
2^b	0.86 (0.10)	0.73 (0.09)
3^c	0.72 (0.18)	0.71 (0.07)
4^d	0.79 (0.05)	0.71 (0.08)
5^e	0.42 (0.18)	0.49 (0.11)
6^f	0.27 (0.16)	0.27 (0.10)
Overall cognitive domain judgment	0.92 (0.07)	0.88 (0.04)

Overall cognitive domain judgment is the item tests factual knowledge only/the item tests application of knowledge.
^aThe central idea is in the stem (i.e., stem is required to answer the item)
^bThe directions in the stem are very clear
^cThere are no obvious cues or item flaws (grammatical cues, conspicuous right answer, etc.)
^dThe length of the choices is about equal
^eAll distractors are plausible
^fThis is a high-quality item