Quiz 3 Measurement And Precision

Shiken: JALT Testing & Evaluation SIG Newsletter
Vol. 11 No. ii. Sep. 2007. (p. 20a) [ISSN 1881-5537]

Suggested Answers for Cess Literacy Self-Study Quiz #three
by Tim Newfields

Possible answers for the questions about testing/cess raised in the
September 2007 issue of this newsletter appear below.

Part I: Open up Questions

1 Q: If you lot could include one additional statistic nearly this exam, which would probably be nearly helpful to full general readers? As well, what other statistics about this exam should probably exist mentioned to the public?

A: Though some bare-bones descriptive statistics are offered, no inferential statistics appear. To assist readers make conclusions that extend beyond the immediate data itself, inferential statistics are needed. Ane statistic that should definitely be mentioned is the standard fault of measurement for this test – the likelihood that a person'south obtained score volition differ from their "true score" due to factors other than their ability. Although the standard fault of measurement (SEM) is mentioned in the TOEIC Technical Transmission (northward.d., p. 24), information technology does not appear in the TOEIC Users Guide nor any full general brochures given to test takers in Japan.
Details of how well the TOEIC^® correlates with other measures of English language proficiency should exist widely distributed, and special care should taken when equating the score of one test with another.

Further reading:

Educational Testing Service. (n.d.). TOEIC Technical Manual. Retrieved September one, 2007 from world wide web.toeic.cl/images/toeic_tech_man.pdf

Simmerok, B. D. (north.d.). Session 6 Lecture: Standard Mistake of Measurement. Retrieved September 2, 2007 from http://home.apu.edu/~bsimmerok/WebTMIPs/Session6/TSes6.html

2 Q: A instructor notices an error in her school archway examination after the exam was administered: i multiple choice question had two possible right answers. What, if annihilation, should exist done?

A: Outset, information technology would be skilful to consider how many items are in the test altogether. Ideally, a high stakes examination such as this should have many items, so that even if a few items do non perform well, the overall test is robust. However, there are many shoddy entrance exams with just 25 - 35 items. In such no really a "dandy" solution exists: any determination fabricated will involve some messy compromises.
If we are dealing with a iv-selection multiple choice question, normally examinees would take a ane:four pick of guessing the correct reply. With two "correct" answers, yet, there is a 1:2 chance of guessing the correct item. This might lead to a slight score aggrandizement. Hence, it would wise to conduct a post-hoc examination of the applicants who just passed the cut-off bespeak for this exam. Were whatsoever of their scores inflated because of this detail? Information technology doesn't actually thing whether students well below or well above the cut-off betoken were affected by the bad item. Notwithstanding, the scores of students who but barely reached the cut-off point should exist examined closely.
At this point, information technology might exist good to look at the item difficulty and particular facility of the problematic particular and as well see how those with the highest scores and lowest scores performed on this particular item. Rather than decide a priori whether to accept or reject that item, it might exist good to encounter how the item is functioning and also reflect on the exam purpose and the social context of the test. Since the population of young people in Japan has been declining, many high school entrance exams concord less and less of a gatekeeping role: for many of the less prestigious schools, entrance exams at present have two bones purposes: (1) income generation, and (2) demonstrating face up validity to the public. In many contexts, the statistical reasons for accepting or rejecting a given exam item matter less to conclusion makers than the bear upon that detail might take on the overall face validity of the test and what information technology says of their institution.
Actually, Section B, Particular 4 of the ILTA Lawmaking of Practice addresses this issue. It states, "Malfunctioning or misfitting tasks or items should not exist included in the calculation of private test taker'due south scores." Past that standard, the item should exist cut.

Further reading:

International Linguistic communication Testing Association. (2007). ILTA Typhoon Lawmaking of Exercise. Retrieved September one, 2007 from http://www.iltaonline.com/ILTA-COP-ver3-21Jun2006.pdf

Miner, B. (Winter 2004/2005). Testing Errors Plague Manufacture. Rethinking Schools. xix (2). http://world wide web.rethinkingschools.org/archive/19_02/erro192.shtml

3 Q: Any problem with the ranking system [proposed past Isao Kobayashi in which universities are rated on the basis of their entrance test questions]?

A: There are, in fact, a number of problems with this ranking system. First, the writer does not explicitly state what the criteria for a skilful English language archway examination are: he is simply rating subjectively and in an opaque manner. Moreover, there appears to be a causality fault in Kobayashi's rating scheme. No bear witness is offered which conclusively links the quality of a university'due south English entrance exams with the quality of their English education. Indeed, they whole effect of how the quality of an English education should be rated is not covered with adequate precision.
From a cultural perspective, what I notice amazing is that a paper of this sort would even exist published. The 2007 Nen Daigaku Rankingu [2007 University Ranking] by Asahi Shinbun is non a minor publication – information technology is a widely read Japanese almanac. For ideas about how to rank universities, information technology might exist helpful to refer to the Berlin Principles on Ranking of Higher Teaching Institutions. Although the system of rating universities proposed by Shanghai Jiao Tong University is now widely cited, it is also problematic in many ways. Their arrangement, for example, favors schools which are strong in science or technology or institutions with Nobel laureates. Though such schools might be stiff in terms of technical inquiry, that does not mean they are good in terms of overall didactics.

Further reading:

Jiao Tong Academy. (2007). Academic Ranking of World Universities. Retrieved September 1, 2007 from http://ed.sjtu.edu.cn/ranking.htm

UNESCO European Heart for Higher Pedagogy. (2006). Berlin Principles on Ranking of Higher Instruction Institutions. Retrieved September three, 2007 from http://www.che.de/downloads/Berlin_Principles_IREG_534.pdf

4 Q: At a university in Japan a placement test was developed to stream science majors into different classes for an EFL reading program. Examinees read a 369 discussion passage most Alfred Nobel, and then answered a series of true/fake statements well-nigh the passage. 1 of the statement from that examination was –

"Alfred Nobel lived his whole life in Sweden."

The response format was to circle either "true" or "false" in the answer sheet. Any problematic points concerning this question? Also, whatsoever issues with using T/F response formats for this sort of exam?

A: Two issues are involved in this question: (ane) the use of absolute "all" or "none" statements in tests, and (2) the content validity of truthful-simulated test items. Each volition be addressed separately.
(one)

Issues with absolute "all" or "none" statements - Essentially, T/F questions are ii-detail multiple choice tests. As such, each pick should roughly as plausible to those who have non actually read the passage. Whereas hedge words such as "some of" or "almost of" have a wide lexical range, statements such every bit "his whole life" have a very narrow lexical range. Even without reading the passage, readers could guess the statement is imitation. For such reasons that Kehoe (1995) recommends avoiding words such words every bit "e'er," "never," "all", (or cognates such as "his whole life") in multiple choice test items. Instead, utilise hedge words with a broader range or consider choices such every bit "over 40 years".
(2) The content validity of true-false test items - The "truthful/false" mode of thinking might is appropriate for some types of information. Notwithstanding, since a lot of scientific data is expanding and unmarried text sources cannot provide sole proof of validity, I advise that a "truthful / imitation / not mentioned" format is more often than not superior to the binary T/F format. To encourage critical thinking skills, it might be skilful to assist readers widen their vision and realize all of the data about a given bailiwick is unlikely to exist in a single text passage. For a statistical signal of voice, a 3-choice "true/false/unmentioned" construct is also better than a 2-choice "true/false" construct considering the chance of guessing is reduced. With well-synthetic true/imitation questions, you accept a i:2 chance of guessing the correct answer. With the "true/false/ not mentioned" framework, the take a chance of guessing correctly is 1:iii. For such reasons, Davies et al (1999) favors 3-to-5 item multiple choice questions to T/F questions.

Further reading:

Kehoe, J. (1995). Writing multiple-choice examination items. Practical Assessment, Research & Evaluation, 4 (9). Retrieved September 2, 2007 from http://PAREonline.internet/getvn.asp?v=four&northward=9

Davies, A. Brown, A., Elderberry, C., Hill, K., Lumley, T., McNamara, T. (1999). True/Imitation Item. In Studies in Linguistic communication Testing, 7: Dictionary of language testing. (pp. 215). Cambridge, Uk: Cambridge University Press.

5 Q: To compare a the mean of a detail sub-grouping to the mean of a larger grouping that is within the same population, what test(south) should be performed?

A: The answer depends on the type of exam data you have every bit well as your sample size and the sub-groups you wish to compare.
If you are dealing with parametric data (data in which a dependent variable is on a interval scale and has a normal distribution) and your sample size is under thirty, a t-test might be chosen for. Garson (northward.d.) offers a good curtailed summary of three major types of types of t-tests and conditions for which they are advisable. With sample sizes over 30, a z-test might be justified. And if you are amid the growing number of people with a Rasch disposition, you might want to compare the overall particular-person fit of the information.
If y'all are dealing with non-parametric information, such as information from a Likert calibration or information with an unknown distribution, so a Mann Whitney U test might be called for when comparison two groups. Nonetheless, depending on the type of data you take and what your exam purposes are, a Kruskal-Wallis Exam or Friedman Test might actually be most advisable.

Further reading:

Garson, G. D. (n.d.). Educatee's t-Test of Difference of Means. Retrieved September ii, 2007 from http://www2.chass.ncsu.edu/garson/PA765/ttest.htm

Cardone, R. (2005). Nonparametric: Distribution-Free, Not Assumption-Costless. Retrieved September 2, 2007 from http://www.isixsigma.com/library/content/c050314a.asp

Function II: Multiple Choice Questions

1 Q: Which of the post-obit is not a common process to investigate examination reliability?

A: The correct answer is (B). There is no documented "equivalent scores method" of ascertaining examination reliability. The "equivalent forms method", however, is widely used.

Further reading:

Garson, Thousand. D. (1998, 2007). Reliability Analysis. Retrieved September 2, 2007 from http://www2.chass.ncsu.edu/garson/pa765/reliab.htm

Mousavi, S.A. (2002). Reliability. In An Encyclopedic Lexicon of Language Testing. (third Ed.). (pp. 580-585). Taipei: Tung Hua Book Company.

2 Q: When reporting test scores to students, which of the following is/are unethical?

A: Option (B) would be unethical considering it involves a violation of privacy. Refer to Principle 2 of the ILTA Code of Ethics for details about privacy.

Further reading:

International Language Testing Clan. (2000). ILTA Code of Ideals. Retrieved September 2, 2007 from www.iltaonline.com/code.pdf

3 Q: To determine a EFL student's progress toward mastery of a classroom content surface area, should be used.

A: Pick (C) is probably the best answer. Information technology seems many university teachers in Japan choose Option (A). This is a valid choice for courses specifically near translation.

Further reading:

Mousavi, S.A. (2002). Formative exam. In An Encyclopedic Dictionary of Language Testing. (3rd Ed.). (pp. 262). Taipei: Tung Hua Book Company.

4 Q: What does this statistical symbol denote: R² ?

A: Unfortunately, there's a bit of inconsistency with respect to the utilise of this term. In Abdi (2007) and Easton and McColl (1997) it stands for Pick (D) - the multiple correlation coefficient. All the same, according to Wikipedia and the Oxford Loftier Beam Encyclopedia, it stands for Option (C) - the coefficient of determination. Rather than fret about the symbol, it is more important thing to understand the concept behind it. JD Dark-brown (2003) offers a good description of the coefficient of determination in this publication. If you accept a stomach for mathematics, you might relish how the Encyclopedia of Mathematics defines the multiple correlation coefficient. Still, most readers will probably prefer the Wikipedia explanation of this concept.

Further reading:

Abdi, H. (2007). Multiple Correlation Coefficient. In N.J. Salkind (Ed.): Encyclopedia of Measurement and Statistics. (pp. 648-651). Thou Oaks, California: Sage.

Brown, J. D. (2003). The coefficient of determination. Shiken: JALT Testing & Evaluation SIG Newsletter 7 (1) 15 - 17. Retrieved September 2, 2007, from http://jalt.org/examination/bro_16.htm

Coefficient of determination. (2007, August 25). In Wikipedia, The Gratuitous Encyclopedia. Retrieved September 2, 2007, from http://en.wikipedia.org/wiki/Coefficient_of_determination

Colman, A. M. (2001). Multiple correlation coefficient R. A Dictionary of Psychology. Oxford, UK: Oxford University Press. Cited in High Beam Encyclopedia under "Multiple correlation coefficient". Retrieved September 2, 2007, from http://www.encyclopedia.com/doc/1O87-multiplecorrelatincffcntR.html

Multiple correlation coefficient. (1997). In V. J. Easton and J. H. McColl (Eds.) STEPS Statistics Glossary. Retrieved September 2, 2007, from http://www.stats.gla.ac.uk/steps/glossary/paired_data.html

Prokhorov, A.Five. (2002) Multiple-correlation coefficient. In Grand. Hazewinkel, (Ed.). Encyclopedia of Mathematics. [Online Edition]. Berlin, Heidelberg, New York: Springer-Verlag. Retrieved September 2, 2007, from http://eom.springer.de/one thousand/m065360.htm