Improving Multiple-Choice Assessments by Limiting Time

Standardized, multiple-choice assessments frequently come under fire because they test rote skills, rather than practical, real-world application.  Although this is a gross over-generalization failing to account for the cognitive-complexity the items (questions) are written to, standardized assessments are designed to evaluate what a person knows, not how well they can apply it.  If that were the end of the discussion, you could be forgiven in assuming standardized testing is poor at predicting real-world performance or differentiating between novices and more seasoned, experienced practitioners.  However, there is another component that, when added to standardized testing, can raise assessments to a higher level: time.  Time, or more precisely, control over the amount of time allowed to perform the exam, can be highly effective in differentiating between competence and non-competence.

The Science Bit

Research in the field of expertise and expert performance suggests experts not only have the capacity to know more, they also know in a way differently than non-experts; experts exhibit different mental models than novices (Feltovich, Prietula, & Ericsson, 2006).  Mental models represent how individuals organize and implement knowledge, instead of explicitly determining what that knowledge encompasses.  Novice practitioners start with mental models representing the most basic elements of the knowledge required within a domain, and their mental models gradually gain complexity and refinement as the novice gains practical experience applying those models in real world performance (Chase & Simon, 1973; Chi, Glaser, & Rees, 1982; Gogus, 2013; Insch, McIntyre, & Dawley, 2008; Schack, 2004).

While Chase and Simon (1973) first theorized that the way experts chunk and sequence information mediated their superior performance, Feltovich et al. (2006) suggested these changes facilitated experts processing more information faster and with less cognitive effort contributing to greater performance. Feltovich et al. (2006) noted this effect as one of the best-established characteristics of expertise and demonstrated in numerous knowledge domains including chess, bridge, electronics, physics problem solving, and medical applications.

For example, Chi et al. (1982) determined that the way novices and experts approach problem-solving in advanced physics was significantly different despite all subjects having the same actual knowledge necessary for the problem solution; novices focused on surface details while experts approached problems from a deeper, theoretical perspective.  Chi et al. also demonstrated the novice’s lack of experience and practical application contributed to errors in problem analysis requiring more time and effort to overcome. While the base knowledge of experts and novices may not differ significantly, experts appear to approach problem solving from a differentiated perspective allowing them more success in applying correct solutions the first time and recovering faster when initial solutions fail.

In that vein of thought, Gogus (2013) demonstrated that expert models were highly interconnected and complex in nature, representing how experience allowed experts the application of greater amounts of knowledge in problem solving.  The ability for applying existing knowledge with greater efficiency augments the difference in problem-solving strategy demonstrated by Chi et al. (1982).  Whereas novices apply problem-solving approaches linearly one at a time, experts evaluate multiple approaches simultaneously in determining the most appropriate course of action.

Achieving expertise is, therefore, not simply a matter of accumulating knowledge and skills, but a complex transformation of the way experts implement that knowledge and skill (Feltovich et al., 2006). This distinction provides clues into better implementing assessments to differentiate between expert and novice: the time it takes to complete an assessment.

Cool Real-World Example Using Football (Sorry. Soccer)

In an interesting twist on typical mental model assessment studies, Lex, Essig, Knoblauch, and Schack (2015) asked novice and experienced soccer players to quickly and accurately decide the best choice of tactics (either “a” or “b”) given a video image of a simulated game situation.  Lex et al. used eye-tracking systems to measure how the participants reviewed the image, as well as measuring their accuracy and response time.  As one would expect, the more experienced players were both more accurate in their responses, as well as quicker. Somewhat surprising was the reason experienced players performed faster.

While Lex et al. (2015) determined both sets of players fixated on individual pixels in the image for nearly the same amount of time, experienced players had less fixations and observed less pixels overall.   Less experienced players needed to review more of the image before deciding, and were still more likely to make incorrect decisions.  On the other hand, more experienced players, although not perfect, made more accurate decisions based on less information.  The difference in performance was not attributable to differences in basic understanding of tactics or playing soccer, but the ability of experienced players to make better decisions with less information and taking less time.

The Takeaway

Multiple-choice, standardized assessments are principally designed to differentiate what people know, with limited ability to differentiate how well they can apply that knowledge in the real world.  Yet, it is also well-established that competent performers have numerous advantages leading to better performance in less time.    If time constraints are actively and responsibly constructed as an integral component of these assessments, they may well achieve better predictive performance; they could do a much better job of evaluating not just what someone knows, but how well they can apply it.



Chase, W. G., & Simon, H. A. (1973). The mind’s eye in chess. In Visual Information Processing (pp. 215–281). New York, NY: Academic Press, Inc.

Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), Advances in the psychology of human intelligence (Vol. 1, pp. 7–75). Hillsdale: Lawrence Erlbaum Associates.

Feltovich, P. J., Prietula, M. J., & Ericsson, K. A. (2006). Studies of expertise from psychological perspectives. In The Cambridge handbook of expertise and expert …. New York, NY: Cambridge University Press.

Gogus, A. (2013). Evaluating mental models in mathematics: A comparison of methods. Educational Technology Research and Development, 61(2), 171–195.

Insch, G. S., McIntyre, N., & Dawley, D. (2008). Tacit Knowledge: A Refinement and Empirical Test of the Academic Tacit Knowledge Scale. The Journal of Psychology, 142(6), 561–579.

Lex, H., Essig, K., Knoblauch, A., & Schack, T. (2015). Cognitive Representations and Cognitive Processing of Team-Specific Tactics in Soccer. PLoS ONE, 10(2), 1–19.

Schack, T. (2004). Knowledge and performance in action. Journal of Knowledge Management, 8(4), 38–53.

Misconceptions about Certification

There seems to be wide misconceptions about what “Certification” and “Licensure” are all about.  Some see it as just the final part of an educational regimen.  Other’s see it as some kind of hurdle imposed by greedy organizations restricting access to some benefit.  These misconceptions and skewed perspectives, lead them to make demands affecting the very heart of what certification is all about and minimizing the value of the very certification they are working to achieve. First:

What Certification IS …

Certification is a legally defined classification stating a certifying body stands behind the capability of a certified individual to perform at some specific level of capability.  Certification is a very simple construct: it is the definition of a standard of performance, and a program assessing individuals to that standard.  That’s it, nothing more and nothing less.

Setting a standard is the first part of any certification.  Ideally, this standard is defined with help of people who perform the job.  In addition, a standard implies that everyone, no matter whether they have done the actual job for decades or participated in the development of the standard itself, must objectively demonstrate their capability in exactly the same manner.  Unless this standard is objectively applied equally to all individuals, it is not a standard.  Maintaining that standard is the foundation of certification value; it is what the certification stands for and how it should be used.

Creating an assessment of that standard is the second part of any certification.  This not only includes a mechanism assessing current capability, but also a program or process to ensure future capability.   Despite what many believe, creating an assessment is not a simple, ad hoc process where someone creates an assessment (test) and makes people take it to prove their ability.  It is, in fact, a highly rigours process backed by decades of scientific research on measuring cognitive ability (psychometrics), the sole purpose of which is ensuring the validity of the decisions made based on assessment performance.  It involves designing the assessment, job-task analysis of the job being assessed, formalized standards of item (question) writing, public beta-testing of items, psychometric evaluation of the item performance, assessment construction, and evaluation of appropriate scoring.  Done properly, it is time-consuming and costly (which is why some organizations don’t do it properly).

Finally, because knowledge and competence are perishable commodities, mechanisms must be put in place ensuring certified individuals remain capable in the future.  This is frequently done by limiting the length of time a certification is valid and requiring periodic re-certification (re-validation) of the individuals capability.  Other methods may include proving continued education and practice of the knowledge.  Regardless, the ongoing evaluation of certified individuals must adhere to the original standard with the same validity, or the standard no longer has value.

There is no hidden agenda to certification.  There is no conspiracy.  It is simply to establish a standard and assess individuals compared to that standard.

Misconceptions about Certification

Really, any belief beyond the design of a standard and the assessment to that standard, is a misconception about certification.  However, there are a number of misconceptions that often drive changes detrimental to the rationale of certification.  The most common ones relate to understanding what an assessment is for, training, and re-certification requirements.

Most people mistakenly assume the items within an assessment must represent the end-all-be-all of what someone should know.  They don’t understand that a psychometric assessment is not about the answers themselves, but the inferences we can make about performance based on those answers.  There is no way to develop an exhaustive exam of all the knowledge necessary to be competent and to deliver that exam efficiently.  However, we can survey a person’s knowledge and through statistical analysis infer whether they have all the knowledge necessary or not.  The answer to any specific question is less important than how answering that question correlates with competence.  Even a poorly written, incomplete, and inaccurate item can give us information about real world performance; in fact, evaluating how a candidate responds to such an item can be highly informative (although this is not a standard, intentional practice).  This focus on the items themselves, rather than the knowledge and competence the answers suggest, is what makes people incorrectly question the validity of the certification.

Similarly, many people think a certification should be able to be specifically taught.  As such, they believe a training course should be all that is necessary to achieve a certification.  However, this does not align with what we know about the development of human competence.  There is a big difference between knowing something, and knowing how to apply that knowledge competently.  Certification is an assessment of performance, not knowledge; and, as such, cannot be taught directly.  If someone can take a class and immediately achieve certification, either: A) the assessment does not evaluate actual performance; or, B) the course simply teaches the answers to the questions on the exam, rather than the full domain of knowledge.  In either case, you have biased the inferences made by the assessment.  Competence begins with knowledge, but must also have experience and practice.  This cannot be gained through a class, but only through concerted effort; you cannot buy competence.

Finally, many people also believe that once a certification has been achieved, it shouldn’t need to ever be evaluated again; or, that taking a course instead of an assessment should suffice.  The former belief simply ignores the fact performance capability is a perishable commodity: if you don’t use it, you lose it.  The latter once again confuses knowledge with performance.  How frequently this needs to happen, or whether continued education is sufficient to demonstrate continued performance is entirely dependent on the knowledge domain the certification attempts to assess.  In highly dynamic environments, this may need to be done much more frequently and rely more on assessments than in other domains; however, ongoing evidence of continued capability is a must if standards are to be maintained.

Leave Certification to the Professionals

The heart of the problem is that everyone seems to believe they are experts in the design of certification programs and assessments simply because they have participated in them.  The reality is that certification is a rigorous, research-based, and scientific endeavor.  The minimum requirement to be considered a psychometrician is a PhD; that’s a great deal of specialized knowledge most people do not have.  The decisions made are not arbitrary, nor are they made with the intention of anything other than maintaining the standard and making valid assessments of individuals according to that standard.

At the end of the day, the value of a certification is whether the people who achieve it can perform according to the standard the certification set forth.  If the certification cannot guarantee that, then it is not valid and has no value.  However, this requires people to actually understand what that standard is, what it means, and why it was created.   It requires people to accept there is a rigorous process accounting for all of the decisions and those decisions all support validity.  Finally, it requires people to understand that just because they may be experts in their field, they are not experts in certification.






Three Reasons Certification is Better than College.

Re-post from LinkedIn – April 8, 2016

Much of the national debate around the value of a college education seems to revolve around the cost of college, particularly in finding ways of making college more affordable and accessible to more people. This frame of reference assumes that a college education is the only means of post-secondary training and education that has value; this is simply not true. According to an analysis of employment during the Great Recession, 4 out of every 5 jobs lost were held by those without any formal education beyond high school; those without post-secondary training were more than three times as likely to lose there jobs than those with even “some college” (Carnevale, Jayasundera, & Cheah, 2012). This, by itself, suggests that even minimal post-secondary training can garner significant benefit; having a job is preferable to not having one. One way to accomplish this is through industry-based certifications (IBCs).

IBCs offer a number of advantages for improving an individual’s employability over simply making college more affordable and accessible. First, in regards to affordability and accessibility, IBCs offer greater return on investment (ROI) than traditional college education. In part because of their far lower time and money commitment, IBCs also provide a more flexible solution either to replace college, or to aid in preparation for later college education. Finally, the dynamic nature and industry relevancy of IBCs provide stronger signals to employers concerning an individual’s ability to actually do the job they need today, rather than months or years down the road. The strong case for IBCs begins with simple economics.

Direct ROI

Given the focus on ROI of post-secondary education, a comparison of the ROI of Certification Earnings Premiumcertification versus a Bachelor’s degree seems relevant. According to the U.S. Census Bureau (Ewert & Kominski, 2014), with the exception of a Master’s degree, there is an earnings premium for achieving certification or licensure regardless of education level. This premium predominantly benefits those with less post-secondary educational investment (see Figure 1). While it is true the earnings premium for having a Bachelor’s degree is much greater (see Figure 2), this is not a measure of return on investment. Return on investment is a measure of what you get for what you put in; i.e. ROI is the amount you can expect to get back for every dollar spent. This is where the ROI of certification is substantially better.

Assuming a cost of $40,000 for a Bachelor’s degree (probably a low estimate) and a cost Education Earnings Premiumof $5,000 to achieve certification (probably a high estimate), the ROI of achieving certification for someone with only a high school education is 2.3 times that of achieving a Bachelor’s degree (see Figure 3). Furthermore, this is just a starting point as it doesn’t account for differences in earning while those achieving a Bachelor’s degree remain in school or the cost of interest on student loans for college tuition. The fact is, certification provides individuals an extremely efficient mechanism to improve their earnings potential, and achieve the post-secondary credentials that improve their ability to get and keep a ROI of Certificationjob, even during tough economic times. The fact that IBCs add value to both those without other post-secondary education as well as those with, also demonstrates the greater flexibility of certifications.


One of the challenges to simply making a college education more affordable (or free), is that cost is not the sole factor contributing to non-participation or non-completion. An analysis of college completion statistics showed a 7% difference in achieving a Bachelor’s degree between students who complete high school with a 3.0 GPA versus those with a 3.5 GPA (Rose, 2013). Rose also reported that family and work responsibilities significantly affect the chances of completing a degree program. In other words, while the cost of a college education might inhibit individuals from starting a degree program, individual preparedness and the time commitment necessary to complete a degree are significant contributors to whether individuals ever actually graduate and garner the benefits. This is likely to be particularly true for low-income or disadvantaged students. IBCs provide more flexibility to address these challenges.

Firstly, IBCs have significantly lower time commitments associated with their completion, making it that much easier for students who must also maintain family and work obligations to complete the requirements for certification. Many IBCs do not even require formalized classes or specific training, allowing individuals to self-study as they are capable or as life permits. Finally, most IBCs award credentials, not based on having completed a regimented program of study, but upon the passing of competency-based exams. This means that students can take as much time, and as many attempts, as necessary without suffering negative consequences; it is not a one-time deal, thus providing greater chances of ultimate success. This applies to both those without any other post-secondary training as well as those with degrees who are simply looking for additional earnings potential.

Secondly, IBCs may provide students not ready for a formal degree program with the knowledge and skill to prepare them for a future degree. IBCs can provide students with exposure to a field of study without the time and financial commitment associated with a formal degree program, reducing the costs associated with choosing a career they ultimate find unsatisfactory or unfulfilling. In addition, this additional knowledge and skill may give students the confidence and ability necessary to complete degrees they would otherwise have been unprepared for.

Not everyone has the time, or capability to commit to formal degree programs. This has a much larger effect on educational outcomes than the cost; simply reducing the cost or providing universal access does not address either of these challenges. IBCs fill a gap between the demands of formalized postsecondary training, and the real world needs of students just trying to stay a head in a highly competitive marketplace while simultaneously making ends meet (Claman, 2012). In addition, IBCs are increasingly more valuable to employers.

Stronger Employability Signals

“The value of paper degrees lies in a common agreement to accept them as a proxy for competence and status, and that agreement is less rock solid than the higher education establishment would like to believe” (Staton, 2014, para. 3).

Despite the nearly $800 billion dollars spent each year in the United States for human capital development beyond primary and secondary education, nearly 70% takes place outside of four-year colleges and universities; of that, U.S. employers spent almost $460 billion on formal and informal employ training alone (Carnevale, Jayasundera, & Hanson, 2012). According to the Economist, only 39% of hiring managers feel college graduates are ready to be productive members of the workforce (“Higher education: Is college worth it?,” 2014).  The Economist further points out the skill gap between college degreed applicants and the needs of employers has left 4 million jobs unfilled. It is no wonder employers are beginning to question whether degrees are appropriate proxies for real world competence; and, some are even seeing advance degrees as a negative hiring signal requiring more cost with little benefit (Staton, 2015).

“The world no longer cares about what you know; they world only cares about what you can do with what you know” (Tony Wagner as quoted by Friedman, 2012, para. 11).

The hands-on, competency-based aspects of IBCs not only create value for individuals directly, but indirectly by providing stronger signals to employers about the actual competence of job candidates. The dynamic and flexible nature of IBCs make them a better reflection of current industry standards and competence even in rapidly changing industries (Carnevale, Jayasundera, & Hanson, 2012). Perhaps even more important, the standards and competency-based testing utilized in IBCs improves the ability to objectively compare applicants, something that has proven extremely unreliable for post-secondary metrics like GPA (Carnevale, Jayasundera, & Hanson, 2012; Swift, Moore, Sharek, & Gino, 2013). IBCs provide employers with highly credible evidence of applicant’s ability to actually do something with their knowledge, not just their ability to know something.

IBCs are increasingly embraced by employers as a more reliable and valid indicator of candidate competence and questioning the value of traditional post-secondary indicators (Carnevale & Hanson, 2015). Because IBCs are, by definition, industry-based, applicants holding IBCs are more likely to have relevant, up-to-date skills meeting national, or international standards. IBCs are not only easier to evaluate, but also provide strong indicators that a prospective applicant will not need additional employer-based training before becoming productive. This is likely why even holders of advanced professional degrees are paid premiums for also having IBCs (Figure 1).


The debate about the current state of education in the United States is a worthwhile discussion, perhaps even a critical discussion in light of the challenges facing us. The problem is the single means of post-secondary education (four-year degrees) that dominates the debate and a singular focus on the cost of educating to this level. This debate fails to account for the many other factors affecting student outcomes, and the actual needs of employers. The reality is that advanced economies are not dominated by high-volume, low-value production, but low-volume, high-value production (Friedman, 2012), and the demand for “middle-education” jobs is growing and will continue to grow for many years (Carnevale, Jayasundera, & Hanson, 2012).   Without addressing these realities, we are only perpetuating a divide between those with degrees and those without, while still failing to meet the needs of business. There will always be a need for formal degrees, but that does not make them the panacea for all people and for all jobs.

At the end of the day, credentialing is an attractive option for anyone looking to improve their employment options.  IBCs provide a greater ROI, in a shorter amount of time than formal degrees.  The flexibility and less structured design of IBCs  make them easier to obtain successfully, especially for students either unprepared for, or unable to commit to formal programs.  Furthermore, IBCs provide strong employment signals to potential employers about the individuals ability to contribute on day-one of employment.  In many cases, the ROI, the flexibility, and the strong employment signals attributed to IBCs may very well be a better option than college; in other cases, IBCs may be an essential stepping-stone to that first degree by providing the skills, and the additional income, necessary to commit to obtaining a formal degree.  AND, if you already have a bachelor’s degree, these same benefits await you compared to getting a graduate degree.  Certification may very well be better than college to many.

NOTE: Anyone interested in exploring how competency-based credentialing is a critical component of the future of higher education should investigate WorkCred (, a non-profit organization working to elevate the visibility of credentialing as an essential ingredient in the future of human capital development in the 21st century. The author is not affiliated with WordCred.


Carnevale, A. P., & Hanson, A. R. (2015). Learn & earn: Career pathways for youth in the 21st century. E-Journal of International and Comparative Labour Studies, 4(1). Retrieved from

Carnevale, A. P., Jayasundera, T., & Cheah, B. (2012). The college advantage: Weathering the economic storm. Retrieved from

Carnevale, A. P., Jayasundera, T., & Hanson, A. R. (2012). Career and Technical Education: Five Ways that Pay. Retrieved from

Claman, P. (2012). The skills gap that’s slowing down your career. Harvard Business Review. Retrieved from

Ewert, S., & Kominski, R. (2014). Measuring Alternative Educational Credentials: 2012, (January), 14. Retrieved from

Friedman, T. L. (2012, November 17). If You’ve Got the Skills, She’s Got the Job. The New York Times. New York, NY. Retrieved from

Higher education: Is college worth it? (2014, April). The Economist. doi:Article

Rose, S. J. (2013). The Value of a college degree. Retrieved from

Staton, M. (2014). The degree is doomed. Harvard Business Review. Retrieved from

Staton, M. (2015). When a fancy degree scares employers away. Harvard Business Review. Retrieved from

Swift, S. A., Moore, D. A., Sharek, Z. S., & Gino, F. (2013). Inflated applicants: Attribution errors in performance evaluation by professionals. PLoS One, 8(7). doi: