Month: January 2024

The Versatilist on Dunning-Kruger

You may not be familiar with the “Dunning-Kruger Effect”; or, you may have only heard the colloquial explanation that “stupid people are too stupid to know they’re stupid”, most humorously explained by Monty Python alum John Cleese here.

In reality, if you read the Wikipedia entry or the actual research, what Dunning and Kruger discovered is that human beings have the tendency to rate our ability, in almost anything, to be at, or slightly above, the average of all people. This is not only an obvious impossibility, it also has some interesting ramifications.

The first, oft repeated, implication is that those with the least capability tend to overestimate their capability the most. That is to say, if we assume 50% is the “average ability” across the population, those with 0% actual capability will overestimate their ability by 50%, (or more) while those with 40% only overestimate by 10%. One explanation for this is that the skills necessary to evaluate capability are exactly the same skills necessary to have the capability; i.e., if you don’t know what you are doing, it is difficult to evaluate that you, or someone else, is doing it wrong. My favorite example of this would be something like English grammar or punctuation: if you don’t have a firm grasp of it, it is impossible for you to evaluate how well you, or someone else, is performing. You must know, in order to evaluate. This is where the “too stupid to know” comes from.

The second, much less discussed, implication is that those with the most capability tend to underestimate their knowledge and competence. Back to the 50% scale, if someone actually performs at the 80 or 90% level, they tend to severely underestimate their performance. This is frequently cited as a contributing factor to imposter syndrome, where those with superior capability don’t necessarily believe they are superior. I attribute this to the colloquial definition of an expert as someone who knows more and more, about less and less (purportedly coined by one of the Drs’ Mayo of Mayo Clinic fame). An extension of this says that an expert is someone who knows more and more, about less and less, until they know absolutely everything about nothing. While this was likely meant to be more humorous than anything, there is a certain kind of meta, philosophical element to it as the process of discovering more and more about an increasingly smaller area of expertise also has the tendency to make it obvious how little you really know about anything else. Experts, while becoming more knowledgeable about their area of expertise, become increasingly cognizant of how little they really know elsewhere.

In either of these situations, overestimating or underestimating, the challenge is that self-reported capability is a very poor predictor of actual ability; and, if you really need an expert because you aren’t one, it is very unlikely you will be able to determine if someone else is one or not.

Hedging Your Bets

Why am I going on about the Dunning-Kruger effect? I point out this well-known characteristic because it touches on my area of expertise … determining the best way to assess expertise, particularly when it comes to augmenting your organization’s capabilities; i.e. this is something we need to think about when we hire people. We need to take this into account and develop strategies to “hedge our bets”.

While resumes are useful, we all know that just because you’ve done something in the past, doesn’t mean you are actually any good at it; and, resumes, although not necessarily outright false, are generally over inflated. Some of this is smart marketing on behalf of the candidate, but some may very well be that the candidate actually believes they are more adept than they are. On the flip side, that expert you’re looking for may be a lot less comfortable touting expertise they don’t feel they actually have. Resumes and interviews are useful, but woefully inadequate and imprecise.

One way to address this is to ensure that the screening/interview process involves some kind of valid psychometric assessment of ability (like respected certifications and licensure) and/or the direct involvement of someone who you know has the appropriate skills to assess the candidate’s ability (if you can find one). You can’t rely on self-reported capability, and you can’t expect someone without that capability to evaluate candidate’s capability … even in the screening process.

Another, perhaps easier, way to hedge your bets is to broaden your horizons. When we post job opportunities, we frequently over estimate the skills required, producing a “wish list” that values “specific” experience over diversity of experience (as I’ve discussed here: Would You Hire Me?). However, if we limit ourselves to one dimension, it can be hard to determine what a candidate’s true capabilities are. If, instead, we look for people who have been successful or demonstrate knowledge of multiple domains, backed by work experience, we may get a better estimation of their knowledge on specific domains. That is to say, a Versatilist, with a broader set of knowledge in multiple domains, is more likely to underestimate their specific domain knowledge than overestimate it. If this doesn’t cause you to overlook these candidates, the only downside is that you may get more than you knew, not less.

Don’t be too stupid to know you’re stupid

The Dunning-Kruger effect is just another factor hindering employers from finding the best people. We all think we are better at everything, including evaluating prospective employees, than we generally are; and, the very people we want are likely to be overlooked because they undersell their capabilities. Using other, valid qualitative criteria like certifications certainly helps, and including experts, instead of AI engines and unqualified HR personnel, in the screening and interview process would also be beneficial.

For my money, until I find a way to fund continued research into better ways, I’ll continue to look for those Versatilists out there who have knowledge and experience, and likely undervalue their true capability.

Standardized Testing in Context of Diversity, Equity, and Inclusion: We need more, not less.

There was recently an article in the New York Times concerning the ongoing debate over standardized testing, specifically about the use of SAT and ACT testing in the college admissions process. The use of these tests has been debated for years, but during the pandemic, when in-person testing became impossible, many educational systems decided to remove this requirement and have simply not reinstated them. 

The point of the article was that, despite many concerns the exams themselves were biased in any number of ways, the use of standardized testing scores in institutions requiring them has actually increased the diversity of the student population (across all factors, race/sex/socioeconomic/etc.) over virtually any other means of admission standard.  In addition, the article points out that what many people see as bias in the tests themselves is likely misplaced: the tests accurately predict what they are intended to predict regardless of race and economics, namely, will the student do well in college or not. 

Herein lies, perhaps, one of the most misunderstood aspects of standardized testing … they can only reliably predict what they are intended to predict and nothing else. As a practicing academic who spends much of his days working on standardized testing programs in the technology industry, I am constantly confronted with these misconceptions. 

What are Standardized Tests?

The first thing to understand is what, exactly, standardized testing actually is.  In short, standardized tests are specifically built to predict some aspect of the individual taking the assessment. In the case of the ACT and SAT exams, they are designed to predict how well the individual will do in the university setting, and nothing more. In addition, by “predict”, I mean that they make a statistical inference, not an absolute determination, as they are based on statistical science which describes a “group”, not any one individual. They do not specifically measure real world capability. They do not measure overall intelligence. They only measure/predict what they are designed to do. 

Two key aspects of this are “Validity” and “Reliability”. Validity is a measure of how well an assessment does what it says it does. Does a high score on the exam actually predict what was intended, or more succinctly “are we measuring what we said we were”.  Reliability is a measure of whether the same individual, taking the same assessment, consistently scores the same without any other changes (like preparation, training, etc.); i.e., does the test make the same prediction every time it is used without any other factors affecting the results. 

Despite what critics say, the SAT and ACT exams have both been proven to be valid predictors of what they measure with high reliability.  My score will accurately (withing statistical deviations) predict my ability to be successful in college and my score will be fairly consistent across multiple attempts unless I do something to change my innate ability.  As the NYT article points out, this remains true: the higher you score on these exams, the better your academic results in post-secondary institutions.  The fact there is a significant discrepancy in scores based on race, socioeconomic situation, or any other factor is, frankly, irrelevant to the validity and reliability of the exam. Using the results of these exams in any context other than how they were designed is an invalid use.

The Legacy of Mistrust

These basic misunderstandings of standardized testing breeds mistrust and suspicion in what they do and how they are used. This is nothing new and likely stems for the development and use of assessments from the past. The original intelligence quotient (IQ) test developed around the turn of the 20th century is subject to the same issues, including suggestions of racial and socioeconomic bias. In part this is because the IQ test is not actually a valid predictor of intelligence or the ability to perform successfully, but like the SAT and ACT exams, research has showed it is a predictor of success in primary and secondary educational environments.  Unfortunately, this was not fully understood when the assessment was born and IQ has been misused in ways that actually have contributed to societal bias.  This is the legacy that still follows standardized testing.

It is bad design, the misuse of standardized testing results, and the misinterpretation of those results that causes such spirited debate.  In the case of the original IQ test, it was originally purported to determine innate intelligence, but was actually a predictor of primary/secondary educational success. Furthermore, research suggests that IQ is a poor predictor of virtually anything else, including an individual’s ability to succeed in life. This is a validity issue; meaning that it did not measure what it purported to measure. Due to the validity issue, IQ testing was then misused to further propagate racial and socioeconomic inequity, by suggesting that different races, or different classes were just “less intelligent” than others, prompting stereotypes and prejudice that simply wasn’t founded. 

Given this legacy, it is easy to understand why many mistrust standardized tests and believe they are the problem, rather than a symptom of a larger problem.

The Real Issue is NOT Standardized Testing

The conversation around standardized testing has suggested the reason for racial and socioeconomic disparity is due to bias within the testing. However, if we can accept standardized tests (at least ones that are well designed to have validity and reliability) simple make a prediction, and that the SAT and ACT, in particular, make accurate predictions of a student’s ability to succeed in post-secondary education, the real question is why is there a significant disparity in results based on race and socioeconomic background? Similarly, why did the original IQ testing accurately predict primary/secondary educational outcomes, but also suffer from the same disparity? The real question is: Why can’t students from diverse backgrounds equally succeed in our education system?

The answer is rather simple and voluminous SAT and ACT data clearly indicate this: there is racial and socioeconomic disparity built into the educational systems. This is a clear issue of systemic bias; your chances of success within the system are greatly affected by race and socioeconomic background. Either what we are teaching, or how we are evaluating performance, is not equitable to all students. This is the issue we should be having conversations about, research conducted, and action taken. Continuing the debate, or simply eliminating standardized testing, is not going to affect the bigger issue. If anything, eliminating SAT and ACT testing will help hide the issue because we will no longer have such clear, documented evidence of the disparity. I don’t want to start any conspiracy theories, but maybe this is one reason so few educational systems are willing to reinstate ACT and SAT testing as part of their admissions requirements, especially when the research suggests they are better criteria for improving diversity than other existing means. They may be imperfect, but it is not the assessment’s fault, it is the system’s fault.

How to Improve? 

First, I want to be clear: I don’t have any specific, research-based solutions. So, before I offer any suggestions based on my years of being in the educational system as a student, my years of raising children going through the educational system, and over a decade working with standardized test design and delivery, I want to emphasize that the best thing we can do to improve is simply to change the conversation away from the standardized tests and focus on the educational system itself. We need research to determine where the issue actually exists; is it what we teach, or how we measure performance?  That MUST be the first step.

That being said, when it comes to “how we measure performance”, based on my background, education, and experience, I’m going to make a radical suggestion: more standardized testing. I know, I know. Our students are already inundated with standardized testing, but hear me out. While standardized test are frequently used in our education system, they are rarely used to measure an individual student’s performance when it comes to grades (the ultimate indicator of success within the system), but as an assessment of the overall school’s performance. My suggestion is that these standardized tests may be a more equitable way to evaluate performance for the individual as well.

From an equity standpoint, while there are some proven correlations between individual test scores on the US National Assessment of Education Progress (NAEP) assessment and those individual’s ACT/SAT scores, the correlations were not perfect. In addition, correlations were weaker across racial/ethnic minorities and low-income students. NAEP scores have also shown positive correlation with post-secondary outcomes, although they were not the only factor. Finally, since the NAEP assessment began in 1990, the disparity in scores based on racial and socioeconomic differentiation has significantly diminished. This suggests the NAEP assessment may actually be better at determining the student’s capability, rather then just predicting their post-secondary success, while also having some ability to predict success. Yet, NAEP assessments are not used in any way to actually grade the student’s performance. At the very least, NAEP results may be a viable way to augment current admissions and similarly reduce the racial and socioeconomic disparity. They may also be a better way to measure “success” in the primary/secondary educational system than current methods, leading me to my next point.

The reality is that well-constructed, standardized assessments with proven validity and reliability are NOT how most of our students are evaluated today. Across the primary, secondary, post-secondary, and graduate levels, our students are routinely evaluated based on individual teacher developed assessments and/or subjective performance criteria. Those teachers are inadequately trained in how to design, make, and validate psychometrically sound assessments (with validity and reliability); and, as such, the instruments used to gauge student performance routinely do not meassure that performance. Without properly constructed assessments, our students are more likely to be measured on their English proficiency, cultural background, or simply whether they can decipher what the instructor was trying to say, rather than the knowledge they have about the topic. Subjective evaluations (like those used for essay responses) are routinely shown to be biased and rarely give credence to novel or innovative thought; even professional evaluators trained to remove bias, like those used in college admissions, routinely make systematic errors in evaluation.  Subjective assessments, in my personal and professional opinion, are fraught with inequity and bias that cannot be effectively eliminated. Furthermore, I can personally report that educational systems do not care, if the reaction to my numerous criticisms is any indication.  Standardized testing would address this issue and, as we’ve seen with the NAEP, likely do a much better job of making more equitable and fair performance assessments across students.

On top of that, our students’ performance is also often judged on things like home work, attendance, and work-products created in the process of learning, rather than on what they have learned or know. This misses the point, and likely exacerbates the disparity in “success” in our educational system. Single-parent and low-economic homes, which also tend to be more racial segmented, can have dramatic effects on these types of assessments. First, you are out-sourcing the learning to an environment you cannot control where some students may gain experiential knowledge growth, but others cannot; second, you compound that by further penalizing those who cannot with poor grades. While some students/parents (regardless of situation) may still engage in learning/experience outside the classroom, making it mandatory and grading on it likely contributes to the disparity giving those students in the best situations with an unfair advantage. Finally, from my own research into the development of expertise, I know that not all students require as much experiential learning to master the knowledge. The development of expert knowledge is idiosyncratic, some require more while some require less. As such, we should not be measuring performance on how the knowledge is obtained, and focus more on whether they have it or not. 

I know the legacy of mistrust will make this a hard stance for people to support, but the use of standardized testing for assessing student performance would address a number of significant issues in current practices. It can be less biased, provide more consistent results across schools, and if used in place of subjective or other non-performance criteria, be a more accurate reflection of student capability.

Conclusion

Standardized testing, especially the behind-the-scenes work done to properly create them, is a mystery to most people. When you add historical misuse and abuse of standardized testing, it is easy to see why many demonize them and question the results. The reality though, is that well constructed assessments, used properly, can not only help us uncover issues in society, but also help us address those issues. The data on SAT/ACT scores, both their ability to predict academic performance, as well as the disparity in scores across racial and socioeconomic background are a clear signal to the real problem: the racial and socioeconomic bias built into the education system. The education systems definition of “success”, or how it is determined is clearly biased. As such, we should not push to eliminate standardized testing, but look to see how we can improve our definition and measurement of success by doubling down on standardized testing instead of how we do it today.

Taking it on the Road?

So, you want to embrace the nomadic lifestyle, but haven’t figured out how to keep it digital? It would seem in today’s day and age, remaining connected while on the road or in the campground, should be an easy thing to do, but I have found that this is not the case. I have found three potential solutions: public WiFi, Satellite, and Cellular hot spots. Each has it’s merits, but they also come with some downsides.

In addition, we identified a few key criteria or use cases we needed to solve:

  1. Sufficient bandwidth with low-latency for making real-time video calls (ala Zoom or other). Ideally, you need the ability to connect just like you would at home or in the office without anyone suspecting that you are nestled by a secluded lake instead;
  2. The ability to connect while in motion. Obviously, when you are a nomad, you are going to be moving around and while you might not be doing a lot of typing or video calls while you are driving, if you are traveling with a partner, they should be able to work like they are in the office, even when the office is moving down the highway.
  3. Reliability and Simplicity. The idea of being a digital nomad, for many at least, is a less stressful way of living. Those hopes can be dashed if your solution is unreliable and/or difficult to maintain.

Public WiFi

It turns out that you can access public WiFi in many places, including truck stops, campgrounds, and the (still) occasional, non-secured WiFi access point found around town. The benefit of this method of connecting is that it is generally free, or low cost; and that’s where the benefits end in my opinion.

The downsides of public WiFi are nearly innumerable; however, besides questionable/inconsistent reliability and speed, I find the biggest challenge is re-configuring all of your devices to leverage the public WiFi. This can be made a bit easier if you have a WiFi bridging device like that found in many modern travel trailers. This allows you to have a consistent “internal” WiFi network that you connect all of your devices to, and you simply have to configure the bridge to connect to the public WiFi you want to use.  This works, well, should work. I had challenges with the one in our trailer that stopped us cold and the bridge itself is only as useful as the networks you attach to. In addition, those dreaded “splash pages” that many public WiFi networks use to authenticate users can be tedious to deal with. 

Lastly, the need to re-configure everything, even if it is just your WiFi network bridge, is exacerbated if you are truly mobile; i.e., if you are driving down the road. Public WiFi is not a mobile solution, but a solution for when you are not mobile. So, on our three use-cases, public WiFi fails #2 entirely, and scores questionably on the remaining two.

Satellite Internet

Satellite internet comes in a couple of flavors, high, geosynchronous orbit and low-orbit; e.g. Dish Network versus Starlink. Personally, I’ve never been a fan of the geosynchronous satellite internet, mainly for its generally limited bandwidth and its latency. While bandwidth has improved, latency is what it is … you cannot speed up the signal. In the past this was just a problem if you were a gamer, but as a digital nomad, it will also affect all your Zoom calls. Going mobile also has its limits as you need a motorized dish that can track the satellite if you want to try using it in motion, which significantly increases the start-up costs. Lastly, as anyone who has satellite TV will know, trees and other obstacles can prevent the “clear view of the sky” that is required to get consistent, reliable service. This is hard enough in a stationary environment, but when you are on the road, living in campgrounds (which often have … trees), it is less than desirable.

Low-orbit satellite, a.k.a. Starlink, does dramatically improve on the geosynchronous option when it comes to speed and latency. These satellites are simply greater in number, and much closer than their geosynchronous cousins providing low-latency and high bandwidth. That being said, you still have challenges when it comes to the “clear view of the sky”. In the standard configuration, with a free-standing antennae you do have a bit more flexibility to move the antennae out from under the tree in the campground, but you also sacrifice the ability to connect while in motion. Starlink does offer a configuration that can be mounted on your vehicle that allows for “in-motion” use, but it is not only significantly more expensive (both in equipment and in monthly fees), it also makes it less flexible when trying to get a clear view of the night sky. 

If we look back at our three use cases, geosynchronous satellite doesn’t quite meet our first one, could meet the second one at additional cost, and only scores half on the simplicity/reliability aspect. Low-orbit only improves by also meeting the first use case, but doesn’t fair much better than the other with the same caveats.

Cellular Hot-Spots

The last option is the good-old cellular hot-spot, which is a solution that we’ve relied upon many times in our preliminary travels. Most people are somewhat familiar with this approach by using their mobile handset as an emergency internet connection for their laptop and/or tablet when nothing else was available; most people are also, then, familiar with many of the challenges including poor signal, usage limits, additional monthly charges, and limits on the number of devices that can connect. While stand-alone hot-spots (either pocket devices or those installed in mobile platforms like cars, trucks, and trailers) do address issues with limits on the number of devices, most providers still severely limit the monthly throughput and you still have the same issues with signal quality as you travel around. 

On the other hand, cellular hot-spots do provide true “mobile” usage as long as you stay on major highways where the signal is consistent (if you don’t drive over 70 miles an hour, which is apparently about the speed at which you cross tower signals too quickly for proper hand-off).  So, cellular generally meets all three of our use cases, with a bit of variability.

An Alternative Solution

After going through the above exercise, I will admit that I was a bit concerned about how we could commit to a nomadic lifestyle while maintaining our digital connections. While cellular seemed to be the best option, our past experiences using hot-spots did not instill a lot of confidence in relying on them for our mobile connectivity needs. I was beginning to think that we would need some combination of low-orbit satellite combined with cellular to be certain we could connect, but didn’t like the idea of having multiple services (and multiple bills), nor did I look forward to doubling the initial start-up costs. That is when I discovered Nomad Internet.

  • Unlimited Data
  • Up-to 200Mbs (currently)
  • No Throttling
  • Nationwide Coverage
  • No Contracts
  • Pause/Restart Service Anytime
  • 7-Day Money Back Guarantee

Nomad Internet is, to be fair, essentially a dedicated cellular hot-spot just like many others. Where it is different, though, is where it counts. Nomad offers unlimited data (with no throttling), speeds currently up to 200Mbs (download), and not only eschews monthly contracts, but also offers a “mobile” option allowing you to pause/restart your service only when it is needed for a small up-charge. (Starlink also offers this on their “mobile” option). Nomad also works on the newer 5G “C-Bands”, which not only allows for higher speeds, but also provides good signal reception even when mobile phone service is dicey.

These changes were enough for me to buy my first Nomad router and put it through the paces. I installed an AC inverter in my truck and for a few weeks, we drove around with a Nomad WiFi network surrounding us at all times. We drove to the backwoods of northern WI where my partner’s mother lives and where both my partner (on Verizon) and I (on AT&T) often have difficulty getting solid cellular reception and our data connections have been slow as molasses. In all these cases, we got solid connectivity, without disruption while driving or at rest.

Out of curiosity, I did a speed check at my partner’s mother’s house. First, I checked her hardwired broadband connection (4Mbs down/.2 up). Then I checked the Nomad connection (40Mbs down/1Mbs up). For Christmas, we replaced her broadband with my original Nomad (this time registering 62Mbs down/ 1.6Mbx up), leaving her with a monthly bill 33% lower than she had before. It’s no wonder Nomad’s original business plan was to provide rural areas with better Internet; here, they delivered exceptionally well. 

Final Thoughts

All-in-all, so far, Nomad has addressed all three of our initial use cases with flying colors. Add that my new Nomad modem came with a 7-day back-up battery, all at less than many others charge for their devices alone, and I’m feeling a lot less anxious about maintaining our digital connections as we explore a more nomadic lifestyle. That’s not to say I am completely at ease and this coming spring/summer we’ll be putting it through its paces in some of those more out-of-the-way places (like Yellowstone and Glacier National Parks), utilizing that back-up battery to see just how far off the beaten track we can go. But, so far, it has shown to be a reliable and simple solution that provides sufficient bandwidth to provide the ability to work on the road, in-motion and at rest.