Also against individual IQ worries
Scott Alexander recently blogged “Against Individual IQ Worries.” Apparently, he gets many readers writing to him terrified that they scored too low on an IQ test, and therefore they’ll never be able to pursue their chosen career, or be a full-fledged intellectual or member of the rationalist community or whatever. Amusingly, other Scott says, some of these readers have even performed their own detailed Bayesian analysis of what it might mean that their IQ score is only 90, cogently weighing the arguments and counterarguments while deploying the full vocabulary of statistical research. It somehow reminds me of the joke about the talking dog, who frets to his owner that he doesn’t think he’s articulate enough to justify all the media attention he’s getting.
I’ve long had mixed feelings about the entire concept of IQ.
On the one hand, I know all the studies that show that IQ is highly heritable, that it’s predictive of all sorts of life outcomes, etc. etc. I’m also aware of the practical benefits of IQ research, many of which put anti-IQ leftists into an uncomfortable position: for example, the world might never have understood the risks of lead poisoning without studies showing how they depressed IQ. And as for the thousands of writers who dismiss the concept of IQ in favor of grit, multiple intelligences, emotional intelligence, or whatever else is the flavor of the week … well, I can fully agree about the importance of the latter qualities, but can’t go along with many of those writers’ barely-concealed impulse to lower the social status of STEM nerds even further, or to enforce a world where the things nerds are good at don’t matter.
On the other hand … well, have you actually looked at an IQ test? To anyone with a scientific or mathematical bent, the tests are vaguely horrifying. “Which of these pictures is unlike the others?” “What number comes next in the sequence?” Question after question that could have multiple defensible valid answers, but only one that “counts”—and that, therefore, mostly tests the social skill of reverse-engineering what the test-writer had in mind. As a teacher, I’d be embarrassed to put such questions on an exam.
I sometimes get asked what my IQ is. The truth is that, as far as I know, I was given one official IQ test, when I was four years old, and my score was about 106. The tester earnestly explained to my parents that, while I scored off the chart on some subtests, I completely bombed others, and averaging yielded 106. As a representative example of what I got wrong, the tester offered my parents the following:
Tester: “Suppose you came home, and you saw smoke coming out of your neighbor’s roof. What would you do?”
Me: “Probably nothing, because it’s just the chimney, and they have a fire in their fireplace.”
Tester: “OK, but suppose it wasn’t the chimney.”
Me: “Well then, I’d either call for help or not, depending on how much I liked my neighbor…”
Apparently, my parents later consulted other psychologists who were of the opinion that my IQ was higher. But the point remains: if IQ is defined as your score on a professionally administered IQ test, then mine is about 106.
Richard Feynman famously scored only 124 on a childhood IQ test—above average, but below the cutoff for most schools’ “gifted and talented” programs. After he won the Nobel Prize in Physics, he reportedly said that the prize itself was no big deal; what he was really proud of was to have received one despite a merely 124 IQ. If so, then it seems to me that I can feel equally proud, to have completed a computer science PhD at age 22, become a tenured MIT professor, etc. etc. despite a much lower IQ even than Feynman’s.
But seriously: how do we explain Feynman’s score? Well, when you read IQ enthusiasts, you find what they really love is not IQ itself, but rather “g“, a statistical construct derived via factor analysis: something that positively correlates with just about every measurable intellectual ability, but that isn’t itself directly measurable (at least, not by any test yet devised). An IQ test is merely one particular instrument that happens to correlate well with g—not necessarily the best one for all purposes. The SAT also correlates with g—indeed, almost as well as IQ tests themselves do, despite the idea (or pretense?) that the SAT measures “acquired knowledge.” These correlations are important, but they allow for numerous and massive outliers.
So, not for the first time, I find myself in complete agreement with Scott Alexander, when he advises people to stop worrying. We can uphold every statistical study that’s ever been done correlating IQ with other variables, while still affirming that fretting about your own low IQ score is almost as silly as fretting that you must be dumb because your bookshelf is too empty (a measurable variable that also presumably correlates with g).
More to the point: if you want to know, let’s say, whether you can succeed as a physicist, then surely the best way to find out is to start studying physics and see how well you do. That will give you a much more accurate signal than a gross consumer index like IQ will—and conditioned on that signal, I’m guessing that your IQ score will provide almost zero additional information. (Though then again, what would a guy with a 106 IQ know about such things?)
Follow
Comment #1 October 1st, 2017 at 1:50 pm
I agree with Scott x 2 on this one. But this is all quite complicated. If you have an easy test that gives you good prediction of “success in life”, it is all so tempting to regard the value of the test as the definition of “success in life.” (This is being done in some circles explicitly or implicitly about richness.) And even if you don’t worry too much about your IQ score (or similar measures) others who need to evaluate, admit, or promote you may. (Which is fairly rational I suppose.)
I have a friend who at age 8 decided not to learn the multiplication table from reasons of principle (that I don’t remember). Subsequently he was considered very weak and his parents were strongly advised by the headmaster against any direction requiring intellectual skills. His parents were a little skeptical also because he liked to perform all sort of experiments and in one of his experiments he even caused loss of electricity to half the city and they managed to get him accepted to a good school were he was considered very slow and did not have to do anything. Then he got some scientific prize and all of a sudden his status changed from “slow” to “genius” which meant that he still did not have to do anything. (And later he became an excellent scientist. And at some point even confronted with that headmaster.)
Still while this is an amusing story, I dont know what we can learn from it, or from the stories in the post.
Comment #2 October 1st, 2017 at 2:41 pm
Calvin (of Calvin & Hobbes) said it best:
“You know how Einstein got bad grades as a kid? Well, mine are even worse!” ( https://i.pinimg.com/564x/48/12/62/48126266d2984602917bc8dd6880fc53–calvin-and-hobbes-comic-strip-avi.jpg )
I’m sure intelligence comes and goes over time. I’m definitely even stupider now than I was when I was younger. It’s not a monotone decline though: it’s not exactly fluid but there are ups and downs. Andre Weil wrote something about that too iirc.
Comment #3 October 1st, 2017 at 2:46 pm
IQ tests are structured tests. Life is unstructured. Success in real life is the real IQ test.
Comment #4 October 1st, 2017 at 4:52 pm
The value of g does have an impact, just not as big an impact as people who have a higher g would like it to have. There are other qualities that probably have a higher correlation with life success than an IQ score.
The analogy I like to use is that using an IQ test to measure intelligence is like using a vertical leap test to measure someone’s height. It is correlated, for sure, but not at all definitive. We can’t really pinpoint, let alone measure, any physiological variables that correlate with it. Like, at all.
So I endorse your program. If you want to know if you’d be good at doing theoretical computer science, start doing some theoretical computer science.
Comment #5 October 1st, 2017 at 5:01 pm
A corollary is that no particular class grade is a perfect indicator of future success. I’m speaking from experience as one who got a D in his first college math course (probably due to stress and having a part-time job cutting away study time) and went on to finish a PhD in math.
Comment #6 October 1st, 2017 at 5:51 pm
I don’t really get it? Childhood IQ tests have very high error compared to those given slightly later in life. This is why selective primary school gifted programs have very mediocre outcomes compared to high school gifted programs.
Comment #7 October 1st, 2017 at 6:33 pm
“IQ tests are structured tests. Life is unstructured. Success in real life is the real IQ test.”
Since success on IQ tests is correlated with life success by many metrics, what is your point?
Comment #8 October 1st, 2017 at 8:02 pm
Scott,
There are several ways that the whole concept of “IQ” can be challenged. The most obvious, from a mathematical point of view, is that it tries to establish a total ordering on humans, whom we all agree are multi-dimensional entities. If the complex numbers cannot be ordered, imagine us, human beings. We would have to be mono-dimensional for “IQ” to even make sense as a measure of anything meaningful.
Then there is the ironies, of ironies: https://alumni-esc.stanford.edu/get/page/magazine/article/?article_id=40678 . Lewis Terman, one of the inventors of IQ-testing as we understand them today, sought to prove the utility of the test as it came to predict high achievement by testing the children in the Palo Alto area and following them up in the upcoming decades. The study is still being carried out. Not only the test produced no Nobel Prize winners, but from the above “we also know that two children who were tested but didn’t make the cut — William Shockley and Luis Alvarez — went on to win the Nobel Prize in Physics”. Shockley later in life became obsessed with race and IQ. I wonder if he not being identified as “high IQ” had anything to do with his obsession, in spite of having won the Nobel Prize in physics.
As with many things that psychology has produced over the years, IQ tests have served only to create harm. Reducing human beings to a number (any number and for any reason, such as “personality”) is the prime example of “information destroying” projection that should be soundly rejected as a measure of anything other than how psychologists -who happen to be those with the lowest grades in every college class- have managed to convince their smarter classmates to care about the junk they -psychologists- produce.
Comment #9 October 1st, 2017 at 8:22 pm
Nearly every measurable outcome we care about is correlated with IQ. These correlations typically range from 0.1 – 0.4. The highest correlations are with various measures of educational achievement, which can get as high as 0.6 or so.
In other words, for most outcomes we’re talking about 1 – 16% of the variance explained by IQ, and education gets up to maybe 1/3 or so. Maybe a little higher.
These are correlations; they say nothing about causality. And further, these are unconditional correlations, whereas an individual cares about the conditional correlations. Realistically, an IQ conveys little marginal information above what can be inferred from one’s own experience and academic record.
But still, given the complexity of life, and the complexity of the human brain, and the nonlinear interactions between the components of each and with each other, it is a wonder that a single number from a short test can explain so much. No other measure that I know of comes close.
But this wonder should not blind us to just how little it explains in the end.
Comment #10 October 1st, 2017 at 9:20 pm
This is the first I have heard of “g”. I might want to appropriate it to start a punk rock band: “Big g and the G – spots”.
Comment #11 October 1st, 2017 at 9:29 pm
Note to editor: I am changing the band name already. “High g and the G – Spots” has a better ring.
Comment #12 October 1st, 2017 at 11:21 pm
@asdf FYI, Einstein having bad grades is a myth.
Comment #13 October 1st, 2017 at 11:40 pm
I was “slightly mentally retarded” at age 3. That didn’t stop me doing a PhD at Harvard.
A good reminder that correlation != causation.
Comment #14 October 1st, 2017 at 11:57 pm
I’ve come to the conclusion that there’s actually two types of intelligence, and IQ really only captures one type.
The IQ dimension is in my view capturing what Robin Hanson calls ‘near-mode’ thinking skills…ability to grasp details, carry through precise logical inferences, and raw mathematical ability.
The second dimension is (in the terminology of Hanson), ‘far-mode’ thinking skills, or creative thinking skills. This is the ability to capture the ‘big picture’ , integrate many concepts into a coherent whole (‘super-clicking’) and ability to think outside-the-box (generation of original angles).
What is called genius is most probably a combination of raw IQ *and* creative ‘super-clicker’ intuition.
Someone can lack raw IQ but make up for it with great ‘super-clicker’ abilities. And vice versa, someone may not have great ‘super-clicker’ intuition, but instead have fantastic math skills (raw IQ).
Comment #15 October 1st, 2017 at 11:58 pm
#8: Well, no one has any problem with a linear ordering on humans if we’re talking about how fast they can run 100m, or how many hot dogs they can eat in a hot dog eating contest. And score on an IQ test, for whatever it’s worth, is a better predictor of economic success in the modern world than either of those, unless perhaps we’re talking about the extremes (Usain Bolt, Takeru Kobayashi). The problem, we might say, is not with the various linear orderings themselves, but with taking any of them as a measure of human worth.
Comment #16 October 2nd, 2017 at 12:10 am
Domotor #12: Yes, Einstein generally did very well in school—especially in math and physics, although not in French. What’s true is that he had many conflicts with teachers he considered authoritarian, and also, that one would never have predicted based on his school performance that he’d be Einstein.
Comment #17 October 2nd, 2017 at 1:51 am
Scot #15,
“The problem, we might say, is not with the various linear orderings themselves, but with taking any of them as a measure of human worth.”
My critique goes beyond that. It is not only a matter of using IQ as a way to measure human worth, it is that for all we know, there are so many variables that influence both a person’s prospects as well as how that person does in IQ tests, that it is a utterly worthless measure. In other words, a true measure of intelligence would need to rank people according to points of a multi-dimensional field, something we know is impossible to do, mathematically. Thus, the most metrics like IQ can do is to tell us something about a single, and narrow, dimension of the human experience. As I said, this is something that only the dumbest students of the class -those who major in psychology- could have produced to torment everybody else.
Let me illustrate what I want to say with a specific example derived from my own personal experience. I went to graduate school to a very prestigious university, but I did my undergraduate in an average school that didn’t require SAT, ACT or any other form of standardized testing. I have never done well in standard tests for the reasons you outline above. Take the typical question “what number comes next in the sequence?”. Anybody who knows anything about mathematics knows that for any given finite sequence, you can always come up with a rationale that is mathematically correct that would allow you to pick any of the offered options. I wanted badly to go to a good graduate school, so submitting GRE scores was not optional this time. In the first test test I took I scored poorly (as I expected). I then bought every preparation guide I could find (Kaplan, Barrons, Princeton Review, you name it). After several months of heavy preparation, I trained my brain to think the way the people who designed the GRE wanted me to think. I aced the test and together with good grades from my less known school and good recommendation letters I got into two top notch schools. After I got in, I went back to my normal way of inquisitive thinking, and that served me better in graduate school than the type of thinking I had to develop to ace the GRE.
What this whole episode showed to me is sure, if you come from a well to do background, with parents aware that investing in their children’s education is the best investment on can do on said children, it is no wonder that these children will do well in life and probably score high in standardized tests they have been coached to do well in. But this is a classic example of correlation, not causation. For all we know, IQ tests might just be a proxy to measure privilege or of parents invested in their children’s education.
Anybody who is distanced enough from his/her high school or college years knows that predicting who will be “better off” 20 years from graduation is a very difficult thing to do. To begin with because “better off” remains a very subjective metric, not an objective one. Take Google for example. It seems to have an addition to Stanford PhD dropped outs, first with its founders, now with its CEO. I am sure that many of Page, Brin and Pichai’s Stanford classmates did better than them using traditional metrics to measure “academic achievement” such as professorships at prestigious universities, maybe awards (both Page and Brin are member of the NAE, so that is less sure). While it is impossible to be sure of these things, I have no doubt that many of the more “academically bright” classmates these three had in graduate schools would trade places with being founders and CEO of a company as successful as Google. Even narrowing down the exercise to “traditional measures of success” such as a awards, etc, I challenge everyone reading this to say they were able to predict who, among their colleges classmates and based on their GPAs alone, was the one better off, 10, 20 or 30 years down the road.
If psychologists want to be in the business of prediction, perhaps they should work on the unification of quantum physics with general relativity and stop tormenting generations of students with this garbage called “IQ”.
Comment #18 October 2nd, 2017 at 2:25 am
“success on IQ tests is correlated with life success by many metrics”
It’s a fact that certain professions require some min IQ, but I define “success” as e.g. making Google, not working for Google.
In this context what is success? Being Feynman with an IQ of 124 or an average post-post-post doc with a childhood IQ of 160?
Or what is e.g. Bill Gates IQ and what is the median IQ of his current high rank employees (which I suppose is greater)?
Comment #19 October 2nd, 2017 at 3:39 am
Jair #5: Yes, but that corollary is much easier. Class grades are not a good indicator because they don’t compare properly among different schools or different teachers, and, in fact, schools were sometimes pushed to inflate them until the college admissions stopped looking at them. As opposed to this, IQ tests are designed to be proper experiments to measure you more or less the same no matter where you take the test. For that reason, IQ tests are a good predictor, but class grades aren’t.
Comment #20 October 2nd, 2017 at 4:32 am
ttt #18: I have two problems with your view.
First, “success” only means becoming Richard Feynman or Bill Gates or Larry Page, and everything else is failure? That sounds like a terrible way to live, and more rigidly elitist than the views of the world’s worst IQ determinist! (Also, are you a success by that standard?)
Second, Bill Gates had a 1590 SAT score. If you’re willing to count the SAT as a disguised IQ test, that would put him comfortably in the 150-170 range (and me too! I’d no longer have to settle for 106 🙂 ). Similarly, I don’t know Feynman’s SAT score, but one could argue that he was clearly, unequivocally one of the smartest people of his generation, not merely “in a broad sense” but in as narrow and easy-to-measure a sense as you like (doing integrals in his head, etc). So if a particular test failed to capture that, then that’s a reason to throw out the test and find a better one.
Comment #21 October 2nd, 2017 at 4:59 am
First I had to laugh after reading this post. As an adult I stopped worrying about IQ since few years ago. That time I took one online IQ test and got an IQ of 86. The test was offered by a free online website, followed by requests to buy the complete test, which were ignored. The test had 20 questions asking such things as – what figure best follows after the following? Or the like what Scot mentioned. And I simply bombed through. That time I was seriously depressed after I got my first rejection of a paper I submitted to a journal hoping to fulfill some requirements to get a PhD at age 36. I was all game (for losing) and trying to see how much it would hurt if I got another disappointment, and sure as hell it did not. After that, I simply did not care. Later I did some online research and found out what Anti-IQ camp had to say about things as emotional-Intelligence, blah blah blah, that it is not important to have high IQ, then I knew and secretly prophesied that to want to find out how good one is at something, then one has to try it. Full stop. Since then, my motto has been to try to learn something new if it fancies me. That made me buy the QCSD-by Scot A and read it in full. I understood some things and did not bother to solve any problem posed as a challenge or exercise. I also bought The Beat of a Different Drum and read it until Chapter 2. I got completely lost and shelved the book. Then I bought Power, Sex, Suicide:-Mitochondria blah blah. I understood great part of this book and used its arguments in several discussions I had with my Doctor friends. They were amazed how I knew so much biology-things while I had shucked it since high school in the 1990s. Too bad, among my friends circle there is no one who is Complex-savvy so I could measure how much I understood the QCSD, Shtetl, and other blogs about Computer Science.
However, I received the PhD nearing 40 with a strong belief that I can learn something new each day. I set goals to wri
Comment #22 October 2nd, 2017 at 5:05 am
“Problem with Gatekeepers”: You argue that IQ, and every other attempted linear measure of intelligence, can safely be ignored, partly on the ground that psychologists “happen to have the lowest grades in every college class.” But in calling psychologists the bottom of the barrel, are you not implicitly appealing to a linear measure of intelligence (or at least academic aptitude) yourself, and is this not a fundamental tension in your view?
Comment #23 October 2nd, 2017 at 7:23 am
It reminds me of BMI, in that it seems useful as a statistical measure for populations, but awful at trying to make individual scale recommendations.
Comment #24 October 2nd, 2017 at 7:49 am
@Scott #20
Broadly speaking “success” has philosophical meaning. Success for a missionary in Africa might be building as many schools as possible and make local communities healthy. For an ordinary woman/man might be to raise a “good” family. For a drug lord to become a monopoly in his area etc.
IQ measurement is helpful to determine if you have the necessary mental capacity to perform a certain task. E.g you might not become a good brain surgeon if you have less than ~110 IQ.
My definition of “success” refers to the “top percentile”. In that respect IQ is irrelevant. Ultimate success does not depend linearly or nonlinearly with IQ, but rather depends on work, persistence and luck. IQ is just a required threshold to succeed.
There’s no point on extrapolating Gates or Feynman’s or Perelman’s IQ.
PS
I always had this question about psychology: how can somebody with an IQ of lets say around 120-140 can devise a test that will correctly measure the IQ of a person with an IQ of >200?
Comment #25 October 2nd, 2017 at 8:42 am
tttt #24:
I always had this question about psychology: how can somebody with an IQ of lets say around 120-140 can devise a test that will correctly measure the IQ of a person with an IQ of >200?
There might be practical problems doing that, but I don’t see any problem of principle. I can pretty easily write a math test that I couldn’t pass in an hour, except for the accident of having written it, but that a better mathematician could pass and that I could grade. As an analogy, a Commodore-64 can easily generate a challenge—say, “factor this 1000-bit number”—that a supercomputer can solve and whose solution the Commodore can easily check, even though the Commodore itself couldn’t have solved it. Indeed in some sense, P≠NP is precisely the statement that such things should be possible… 🙂
Comment #26 October 2nd, 2017 at 8:43 am
[…] Also Against Individual IQ Worries by Scott Aaronson – IQ tests tend to ask unclear questions and require you to reverse engineer what the test maker meant. Scott’s own IQ was once measured at 106. […]
Comment #27 October 2nd, 2017 at 10:20 am
>> dumb because your bookshelf is too empty
So I should not use Kindle ?
Comment #28 October 2nd, 2017 at 11:27 am
Scott #22
I do think that there are people that are more talented than others. In other words, I am not defending the notion that all human beings are equal in terms of talent and ability. I do believe that we are equal in terms of “worth” and I believe that one of the geniuses of the American system was the creation of the notion of “citizenship” as a concept of equality before the law -as opposed to the hierarchical system that existed in Europe that had kings and queens at the top, and common people at the button. The notion of equal value of all humans is one of the most noble aspirational values of the United States even if in practice, the country has not always lived up to this ideal -starting with the obvious issue of slavery and Jim Crow.
This said, it would be foolish to pretend that all humans are capable of the same feats. But there is also the fact that as “tttt” says in #20, “success” is a very subjective thing. I take a lot of issue with defining “success” as the ability of scoring high on a multiple choice test. Second, “IQ” tests say nothing about actual achievement, only about what psychologists believe to be “potential”. IQ tests are, if you will, “psychotic” in the sense that the play in the realm of “potential” instead of “actual” achievement. I personally have no respect whatsoever for people who score high in IQ tests if that’s the only thing that they have to show off. I have a lot of respect for people who have achieved great things in life. Every single high achiever whole life I have studied in detail, has had a life full of trials and tribulations. John Von Neumann? The story goes he came to the US convinced that he would be unable to find a job in his native Hungary. Einstein? Behind the fable of how general relativity was easily tested through an eclipse, there is a life of being considered a “trouble maker” , “no good”, that Hilbert reached the final general relativity formulation before Einstein did -although Hilbert was gracious enough not to get into a priority dispute-, and that the empirical verification of general relativity ran into a lot of problems -including the outbreak of World War I- before it was finally successfully tested in 1919.
On the other hand, I have little respect for people like Marilyn vos Savant whose only claim is “dude, I tested high in this IQ test”. It takes a lot more than that to impress me.
If history is any guide, the geniuses of the future will be the people who “escape” measurement using our contemporary metrics of “genius”. That has been a constant in human history. The obsession with “potential” will be detrimental, in the long run- for human civilization.
tttt #24:
I agree that one of my biggest beefs with IQ tests is that their biggest defenders are generally the losers in high stakes academia. Since they -their defenders- have generally achieved little of significance, they torment others with their psychotic invention. One of the life long conclusions I have reached in general is that there is a very insidious fact of life: the decisions that are of utmost important for us, as individuals, are usually made by people who are generally the most stupid (using some sort of “non linear” measure of stupidity). The importance our society places in things like IQ tests, SAT tests, GRE tests and the like is an example of this phenomenon but there are others.
Comment #29 October 2nd, 2017 at 11:34 am
Scott #25
I think that tttt #24 refers more to the issue that if IQ were measuring anything of significance, you would need people of a higher IQ telling us how to measure IQ. To put an analogy with AI: one of the arguments against strong AI and the “emergence theory” of computing is that the godel statement is something our minds can see as true because something our minds have -more correct would be to say that Godel’s mind had when he constructed Godel’s encoding to come up with “this statement cannot be proved”- escapes Turing computation. In other words, the ability of Godel to design such an encoding and of coming up with such a sentence escapes “mechanical computation”, irrespective of how many cores you have doing parallel computation -since after all all parallel computation can be reduced to serial computation via a small enough computing interval- or how much memory you have to store intermediate results. Coming up with “Godel encoding” and “Godel statements” is exclusive to minds, not to “computing”.
Using this analogy, being “faster” at doing some well defined standard tests is not necessarily a measure of anything other than “being faster”.
Comment #30 October 2nd, 2017 at 11:58 am
“On the other hand … well, have you actually looked at an IQ test? To anyone with a scientific or mathematical bent, the tests are vaguely horrifying. “Which of these pictures is unlike the others?” “What number comes next in the sequence?” Question after question that could have multiple defensible valid answers, but only one that “counts”—and that, therefore, mostly tests the social skill of reverse-engineering what the test-writer had in mind. As a teacher, I’d be embarrassed to put such questions on an exam.”
Scott… you’ve captured my sentiments exactly… and the sentiments I had when I wrote such tests in grade 3. This is also an issue that occurs in high school and University exams. The “best” test takers learn the “reverse engineering skills” to guess what the professor or teacher had in mind.
The challenge is when they say something unclearly, and you “think” they meant something else, but they actually “meant” a different thing. But actually what they described was incoherent.
Comment #31 October 2nd, 2017 at 12:33 pm
I find the IQ test being one number to already be a problem.
consider:
If Alice gets an 800-math and 500-Verbal on her SAT
Bob gets a 650-math, 650-verbal on his SAT
Carol gets a 500-math and 800-verbal on her SAT
They all got a 1300 but what its telling you is very very different.
If (1) it was a better test, and (2) it gave you a set of numbers for
different skilss, then the IQ test might have some value.
But even then- within (say) math there are good problem-solvers and good problem-makers. Perhaps the IQ test can tell you if you would be good at making up IQ tests.
Comment #32 October 2nd, 2017 at 12:34 pm
#28: I’m in enthusiastic agreement that the real question is not how well people score on a test; rather, it’s what they can create or discover or otherwise contribute to civilization.
Or more precisely: if someone’s only claim to fame is scoring 200 or whatever on some IQ test, then that’s “merely” as interesting as their winning at Jeopardy!, or breaking a bicycling record, or typing 100 words per minute with their toes. I.e., interesting only because it’s rare and difficult, not because of some independent value.
On the other hand, when colleges (for example) are choosing which students to admit, potential is pretty much all they have to go on. Even at the most elite colleges, what fraction of applicants have already published a serious research paper, started a serious company, or done anything else of that kind?
In those cases, I’ve previously argued that relying on SAT scores, while far from perfect, is less bad than any of the obvious alternatives, such as grades (completely different from one school and teacher to the next), or extracurriculars and “leadership” (things that function, in practice, mostly as proxies for affluence and social well-connectedness).
Comment #33 October 2nd, 2017 at 12:43 pm
Scott, this is just a brilliant piece of writing on every level. Thanks for writing it.
I agree with it completely, but also think that most people won’t even understand it. I’m not sure how to communicate it to regular people. But, I think that the people who do understand it should rest assured that there should be no doubt that they have a reasonably decent g-factor.
I think for them the fact that they even read your blog is probably a bigger piece of evidence for it than any IQ test ever could.
Comment #34 October 2nd, 2017 at 12:55 pm
@Scott: to counter the SAT scores argument, I can think of some problems with it. Suppose there is a person who is OBVIOUSLY capable, like has perfect marks on every course at a high school with a reputation for producing solid students. Suppose that person bombed the SAT? Does it make sense for this person to not be admitted because of this? In this case we actually DO have enough information to conclude they will obviously do well in University. The SAT is just a very noisy in this respect. Aren’t we just trading off one form of reliability (the standardization part of the SAT) for another (the fact that high school marks have lots of samples and so are less susceptible to random noise)?
By the way, another reason why schools should not use the SAT is that they are competing for students with Canadian schools which don’t require it. I suspect other international schools do not as well. The SAT is a lot of work to reasonably prepare for, is very stressful, and pointless if you can just go to a Canadian school. In fact, this is why I never wrote it.
At the graduate level, the GRE is also something like this. But I never wrote it because it was a waste of my time. Fortunately for me MIT at the time did not require it and I just went there.
Putting pointless barriers to going to a school in front of top students (who you want to attract) just makes them not bother applying in the first place.
Comment #35 October 2nd, 2017 at 1:10 pm
Not that the SAT doesn’t also track affluence.
Comment #36 October 2nd, 2017 at 1:12 pm
I think part of the problem might be that smartness can be self-defeating, in this way: smart people are good at coming up with reasons to challenge someone else’s solution to a problem and defend their own. This could explain in part why smart people don’t always give the expected answer on an IQ test, why the expected answer isn’t always the only right answer, and why those who do score high don’t always “succeed” (i.e., why the correlations aren’t higher). Although, there are lots of other reasons for the last case.
My IQ test in high school (long, long ago), or the parts I remember of it, consisted of being told longer and longer strings of words and having to repeat them backwards, and then a series of problems of the form, “You have a 5-quart jar, an 8-quart jar, and a three-quart jar. How can you use them to get exactly four quarts?”, which were timed with a stop watch. (It was a one-on-one test, myself and a professional examiner.)
For a written test, my money would be on the SAT math test, except that it does require some background – you need to have studied math and worked with math previously. Given that, I believe math is thinking, thinking is math, so the best at math are the best thinkers (and Feynman was a genius).
Of course the best thinkers don’t always win, in our society or any other one so far.
Comment #37 October 2nd, 2017 at 1:15 pm
Chris #34: Yes, if you have separate evidence that an applicant is really extraordinary—like, a company they started, major piece of software they wrote, adversity they overcame, etc.—I said in my previous post that my ideal university would just ignore test scores and admit the person. Test scores are simply what you’d fall back on for the applicants who don’t have such extraordinary achievements, which by definition means the majority of applicants.
As for high-school grades, those are sufficiently arbitrary and unmonitored that teachers can use them just to reward students they like and punish ones they dislike (ask me how I know 🙂 ). Indeed, my ideal high school would relieve teachers of the burden of giving out grades entirely (making evaluation an outside entity’s job), so the teachers could focus on actually teaching and the students on learning.
Comment #38 October 2nd, 2017 at 1:26 pm
Doug #35: From what I understand, the SAT is somewhat coachable—like, taking a prep course that only richer kids have access to might give you ~100 points (though there’s also free prep available online). By contrast, it seems to me that many of the other factors considered in elite universities’ “holistic” admission processes, have evolved to be almost entirely proxies for affluence, conformity, and status-striving skill. (See the previous post for more.)
Comment #39 October 2nd, 2017 at 1:26 pm
Scott: maybe there’s just a difference between high schools in Canada and the USA. One thing is that we tend to pay our teachers a lot more, which just ends up attracting the highest quality of people. I can surely attest to that in my Ontario, Canada public education. In spite of all the flaws the teachers DO care about giving you a grade that is used to get you into University, so it does mean something. I think because in Canada it is KNOWN that grades define the credentialing system to get into Universities, that MAKES the high school teachers take them seriously. Otherwise, they will damage the credibility of the grades they give out and damage the ability of their good students to get into good schools.
You have not addressed the problem of US schools requiring the SAT as a requirement for applying (I think this is true, right?) If bad SAT marks are just going to be ignored if an applicant is excellent anyways, why make that applicant do the stressful SAT if they know they’d obviously be qualified to go to the school in question? You are just weeding out candidates who are totally qualified but don’t want to waste their time.
In grad school, MIT has already figured this out.
Comment #40 October 2nd, 2017 at 1:37 pm
Scott #32
“I’m in enthusiastic agreement that the real question is not how well people score on a test; rather, it’s what they can create or discover or otherwise contribute to civilization.”
Great! Believe or not, not everybody thinks this way.
As to standardized testing being the lesser evil, I tend to disagree. I read your post -and Steven Pinker’s New Republic essay- and while I do not disagree with the diagnosis, my own experience coaching myself to do well in the GRE disabused me from the notion that these tests are not coachable. The well to do, well connected and the children of academics would still have a “built-in” advantage if standardized testing were to be the main criteria to select applicants. From my own experience, I would say that the main remedy for the situation Pinker denounces is not to come up with a different game that the well to do would try to game but rather to expand the unique features of the American system of higher education that are completely absent in the rest of countries that I know (other than the American system I am familiar mostly with the systems of several European countries): one can start his academic career in his mid 20s at a community college and end up achieving great success both academically and economically as Craig Venter did. In other words: maximize access and opportunity at every step of the process.
In the long run, as Steven Pinker notes, the competition is not as severe as it seems. On a personal level, when I myself went to grad school a significant portion of admits had a profile like mine: we went to not so famous schools but we were very motivated in aiming high. And we did. So the question is more to create opportunity at each step along the way so that motivated people who had a less than ideal start in life can join the party to contribute. In that regard, fighting against things like “ageism”, perceptions that were you went to undergraduate school is your more important asset, and yes, how you scored at the age of 16 in the SAT is also you most important asset are the things I would prioritize to create a fairer environment.
The reality is that the well to do will always game the system no matter what. It is more about reaching out to the less well to do to tell them that the paths to success are many and that the limits lie more in themselves than in built-in obstacles of society. For that to be the case, higher education should work hard at expanding access, not limiting it. Creating complicated undergraduate admissions procedures that benefit those with the time and knowledge to get through them together with the impression that where you got your undergraduate degree is all that matters in life goes in the opposite direction of fairness. If you want a couple of countries that live under the illusion of “meritocracy”, study how the elite academic institutions in France and the United Kingdom select their students. Legacy admissions are a big factor there not for the reason they are in the US -ie, money- but rather because a big chunk of those admitted every year to their best schools happen to be the children of those who went to the same schools in previous generations. Their parents have learned to “game” a system that on the surface looks very meritocratic.
There are no easy solutions and I would say that the US is doing better when it comes to offering opportunities than other advanced countries. More standardized testing will bring us more misery, not less.
Comment #41 October 2nd, 2017 at 2:11 pm
This is the situation the US should try to avoid at all costs,
http://www.nytimes.com/2010/07/01/world/europe/01ecoles.html
“France imagines itself a country of “republican virtue,” a meritocracy run by a well-trained elite that emerges from a fiercely competitive educational system. At its apex are the grandes écoles, about 220 schools of varying specialties. And at the very top of this pyramid are a handful of famous institutions that accept a few thousand students a year among them, all of whom pass extremely competitive examinations to enter.
“In France, families celebrate acceptance at a grande école more than graduation itself,” said Richard Descoings, who runs the most liberal of them, the Institut d’Études Politiques de Paris, known as Sciences Po. “Once you pass the exam at 18 or 19, for the rest of your life, you belong.”
The result, critics say, is a self-perpetuating elite of the wealthy and white, who provide their own children the social skills, financial support and cultural knowledge to pass the entrance exams, known as the concours, which are normally taken after an extra two years of intensive study in expensive preparatory schools after high school.”
Comment #42 October 2nd, 2017 at 2:11 pm
Regarding the usage of SAT/IQ tests for admittance there are various issues (which are especially bad combined).
1) You can prepare to the tests. So the success expresses resources (time + money) spent for preparation. In particular it is biased towards the rich. (And preparations waste a lot of human resources on nonsense.)
2) In quite a few cases the fine tuning of selecting among the very high achievers of SAT and IQ is rather irrelevant to the actual intellectual skills required for the study. This is especially problematic if the study itself gives you large social advantage.
3) There are known systematic biases against various groups in the tests.
4) There are cases of very talented/original people that do bad in these tests. In other words, occasionally not doing so well in the tests is the outcome of features that are neutral to academic “promise” or even positively correlated to some forms of promise. [of course this is trivial in view of the actual correlation giving the predictive power.]
Having said all that I don’t know what should be done if anything to rely less on IQ/SAT tests. (Making lotteries among a larger pool of candidates might be an idea.)
Comment #43 October 2nd, 2017 at 2:44 pm
#41: But we’re already in that situation in the US, or worse! I.e., it’s already the case that getting into a place like Harvard, Yale, or Stanford is far harder and more important than graduating from them, and that the undergraduate admission process is heavily gamed by the elite.
If there’s a difference between the countries, as far as I can see, it’s that France can at least restrict admission at the Ecoles to those children of the elite who are able to pass a difficult test, whereas America’s admission barriers are porous to the … “well-rounded” children of the elite with hard-to-define … “leadership skills,” like George W. Bush or Jared Kushner. Could that have anything to do with why France currently has Emmanuel Macron whereas we’re stuck with Donald Trump? 🙂
Now, the fact that so many well-meaning, progressive Americans see the problem with university admissions, and then advocate “solutions” that would make it even worse—i.e., even less intellectual, even less meritocratic, even more gameable by status-striving elites—just makes me bang my head on the desk.
Comment #44 October 2nd, 2017 at 3:40 pm
Scott #43:
“Now, the fact that so many well-meaning, progressive Americans see the problem with university admissions, and then advocate “solutions” that would make it even worse—i.e., even less intellectual, even less meritocratic, even more gameable by status-striving elites—just makes me bang my head on the desk.”
This is an excellent point and one that frustrates me too. People keep jumping to the fact that standardized tests are gameable as if that means anything by itself. The question isn’t whether or not they’re gameable – OF COURSE they are – the question is if they’re more or less gameable than other criteria by which college students are filtered for acceptance/rejection.
It seems obvious to me that privileged rich people have massive advantages over less privileged poor people in things like pursuing community service, leadership positions at clubs and athletics, and other “soft” measures like extracurricular activities or arts and music. But I don’t know, maybe things like that which require lots of extra $, time, and energy outside of academic pursuits, with results subjectively judged by people who already have institutional power are actually less gameable by affluent families than a test is. I don’t think that’s the case, though.
Comment #45 October 2nd, 2017 at 3:53 pm
Scott #43,
Let’s leave politics aside, shall we :-).
Seriously, I see a big difference between the French system and the American system. I don’t deny that what you say is true for undergraduate admissions in the United States. Here is the difference though: undergraduate degrees only tell part of the story in the US.
Say for example your dream is to work for Goldman Sachs or McKinsey. In the US, if you miss the first round (ie, getting your undergraduate degree from one of the schools you mention), you can always go the path of getting to an OK undergraduate school, go to graduate school to a better school -like any of the ones you mention- and by the age of 30 in your Goldman Sachs or McKinsey job, nobody gives a freaking damn about where you did your undergraduate degree. From personal knowledge, in France, your undergraduate degree follows you -for better of for worse- anywhere you go in life, even if after graduation you lead a pretty mediocre career. You learned the rules of the game at age 25 because you were not born in the right family? Too bad, it is too late for you then. I think we are not in this situation by any means in the United States.
One way to see what I am talking about is to compare these two lists,
https://en.wikipedia.org/wiki/List_of_French_people_by_net_worth
https://en.wikipedia.org/wiki/List_of_members_of_the_Forbes_400
One of the two lists reflects a clear dynastic bias while the other reflects a clear dynamism in the acquisition of wealth. It is obvious to me which one is which. That’s only one data point, of course, but the fact that the elites in the United States renew themselves with new blood every couple of generations (if by generation we consider the span of 15 to 20 years) and that in Europe their elites tend to perpetuate themselves in perpetuity is a clear sign that the United States and Europe are still worlds apart. We could get closer, if we insist in putting in place games that favor the well to do and well connected. It is the same reason why when companies become big they are the biggest proponents of government regulations. Heavy regulations favor incumbents and put barriers of entry to newcomers.
No matter which mechanism of admission you put in place at the undergraduate level, any time you add complexity, it will be gamed by the same people: the well to do and well connected. They will adapt to whichever mechanism you pick: holistic admissions, standardized testing, SAT-II/AP type of exams, etc, etc. The only solution to the problem of inequality is increased access and increased fight against ageism, against preference for testing, against preference for undergraduate degrees from certain schools, etc. Let’s focus on results and achievement not on privilege and potential, even if that privilege is “I scored 2400 in the SAT when I was 16”. Focus on results and access is the best antidote to inequality.
Comment #46 October 2nd, 2017 at 5:34 pm
Now, for scientific reasons, we need you to make an IQ test. Childhood IQ tests have much greater variance.
Comment #47 October 2nd, 2017 at 6:00 pm
I live in a country which SAT-like tests have a huge impact in someone’s life (entrance on public and free universities), and I don’t understand why many people here think the elite can game SAT-like tests so much? Unless you think there is outright manipulation of scores going on…
The reality in this country is that the economic elite can increase their children scores with prep courses and longer preparation by something like 5-10% but rarely that will make someone of average IQ pass the exam for medical studies (the most sought after profession) or even for engineering…
Average children from the economic elite usually end up going to private universities that will be considered much worse in their resumes…
Comment #48 October 2nd, 2017 at 10:14 pm
This whole discussion about IQ reminds me of Asimov’s disagreeable experience as a member of MENSA (of which I was embarrassingly once a member too):
“It was not on the whole, a happy experience. I met a number of wonderful Mensans, but there were other Mensans who were brain-proud and aggressive about their IQs, who, one got the impression, would like, on being introduced, to be able to say, ’I’m Joe Doakes, and my IQ is 172,’ or, perhaps, have the figure tattooed on their forehead. They were, as I had been in my youth, forcing their intelligence on unwilling victims. In general, too, they felt underappreciated and undersuccessful. As a result, they had soured on the Universe and tended to be disagreeable.”
Comment #49 October 2nd, 2017 at 10:36 pm
“Random Critical Analysis” has an excellent post on test scores titled “No, the SAT doesn’t just measure income” – https://randomcriticalanalysis.wordpress.com/2015/11/25/no-the-sat-doesnt-just-measure-income/. Its a long post with lots of data, but I think its well worth reading. Scott Alexander of SSC has a link to that blog on his sidebar. Some of the hypotheses he (or she? – the author is understandably anonymous) presents evidence for include, “The SAT’s association with family income is almost entirely mediated by family education” and “SAT test prep has little effect.”
Comment #50 October 2nd, 2017 at 11:10 pm
Curious Wavefunction #48: There are certain temptations that I seem completely to lack, and barely even understand—like gambling and dangerous sports—and one of them is joining an organization like Mensa (well, not that they’d accept me anyway, without taking some other IQ test…). I’ve wondered: why would a person bother, when there are so many organizations, causes, academic fields, etc. that actually do something, stand for something, discover something, are about something, and that incidentally select for high intelligence anyway, without even needing to make it a requirement? 🙂
Comment #51 October 3rd, 2017 at 1:00 am
#50
Because we just want to be, without having to do something, stand for something, discover something, be about something. We just want to be, and be with our peers.
Just joking, I’m not part of Mensa, but I thought that would be a fun answer by a Mensa member with some poetic inclinations.
I don’t really know what they do when they meet. I always thought it worked like a dating website. The organization just helps you to find a date. What you do at dinner, or after dinner, is up to you and your imagination 🙂
Comment #52 October 3rd, 2017 at 1:44 am
Right, I guess Mensa is for people who have high IQs… and nothing else.
Comment #53 October 3rd, 2017 at 1:50 am
I found the following interesting,
https://en.wikipedia.org/wiki/Mensa_International
“Berrill and Ware were both disappointed with the resulting society. Berrill had intended Mensa as “an aristocracy of the intellect”, and was unhappy that a majority of Mensans came from humble homes, while Ware said, “I do get disappointed that so many members spend so much time solving puzzles”.”
While I understand that the demographics of Mensa do not necessarily match the demographics of those who score high in IQ tests at large, the above situation goes to show that those who are more likely to find meaning in having scored high in IQ tests are those who have little to show for other than having scored high in these tests.
Comment #54 October 3rd, 2017 at 4:48 am
#53: I mean, if (hypothetically) someone lived in a place where there was nothing for them except Mensa, if they met their first friends or spouse or whatever through the organization, then I’m not going to knock their experience—any more than I would knock an atheist who goes to church or synagogue or whatever only because it’s where they found a community. But it’s even better when you can find a community that’s organized around something you believe in.
A good modern alternative might be the rationalist / LessWrong / SlateStarCodex meetups, which welcome all, with no IQ or educational or any other requirement, but do tend to appeal most to the extremely nerdy. 😀
Comment #55 October 3rd, 2017 at 5:10 am
Just found this, from Jessica Su on Quora, which says it better than I did:
I would never join Mensa, because I meet so many smart people in real life that joining would be superfluous. Silicon Valley is basically Mensa, except you pay rent instead of the membership fee.
But I could see how Mensa would be a value-add if you were a smart person who, for whatever reason, didn’t know any other smart people.
Comment #56 October 3rd, 2017 at 5:13 am
Going back to children, let me mention that overall, the IQ tests for children give a fairly good and useful tool to evaluate abilities and weaknesses. The performance on the various sub-tests is often more helpful than the combined IQ (which can also be useful while problematic). A main purpose of these tests, along with other means, is to give a wide psychological evaluation of a child which can play important role in helping children and parents. A phenomenon of gifted children that do extremely well on some tests and badly on others is quite familiar.
Comment #57 October 3rd, 2017 at 8:32 am
Comment #8: ”.Shockley later in life became obsessed with race and IQ. I wonder if he not being identified as “high IQ” had anything to do with his obsession, in spite of having won the Nobel Prize in physics.”. Schockley’s apparent obsession with IQ has been attributed to his being a racist. His view on the subject are best expressed in his own words: https://www.youtube.com/watch?v=sAszZr3SkEs
Comment #58 October 3rd, 2017 at 10:30 am
Richard Gaylord #57
Oh, don’t take me wrong. I am on record, in a different thread, saying that not only I believe Shockley was a racist, but in fact I believe he was “real Nazi”-like, the kind of Nazi I would be terrified of having running the government. You see, the context of that discussion was how I differentiate between the losers that call themselves “neo-Nazi” that are not a cause of concern for me and “real Nazis” like Shockley who would use the political power at their disposal to implement Nazi Germany type of policies. I also said then, and I reaffirm now, that if you want to find real Nazis, you are more likely to find them in meetings of the National Academy of Sciences or high profile academic conferences than in violent demonstrations like Charlottesville’s. I also said then, and I repeat now, that I am not saying that all uber-smart people are Nazis, but only that that’s the group you should scrutinize if you really want to find the Nazis you should be afraid of.
With all this said, the whole Terman experiment shows better than any empirical evidence I can provide, that the concept of IQ is baloney. Or, if you will, that IQ tests only measure how well you take IQ tests and little else. Here is another analysis of the Terman children’s achievements,
https://www.psychologytoday.com/blog/beautiful-minds/200909/the-truth-about-the-termites
“Let us give Terman the benefit of the doubt and post that all 2,000 scientific and technical publications were produced by the 70 who made it into American Men of Science. That implies that, on average, Terman’s notable scientists produced about 29 publications by the time they had reached their mid-40s. In contrast, American Nobel laureates in the sciences averaged about 38 publications by the time they were 39 years old, and claimed about 59 publications by their mid-40s. THat amounts to a twofold disparity in output. Hence, Terman’s intellectual elite was not of the same caliber as the true scientific elite of the same nation and era.”
Or,
“Another analysis shows that the accomplishments of the “Termites” could have been predicted on their socioeconomic status alone. These were mostly white, middle to upper middle class men with opportunities and resources for success. Some argue that it wasn’t even necessary for Terman to analyze the IQ dimension–he could have stopped with SES and call it a day.”
I have several heuristic rules that I have “discovered” throughout my life. One of them, as I have mentioned, is that the most important decisions in society that affect us as individuals are made by the most stupid members of society. A closely related one, but different, is that most problems in society are caused by lack of mathematical literacy among the general population. Using IQ testing as a predictive tool is a clear example of confusing correlation with causation. In addition, the best correlations people have been able to come up with are around 0.5, a far cry of 0.99 which is what any serious correlative measure should have to be taken as a proxy of something. Imagine if medical testing of infectious disease or cancer were performed the same way: we have not detected you have HIV antibodies, but you know, only at most 50% of HIV positive people develop antibodies, so we cannot tell you much about whether you are infected with HIV. Or, gee, we detected anomalous cells in your pancreas, but you know, only up to 50% of people with pancreatic cancer develop those cells and there are other reasons those cells might have been developed so we cannot tell you whether you have pancreatic cancer. I hope people get an idea. Society as a whole gives a pass to IQ tests that does not give to other measurements.
As I said, psychology must be the biggest scam the losers of the academic game perpetrate in society at large.
Comment #59 October 3rd, 2017 at 10:42 am
I’d like to share my experience with IQ testing in elementary school and some thoughts.
I was about 15 and it was part of a more general voluntary psychological examination, so they could provide you with a ,,professional orientation” ( career choice etc ) . The IQ or ,,aptitude” test had three parts (perceptive, verbal and spatial),
on the first one you had pairs of pictures and had to tell if they were the same or not or something like that,
on the second one there were words and you had to write the words opposite in meaning next to them,
and on the third one, you’re given a 3D arrangement and have to reconstruct it from a different angle.
I got an average, maybe a bit above average score, I did terribly on either one of the first two ( verbal part I think),
but exceptionally on the third one. On the verbal part I didn’t even know many of the words, and of the ones I did know I either didn’t know their opposite or took a long time to retrieve it from memory. On the perception part I was overly cautious and paranoid, mind started playing tricks on me, started seeing things, I don’t think it affected my ,,performance,, but I bet it could have.
I believe these kinds of tests are mostly beyond ridiculous for obvious reasons. There are others like Raven’s matrices that can be taken a lot more seriously. Some tests emphasize ,,the raw working memory’’ ,,processing speed’’ aspects; others have more to do with ,inductive leaps, etc. The problem with the first ones is these things don’t seem that important when it comes to human intellectual achievements, certain animals seem to have better ,,raw,, capacities of that sort. When Archimedes (and probably many before and after him) sat in his bathtub and came to his realization, it was not a feat of working memory ,processing speed, some combo or whatever. The problem with the second ones is, well for a given problem there are many routes the mind checks, which routes it checks is a result of many factors, everyone’s inner wiring and experience is a bit different, some routes are quicker, and you’re just lucky if you stumble upon them, or you’re lucky that the problem suits your wiring and experience, ( sometimes we instantly give elegant solutions to problems, other times we are clueless ) it doesn’t really matter if there are more solutions or just one. These things I mention don’t really matter that much because psychometrics relies on statistics. But at this stage all the research done is weak from a ,,hard science’’ point of view, and it doesn’t say much things that we in fact did not already know in one way or another. The specifics are nowhere near where they should be. People are silly to even bother themselves with it. If a time comes when intelligence is truly understood, and if certain individuals happen to be a lot more disadvantaged than they would like, there will probably be treatments/augmentations of some sort. This probably assumes more rigorous definitions and an understanding of the processes and mechanisms, or equations that describe the phenomenon.
Comment #60 October 3rd, 2017 at 12:23 pm
My 4 kids were given IQ tests when they were young. They all took them about five years later and got almost identical scores. The tests were given by different professionals.
So Scott, if you were to take the IQ test today, do you think you would get a 106?
Comment #61 October 3rd, 2017 at 12:31 pm
Craig #60: For whatever it’s worth, my parents tell me that I was retested slightly afterward by a different psychiatrist, and on the second test I was more like 150. I also had a perfect SAT, which is supposedly consistent with that. But honestly, I have no real idea, there doesn’t seem to be a great deal of inter-test reliability in my case, and for reasons spelled out in the above exchange with “The Problem With Gatekeepers,” it’s not a question that interests me greatly.
Comment #62 October 3rd, 2017 at 1:58 pm
Scott #55,
yes, but have you checked the rents in Silicon Valley recently?
That’s a pretty expensive membership fee.
Not exactly for everyone 🙂
I do think it’s worth it though.
Comment #63 October 3rd, 2017 at 2:22 pm
Craig #60
If they took the same ,kind’ of test is it really a surprise ? It doesn’t matter if it was different people who gave the tests.
If in those 5 years the kids didn’t change in any other way, it is also not expected for the scores to change on their own, it depends how young they were as well. My personal experience is almost the same as Scott’s, no inter-test reliability, Raven’s matrices for example I did a lot better than my elementary school test. In my opinion, the Raven’s is the better test ( unlike Scott, I am not humble ), but opinions of the experts seem to be that they are both good. More problems for them then.
I just don’t think these fields are to be trusted, yet. Some of my friends in fact had similar/same experiences.
Comment #64 October 3rd, 2017 at 3:59 pm
Speaking of clubs for smart people, I love Richard Feynman’s take:
“When I was in high school, one of the first honors I got was to be a member of the Arista, which is a group of kids who got good grades—eh?—and everybody wanted to be a member of the Arista, and when I got into the Arista I discovered that what they did in their meetings was to sit around to discuss who else was worthy to join this wonderful group that we are—okay? So we sat around trying to decide who it was who would get to be allowed into this Arista. This kind of thing bothers me psychologically for one or another reason I don’t understand myself—honors—and from that day to this [it] always bothered me. When I became a member of the National Academy of Sciences, I had ultimately to resign because that was another organization most of whose time was spent in choosing who was illustrious enough to join, to be allowed to join us in our organization, including such questions as [should] we physicists stick together because they’ve a very good chemist that they’re trying to get in and we haven’t got enough room for so-and-so. What’s the matter with chemists? The whole thing was rotten because its purpose was mostly to decide who could have this honor-okay? I don’t like honors.”
Comment #65 October 3rd, 2017 at 4:13 pm
Also, as for verbal skills, spelling, vocabulary, and other acquired knowledge that might help you on the non-math portion of the SAT (or possibly an IQ test, I’m not sure), what role does this accumulation of rote-memorized knowledge play in accessing intelligence? (from the kid who did poorly on the verbal half of the SATs 😉
Comment #66 October 3rd, 2017 at 5:50 pm
Re Mensa.
I infiltrated Mensa as a junior high schooler. At the time, Frank Ryan, the QB of the Cleveland Browns, and later a Math Prof at CWRU, was promoting it, so I was psyched.
The hardest part of the test was working out money transactions in UK money. They explained the rules for pounds and schillings. etc., and you had to figure out how to make change.
I went to a couple meeting, expecting to be initiated into cool secret grownup knowledge, but if any such was there, it was cleverly disguised as adults getting drunk and trying to sound profound.
There is a secret Mensa symbol that you wear in a particular spot on your sports jacket. This allows you to spot other Mensa types in the wild. I never wear a sports jacket, so I missed out on that. I have been at plenty of scientific meetings where a lot of people were wearing jackets, but have yet to see the secret symbol. Maybe they got a new one and didn’t tell me — I guess I am a half century + behind on dues.
Comment #67 October 4th, 2017 at 1:37 am
What were you supposed to say about the smoking house?
Comment #68 October 4th, 2017 at 4:37 am
Eric #67: I think, that you’d call firefighters or other adults, not conditional on how much you liked your neighbors. 🙂
Comment #69 October 4th, 2017 at 5:04 am
“Richard Feynman famously scored only 124 on a childhood IQ test… ”
Well, given the Flynn Effect, perhaps we should apply a correction to this if we want to compare his result to yours. Since there were about five or six decades in between his test and yours, and assuming a typical rate of 2.9 points per decade, his raw test score would translate to a normalized score of about 105 or 110 for the decade in which you took the test.
Also, before you get too excited about g, I’d recommend reading Cosma Shalizi’s discussion “g, a Statistical Myth”.
Comment #70 October 4th, 2017 at 9:07 am
Scott #61: So if you got 106 on one test and about 150 on the other, that averages to around 124, so you are in Feynman’s league. Not bad.
Comment #71 October 4th, 2017 at 1:43 pm
To anyone with a scientific or mathematical bent, the tests are vaguely horrifying. “Which of these pictures is unlike the others?” “What number comes next in the sequence?” Question after question that could have multiple defensible valid answers, but only one that “counts”—and that, therefore, mostly tests the social skill of reverse-engineering what the test-writer had in mind.
Just to amplify this point — this problem is most glaringly apparent in the tasks that want you to “continue a pattern,” like the ones you mention here, but it’s not limited to them. My “favorite” example of an absurd IQ subtest is the Wechsler Similarities subtest, which I ranted about here.
Comment #72 October 5th, 2017 at 2:21 pm
IQ tests are 100% bunk. The idea of IQ was developed in the era of phrenology, long before we had any reasonable knowledge about the human brain. We still don’t know a lot about the brain today. IQ tests should be made obsolete.
Comment #73 October 5th, 2017 at 5:18 pm
Why do you write ““g“, a statistical construct derived via factor analysis: something that positively correlates with just about every measurable intellectual ability, but that isn’t itself directly measurable (at least, not by any test yet devised).”? The Raven test (which has been around for 80 years) is designed specifically to measure g.
https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices
Comment #74 October 5th, 2017 at 8:57 pm
Andras #73: In some sense, every IQ test is designed specifically to measure g; that’s what makes them IQ tests as opposed to any old tests of knowledge or puzzle-solving ability that happen to be g-loaded. But no single test, including Raven’s, measures g perfectly (or even “perfectly modulo random noise,” with no test-specific systematic contribution). That’s because g is only defined by looking at hundreds or thousands of tests and then doing some linear algebra on them to extract the linear combination of them that explains the largest amount of correlation in scores.
So in particular, I agree with Cosma Shalizi, in the link given in comment #69—and, earlier than him, with Stephen Jay Gould and others—that there need not be any natural “thing in the brain” that corresponds to g. (Gould rather portentously called this “the error of reification”—doing a little reifying himself! 🙂 ) Instead, g is something that you could necessarily extract, as a matter of mathematics, as long as scores on all the different intelligence tests are fairly well correlated with each other (which they are). And g could just as easily be—and probably is—a sum of hundreds of unrelated “things” as a single “thing.”
OK, but that still doesn’t mean that it’s useless to do factor analysis to extract a factor that explains the maximum amount of correlation, when you see hundreds of correlated things. It’s just that, as Scott Alexander pointed out, it’s more useful for population-level research than it is for individuals. As an individual, if you want to know whether you can be a good engineer, then just see how good you are at engineering! Who cares what fraction of the total contribution to your engineering ability is given by “g”, and what fraction by other components?
So alas, I don’t see how the debate between “liberals” and “conservatives” in the g debate can ever be definitively resolved, because there’s no experiment to resolve it, and also no simple mathematical point that the smart people on either side don’t understand. It’s all just a question of how much you expected “g” to do for you in the first place, and whether it’s surprising or unsurprising, interesting or uninteresting, that there’s a statistical construct that does that. The two sides are talking past each other.
Comment #75 October 6th, 2017 at 3:18 am
A small clarification for #1. I wrote “I have a friend who at age 8 decided not to learn the multiplication table from reasons of principle (that I don’t remember).” After checking the facts it turns out that this guy, did not mind learning multiplication table but insisted on making computations based on the value of squared that he quickly rememberd. So when he had to compute 7 x 8 he would compute 7 x 8=64-8, which immensly annoyed his teacher. 🙂
Comment #76 October 6th, 2017 at 3:44 am
I have had heated arguments on the internet about IQ, and I wish I could have pointed my opponents in the direction of this post. I would go a little further and say that individual results are almost meaningless. Most of the questions boil down to “what number am I thinking of” and similar things. But IQ people can be as bad as Myers-Briggsists and Astrologers, so good luck with that!
Comment #77 October 6th, 2017 at 3:46 am
Scott #74 Reification is a charge easily leveled against any abstract concept — take for example temperature. Ultimately, there is no such thing, there are just highly correlated measurement protocols (actually really tricky in the low temp range), and of course a skeptic can object to any “definition” that refers to average molecular speeds the same way Gould objects to g. Measuring things is hard. Even our fundamental units are computed by least squares adjustment over many measurements.
It should come as no surprise that mental abilities are hard to measure. That said, there are tests that measure g quite directly (correlation coefficients in the high nineties, about as good as you can expect in a messy biological system) and even if we take a broad humanistic view of the matter (individuals should not be judged on the size of their X) to pretend that such measurement is not possible for some deeper statistical reasons is pretty silly.
Comment #78 October 6th, 2017 at 3:29 pm
#74
“it’s more useful for population-level research than it is for individuals.”
“As an individual, if you want to know whether you can be a good engineer, then just see how good you are at engineering!”
Of course, those two things are not independent. If inferences about group differences are used to justify less investment in opportunities for individuals in those groups to explore their own potential for engineering etc, then there is a problem.
#77
Shalizi makes the point that “reification” [thanks Professor Gould!] of “g” would be justified if it lead to some deeper insight into the structure and function of the brain in relation to intelligence, but so far that hasn’t developed. Maybe a causal connection will be made someday, but from a genetic and physiological point of view the failure of “g” to “reify” is not surprising. That’s an inherent limitation of factor analysis.
Anyway, I was fortunate to grow up in an era when IQ was out of favor and so I have never been burdened with having one. Of course, there were numerous other scholastic achievement and aptitude tests along the way that had less ambitious goals. Those scores (typically percentile based) were actually important for confidence building, etc without being freighted with the baggage of an IQ.
Comment #79 October 7th, 2017 at 1:47 pm
@Chris: Not to mention Anti-IQ and Anti-MBTI.
Remember that there is a lot of gray area between “cleaves reality at its joints” and “complete bunk”, and in that area lies any tool that exists in the real world. If you throw them away because they are not perfect, you’re left without any tools.
I’m even ready to admit that there are probably better measures than IQ and definitely better measures than MBTI for what they proclaim to measure. What makes them so popular is that they are easy to use and good enough most of the time. You should just be aware of the limitations (especially when looking at individuals).
Comment #80 October 23rd, 2017 at 7:37 am
[…] 124 is about 30 points off the lowest remotely plausible value. Scott Aaronson writes about his own similar experience taking an IQ test at age four and getting a 106 — right about average. Aaronson is a computer […]
Comment #81 October 23rd, 2017 at 5:14 pm
[…] on personal scale: http://slatestarcodex.com/2017/09/27/against-individual-iq-worries/. The other Scott A. […]
Comment #82 November 17th, 2017 at 6:57 am
All I really needed to see in the “g, a Statistical Myth” post was the declaration “Since intelligence tests are made to correlate with each other, it follows trivially that there must appear to be a general factor of intelligence.”
That in itself absolutely proves the author is writing in ignorance; there is lots of psychometric literature of people trying to disprove g‘s existence by trying to design cognitive tests that fail to correlate. The paper is accordingly very obviously a critique from someone who does not actually know the field. It is not surprising it accordingly makes old, discredited arguments.
Sampling theory is thoroughly discussed in the literature, and has serious defects. For example, it would predict higher correlations between closely-related tests, and weaker ones between distantly-related ones, since closely-related tests would sample more of the same abilities. Yet actual testing disagrees. Tests as distinct as vocabulary tests and Raven’s Progressive Matrices are strongly correlated, while memory span tests forward and backward are comparatively weakly correlated.
It is possible that apparently-unrelated tests are actually closely-related, and apparently-similar tests are not. But after enough efforts to deliberately choose tests that have no obvious relations are done, and the correlations consistently fail to fit the pattern expected by sampling, it’s time to admit that the sampling approach is a failure. And that time was well before 2007.
It is true that the science can’t be complete until the neurologists actually find what g corresponds to in the brain. But that’s the same as saying genetics couldn’t be complete before molecular biologists found what genes corresponded to in cells. The lack of identification of the underlying cause is not an excuse to go and invent more ideologically-congenial theories that contradict empirical findings in the field of human intelligence than it was an excuse for Lysenkoism in the field of heredity.
Comment #83 December 2nd, 2017 at 10:02 am
[…] Scott Aaronson had an IQ of 106 […]