23, Me, and the Right to Misinterpret Probabilities
If you’re the sort of person who reads this blog, you may have heard that 23andMe—the company that (until recently) let anyone spit into a capsule, send it away to a DNA lab, and then learn basic information about their ancestry, disease risks, etc.—has suspended much of its service, on orders from the US Food and Drug Administration. As I understand it, on Nov. 25, the FDA ordered 23andMe to stop marketing to new customers (though it can still serve existing customers), and on Dec. 5, the company stopped offering new health-related information to any customers (though you can still access the health information you had before, and ancestry and other non-health information is unaffected).
Of course, the impact of these developments is broader: within a couple weeks, “do-it-yourself genomics” has gone from an industry whose explosive growth lots of commentators took as a given, to one whose future looks severely in doubt (at least in the US).
The FDA gave the reasons for its order in a letter to Ann Wojcicki, 23andMe’s CEO. Excerpts:
For instance, if the BRCA-related risk assessment for breast or ovarian cancer reports a false positive, it could lead a patient to undergo prophylactic surgery, chemoprevention, intensive screening, or other morbidity-inducing actions, while a false negative could result in a failure to recognize an actual risk that may exist. Assessments for drug responses carry the risks that patients relying on such tests may begin to self-manage their treatments through dose changes or even abandon certain therapies depending on the outcome of the assessment. For example, false genotype results for your warfarin drug response test could have significant unreasonable risk of illness, injury, or death to the patient due to thrombosis or bleeding events that occur from treatment with a drug at a dose that does not provide the appropriately calibrated anticoagulant effect … The risk of serious injury or death is known to be high when patients are either non-compliant or not properly dosed; combined with the risk that a direct-to-consumer test result may be used by a patient to self-manage, serious concerns are raised if test results are not adequately understood by patients or if incorrect test results are reported.
To clarify, the DNA labs that 23andMe uses are already government-regulated. Thus, the question at issue here is not whether, if 23andMe claims (say) that you have CG instead of CC at some particular locus, the information is reliable. Rather, the question is whether 23andMe should be allowed to tell you that fact, while also telling you that a recent research paper found that people with CG have a 10.4% probability of developing Alzheimer’s disease, as compared to a 7.2% base rate. More bluntly, the question is whether ordinary schmoes ought to be trusted to learn such facts about themselves, without a doctor as an intermediary to interpret the results for them, or perhaps to decide that there’s no good reason for the patient to know at all.
Among medical experts, a common attitude seems to be something like this: sure, getting access to your own genetic data is harmless fun, as long as you’re an overeducated nerd who just wants to satisfy his or her intellectual curiosity (or perhaps narcissism). But 23andMe crossed a crucial line when it started marketing its service to the hoi polloi, as something that could genuinely tell them about health risks. Most people don’t understand probability, and are incapable of parsing “based on certain gene variants we found, your chances of developing diabetes are about 6 times higher than the baseline” as anything other than “you will develop diabetes.” Nor, just as worryingly, are they able to parse “your chances are lower than the baseline” as anything other than “you won’t develop diabetes.”
I understand this argument. Nevertheless, I find it completely inconsistent with a free society. Moreover, I predict that in the future, the FDA’s current stance will be looked back upon as an outrage, with the subtleties in the FDA’s position mattering about as much as the subtleties in the Church’s position toward Galileo (“look, Mr. G., it’s fine to discuss heliocentrism among your fellow astronomers, as a hypothesis or a calculational tool—just don’t write books telling the general public that heliocentrism is literally true, and that they should change their worldviews as a result!”). That’s why I signed this petition asking the FDA to reconsider its decision, and I encourage you to sign it too.
Here are some comments that might help clarify my views:
(1) I signed up for 23andMe a few years ago, as did the rest of my family. The information I gained from it wasn’t exactly earth-shattering: I learned, for example, that my eyes are probably blue, that my ancestry is mostly Ashkenazi, that there’s a risk my eyesight will further deteriorate as I age (the same thing a succession of ophthalmologists told me), that I can’t taste the bitter flavor in brussels sprouts, and that I’m an “unlikely sprinter.” On the other hand, seeing exactly which gene variants correlate with these things, and how they compare to the variants my parents and brother have, was … cool. It felt like I imagine it must have felt to buy a personal computer in 1975. In addition, I found nothing the slightest bit dishonest about the way the results were reported. Each result was stated explicitly in terms of probabilities—giving both the baseline rate for each condition, and the rate conditioned on having such-and-such gene variant—and there were even links to the original research papers if I wanted to read them myself. I only wish that I got half as much context and detail from conventional doctor visits—or for that matter, from most materials I’ve read from the FDA itself. (When Dana was pregnant, I was pleasantly surprised when some of the tests she underwent came back with explicit probabilities and base rates. I remember wishing doctors would give me that kind of information more often.)
(2) From my limited reading and experience, I think it’s entirely possible that do-it-yourself genetic testing is overhyped; that it won’t live up to its most fervent advocates’ promises; that for most interesting traits there are just too many genes involved, via too many labyrinthine pathways, to make terribly useful predictions about individuals, etc. So it’s important to me that, in deciding whether what 23andMe does should be legal, we’re not being asked to decide any of these complicated questions! We’re only being asked whether the FDA should get to decide the answers in advance.
(3) As regular readers will know, I’m far from a doctrinaire libertarian. Thus, my opposition to shutting down 23andMe is not at all a corollary of reflexive opposition to any government regulation of anything. In fact, I’d be fine if the FDA wanted to insert a warning message on 23andMe (in addition to the warnings 23andMe already provides), emphasizing that genetic tests only provide crude statistical information, that they need to be interpreted with care, consult your doctor before doing anything based on these results, etc. But when it comes to banning access to the results, I have trouble with some of the obvious slippery slopes. E.g., what happens when some Chinese or Russian company launches a competing service? Do we ban Americans from mailing their saliva overseas? What happens when individuals become able just to sequence their entire genomes, and store and analyze them on their laptops? Do we ban the sequencing technology? Or do we just ban software that makes it easy enough to analyze the results? If the software is hard enough to use, so only professional biologists use it, does that make it OK again? Also, if the FDA will be in the business of banning genomic data analysis tools, then what about medical books? For that matter, what about any books or websites, of any kind, that might cause someone to make a poor medical decision? What would such a policy, if applied consistently, do to the multibillion-dollar alternative medicine industry?
(4) I don’t understand the history of 23andMe’s interactions with the FDA. From what I’ve read, though, they have been communicating for five years, with everything 23andMe has said in public sounding conciliatory rather than defiant (though the FDA has accused 23andMe of being tardy with its responses). Apparently, the key problem is simply that the FDA hasn’t yet developed a regulatory policy specifically for direct-to-consumer genetic tests. It’s been considering such a policy for years—but in the meantime, it believes no one should be marketing such tests for health purposes before a policy exists. Alas, there are very few cases where I’d feel inclined to support a government in saying: “X is a new technology that lots of people are excited about. However, our regulatory policies haven’t yet caught up to X. Therefore, our decision is that X is banned, until and unless we figure out how to regulate it.” Maybe I could support such a policy, if X had the potential to level cities and kill millions. But when it comes to consumer DNA tests, this sort of preemptive banning seems purposefully designed to give wet dreams to Ayn Rand fans.
(5) I confess that, despite everything I’ve said, my moral intuitions might be different if dead bodies were piling up because of terrible 23andMe-inspired medical decisions. But as far as I know, there’s no evidence so far that even a single person was harmed. Which isn’t so surprising: after all, people might run to their doctor terrified about something they learned on 23onMe, but no sane doctor would ever make a decision solely on that basis, without ordering further tests.
Follow
Comment #1 December 11th, 2013 at 7:44 pm
Ironic that it is considered better to let doctors interpret the probabilities, given their documented inabilities to do so correctly. See Gigerenzer’s work, e.g.,
http://psi.sagepub.com/content/8/2/53.short
or Noguchi et al.,
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1495132/
to give just a couple of examples.
Comment #2 December 11th, 2013 at 8:26 pm
From what I’ve been reading, a large part of the problem with 23andme seems to be lack of information provided to customers about multiple hypothesis testing (for instance read http://liorpachter.wordpress.com/2013/11/30/23andme-genotypes-are-all-wrong/)
This seems like a very legitimate concern to me especially since most customers are not going to have the background to know something like this a priori.
Comment #3 December 11th, 2013 at 9:00 pm
Regarding your regulatory question, it’s not an entirely new question–New York had some laws against direct-to-consumer genetic testing, so I had to mail my sample from New Jersey.
Comment #4 December 11th, 2013 at 9:15 pm
Signed the petition, though my understanding is that 23andMe failed to communicate with FDA properly and this is why they have this issue. Ancestry-only services like ftdna didn’t have FDA problems, though in principle you can download your results from them and then run some software to make disease-related predictions.
I did my test in 2011 after you mentioned 23andme on this blog. It was really amazing to find that my Y-chromosome haplogroup is extremely rare in the area where my paternal grandfather is from and that I may have common ancestors with some German and Dutch people just a few generations away. BTW, whoever is on 23andme, feel free to send me sharing invite:
https://www.23andme.com/user/?profile=3e23e70cbe6ae1c6&view_public
Comment #5 December 11th, 2013 at 10:26 pm
The FDA has not banned consumer DNA tests. You can still get the raw genomic data from a test. But the 23andme product is within the scope of the medical device regulated by law, and 23andme has not submitted the data showing the accuracy and reliability of the consumer medical info that it provides. The petition would make more sense if 23andme had complied with basic requirements.
Comment #6 December 11th, 2013 at 10:27 pm
Scott:
I don’t know what the issue is, but I don’t think this line of reasoning is correct. CLIA regulation of labs is pretty mild. It’s about whether the lab is handling samples correctly, not whether the chip is accurate. It may very well be that the FDA wants some kind of proof of the accuracy of the tests.
Anon: Do you have an independent source? Pachter doesn’t even say that he thinks that is the issue, only that it should be. I think it’s pretty implausible that this is the issue. Indeed, that FDA singled out two of the very few useful measurements is evidence that they are not worried that most people will get a measurement claim on one of the many useless claims.
Comment #7 December 12th, 2013 at 12:56 am
Here’s a story of a bad health scare due to a 23andme bug: http://mntmn.com/pages/23andme.html
23andme is also scary from a privacy standpoint. Sort of like Google scooping up everyone’s personal data, except it’s genetic data which is even more personal : http://www.scientificamerican.com/article.cfm?id=23andme-is-terrifying-but-not-for-reasons-fda
Do you really want this Google alter ego figuring out if you’re related to (say) Geordie Rose, especially if you don’t know that info yourself? I know enough history to not want such a database to exist. What happens if there’s a gene for religious whack-jobbery, and then the religious whackjobs use the database to round up everyone who doesn’t have it?
Here’s 23andme patenting a designer baby scheme: http://www.wired.com/wiredscience/2013/10/23andme-patent/
Frankly they creep me out.
The way I heard it about the health info they gave, was that it was presented in the form of suggestions of health measures based on their analysis of your bodily fluids, i.e.medical advice. Of course there’s a libertarian argument that one shouldn’t need a license to dispense medical advice. If one rejects that argument though then I don’t see why 23andme should be exempt.
Comment #8 December 12th, 2013 at 1:10 am
Consider the bigger picture of alternative approaches to medicine. Some turn out to be worthwhile, some so-so, and many worthless. Frequently they get a large group of enthusiasts, at which point they are hard to control, let alone root out.
New approaches are all over the map:
Some are 100% crap, Homeopathy comes to mind.
Some work pretty good, but the theory is mostly crap, chiropractic being the poster child.
Some might slightly work, but the theory is totally crap, acupuncture is the poster child.
Prayer rather than medicine? I’m not touching that one.
If I had time right now, I could make a longer and more entertaining list.
Anyway, it is always much easier to loosen the rules if appropriate that to tighten rules up after a cult following has developed.
23&me is a well funded player in new area. Maybe their info will often be useful. But maybe they will make $Billions by scaring people into worthless and dumb stuff. Who knows? But that is the job of regulators to make an attempt to figure this out first. The fact that regulators often do a crappy job for a variety of reasons does not mean there should not be any regulations.
Comment #9 December 12th, 2013 at 1:30 am
Scott says: “I understand this argument. Nevertheless, I find it completely inconsistent with a free society. “
Well, some libertarians might say they find the entire concept of an FDA inconsistent with a free society.
Question is, at what point does one draw the line? There’s always a tension between protecting people and allowing them the freedom to harm themselves.
Comment #10 December 12th, 2013 at 1:51 am
From your description, 23andMe sounds reasonable and above board, but I can definitely understand the desire to be cautious about this. There are lots of opporunities for unethical behavior or even outright fraud, and no code of professional ethics to try to reign it in.
To be blunt, if Pfiezer were selling this product directly to consumers my default position would be that they should be shut down. The conflict of interest and the incentive to mislead to cross market pharmecuticals would be too great. Of course 23andMe isn’t Pfeizer, and there is no apparent conflict of interest, so I am on the fence.
Probably I actually agree that we should let 23andMe do this now rather than banning it until some big company lobbies to get the law changed, then does the job worse in a field cleared of the hungry startups by preemptive regulation. My point is just that I think there are situations where I would have the opposite conclusion that are not far fetched.
Comment #11 December 12th, 2013 at 3:14 am
My impression was 23andMe is not handing over the necessary data to FDA at least till recently?
Am I misinformed?
Comment #12 December 12th, 2013 at 4:54 am
That is an outstanding comment by Evan (as it seems to me).
Moreover, it is exceedingly unlikely that any one person, or any one professional society, can conceive of all the possible unethical/fraudulent enterprises that are attendant to these new technologies.
Rather, history shows us plainly that the opposite is true: the ingenuity of crooks, con-men, and crime-bosses generically (and often hilariously) outstrips the ingenuity of the agencies that attempt to restrain their activities.
None-the-less, Scott is entirely right (as it seems to me) that principles of liberty and democracy call for maximal choice, and the medical information that guides that choice, to rest with individuals and families, not with states, corporations, religions, and professional guilds (and criminal enterprises).
Question Markets are most efficient, when information is most widely disseminated. Does strict libertarianism therefore demand that corporations, states, and guilds enjoy unrestricted access to individual genomic information, including unrestricted market freedom to sell that information to highest-bidders?
More broadly, when information-markets fail — in health-care for example — does dogmatic libertarianism fail with them?
Answer Yes.
Comment #13 December 12th, 2013 at 5:28 am
Thank you for this information. I have signed the petition.
Comment #14 December 12th, 2013 at 6:14 am
Rahul #9:
Well, some libertarians might say they find the entire concept of an FDA inconsistent with a free society.
Question is, at what point does one draw the line?
Yes, I agree that that’s the question! The purpose of this post was to give some indication of where I would draw the line. Namely, I’m OK with heavy government regulation of drugs (in principle—I have enormous problems with how the FDA does things in practice today, but that’s a separate discussion!). I’m also OK with regulation of medical information if it’s extremely mild (e.g., attaching warning labels), or perhaps in exceptional circumstances (compelling evidence that lives are in danger). However, suspending much of 23andme has the actual effect of removing people’s freedom to learn (mostly) accurate, (sometimes) relevant information about their own bodies; there’s simply no other service right now that’s comparable. And no rationale has been offered that doesn’t strike me as creepily paternalistic (“maybe someone could misinterpret the results and overreact”). Thus, for me, it crosses the line.
Comment #15 December 12th, 2013 at 6:45 am
Raoul Ohio #8:
The fact that regulators often do a crappy job for a variety of reasons does not mean there should not be any regulations.
Completely agreed! But citizens should also point out when they think regulators are doing a crappy job.
Comment #16 December 12th, 2013 at 8:12 am
Scott #15:
“However, suspending much of 23andme has the actual effect of removing people’s freedom to learn (mostly) accurate, (sometimes) relevant information about their own bodies;”
I am not sure about the “mostly accurate” bit. If they merely printed out your chromosome fine. But the bits where the correlate it with chance of a disease etc. we only have their word for it right now, right?
I’m not saying it’s a fraud and there indeed may be studies & peer reviewed literature that they base it on but if you look at conventional FDA reviews the bar is a lot higher than just having a study.
IMHO these are very tricky issues especially about causality and the chance of spurious correlations when mining large datasets. Or plain old “bugs”.
Comment #17 December 12th, 2013 at 8:57 am
Rahul #16:
But the bits where the correlate it with chance of a disease etc. we only have their word for it right now, right?
In each report, they give you links to the research papers that that particular correlation comes from. They also give you their subjective rating of the reliability of the correlation (1 to 4 stars).
Apart from that, you can of course form your own crude estimate of the reliability of their information, by seeing how many things about you they got right (and how many wrong) that you already knew.
Yes, there remains a considerable chance of error. But I’d say the relevant question should be how that chance compares to the chance of error in conventional medical tests (i.e., the kind the FDA knows how to review and approve). As I said before, I felt better about the level of context I was getting than I did with most medical information (e.g., the many official-looking pamphlets and websites I read when Dana was pregnant, which said to do this or not worry about that but didn’t explain the reasoning or give you the statistics).
Comment #18 December 12th, 2013 at 9:30 am
asdf #7:
Here’s a story of a bad health scare due to a 23andme bug: http://mntmn.com/pages/23andme.html
Wow, that story is extremely interesting! It underscores why you should absolutely consult a doctor / get retested / investigate further given any 23andMe results that shock you. (Maybe the best situation would be if there were several competing services, all cheap, and you only had to worry about something if they all found it independently.)
Of course, the story also illustrates that
(1) when 23andMe did badly screw up, it wasn’t for a completely doofus reason (e.g., switching his sample with someone else’s), but for a much more interesting one (the guy really did have multiple mutations linked to limb-girdle muscular dystrophy, but not in the same gene), and
(2) when notified of the error, they fixed it and apologized within a few days.
As I said, what would be interesting would be to compare the error rate to that in conventional, FDA-approved testing services.
Anyway, I part ways with you when you start talking about privacy:
Do you really want this Google alter ego figuring out if you’re related to (say) Geordie Rose, especially if you don’t know that info yourself? I know enough history to not want such a database to exist.
Lord knows, I’ve heavily criticized Google on this blog, for everything from their absurd claims about the D-Wave device, to capitulation to Chinese censorship, to their taking away services people had grown to rely on in an attempt to shepherd them onto Google+, to my having had no one to contact when my Gmail account went down for days without warning. And they could rightly be criticized over privacy issues as well. And I’ll continue to criticize them in the future if I think they deserve it, as befits a company of their size and power.
But let me answer your question with another question: would the world be better off if Google didn’t exist, if the government shut it down? Here it seems obvious to me that the answer is no: a world without them would be poorer, less open, and less free. Well, I feel roughly the same way about 23andMe, except more so—since unlike Google, 23andMe doesn’t yet have any serious competitors.
Incidentally, if Geordie Rose were my uncle, I’d see no particular reason not to know that, or not to tell others—and (I hope) I’d continue to treat his claims with skepticism, just as I do now. 😉
Comment #19 December 12th, 2013 at 9:55 am
Question is, at what point does one draw the line? There’s always a tension between protecting people and allowing them the freedom to harm themselves.
The answer to this question in my opinion cannot be a general one. It depends heavily on the specific social/political/economic context you consider. For example, I agree with the point of this post that the limitations on 23andme and similar services can be seen as limitations on people’s freedom if you are a usa citizen. If you “misinterpret the results and overreact”, it’s your problem, the cost of your (re)actions is minimal for the general community because you pay almost entirely for these actions (as far as I know, I’m italian…) In this case I agree that this kind of rationales are pathernalistic.
But if you are in europe or in countries with a strong welfare state, where the state provides for example healthcare services free of charge for everyone, I’m inclined to think the opposite, because the cost of your actions (possibly unjustified and irrationale) is almost totally paid from the state. In this case, the rationale is clear and restrictive regulations can have a very practical purpose.
Comment #20 December 12th, 2013 at 9:55 am
Wow, there is a lot of “protect people from bad ideas” sentiment out there. If 23andme were committing fraud, then it would be possible to sue them for fraud. If this fraud hurt someone, then the lawsuit might put 23andme out of business. Outright quackery isn’t really possible, unless you’re selling something harmless, like echinacea.
I think part of what we’re seeing is what Raoul Ohio said, which it’s harder to regulate something after a lot of people have gotten used to using it. One amazing example of this is acetaminophen:
http://www.thisamericanlife.org/radio-archives/episode/505/use-only-as-directed
Comment #21 December 12th, 2013 at 10:18 am
>> What happens when individuals become able just to sequence their entire genomes, and store and analyze them on their laptops?
How far are we from that? Is there any serious ongoing project that aims at that? I assume all the 23andme algorithms/technologies are closed source, is that correct?
Comment #22 December 12th, 2013 at 12:11 pm
My comment on the difficulty and need for attempting to “draw the line” went into a space warp. Just as well; it was poorly articulated.
Comment #23 December 12th, 2013 at 12:24 pm
Dog Breeds.
I often take my pack to a local dog park. This leads to an hour or two of socializing with a cross section of people, often discussing dogs. Many have spent some money getting printout of their dog’s mix of breeds.
The results quoted sound rather specific; “22.3% lab, 1.4% chow, …”. Being a born skeptic, I wonder if these numbers are slightly off, or perhaps total garbage. For example, what is a breed? Probably many breeds are marketing scams by dog breeders, who can perhaps profit by fraudulent documentation. Other breeds might be 1000’s of years old, with lots of local variations.
Maybe someone can inform me: Where do the companies offering this service get their data? Is the data “good”, BS, or somewhere between? Is there even a meaningful definition of what is a breed (other than what some dog club says)? Is there a standard for “solid data” in the genetics testing industry?
Has anyone ever tried sending a sample to the various companies and compared the results? If the results all agree, this might indicate they all farm out the work to the same lab; if they do not agree, …
Comment #24 December 12th, 2013 at 12:37 pm
The contempt with which 23andme treated the FDA smacks most unpleasantly of corporate arrogance; the company seems to believe itself above the law. From the SA article linked above,
“what a veteran Forbes reporter calls “the single dumbest regulatory strategy [he had] seen in 13 years of covering the Food and Drug Administration,” 23andMe changed its strategy. It apparently blew through its FDA deadlines, effectively annulling the clearance process, and abruptly cut off contact with the agency in May. Adding insult to injury the company started an aggressive advertising campaign (“Know more about your health!”), leaving little doubt about the underlying medical purpose of 23andMe’s Personal Genome Service. ”
This suggests that the product of 23andme is not the $99 test, but the data warehouse they are able to build from the results of all those tests.
Also, it seems there is more than the interpretation of probabilities to be considered. There is some evidence that our current understanding of genes is overly simplistic. Predictions made on the basis of the genome may not be reliable.
http://aeon.co/magazine/nature-and-cosmos/why-its-time-to-lay-the-selfish-gene-to-rest/
How many customers of 23andme know enough statistics or gene theory to make an informed judgement ? I have degrees in mathematics, operations research, computer science; I pay attention to this stuff and I’m confident I don’t know enough. Patients are not consumers.
http://allbleedingstops.blogspot.com/2011/04/why-patients-are-not-consumers.html
To hull’s point that in a rational society with healthcare, “the cost of your actions (possibly unjustified and irrational) is almost totally paid from the state”, it’s worth noting that in our alternate society, the US government pays as much or more in healthcare costs than in those rational societies.
http://theincidentaleconomist.com/wordpress/public-vs-private-health-spending-in-oecd-countries/
Comment #25 December 12th, 2013 at 12:41 pm
Alessandro, as Scott keeps saying, they publish all of their health analysis algorithms with citations to the original research. The vast majority of them are simply logistic effects, because researchers all use logistic regression; and because you can combine papers easily. There is a lot of opaque and labor-intense work where they pick which papers to trust, which is also the main difference between them and their competitors. I don’t know if they publish their ancestry algorithms, but there are similar open-source algorithms and the database is far more important.
In the current situation, with just SNP data, there isn’t really room for fancy algorithms. Myriad claims to do fancy analysis of the sequenced BRCA gene including introns and exons, but I’m skeptical that they do and I certainly don’t believe that it’s useful.
Comment #26 December 12th, 2013 at 12:52 pm
aram #20 says:
“If 23andme were committing fraud, then it would be possible to sue them for fraud. If this fraud hurt someone, then the lawsuit might put 23andme out of business.”
Yes but do we apply this standard to say medicines or investments or bank deposits or public company stock?
Although post facto lawsuits are almost always available as a final redressesal mechanism, we as a society have decided to also proactively police, validate & license certain activities.
Why don’t we let just anyone perform surgeries and when a quack botches it up they can sue?
Comment #27 December 12th, 2013 at 12:53 pm
Rahul, yes, you are misinformed.
Comment #28 December 12th, 2013 at 12:59 pm
Scott #17 says:
I felt better about the level of context I was getting [from 23andMe] than I did with most medical information (e.g., the many official-looking pamphlets and websites I read when Dana was pregnant,…)
To be fair 23andMe is actively trying to sell this to you. Most medical companies sell to the doctors. The audience differs. The dense drug pamphlets are a cross between an afterthought and legal liability backstops.
There’s a fundamental difference here.
You may fault drug companies for being bad marketeers (based on those ugly pamphlets at least) but it’d be a mistake to use that to judge the quality of the underlying product. This really isn’t apples for apples.
Comment #29 December 12th, 2013 at 1:59 pm
Rahul #28: I see. So the argument seems to be, I should forgive medical services that treat me like a 4-year-old (“do this and don’t ask us for explanations”), because they weren’t trying to do anything beyond that, because they weren’t legally required to.
This is one case where I think I prefer to be marketed to—i.e., to have medical providers who feel like they need to compete for my business.
Incidentally, I think this point is often missed when people debate alternative medicine. Yes, alternative medicine is 97% voodoo (compared to conventional medicine, which is merely 50% voodoo). But it would never have gained a toehold, were the conventional medical system not so frustrating and opaque so much of the time.
Comment #30 December 12th, 2013 at 2:05 pm
@Doug K #24
In the context of this post I would not consider the healthcare costs tout court but only those related to preventive actions such as a complete medical check up or clinical analysis. Don’t know the real situation in US but here in Italy for example you can easily make very expensive exams almost for free (as an example a TAC exam, maybe you say CAT in english, costs 50 euros but the real cost for the state can be about 1500 euros) and if people abuse of this service because some sort of unjustified panic… It’s just an economical consideration, some freedom restirctions *may* be in principle accepted in some cases, so the line can be drawnat a higer point. For the same reason we are forced to put helmet on motorbike, because if you break your head the costs of your incident are paid from others.
Comment #31 December 12th, 2013 at 2:50 pm
Raoul (dog breeds): The input data is probably not very good. In particular, they probably do not understand the diversity of each breed because they’d rather test new breeds because they can advertise their breed count. The precision is certainly garbage. Yes, breeds are defined by clubs, these days mainly interested in pedigree, which is bad for the dogs and owners, but well-aligned with genetic testing.
A quick search suggests that different testing brands don’t agree with each other. That’s probably because they are all nonsense. Most breeds are quite new, making them hard to distinguish genetically. Older breeds, especially landbreeds, should be easy to distinguish, but I don’t know if different companies are more consistent for them. A better methodology would be not to consider all breeds as peers, but to have a genealogy of breeds and a hierarchy of claims of origin, the claims about the older categories being more sure. In principle, the same is true for human ancestry, but human categories are much older than dogs and they should be distinguishable with enough data. The problem with human ancestry is getting enough samples into the database, a problem which is surely much worse with dogs.
Comment #32 December 12th, 2013 at 3:26 pm
Scott #29:
Not at all.
You should be annoyed at conventional medicine for “treating you like a 4 year old”. And for being opaque and frustrating.
But that doesn’t change the crux of the FDA vs 23andMe story.
Comment #33 December 12th, 2013 at 3:37 pm
Douglas Knight: Oh, I agree that that’s not the reason why the FDA intervened. But it is a concern and while the reasons behind the FDA decision might have been wrong, it’s less clear to me whether the outcome is wrong…
Comment #34 December 12th, 2013 at 4:53 pm
To follow up on my first comment about whether FDA’s concern is about accuracy of measurement, interpretation, bribery, paperwork, or what, the letter singles out BRCA. 23’s BRCA test is for 3 particular variants common in Ashkenazim. Myriad sells a $300 test for the same variants. So the two companies are doing exactly the same interpretation. I suppose this is evidence for Scott’s reading, in that Myriad only sells through doctors.
The letter brings up the possibility of a false positive on BRCA. This is ridiculous unless you doubt the accuracy of the test. Probably it is just ridiculous. Actually, in addition to the three nonsense mutations, 23 tests for lots of small mutations in BRCA that appear to have a small effect on the chance of breast cancer. This is worthless information and you could call it a false positive. But a false positive is certainly not going to result in a mastectomy without going through a doctor.
Comment #35 December 12th, 2013 at 4:59 pm
Scott #29: Given that you answer a lot of questions quickly, in addition to having a real job, you obviously don’t have much time for rethinking and fine tuning postings; you might request a do-over on some of your points in #29.
Consider the problem of the medical profession communicating with the general public. The problem is similar to if you had to teach C programming to an group of people, few of whom are aware that * has higher precedence than +. You would not find yourself talking about complexity classes.
I find that most medical professionals are happy to learn that I am a scientist and that they can upgrade the level of discourse.
My estimate for conventional medicine as only about 15% voodoo (not counting psychiatry, of course). The much bigger problem (in the US) is how various cartels are entrenched in the system. They rake in huge $$ with little or no checks, balances, or competition. Thus we have $2E1 aspirins, $1.2E4 shots, $3E3 stitches, and probably worse. Most people cannot afford it.
Comment #36 December 12th, 2013 at 5:04 pm
Douglas #31: Thanks, good info, well summarized.
Comment #37 December 13th, 2013 at 12:23 am
Scott says:
“But as far as I know, there’s no evidence so far that even a single person was harmed. Which isn’t so surprising: after all, people might run to their doctor terrified about something they learned on 23onMe, but no sane doctor would ever make a decision solely on that basis, without ordering further tests.”
Here’s a thought experiment, say someone started a firm claiming they had a super-duper algorithm to scan you and very accurately tell if you’d die of an obesity induced heart attack. They charged $2000 a scan and then just gave everyone colorful authoritative looking charts and told everyone they had a super high risk.
Assume somehow they do this discreetly enough that people don’t talk among themselves and no one’s wiser to their scam.
Now medical no harm was done clearly. Probably significant good if you scared all those people into exercising a lot.
Would we or should we tolerate this fraud? Harm isn’t the only metric here. Preventing fraud is another.
Comment #38 December 13th, 2013 at 7:38 am
Rahul #37:
Harm isn’t the only metric here. Preventing fraud is another.
Completely agreed, but I don’t think there’s any evidence of fraud either…
Comment #39 December 13th, 2013 at 7:40 am
Wait. So for this genetic analysis, you don’t have to go to their lab personally and have then prick you with a needle, instead you just send in some spit in a jar? I’ll go and scrape some hair of my employees from the chairs then, send them for analysis, and fire anyone with bad health results.
Comment #40 December 13th, 2013 at 9:59 am
jonas #39: Well, they don’t do it from hair, so you’d have to collect their spit somehow. And I’m pretty sure you’d violate some law along the way. 🙂
Comment #41 December 13th, 2013 at 1:02 pm
@Douglas Knight #27:
Why my impression is that 23andMe is intentionally antagonizing FDA:
(1) FDA has not received any communication from 23andMe since May 2013
(2) The FDA references 14 face-to-face meetings and telephone calls, hundreds of emails and dozens of written communications during which the agency said it provided specific feedback on study protocols that would prove 23andMe’s tests were useful as diagnostics and other tests to prove the 23andMe kits gave consistent results.
(3) In January, the FDA says the company promised that it was doing extensive studies that would take months to complete – and then that promised data never materialized.
(4) The FDA’s account seems to show the agency giving 23andMe a lot of chances, and to show a willingness to find a way to work with the startup.
Source:
http://www.forbes.com/sites/matthewherper/2013/11/25/23andstupid-is-23andme-self-destructing/
Is any of this factually wrong?
I like how the Forbes article sums this up:
This is not the story of a big regulator choosing to squash a small company, but of a company that decided that it didn’t have to follow the rules.
Comment #42 December 13th, 2013 at 3:29 pm
Sorry, you are right. You are correct in your original point, which matches point (3) in the recent comment, that 23 failed to deliver data it promised. You are also correct to single out that point.
I had attempted to read the FDA letter which emphasized (1) and (2), which seem to me like small potatoes (or at least lacking context); but I missed (3). I reject the interpretation (4) and the interpretation that 23 is trying to antagonize FDA. Do you really think Herper thinks they are trying to antagonize FDA?
Comment #43 December 13th, 2013 at 5:22 pm
Douglas Knight, About point (4) above.
I doubt they were TRYING to antagonize the FDA. More likely they thought they could ignore the FDA, or at least did not put much resources into jumping through the FDA’s hoops in correct order. Anyone with a Ph.D. knows the importance of jumping through the hoops, some of which are meaningful.
Comment #44 December 14th, 2013 at 2:03 am
Scott #38:
“Completely agreed, but I don’t think there’s any evidence of fraud either…”
No, not fraud yet, I agree. But there seems evidence of evasiveness & stonewalling.
We, as a society, give FDA the privilege of examining medical interventions before they get to the market. So anyone within that domain trying to circumvent or evade that oversight might cause some legitimate concern, yes?
A watchdog might harbor some presumption of fraud in such cases?
Comment #45 December 14th, 2013 at 8:09 am
Regrettably but foreseeably, history provides plenty of examples of proven corporate fraud amounting to global-scale criminal conspiracy, and even more commonly (in both individuals and corporations), sustained willful ignorance of lack of efficacy.
Example Semiconductor manufacturers commonly sell high-quality batches of chips to elite customers at a premium price; while selling low-quality batches of unreliable chips to third-world customers at a discount.
What obstructs gene-chip manufacturers from selling high-quality batches of genechips to elite customers at a premium price; while selling low-quality batches of unreliable genechips to individual customers … also at a premium price?
That would be a *terrific* high-profit business model … exploiting market inefficiency at its best! … and how would individual customers ever know?
Conclusion At a minimum, 23andMe appears to have significantly underestimated the challenge of verifying, validating, and documenting that their genomic products provide high-quality information in regard to health-care.
Comment #46 December 16th, 2013 at 11:42 am
As near as I can tell, no one has presented evidence contra Scott’s related claims:
(2) FDA does not have a policy covering personal genome devices
(1) 23andme has not violated FDA policies covering personal genome devices. If (1) is true, (2) is true.
Also, genetic correlation is a fairly “low-impact” feature for a medical device to have. I could understand the FDA having a problem with 23andme if the personal genome devices came bundled with self-mastectomy kits. But this seems like a reflexive anti-innovation / attention-grabbing response to me.
Comment #47 December 16th, 2013 at 5:14 pm
The National Human Genome Research Institute hosts a diverse body of scientific, legal, economic, and ethical resources will leaven anyone‘s opinion that these issues are simple.
Comment #48 December 18th, 2013 at 12:36 pm
@marris:
Fine. Then why won’t 23andMe take that stand. Let them say outright to FDA, look guys you have no authority over this.
Comment #49 December 30th, 2013 at 1:38 pm
There’s a good article in the NY Times today (well, tomorrow) on SNP-based genetic testing: http://www.nytimes.com/2013/12/31/science/i-had-my-dna-picture-taken-with-varying-results.html
A few quotes:
“23andMe said my most elevated risks — about double the average — were for psoriasis and rheumatoid arthritis, with my lifetime odds of getting the diseases at 20.2 percent and 8.2 percent. But according to Genetic Testing Laboratories, my lowest risks were for — you guessed it — psoriasis (2 percent) and rheumatoid arthritis (2.6 percent).”
“The major issue, experts say, is that the causes of most common diseases remain unknown. Genes account for just 5 to 20 percent of the whole picture.”
“There are only 23 diseases that start in adulthood, can be treated, and for which highly predictive tests exist. All are rare, with hereditary breast cancer the most common.”
Comment #50 January 13th, 2014 at 4:06 pm
[…] A similar discussion arose recently in the FDA induced shutdown of 23andMe. Lior Pachter started a heated discussion on his blog about why the findings of of 23andMe are unreliable. In short: Too many hypotheses evaluated against a single genome. Meanwhile, Scott Aaronson asserts his right to misinterpret probabilities. […]
Comment #51 February 17th, 2014 at 9:44 pm
I don’t see what’s so confusing. The concern is that the 23andme results would cause:
1. Irrational decisions, which anyone can make worrying about a possible serious illness no matter how smart they are.
2. Events the doctor and/or patient might regret.
3. Legal repercussions that might bilk or illogically sanction US health care.
Comment #52 July 22nd, 2015 at 2:26 am
23andme is in the news again:
http://science.slashdot.org/story/15/07/22/0146236/genetic-access-control-code-uses-23andme-dna-data-for-internet-racism
Apparently there is a Github project to use 23andme genetic data for racial profiling. Terrific :(.