Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.

I post quite a bit about one of my favorite AI companies, Pangram. I think Pangram makes an amazing product: they’ve got the only working AI text detector, and they’re also quick to update it in response to malicious text humanizers that crop up. I trust the Pangram team and I know their product is real and works as it’s says on the tin. Why, then, do so many writers seem to disagree?

If you read about Pangram, you’ll quickly see a bunch of people claiming that it doesn’t work. Worse, they’ll sometimes claim that it’s biased. Pangram allegedly discriminates against autistic authors, against anti-vaccine authors, against people I’d politely describe as ‘conspiracy theorists’ on both sides of the aisle, and, most prominently, against African writers. How? Well, it supposedly flags their work as AI!

There are a lot of articles about Pangram and other AI detectors ‘mistakenly’ flagging Black and African authors as using AI to do or assist with their writing. The most prominent piece that comes to mind for me was published in the Pan African Review. The piece was an unusual breakout for the Review: the journal generally enjoys a pretty limited circulation. I suspect this is because of the import of the topic area and the utility of providing an example to point to when people want to explain why they were ‘wrongly’ flagged as using AI for writing. So, I’ll consider the perspective of this piece to be emblematic of a community’s wider concerns given how much it resonated.

The piece is authored by one Timi Olagunju. Olagunju is “a lawyer and public policy expert working at the intersection of tech, governance, and development.” His claim is that writing in formal English will be flagged as AI-generated by AI text detector:

Consider an international student in the United States. She has spent years being told that her English must be grammatically correct. She avoids slang, checks every verb and structures each paragraph carefully. Then a detector labels her work artificial because it is controlled and predictable. First, the institution tells her to master standard English through IELTS and TOEFL. Then it treats mastery as suspicious.

This allegedly explains why “African (or black) writers” are liable to have their work flagged by AI detectors:

For many African writers like Falade, English is both a language and an inheritance shaped by colonial history. In Nigeria, Kenya, Ghana and other former British colonies, generations of students were taught forms of English influenced by British education. Teachers emphasized correctness, formal composition and structure.

Is this the case? We can test this right away by inputting Olagunju’s own article into Pangram and… it receives a 100% AI score with high confidence. Surely he wouldn’t use AI to write, right? Let’s consider his thesis in good faith and perish the thought—for now. Surely the other writers in the Pan African Review wouldn’t use AI to write either, right?

To test the extent of AI adoption in the Pan African Review, I looked at how much Pangram flagged articles there over time. Unfortunately, the site is currently down, so I had to extract whatever articles I could from the Internet Archive.¹ What we see right away is a total rejection of Olagunju’s thesis: articles were never flagged as AI-generated before the advent of LLMs, and the share flagged as AI-written has rapidly creeped up afterwards. This is not what we expect if African writing generally is to be considered liable to be flagged as AI.

Another test of Olagunju’s thesis is to simply look at his writing. He wouldn’t lie about this stuff, would he? All of his past work before LLMs came out should return a 100% AI score with at least moderate confidence. Alas, quite literally none of it that I could find does. All of it shows up as human-written. It’s only after LLMs came out that his work started being flagged as AI-generated.

But maybe we’re at fault here. Maybe Pangram trained on the Pan African Review and Olagunju’s work and they flag pre-LLM work as AI-generated because it’s in the training set. To get at this question, I asked Pangram CEO Max Spero if that was the case: was “Pangram trained on the articles in the Pan African Review?” “It was not.”

Pangram doesn’t discriminate—not against Africans, not against the disabled, not against people who think mRNA vaccines contain nanobots. Pangram detects AI.²

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

2

Let those who disagree bring forward evidence like I did. To check all the Pangram results I ran, open up this PDF generated by our friend DeepSeek V4.1 Flash:

	Pangrammed 244KB ∙ PDF file
Download

If you like my writing, consider getting a paid subscription. Every paid subscription keeps the blog active, and the more paid subscriptions I have, the more active I can be.

Paid subscribers can suggest future content, I’ll make sure to read their comments, and they can communicate with me via direct message. If any of that sounds interesting to you, or you just want to support the blog, enjoy this coupon for 15% off a subscription by clicking this button (or this link):

Get 15% off for 1 year

Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.

It’s widely understood that anti-vaccine activism is predicated on illiteracy, conspiracist ideation, and a resistance to updating in response to empirical findings, but it’s less well-known that anti-vaccine activists (anti-vaxxers, colloquially) are also generally innumerate and—despite their invocation of drawn-out arguments, biological mechanisms, and chains of causation—they don’t think through the numbers they like to cite. The evidence of this is everywhere in their arguments, but nowhere is it more distinct than when they try to invoke statistics in support of anecdotes. Consider the McDowell triplets.

The story goes:

All three of Brenda McDowell's triplets, she attests, 'shut down' within hours of each other after receiving their routine pneumococcal vaccines — later to be diagnosed with Autism Spectrum Disorder.

Alternatively:

Healthy triplets. One pneumococcal vaccine. One single day. Within hours, all three babies changed forever.

Claire screamed uncontrollably… then went completely silent.

Her brothers Richie and Robbie lost their smiles, their sounds, their milestones.

All three diagnosed with severe autism. Only one has ever partially recovered.

And on the odds, anti-vaccine activists have provided three estimates. For example, “A geneticist explained that the chance to this happening to 2 of our triplets would be 1 in 4 million. It happened to all 3.” Alternatively, anti-vaccine activist Steve Kirsch has proffered two probabilities: “The chance of that happening if autism is happens randomly to kids is less than 1.66e-13” and, from the same person:

To see 3 (or more) events in a specific 24-hour window when you expected to see 3e-5 events is:

>>> poisson.sf(2,3e-5)
4.499898751214992e-15

We’d want to multiple this by 1,000 since it could happen on any of the days for the first child. This is still orders of magnitude from ever happening.

All of this might sound convincing if you ignore the discrepancies¹ and also don’t think about it for even a few seconds. But if you put any amount of thought into this, it falls apart. It obviously does not demonstrate any risk from vaccines. Think about it: if this is true, and vaccination caused all of these children to suddenly become autistic, what would the implied effect of vaccination be on the odds of autism? We can calculate that!²

If 90% of people get vaccinated and the prevalence of autism is 1%, the implied odds ratio is 4.89³. Moreover, the rate of autism among the unvaccinated should be 0.22% versus 1.09% for the vaccinated, meaning that 77.6% of all autism is directly caused by vaccines, and in particular, by the vaccine these children received, the pneumococcal conjugate vaccine (PCV). Is this prediction even remotely plausible? No.

We know that PCV is unrelated to autism in individual-level data—a clear prediction failure—and that autism doesn’t spike in the years after the vaccine is introduced—a causally-informative prediction failure!⁴ Take a look:

Where does this leave the anecdote? Well, it’s obviously wrong and the extreme numbers cited by proponents of it must be off. We know this is the case, because it must be the case given the observed numbers. For it to be any other way, the autism rate would have to be enormously understated or the vaccination rate overstated or… something else that’s obviously not happening.

What’s really happening in this anecdote is that autism runs in families—it’s highly heritable—to the tune of younger siblings of autistic children having a 10-20% chance of having autism themselves (this study estimated 18.7%) and the timing of vaccination coincides with roughly the time when many parents notice their kids are autistic. Across millions of vaccinated families, large numbers of parents will begin to notice their kids’ autism around the time they get shots, and if they’re all getting their appointments together like these triplets were, then a coincidence of diagnostic timing is to be expected.

The story of the McDowell family relies on vividness to make it seem like something is amiss and vaccines are to blame, but it’s statistically quite probable. In fact, we can just work through Kirsch’s spectacular numbers to get at why we do expect to see examples like this. Here’s a diagram:

From left to right, we have estimates from different scenarios. On the left, you have Kirsch’s implausibly small probability estimate that he—as we’ve shown, improperly—believes establishes that vaccines cause autism. This estimate comes from his tweet and it’s based on taking an autism risk of 1 per 100 by age 4 in 2007 with onset somewhere between ages 1 and 4 (i.e., 1,095 days), noting three kids means 3e-5 expected cases per day, and then computing the Poisson chance of three or more events on one day as poisson.sf(2, 3e-5) and multiplying by 1,000 for “any of the days”, giving roughly 4.5e-12, but using the per-child rate, giving 1.66e-13. This estimate assumes children are independent, that autism begins on a specific, datable day, that every day in the window is equally likely and dated separately per child, and that nobody went looking for the family—all incorrect.

The next estimate to the right simply takes his inputs and computes the number based on his assumptions correctly. The assumptions remain wrong; the point, however, is to show that an implausibly small number still follows from them even when you do the calculation correctly. But what happens when we start correcting the erroneous assumptions? The next estimate removes independence: given the first child is autistic, the next child has an 18.7% chance of also being autistic. From the same study, we also see that, given two siblings are affected, the risk for the third one is 32%. Incorporating this alone raises the probability by about 600 times, and this understates how much the probability should improve, because triplets also share a womb and birth timing.⁵

The number to use isn’t 1/1,095²—”three independent onset days”—, but q, the chance that parents tie three events to one visit. This means, given three autistic, same-age siblings vaccinated together, we expect this outcome to happen in approximately 1-in-6,700 triplet sets. We know this for a few reasons: autism doesn’t begin on a single day (it’s developmental); triplets share developmental timing and appointments; and parents date the onset of autism after diagnosis and anchor to memorable events like well visits. In other words, diagnoses are not independent draws!⁶

Finally, we have the probability of at least one family like the McDowells. This is based on an extremely conservative calculation taking 30,000 U.S. triplet sets born between 2000 and 2020⁷ and multiplying by the previous probability of 1.5e-4. P(at least one) = 1 - (1 - p)ᴺ ≈ 1 - e⁽ᴺᵖ⁾ ≈ 0.99 (0.83 to >0.999). Even this is conservative, as it assumes families are independent, it leaves out quadruplets, other countries, and it omits both earlier and later years.

We can confidently state that this anecdote is uninformative because a case like this is expected by chance. At most it could move the odds very slightly. Once we also acknowledge that its implied effects aren’t realized in the general population data, we’re left with the realization that this whole thing was pointless to bring up.⁸

This is All They Have

This sort of poor-quality evidence stated without any caveats or rigorous interrogation is all that anti-vaccine activists have to go off. They cite lurid anecdotes and don’t check their implications; they get confused by confounded and selected associations; they fail to recognize the role of institutions in driving reporting on disease; and they very generally do not understand the conditions they’re talking about. To end on an example, consider the fact that the apparent ‘rise’ in autism is entirely due to diagnostic drift. This is well-established, and yet anti-vaccine activists try to explain it: they blame it on vaccines, even though there’s nothing to explain!

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

implied_or <- function(prev = 0.01, cov = 0.90, n_kids = 3, thr = 0.5) {
  rr <- 1 / (1 - thr^(1 / n_kids))        # RR at which P(all n cases vaccine-caused) = thr
  p0 <- prev / (cov * rr + 1 - cov)       # unvaccinated risk, holding overall prevalence fixed
  p1 <- rr * p0                           # vaccinated risk
  data.frame(coverage   = cov,
             RR         = rr,
             OR         = (p1 / (1 - p1)) / (p0 / (1 - p0)),
             risk_unvax = p0,
             risk_vax   = p1,
             PAF        = 1 - p0 / prev,          # share of all autism caused by vaccines
             excess_pp  = 100 * (prev - p0),      # percentage points of the overall rate
             max_BF     = (1 / cov)^n_kids)       # most the anecdote can favour any effect over none
}
rbind(implied_or(cov = 0.90), implied_or(cov = 0.92))

Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.

On June 29, 2023, the Supreme Court declared by a 6-2 vote that American universities that accept federal funds were no longer allowed to racially discriminate when making admissions decisions. The case—SFFA v. Harvard—concerned the fact that Harvard was making decisions using students’ race as a variable in admissions.

Advocates on the side of Harvard argued that racial discrimination was necessary practice to preserve diversity. Opponents argued that racial discrimination was wrong and illegal. The opponents are best described as ‘correct’; but the advocates do also have a point, albeit one that should only be responded to with ‘So what?’: racially-fair admissions should undermine diversity, because certain groups—namely, Asians and Whites relative to Blacks and Hispanics in the U.S.—tend to have higher levels of academic achievement.

So, three years on, has the verdict in SFFA v. Harvard resulted in universities becoming less ‘diverse’ and more reflective of what a fairly-admitted student body would look like? The answer is ‘partly’, not entirely. A recent article in Econ Journal Watch looked into this and reached just that conclusion. But why? From the changes or lack thereof alone it’s hard to tell. But the most likely possibility is that universities are still discriminating even though the Supreme Court has declared doing that to be illegal.

The Department of Justice has been looking into this issue using the methods I outlined when I showed that Columbia and NYU were still discriminating. They’ve recently opened cases against UCLA’s medical school, Yale’s medical school, UC Davis’ medical school, UC San Diego’s medical school, Duke Law School, George Washington’s medical school, UC Berkeley Law School, UC San Francisco’s medical school, and UCLA Law School. They’ve also sued Harvard for withholding race-related admissions documents, run broader compliance reviews at Stanford, Berkeley, UCLA, and UC Irvine, and began a broader investigation into fifteen medical schools.

Every university the DOJ has opened an investigation into so far is almost-certainly discriminating. We know this because each of the universities shows a considerably higher odds of admissions for students from favored racial groups—Blacks and Hispanics—as opposed to the ones that universities have historically wanted to discriminate against—Whites and Asians:

• Berkeley gives Blacks ~6-times higher odds of admissions at the same LSAT and undergraduate GPA (UGPA) levels.

• UCLA Law gives Blacks 7-times higher odds of admission and Hispanics 2-times higher odds of admission than Whites.

• Duke gives Blacks 3.5-times higher odds of admission than Asians at the same LSAT and UGPA levels.

• Yale invites Blacks as much as 29-times more often for interviews than it invites similarly qualified Asians.

• UCSF is 12.6- and 4.6-time as likely to admit Blacks and Hispanics, respectively, as it is to admit Whites with comparable MCAT scores, UGPAs, and socioeconomic characteristics.

• UC San Diego was nearly 3-times as likely to admit Blacks as Whites.

• UCLA Medicine admits much less qualified Blacks and Hispanics than Whites and Asians.

• George Washington Medicine admitted less qualified Black and Hispanic applicants than applying Whites and Asians.

There are only two ways to obtain these patterns of results:

1. Discrimination; or,

2. There are compensatory factors that make less academically qualified Blacks and Hispanics as worthy of admissions as their more academically qualified White and Asian peers.

Proponents of racial discrimination frequently argue in favor of the latter idea. But it’s absurd. For one, they’re forced to argue for an incredible coincidence: that admissions departments are making the same sorts of decisions without explicitly using race that they made when we know they explicitly used race. People are free to argue that, but it’s obviously unlikely given that there are no such factors that reliably proxy race in the way they’d need. They also don’t have any room to stand on the idea that there are compensatory factors generally. The reason for this is simple: admissions departments don’t collect information about factors that are plausibly compensatory.

A valid non-discrimination explanation for universities continuing to admit student populations that look demographically like their pre-SFFA ones requires a few things:

1. It needs to use variables that are available to admissions departments: whatever admissions decisions are based on must be collected and available before the admissions decision;

2. Variables need to be acceptably measured: they should be assessed consistently across races, without functioning as a racial preference or identity substitute;

3. They need to show the necessary racial disparity: they should favor the group receiving higher admissions odds, conditional on tests, grades, and other academic achievement-related controls;

4. They need to be strong enough: the conditional advantage the variables provide should, when combined with their actual admissions weight, explain the gap.

And though this isn’t required, to be considered valuable:

5. They need incremental predictive value: they should predict relevant outcomes like student GPAs, course completion, in-class behavior, etc. beyond the purely academic criteria they’re being weighted against.

Every college the DOJ is investigating is an open-and-shut case because none of them has an admissions department that gathered data on variables fitting these criteria. We know what they collect because it’s public; we know how the differences in what they collect tend to vary across groups; and we know how much predictive value they have in the general population. There’s no explanation besides discrimination.

Take Berkeley Law as an example. Their admissions department collects 2-4 letters of recommendation, a personal statement, a résumé, and they don’t have a general interview to speak of. Each of these things tends to favor White and Asian students, which means that if they were used, they would actually make it harder for Berkeley to make its case unless they were an outlier school in terms of how these things correlate with grades, test scores, and demographics. To make things worse, most of what Berkeley Law collects data on is unreliable. In Kuncel et al.’s meta-analysis of letters of recommendation, for example, they found that to achieve a reliability of 0.80—roughly the bare-minimum for how reliable something should be—students would need to submit six letters, but Berkeley Law collects at most 4 per student

Unlike test scores, the variables Berkeley Law collects are also usually racially biased, and in any case, they can be easily used to infer student race even if admissions departments never explicitly ask for students to report their race. For good measure, the Supreme Court ruled that sort of proxying illegal, but—and this is part of the evidence—Berkeley Law has decided to engage in this practice anyway.

In the case of Berkeley Law, the DOJ noted that the school’s dean Erwin Chemerinsky explicitly and publicly discussed replacing explicit racial preferences with proxies and presenting such a policy with race-neutral justifications in order to keep discriminating. Communications within the admissions department were uncovered and they explicitly called for maintaining diversity despite litigation risk. Application questions created by these people and their coworkers also expressly invite students to disclose their race. Finally, to make things clearer with some contextual evidence, Chemerinsky explicitly discussed concealing racial considerations in faculty hiring, suggesting that he wants racial discrimination in multiple areas, not just admissions.

Evidence like this was present in every case the DOJ has opened so far, from Duke Law to George Washington’s medical school. The DOJ is not resting on the strong statistical evidence that universities are discriminating; they’re also showing that we know what they’re doing. And the truth is, they know that we know what they’re doing, too. They discuss it openly! The fact that we know that they know that we know what they’re doing should make these cases easy to prosecute¹, but only time will tell.

It’s clear now that the universities cannot be trusted. The documentation from the universities themselves indicates beyond any reasonable doubt that they have continued engaging in racial discrimination. In several cases, they’ve been caught planning to do so, admitting to were doing so, hinting at it, and so on. In each case, it’s clear: making racial discrimination illegal was insufficient. We must go further.

Universities must be compelled to affirmatively justify their admissions decisions. They need to publicly provide the data and the model used to select students, and they need to tell us how it works—explicitly—and then show us—verifiably—that they used only the information provided on students’ applications, without proxying race. If they find that to be a too great a burden, then we only have one course of action: defund them. Ban their researchers from grants, ban their students from public loans, take away their tax benefits and accreditations and confiscate their endowments until they become compliant or voluntarily cut themselves off from the public teat. The universities will only learn if it hurts.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.

Growing up, it was a common trope that cultures that we now associate with poverty, violence, and oppression were once world-leading in promoting prosperity and moral progress as well as the quintessentially human act of inventing. This popular belief was ultimately not driven by an earnest desire to promote awareness of empirically true but neglected facts. Instead, it was supported by anecdotes and quotations of dubious reliability and relevance, by scholarship that has turned out to be exaggerated or fraudulent, and by the promotion of these things in popular media.

My favorite example of this has to do with the invention of many of the inventions credited to the so-called Islamic Golden Age by a rogue Wikipedia editor.

Have you ever heard the name “Jagged_85”? If you’re a history buff, you’ve probably at least heard of some of his claims. For example:

• Muslims invented funnels and sieves

• Al-Razi invented dissolving substances and sublimation

• Jabir ibn Hayyan invented cheese glue

• Ibn Sahl first described the parabolic mirror

• An Islamic water clock from the 12th century had the earliest copper pipes

• Avicenna formulated momentum as weight times velocity

• The Brethren of Purity proposed a heliocentric universe in the 10th century

• Darwin drew on Ibn Miskawayh’s work to develop the theory of evolution

• Al-Ghazali directly influenced Descartes’ later philosophical work

There are many other claims I could add to this list: Taqi al-Din invented the telescope sometime in the 1570s; Avicenna discovered microorganisms; Muhammad ibn Musa discovered gravity between planets; Al-Biruni discovered elliptical planetary orbits; Al-Jazari invented flow regulators and feedback control; Raqqa had a major glass factory where 300 different glass recipes were used, or—most amazingly—Al-Battani discovered relativity a millennium before Einstein.

Jagged_85 vandalized thousands of articles while making over 87,000 Wikipedia edits in the time between starting in 2005 and being caught in 2012. Those edits have led to hundreds of false claims misattributed to history books and papers, and now, those claims are alive on the internet, being repeated carelessly and promoted as if they’re true, when they are anything but. Several of these claims have even seeped into the scholarly literature, and now they’ve gained some credence as true facts by virtue of being repeated by scholars misled by what they read in Jagged_85’s Wikipedia articles!

How Do False Claims Make it to Wikipedia?

The way a Jagged_85 claim works is straightforward. He rarely ever fully fabricated claims and they always had some sort of source document attached to them, from which you could usually sort-of see what he was getting at. As an example, take the claim that Muslims invented funnels and sieves between 721 and 925. Jagged_85 did cite a source, it just didn’t say what it was cited for. What it actually provided readers was a list of laboratory equipment used by Muslim scholars, among which one could find funnels and sieves. The crucial issue, however, is that it never said they invented them, and in fact, it showed that they had been in use since Antiquity by Roman and Greek scholars. That last bit went unmentioned in places where the work was cited.

What about the claim that Al-Razi invented dissolving substances and discovered sublimation? There was a source cited for this claim too, but what it said was that Al-Razi described their operations, not that he discovered them. That achievement was given to the Greeks. Jabir ibn Hayyan’s invention of cheese glue? Sourced to a document that said he translated a medieval recipe; other sources show us that adhesives of this variety were in use in ancient Egypt. Ibn Sahl inventing the parabolic mirror? This came from a source that mentioned plenty of earlier investigators. And funny enough, when Jagged_85 made that citation, he didn’t understand what he was talking about, as he called the mirror “refracting”, as opposed to ‘reflecting’.

The edits provided by Jagged_85 had every appearance of being scholarly, but the citations were nevertheless stretched and exaggerated, and sometimes contradicted the things they were cited to say. This was noticed in 2010 in a prominent case where where Jagged_85 invented the concept of Avicennian logic and then conflated Avicennism with that concept:

The article on 'Avicennism' says that "Avicennian logic had an influence on early medieval European logicians such as Albertus Magnus". Yet the idea that most influenced these scholars, including Albert, was Avicenna's distinction between essence and existence, which was a genuine innovation, and a departure from Aristotle. This idea had a profound impact, in different ways on Aquinas, Giles of Rome, Henry of Ghent, Scotus and eventually Ockham. But it is a metaphysical idea, not a logical one. As far as I am aware, no medieval writer discusses Avicenna's logic (as opposed to his commentary on the Metaphysics, which they frequently cite). Kneale's great work The Development of Logic mentions Avicenna twice, neither in connection with logic. Bochenski does not mention him at all. None of the articles in the main secondary sources on medieval philosophy mention Avicenna's logic (although they discuss his other achievements at length).

[This] proves a clear weakness in the Wikipedia 'verifiability principle'. [… Jagged_85 frequently used] scholarly works not available on the internet. Most of Wikipedia's editors are amateurs who have no access to a university library. Thus they cannot check a source from a journal, or an old or obscure book that would only be found in a library.

Despite this, if you look up Avicennian logic or Avicennism today, you can find distorted notions of what Avicennism is, and plenty of people talking about Avicennian logic, as if it’s a real thing, with attributions to the sources used by Jagged_85. But these attributions are false; the sources don’t support the claims scholars have made about them, and because they were miscited in the ways Jagged_85 cited them, we know they got their backing from his Wikipedia edits.

How Do False Claims Make it to Other Scholars?

Scholars are, in a word, lazy. But the reason for this isn’t some especial problem with scholars, more so, it’s a reflection of the fact that most humans are lazy. If you can read a Wikipedia summary to get your information, why not do that? It’s far easier and less complex than going to the secondary sources Wikipedia cites, and it’s easier yet than going to the primary sources those secondary sources cite. In some cases, it’s easier to use Wikipedia and to rip off its bibliography section than it is to access the relevant works. If you need a citation, that’s a commonly used route, unfortunately.

It’s unsurprising that so many scholars have parroted the confabulations Jagged_85 lied about.¹

The scholarly influence of these false claims hasn’t gone unnoticed by academics. There are papers and articles that have specifically noted the influence of Jagged_85 on literatures as wide-ranging as psychology and medicine. In the most bizarre cases, scholars get misrepresented by Jagged_85 and then update their own claims to support the misrepresentations.

Consider University of Manchester professor emeritus Salim Al-Hassani, the maintainer of the 1001 Inventions project and MuslimHeritage.com website. In an article dated 2004—well before the Jagged_85 editing spree began—he described the inner workings of the elephant clock, flow regulation, closed-loop operation, etc., but nowhere did he say these were the first appearances of the concepts. In 2007, Jagged_85 edited Wikipedia to claim that Muslims invented these things with the elephant clock, citing al-Hassani. Not long after, in 2008, al-Hassani published another article that made the same sorts of claims, with similar wording! These errors were then reprinted widely, including by businesses that still have the claims up today.

How Do False Claims Make it to You?

False claims laundered from Wikipedia make it into academic circles and further into the zeitgeist, because they inform people about things they can say and should believe. They go further than that because they are believed, and because, in these cases, they are plausible—by virtue of academic acceptance, Wikipedia presence, scholarly bona fides, etc.—and people want to believe them; it is fashionable to talk about an exaggerated Muslim Golden Age and era of invention. Muslims are newly being introduced into Western societies, so just as Christopher Columbus began to be celebrated to welcome Italians into America, we now find it worthwhile to talk about how Western scholars and historical geniuses were actually inspired by Muslims in ages long past, forgotten, and perhaps even intentionally obscured.

The most widely circulated false ideas achieve their prominence through becoming educational: they are now the sorts of facts people should know, not just academic obscurities. For example, Salim al-Hassani set up an exhibit at London’s Science Museum, and London journalist Olivia Sterns saw the exhibit and decided to write about it for CNN. The article included the false claim ultimately sourced from Jagged_85, that the crank was invented by Al-Jazari in the 12th century.

In a similarly poor and viral showing, Sean Carroll evidently used Wikipedia to try to figure out who had coined momentum, and in doing so, he quoted an error that originated with edits made by Jagged_85. He attributed the thing to Avicenna, using wording derived from Jagged_85. Avicenna, as it turns out, was miscited, and this wasn’t the first time. People might also have been exposed to Jagged_85 edits about Avicenna by way of Google Books descriptions, listings on sites like Booktopia and Goodreads, and so on, which have all featured improper descriptions of translations of Avicenna’s Canon of Medicine, claiming that the author—Avicenna—invented randomized controlled trials and risk-factor analysis.

What the Jagged_85 saga shows us is that with a little gumption, anyone can flood our shared information environment with garbage. They can have lies and exaggerations cited by thousands of scholars, educational programs and pamphlets, museum exhibits, journalists, and search engines alike, because no one checks, and people get lazy.

Who knows how many Jagged_85 edits have survived in the public consciousness, making people feel wrongly ‘informed’ about Islam and the Muslim world? Knowing just a few dozen of them, I can personally attest that I see them multiple times a year. I’m also occasionally privy to other famous Wikipedia-derived false claims like those about the electric toaster inventor Alan MacMasters—who never existed despite making it onto multiple BBC programs—, the ones about the nonexistent gas chambers in Warsaw (the real gas chambers were elsewhere), or the various articles on Chinese deportations and nonexistent silver mines in Russian history invented by the editor Zhemao. Who knows how much literature, how many TV programs, how many ‘fun facts’ are nothing more than the invention of some unsavory Wikipedian?

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.

What statistics belong in the Annals of Fake Statistics? Healthcare administration costs ballooning tens of times over in the past few years? Nearly 70,000 deaths a year due to lacking insurance coverage? You’re more dangerous to schizophrenics than they are to you? Men kill their spouses far more often than women do? (Although see.)

There are many candidates, but I would definitely tender the following fake facts:

1. Medical error is responsible for 251,000 American deaths a year, making it the country’s third leading cause of death;

2. Misdiagnosis kills 371,000 Americans per year;

3. Misdiagnosis leaves 424,000 Americans permanently disabled each year.

I recently saw these claims being made in a clip from an episode of the Joe Rogan Experience podcast. These claims are false, but they’ve permeated the discourse and found a way to survive despite their obvious absurdity; in fact, without proper prompting, several frontier LLMs will faithfully regurgitate these erroneous figures.

The Vigilant Fox 🦊 @VigilantFox Joe Rogan and Andrew Huberman look up how many Americans die from medical error each year, and the answer floors them both. ROGAN: “Okay, so medical error is responsible for 251,000 deaths a year.” HUBERMAN: “Whoa, that’s crazy.” ROGAN: “Do you understand how insane that is? … 8:18 PM · Sep 22, 2026 · 619K Views 585 Replies · 5.16K Reposts · 20.4K Likes

A Quarter-Million Medical Error Deaths?

This infamous estimate comes from a 2016 British Medical Journal publication by former FDA Commissioner Marty Makary and researcher Michael Daniel. The article achieves an estimate of 251,454 deaths from medical error each year by averaging four independent estimates of different quantities. I say ‘different quantities’, because none of the numbers being averaged was an estimate of the rate of death from medical error.

The first estimate comes from a 2004 estimate by HealthGrades. For Makary and Daniel’s purposes, this study looked at Medicare enrollees admitted to hospitals between 2000 and 2002 and provided an estimate of the number of deaths that could be flagged for investigation for “failure to rescue” and because the death took place in a member of a group that ordinarily has a low-mortality rate. Makary and Daniel then extrapolated the death count from this study forward based on the increase in the number of total hospitalizations by 2013, arriving at a figure of 251,454

The second estimate is taken from Classen et al.’s 2011 study on treatment-related adverse events at three tertiary care hospitals in 2004. This study involved counting up potentially treatment-related adverse events in hospitals, without evaluating whether those events were preventable or if a different pattern of care would have made them less severe. The study looked at 795 hospital admissions, found 9 fatal events, and from this, Makary and Daniel concluded that 1.13% of hospital admissions resulted in a preventable fatal event, which they chalked up to medical error and then extrapolated to total U.S. hospital admissions in 2013. But in addition to being based on an unwarranted assumption of preventability, this number is explicitly wrong and the original article’s count was corrected from 9 to 8 in a 2011 errata that Makary and Daniel should’ve seen. Using this number would drop the count from Makary and Daniel’s 400,201 to 357,502 with rounding and 356,388 without it, dropping estimated medical error deaths to 240,829 (i.e., roughly 10,000 fewer deaths).

The third estimate comes from a 2010 HHS Office of Inspector General report on a small number of Medicare beneficiaries. The report reviewed adverse events and recorded 12 fatalities among 838 events, with Makary and Daniel choosing to use the study’s 44% preventability number for adverse and temporary harm events to adjust this to an estimate of 219,579 medical error-attributable deaths. The study actually said 5 of 12 deaths were clearly preventable and 1 was likely preventable, so they should’ve chosen a rate that was either slightly higher (6/12) or slightly lower (5/12), for estimated death counts of 253,575 or 211,313. But this is a curious calculation, since it assumes the fatal error rate is the same in Medicare and non-Medicare populations, implying that, against an all-cause mortality benchmark of 3.56% for Medicare and 1.01% for non-Medicare, the fraction of deaths attributable to error is 17% versus 62% for non-Medicare, and if we bring that forward, the error-attributable fraction should have fallen for compositional reasons, contrary to their calculations.

The final estimate comes from Landrigan et al.’s 2010 estimation of temporal trends in patient harm in 10 North Carolina hospitals. The authors of this study identified 14 fatality-contributing events across 2,341 hospital admissions, judging 9 of these to be preventable. From this, they got 134,581 medical error-attributable deaths, even though that doesn’t follow from the study’s fatality-related numbers. This is because, instead of using the rate based on the 9/14 preventable events, they used the study’s overall event preventability, which included non-fatal events. The real estimate—assuming we can generalize from this sample, which we cannot—would be 136,157.

Can you grok what’s going on here? Makary and Daniels took four different rates, from four populations that weren’t nationally-representative, generally with very imprecise estimates, understood that their derived rates had to be caveated but despite that, only caveated two of four estimates, then extrapolated each rate forward in time, and called the mean of those numbers the medical error-attributable death rate when not even one of those was the medical error death rate in its study! They then stated that this was likely an underestimate because “the studies cited rely on errors extractable in documented health records and include only inpatient deaths.”

I’m going to note a few concrete issues with this. I’ll start with the claimed sources of underestimation that Makary and Daniels suggested. The first one, about errors, is almost-certainly wrong and they got the bias backwards. There is substantial empirical work on this. For example, Hayward et al. reported that, absent a large number of reviewers, rates of preventable death in U.S. hospitals tend to be overestimated. The low reliability does not result from classical error, it results in cases of non-preventable death being wrongly classified as preventable based on unreliable judgment. For two, there’s much more harm from care not delivered than from harm delivered, as out-of-hospital cardiovascular event, violent death, (etc. etc. etc.) rates make very obvious, unless we assume that inpatient deaths are actually far rarer than Makary and Daniels suggested, and medical error-attributable deaths outside of hospitals, very common. (This is inherently absurd given how little medical care occurs outside of hospitals and especially so given how little can be linked in any way to subsequent fatalities, as well as how low the post-discharge fatality rate is.)

Makary and Daniels’ underreporting objection is also silly on its face because the studies they cited explicitly dealt with it. They are correct that underreporting to incident systems does occur, but the studies they cited dealt with research reviews. The OIG explicitly found that 14% of its harm events were identified by hospital incident systems, but you can’t then extrapolate upwards, because the OIG’s estimate is based on the full set of identified events, not just the 14%.

Makary and Daniels also should not have extrapolated high medical error-attributable fatality rates forward like they did. The reason why this is wrong is because rates have almost-certainly fallen over time. Using just studies Makary and Daniels should’ve been familiar with, we know, for example, that AHRQ recorded a 17% decline in hospital-acquired conditions between 2010 and 2013, Wang et al. recorded a 4.7% annual decline in adverse events per 1,000 hospitalizations between 2005 and 2011, and their cited Landrigan et al. study discussed possible preventable harm rate declines, although it didn’t find any in its own nationally-non-representative sample. If you were to extrapolate from Makary and Daniels’ studies with improvements in care in mind and with consistent deflation of estimates to account for preventability, then medical error deaths would drop somewhere between COPD and suicides to the fourth-leading cause of death. And Makary and Daniels should’ve known this!

The most fundamental issue with Makary and Daniels’ estimate is just that if we actually try to estimate the medical error-attributable fatality rate, it’s clearly much lower than what they estimated.¹

The best estimate comes from Hayward et al. This estimate was based on a review of medical records produced between 1995 and 1996 across 7 different VA medical centers. The records reviewed were oversampled for conditions with high rates of preventability. This study doesn’t have a particularly good sample, but it does have particularly good handling of its data: the authors generated data with 14 separate reviewers, standardized reviewing instruments, and then they modeled the effects of unreliability and skewness. The estimate the authors achieved was that 6% of patients would’ve left the hospital alive had they been provided the best possible care, but with 3-month prognoses and review skew taken into account, “only 0.5% of patients who died would have lived 3 months or more in good cognitive health if care had been optimal”. If we call it 2-3 preventable deaths per 10,000 admissions, the annual death count probably clocks in under 10,000, and that’s with the 1990s’ high error rate.

Other studies have provided estimates that are more relevant to medically attributable deaths than Makary and Daniels’. Rhee et al., for example, reported that, across 6 U.S. hospitals in 2014-15, just 3.7% of sepsis-associated deaths were definitely or moderately likely to be preventable. Tatara et al. found that 8.6% of deaths from non-ventilator hospital-acquired pneumonia across 4 hospitals 2016-21 were likely or very likely preventable. Bates et al. reviewed 2,809 admissions across 11 hospitals in Massachusetts and concluded that of 7 fatal adverse events, just 1 was preventable.

The most defensible recent estimate that I can come up with comes from applying Rodwin et al.’s 2020 systematic review estimate—3.1% preventability—with the AHRQ’s estimate of 769,460 inpatient deaths in 2023. This combination leads to an estimated 23,853 medical error-attributable deaths (~17-32k). If you were to apply Bates et al.’s numbers to the AHRQ’s, you’d end up at 12,004 instead. This discrepancy should give some indication of how weak estimates are, given the combination of rare events and unreliable classifications.

In short, Makary and Daniels did not produce estimates that hold up to scrutiny. They took unrepresentative and extreme samples, irrelevant estimates, figures that were later revised, numbers they—conveniently—knew had to be adjusted downwards instead of being kept as they were, and they produced an estimate that was too high. What’s worse is that the concepts they dealt with were not even coherent: to get even remotely close to their estimates, we must assume that a failure to provide optimal care is a “medical error” that realistically—not just ‘theoretically’—could’ve been prevented in an active care setting. Given the reality of hospitals, where there are not unlimited human or capital resources to get everything that could be right done correctly, the theoretical maximum estimates from ‘optimal’ care projections are still too high.

A Third of a Million Misdiagnosis Deaths?

The 371,000 deaths from misdiagnosis figure comes from Newman-Toker et al. Thsi study was a review of disease incidence data from 2012-14 that was paired with literature-derived error error and harm rates to project that each year, diagnostic errors led to 795,000 deaths or permanent disabilities in the U.S. Neither the error counts, nor the implications of the projection forward, or the split between disability and death are empirically supportable or interpretable as the authors did.

As already noted: conditions regarding error have gotten better with time and counts tend to be inflated by unreliability that’s typically unaddressed in studies. Now, more specifically, Newman-Toker et al. cited two studies—Singh et al. and Zwaan et al.—to support their conclusions about death and disability.

Singh et al. rated the potential severity of 190 diagnostic errors, and put 27 cases in the potentially fatal category—it did not report on 27 observed deaths. Most of Singh et al.’s diagnostic errors were identified through subsequent hospitalizations that were considered unexpected, meaning that—as Singh et al. noted—, the errors were likely more severe than errors generally. The estimate from Singh et al. cannot be paired with national disease counts to obtain a representative estimate of harms because it does not supply representative harm probabilities. Nor can Singh et al. be used to determine the split between error-associated deaths and disabilities, because it skews towards greater-than-expected harms to an unknown degree. Additionally, deaths following diagnosis are not necessarily deaths that a proper diagnosis would have prevented or slowed down at all.

Zwaan et al. reviewed 7,926 records sourced from 21 different Dutch hospitals, concerning admissions in 2004. They identified 80 diagnosis-related adverse events, with reviewer adjudication about the presence and preventability of adverse events, but not about their consequences. 29.1% of events were labeled as contributing to deaths, but it wasn’t clear if they were causally involved. In any case, this is an old estimate from a different country, with limited review reliability, being combined with a known-unrepresentative estimate to get a split between disabilities and deaths to somehow come together and provide us with an estimate of America’s misdiagnosis death rate.

Fundamentally, this estimate is not based on numbers that actually establish anything like the misdiagnosis-related death rate. That’s not what these sources can provide.

Half a Million Misdiagnosis Disabilities?

The figure of 424,000 permanent disabilities from misdiagnosis is the flip side to the 371,000 death figure. It is based on the same studies, and thus based on an exaggerated baseline count and an inaccurate split that leads us the Newman-Toker death and disability numbers from said count. Worse is that Singh et al. merely classified 36 of 190 diagnostic errors as potentially causing permanent damage; they did not find that 36 patients wound up with permanent disabilities due to misdiagnoses. Zwaan et al. also had a classification problem because their numbers had nothing to do with permanence: they were about disability at discharge!

The outcome that should have been measured here is new or worsened persistent disability attributable to a diagnostic error. That was not measured.²

Each of these mistaken estimates is an example of the core problem with what I’ve dubbed ‘attribution studies’. The way attribution studies work is by attempting to project the numbers of uncounted things—deaths, disability, displacements, bankruptcies, whatever—based on proxy inputs gathered from elsewhere. The problem is that if you control the inputs, you control the outputs, and if you want a given output, it’s not hard to get it. With each of these examples, the inputs were dubious, so the outputs were too, and our public discourse has been tainted as a result.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.

Thomas Jefferson is the well-known Founding Father who wrote the Declaration of Independence. Sally Hemings was his three-quarters White (‘quadroon’) slave and his wife’s half-sister. She was tasked with being the attendant to his daughter Martha.

In 1787, Martha took Sally on a trip to Paris, where Hemings’ slave status was not recognized, making her legally free. During that time, she stayed on as Jefferson’s servant, and over the course of their 26-month French sojourn while she was a free woman, Jefferson and Hemings began a romantic relationship. At the time, she was roughly 15 years old and Jefferson was a mid-40s widower.

When it came time to return to the States, Jefferson negotiated with Hemings: come home, return to slavery, and you’ll be afforded considerably leniency in work, and your kids will all be manumitted when they come of age. After returning to America, Hemings had four children and Jefferson and his estate eventually freed all of them.

This bit of history has caused considerable political consternation. There are those who want to tar Jefferson for this relationship for all manner of reasons. At the time, the main focus was on the relationship being miscegenation, and thus illicit for that reason. Nowadays, the main focus is on the relationship being necessarily coercive due to Hemings’ status as a slave. Regardless, there was and remains controversy.

Part of the controversy about the relationship between Jefferson and Hemings was about whether it even resulted in children. It’s certainly the case that Jefferson and his estate freed Hemings’ children, and it’s also the case that slave-owners disproportionately freed slaves who were their children. However, this on its own does not establish that Jefferson was the father. Jefferson may have simply kept his word and freed Hemings’ children regardless of parentage.

This perspective runs into several issues. For example, people at the time frequently attested to and mocked Jefferson for his involvement with Hemings, and they even claimed that Hemings’ kids had a striking resemblance to Jefferson. Moreover, we know from genetic tests that there are certainly many people alive today who are related to Jefferson.

In 1999, a genealogical project established that Jefferson’s Y-chromosome was passed down to the modern-day, establishing for many that he had fathered Hemings’ children. But this finding is not unambiguous. Two years later, the Thomas Jefferson Heritage Society published a report noting that the Y-chromosomal evidence was consistent with Hemings also being impregnated by a different Jefferson man. This perspective deserves some credit precisely because there were candidate Jefferson men in contact with Hemings beyond Thomas: his brother and his nephew.

That back-and-forth leaves us where we were until a few days ago: unsure if Jefferson was actually the father of Hemings’ children, or merely a bystander to the impregnation of one of his favored slaves by a male relative. But now, the Smithsonian has sequenced Jefferson’s whole genome—mystery no more!

We now have percentages of autosomal DNA sharing between Thomas Jefferson and a handful of Hemings’ descendants. This is the DNA that’s not mitochondrial or from the sex chromosomes. You share 50% of it with either of your parents and your full siblings share an average of 50% with you as well (vs. 25% for half-siblings and avuncular and materteral relatives, and 12.5% for first-cousins), with a range of relatedness due to the randomness involved in inheritance.

The new paper on the Jefferson descendants lists Hemings descendants (with identification via labels like ‘M5’ and ‘E9’) who share 5.01%, 4.27%, 3.66%, 3.64%, 3.50%, 2.09%, 1.01%, and 0.55% of their DNA with Thomas Jefferson. The expectation for fifth-generation direct descendants is an average of 3.125%. For a brother, it’s 1.563%, and for a nephew, it would ordinarily be half as much, but Thomas Jefferson's nephew was more related to him than an ordinary nephew because Randolph married his and Thomas’ first-cousin. This made Thomas’ nephew 31.25% related to him rather than the usual 25%, making the expectation about 0.977% rather than 0.781%. For comparison, we also have DNA from two descendants of Jefferson and his wife—not Hemings—, and they share 2.57% and 2.69% of their DNA with Jefferson.

Because the expected levels of relatedness differ between descent from Thomas or his brother, we can categorize each of these observations based on their consistency with either option to obtain a likelihood that the kids came from either potential father. Arranging each observation by the order they’re presented in the paper, they’re positioned like this relative to the expected parental distributions:

If we go from the least-Thomas Jeffersonian person to the most-Thomas Jeffersonian person, we can see the likelihood of a match to Thomas rising to a very high level. The probability that Thomas was the father rather than his brother reaches 851:1, and the probability of Thomas instead of his nephew reaches an even higher 6.8m:1.

These results directionally match the more complicated simulation result presented in the paper, which makes sense given that they’re both driven by the same underlying logic. The authors’ preferred likelihood of Thomas compared to his brother is 2,213:1.¹ Clearly it’s highly likely Thomas Jefferson had kids with Sally Hemings.

To alter the judgment that Jefferson fathered at least one of Sally Hemings’ children, one or both of two things would’ve needed to happen: gross error in DNA sample identification and/or handling; or major problems in sampling. I seriously doubt either possibility. It seems one of the country’s historical mysteries is now solved.²

You can find my replication package for this project by clicking the link here. Enjoy!

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.

Why are men more violent than women? ‘It’s the testosterone.’ Why do some men end up in jail? ‘They’re high-test.’ Why do teens commit more crime than 40-somethings? ‘Testosterone decline.’ Why are bodybuilders always flying into a rage? ‘Roid rage!’

These are all popular beliefs about testosterone, and there are many more like them. Many—and I’d argue a majority—of the public believe that testosterone is some sort of rage or aggression chemical, that it radically increases risk-taking, aggravation, the propensity to commit violent acts, and so on. But is there much evidence for this?

The first thing to establish is the baseline for testosterone. Male testosterone levels are generally not very high, and they’re quite stable even into the beginning of old age. The big decline comes in the elderly years, and its cross-sectional shape is likely impacted by selection.¹ take a look:

Given how little testosterone levels appear to decline with age, it would be surprising if testosterone explained the age-crime curve—the well-known relationship whereby the propensity to crime jumps up after puberty and into early adulthood and declines into middle age and beyond. In fact, they do not line up:²

An even simpler method to understand the relationship between testosterone and crime is to simply compare members of the general population to criminals. If we take a look across the whole literature, we get two pictures: violent criminals have about 18% higher testosterone levels than members of the general population (i.e., ~71st-percentile testosterone levels) and sex offenders have completely normal testosterone levels.

One possible issue with this comparison is that incarceration—where testing generally takes place—could lead to reduced testosterone levels due to the diet, exercise, and social conditions of prison. This is not a major worry. There’s one study of the effects of incarceration on testosterone levels, and even though it used salivary testosterone measurements—which are more context- and obesity-, aging-, and insulin-sensitive than serum estimates—it found that changes after admission largely reverted after three months. Most prisoners tested in the literature are not new, so whatever underestimation happens due to incarceration lowering testosterone is not a very large problem.

But is this even interesting? Is testosterone correlated with criminal behavior in the general population? What about among prisoners? Looking at the whole literature including the various meta-analyses that have been done, the correlation between testosterone and criminal violence is just not strong. The correlations are dragged up by old, imprecise estimates from select samples; even then, they’re small, around 0.10.

Notice: I alluded to the fact that the meta-analytic estimates are dragged upwards by publication bias. If we correct for this, the correlations are very small indeed. Summarizing it, the correlation between aggressive behavior and testosterone is about 0.05 to 0.10 among the general population, and it appears to be a bit higher among offenders, at about 0.15 to 0.20—still quite a small correlation, especially given the stereotypes associated with testosterone.

Even this is overstating things. The extent to which criminals have higher testosterone levels is not that great, and it’s easily explicable by confounders. For example, they could be in status competitions or competing for sex more, they could have different diets, their violent surroundings could lead to higher testosterone, etc. The E-value—a measure of how large a confounder would need to be to overturn a result—for the prisoner-general population difference is small, at about 1.65, where anything below 2 is considered fragile. This could be easily be overturned by controlling for differences in substance use, the use of age-mismatched controls, smoking, and so on.

Another way to get at criminality’s relationship with testosterone is to look at recidivism. For the purposes of predicting sexual reoffending at least, testosterone seems to fail. It doesn’t make models any better than chance. This helps shore up our inferences about criminals, because this is about post-conviction behavior, and even this is unduly optimistic since the AUC is from the testing data.

These correlations are useful, but there is more, better evidence, too: there are lots of testosterone administration experiments! The ones that look at aggression turn up results that are remarkably similar to the general population correlations. That is, there’s not that much evidence of effects on aggression. Moreover, we also have some experiments on testosterone administration where the object wasn’t to measure aggression, but instead to assess impacts on health. In those studies—notably, TRAVERSE—, aggression incidents were recorded as side effects. They didn’t happen.

Let’s say you were to apply these numbers to explaining why men are more violent than women. Could it be done? Linear extrapolation suggests ‘no’—the gap in violent behavior is far too large to be explained by testosterone levels.

What about race? Could differences in testosterone levels explain why Black men are more violent than White men, for example? The answer to that is also no, for three reasons: total testosterone levels are highly similar with only a slight Black edge; free testosterone levels are highly similar with a slight White edge—the behaviorally-affective type, appearing contrary to predictions; and, the Black-White violent behavior gap is far larger than this could explain, much like the gap between the sexes.

We also have cohort studies available to see how testosterone levels affect men over time. In the CDC’s Vietnam Experience Study—one of their archival datasets put together to assess the impacts of wartime exposures to combat experiences and chemicals like Agent Orange—we have serum testosterone levels followed by years of subsequent measurements of behavior and self-reports of rage and short temper. There’s no correlation with self-reported anger, and the associations with dishonorable discharge, drug abuse, drunkenness, drunk driving, and remarriage are pitifully small at most.

One way to get at the effects of testosterone that I’ve seen suggested is to look at the effects of castration on sex offenders. It’s well-known that sex offenders who are castrated are far less likely to re-offend. But castration in studied settings has been voluntary, opening the reuslts up to self-selection problems.

If we compared hormonal castration to other treatment methods, the possibility of a self-selection issue becomes apparent: hormonal treatments also lower testosterone, but they don’t result in nearly as large an effect. In fact, without this issue, the apparent reduction in reoffending is just not that large at all, like a Cohen’s d of 0.19.

The studies on castration—chemical or surgical—are ambiguous because the reason for the reduced rates of reoffending could be due less to reduced aggression and more due to reduced desire—a separate issue. No one contests that testosterone makes people much more aroused or that it helps with motivation. But those are distinct from aggression and could be independently related to certain types of criminal offending. And, to make this less ambiguous, recall that sex offenders’ testosterone levels are not unusually high or low at baseline.³

There is a group that has unusually high testosterone levels, and it’s bodybuilders! Anabolic steroid users, most of whom bodybuild, do have elevated rates of criminal offending. But this is because they’re selected: the people who take anabolic steroids are people who engage in other risky behaviors, have preexisting anger issues, etc. And in fact, despite having testosterone levels far above most criminals, they tend not to be criminals themselves.

Studies that control for polysubstance use—meaning drugs used beyond anabolic steroids—find that the risk associated with steroid usage mostly evaporates. I doubt that it’s just the other drugs driving this, and I see the use of other drugs mostly as an indication of other behaviors, but nevertheless, surely residual confounding remains, making the observation of steroid users, at most, consistent with small effects.

We can also make this less ambiguous by looking at trial results. In those, steroid users don’t appear to hulk out and fly into a rage that observers detect. The self-reported bump is also small, and it could just be expectancy, meaning that people who take steroids expect to be angrier, so they report being a bit angrier. Or, they could even be confused about their emotions, mistaking motivation for aggression. Regardless, there’s just very little evidence of strong ‘roid rage’ effects.

What about games? In the setting of games, we can see if testosterone administration or baseline levels predict aggressive behavior, risk-taking, teamwork, competitiveness, and so on. With acute administrations, basically nothing shows up in economic games. This matches another meta-analysis I covered recently, and a true (near?) null aligns with the observation that the various small studies on this topic have effects in all directions for outcomes like cooperation, reciprocity, generosity, and so on.

What about so-called ‘supermales’, men with two Y chromosomes? They are disposed to higher levels of criminality, but curiously, this is not due to testosterone. Their levels are actually not meaningfully elevated, and their heightened propensity for crime is exactly what’s predicted from their much lower than average IQs!

I think I’ve basically covered everything now, but let’s look at yet another route through which androgens might affect the propensity to be aggressive, do crime, etc. That is, prenatal exposures and the so-called ‘dual-hormone hypothesis’, whereby testosterone only affects aggression when cortisol is low. We should have a low prior on both things mattering for a few reasons. For one, prenatal testosterone doesn’t track adult testosterone very well for either sex. For two, cortisol is highly unreliable, so for this to be true and recorded as such in a meaningfully-sized sample would be incredible. For three, the dual-hormone hypothesis concerns an interaction for a small effect. We’re hoping for a suppressor effect, and those are rare in general. These fail:

Even natural experiments like (Complete) Androgen Insensitivity Syndrome, Twin-Testosterone Transfer, and Congenital Adrenal Hyperplasia all also fail to support the idea that testosterone is strongly involved in aggressive behavior. We can also go beyond those natural experiments and look at a much more permanent, lifelong testosterone effect via genes known to be associated with testosterone. That is, we can use Mendelian randomization. When we do this in the U.K. Biobank, what comes up is that genetically-predicted testosterone levels aren’t related to risk-taking, nervousness, mood swings, neuroticism, or irritability. Bupkes! Long-term bupkes!⁴

Testosterone is the ‘male hormone’. It’s stereotypically associated with everything to do with men: risk-taking and derring-do, aggression and violence, willpower and strength, balding and boners, and so much more. But much of this is not because testosterone does or causes those things, but rather, because it is the male hormone.

Men produce lots of testosterone; it masculinizes the body, it invigorates those who take it, and it can even cure depression. It can provide motivation, it can get you off the couch, and it can clear your mind. But taking it does not make one do all the stereotypical things men do. Men make testosterone; testosterone doesn’t make men.

As to the question of why men are more violent than women, I don’t have a great answer. But I do have an idea: what if it has to do with capability? What if the reason men are so much more aggressive is because it’s instilled in people that if they can be, they ought to be? What if it’s a social interaction, where people recognize that someone is capable of violence, so they fill the role?

I referenced the following studies in this piece: Geniole 2020; Book 2001; Archer 2005; Rada 1983; Cadena-Molina 2025; Cantos 2024; Giotakos 2004; Seim & Dwyer 1988; Wong & Gravel 2018; Dabbs & Morris 1990; Mazur & Booth 2014; Weinberger 2005; Lösel & Schmucker 2005; Schmucker & Lösel 2017; Kristensen 2021; Lundholm 2015; Klötz 2006; Chegeni 2021; Pope 2000; Tricker 1996; TRAVERSE / Lincoff 2023; Huit 2026; Dekkers 2019; Nave 2020; Pratt, Turanovic & Cullen 2016; Kingston 2012; Studer 2005; Witkin 1976; Götz 1999; Schiavi 1984; Stochholm 2012; Raznahan 2023; Zhao 2022; Cannarella 2025; Leggett 2010; Pasterski 2007; Robinson 2013; Reinisch 1981; Harrison 2021; Hanson & Morton-Bourgon 2009; Thompson, Dabbs & Frady 1990; Dreber, Nave et al. 2025; Sánchez Rodríguez 2026; Eisenegger 2010.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

4

When we do the Mendelian randomization across samples using iPSYCH+PGC outcomes with the same instruments from the U.K. Biobank, we can avoid any issues with the IV becoming OLS since the samples don’t overlap, and in doing so we arrive at the result that testosterone predicts autism, but not ADHD. ADHD is a better violence predictor than autism, and there is at least one theory (the Extreme Male Brain Hypothesis) that predicts the autism result. Curiously, a female testosterone GWAS does not result in predicting autism. The architectures by sex are quite distinct, so I’m not sure what to think of this result.

Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.

Zach Goldberg’s most famous graph depicts the strange phenomenon whereby every racial group in America likes themselves, except for White liberals, who like other groups more than they like Whites.

I’ve seen this graph enough times that I got curious about how well it replicated, and what else we could learn from this sort of data. So, I grabbed the data and set about answering some questions.

#1: Does it replicate? Yes. The plot itself holds up in adjacent waves of the same survey.

#2: Does a self-derogating split across ideological lines extend to other races? No. Other races consistently like themselves, regardless of their politics. It’s just White liberals.

#3: Has it always been this way? No. White liberal self-derogation is a recent phenomenon, timed after the so-called ‘Great Awokening’ that came with Obama’s re-election and fomenting of race relations.

#4: Do White conservatives show an opposing trajectory shift? No. White conservatives are about as distinct from one another as they’ve always been. They even seem to have started going in the direction of their liberal counterparts, with weak evidence that such a shift has been larger for the less conservative among them.

#5: Do White liberals show a dose-response? Yes. More liberal Whites show greater racial outgroup preference.

#6: Do the results for ideology transfer to party? Yes and no. The more liberal party does show the same trajectory as liberals more generally.

The ‘No’ has to do with the dose-response: the split among Democrats is about ideology, not party attachment, and party attachment and ideology are only weakly correlated (r = ). As a result, we get this quite different-looking graph:

#7: Has this happened before? Not in recorded data. If we use the older ANES data going back to 1964, we don’t see this happen any other time. We also see that Blacks have reverted from the trajectory other groups were on, where they were heading towards having less ingroup bias over time. They are back to 1960s levels.¹

#8: Is this because White liberals began liking Whites less or liking non-Whites more? Both.

#9: Does this replicate in other U.S. datasets? Yes. This seems to replicate whenever we have representatively-sampled data from the U.S., in which participants are asked a substantially similar question. For example, here’s alternative ANES results, the VOTER survey, and mention of two additional sources and one dead-end.²

#10: Is this responsive to events? Yes. We generally do not have granular enough data to see whether particular events drive this reporting, but we do for one event: George Floyd’s death. We have weekly polling from the time, and right after it happened, White liberals’ anti-ingroup bias doubled. White moderates were affected in the same way to a smaller extent, whereas conservatives were unaffected.

#11: Did the Floyd effect on warmth affect members of other races? Yes. From the same dataset:

#12: Does White liberal outgroup preference replicate in international datasets? No. Across Europe, Australia, Canada, etc., attitudes have trended in the same direction, but they have not become as severe as in America: on net, Whites abroad still tend to prefer their own national and racial ingroups, even if said Whites lean left. We see this replicate in the European Social Survey, Australian Election Study, the British Election Study, and so on. I’ll show two charts to make some points.

First, compare the European Anglosphere to the Continent in terms of views on other-race versus same-race immigrants. The preference for own-race immigrants is smaller in the Anglosphere:

Second, in Britain, the post-Brexit trajectories on immigrant attitudes are curious:

And I’ll toss in a third chart to make a contrary point: where America, Europe, and the rest of the Anglosphere differ: the various in-group attitude measures are most differentiated by politics in the non-U.S. Anglosphere, despite how different the patterns look.

Bonus: The Great Awokening had a lot of focus on Black-White relations. How do different groups feel about Whites relative to Blacks? Outside of non-liberal Whites, everyone now prefers Blacks, at least in polling.³

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.

The Wordsum is a 10-item vocabulary test where you’re presented with a word and you have to then select the closest word to that one from a list. For example, you could be given the word “concern” and your options would be:

• See clearly

• Engage

• Furnish

• Disturb

• Have to do with

This brief test has been instrumental for understanding the relationship between intelligence—which, like all tests, it taps—and all sorts of social variables, from political views to incomes and education. But, despite its repeated administration and the considerable amount of use it’s seen, it’s been decades since it’s been properly validated. Accordingly, we don’t know how g-loaded Wordsum is.

The easiest way to figure out how g-loaded Wordsum is would be to administer a battery of tests with the Wordsum as one of the tests. From there, you would do a factor analysis and the loading would pop right out—simple. But in lieu of such a test, we can still validate the Wordsum through proxy methods. The way this works is simple:

• The National Longitudinal Survey of Youth (NLSY) 1979 and the 1997 version both administered a Word Knowledge test

  • This test is substantially the same as the Wordsum, it’s just longer

  • We have the actual test items for the 1979 cohort

• Both NLSYs (‘79 and ‘97) also administered a more extensive cognitive battery

• Both NLSYs and the GSS have extensive sets of demographic variables

• We can match the samples demographically

  • If we do this in time and place, it’s plausible that conclusions from the NLSY and the GSS can be transferred between one another

• We can reconstruct a ‘quasi-Wordsum’ in the NLSY ‘79 by matching items to the difficulty of the Wordsum items in the GSS

  • We can validate the similarity of this quasi-Wordsum to the real one by examining how it correlates with variables found in both the GSS and the NLSYs

• We can assess the g-loading of the quasi-Wordsum

• Thus, we can pretty much get a measurement of the g-loading of the real Wordsum

I noticed this was possible while working on a post about the relationship between intelligence and political alignment and how it’s changed over time. I was particularly inspired when I noticed that the IQ-by-politics gradient was four-times steeper in the NLSY ‘79 than in the GSS in the time around Obama’s election. Look:

I figured this was due to the fact that the NLSY ‘79’s testing was much more extensive, so I thought of how I could test that, and the fact that we had item data immediately came to mind because I had worked with it before. Accordingly, I took this snapshot from this era and began the demographic matching to the GSS before attempting to construct a Wordsum analogue. Remaking the plot with the matching plus the quasi-Wordsum ready to go, the difference is readily apparent, as is the validity of the quasi-Wordsum as a proxy for the genuine article:

After more extensive vetting, including looking at how well income and education were predicted, I concluded that the quasi-Wordsum does function like Wordsum, and the linkage with between Word Knowledge and other tests more generally is similar enough across the NLSY ‘79 and ‘97 that we can use the latter—which lacks item-level data for proper quasi-Wordsum construction—to validate the GSS’ Wordsum, too. All of this will be viewable in the replication package with the politics post I’m about to make.

For now, let’s look at a comparison of the AFQT and the quasi-Wordsum to figure out why it predicts politics worse. The answer is about half reliability differences (quasi-Wordsum is a short test with range restriction, like the real Wordsum is in the GSS), most of the residual is about low g-loading, and a tiny share is due to non-g ability variance. This looks like so:

As to the question behind this post, about how g-loaded the Wordsum is, the answer in the NLSY ‘79 is that the g-loading is 0.68 if we include the rest of the Work Knowledge items as indicators for the g factor, and it’s 0.62 without that, indicating a role for non-g ability variance. Disattenuating leads to g loadings of 0.83 and 0.76. In the NLSY ‘79 and ‘97, the g-loadings for the Work Knowledge subtest are 0.84 and 0.82, respectively (not significantly different), so we can reasonably assume psychometric comparability, and thus that the quasi-Wordsum g-loading in the NLSY ‘97 would be the implied Word Knowledge loading if the test had the reliability of a version with ten items: 0.66 (0.81).¹

The Wordsum test is highly g-loaded and is a suitable proxy for intelligence, even if it would be nice to have a broader, more reliable, and less range-restricted test score.²

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.