Elizabeth Warren Academic Research Controversies

In early June 2012, investigative reporter Michael Patrick Leahy at Breitbart.com^[1] revealed a scathing critique of Warren published by highly regarded Rutgers Law Professor Philip Shuchman which was uncovered in the 1990-1991 edition of the Rutgers Law Review.^[2]

The 60-page analysis concerned a book Warren co-authored, As We Forgive Our Debtors: Bankruptcy and Consumer Credit in America.  Shuchman wrote:

Most of their study replicates several earlier research publications. These are hardly mentioned. The writers make extravagant and false claims to originality and priority of research. There appear to be serious errors in their use of statistical bases which result in grossly mistaken functions and comparisons. Some of their conclusions cannot be obtained even from their flawed findings. The authors have made their raw data unavailable so that its accuracy cannot be independently checked. In my opinion, the authors have engaged in repeated instances of scientific misconduct. [emphasis added]

Professor Shuchman argued that Warren and her co-authors jumped to conclusions, proclaimed new findings which were not new, and most importantly, ignored or did not accurately reflect data.

On page 240 of the Rutgers article, Schuchman asserted that some of the finding reached reached conclusions which were  not supported by the data:

Burgeoning consumer credit and increased volatility, they say, are the two primary systemic factors that caused the “startling increases” in consumer bankruptcy in the 1980’s. But these increases are less than in other recent periods and are only about one-sixth more than the increases in business bankruptcies, which suggests that consumer credit is not the cause of the increase in personal bankruptcy filings. [footnotes omitted]

Professor Shuchman went even further (at pp. 243-244), and suggested that the data was presented in such a way as to preclude verification:

This book contains so much exaggeration, so many questionable ploys, and so many incorrect statements, that it would be well to check the accuracy of their raw data, as old as it is. But the authors arranged matters so that they could not provide access to the computer printouts by case, with the corresponding bankruptcy court file numbers, thus preventing any independent check of the raw data in the files from which they took their information. [footnotes omitted]

Leahy followed up with an examination of whether, as some had claimed, Warren and her co-authors were cleared of the charges made by Professor Shuchman:^[3]

Twenty-two years later, Professor Shuchman’s charges of “scientific misconduct” against Elizabeth Warren and her co-authors remain publicly unanswered and unresolved. These unresolved charges associated with her first major book raise continue to raise questions that hang over Elizabeth Warren’s entire body of academic work.

The next three articles in this series will address the conduct of three institutions enmeshed in this scandal: The University of Texas, the National Science Foundation, and Harvard University.

Leahy looked into investigations conducted by The University of Texas, and concluded that the investigations were inadequate:^[4]

The lack of serious scrutiny of Ms. Warren’s academic research has continued for the subsequent two decades. Questions about Ms. Warren’s empirical studies have not been fully explored, and specific policies she has promoted–in particular those that led in 2010 to the passage of the Dodd-Frank Act and its onerous Consumer Financial Protection Bureau–have been imposed upon the public.

Breitbart News is not alleging that Sullivan, Warren, and Westbrook engaged in scientific misconduct. We are, however, presenting evidence that suggests the 1991 investigation conducted by the University of Texas into the allegations brought by Philip Shuchman in his scathing sixty page review of the book Sullivan, Warren, and Westbrook co-authored in 1989 was neither thorough nor exculpatory.

Warren’s research also has been criticized by law professor Todd Zywicki.^[5] In September 2010, before Warren was a declared candidate for political office, Zywicki wrote^[6] that Warren’s much publicized findings as to medical bankruptcies was suspect:

Consider Ms. Warren’s much-ballyhooed study on the alleged link among health problems, medical expenses and personal bankruptcy filings. Published in the February 2005 issue of Health Affairs, the report was timed to head off bipartisan bankruptcy legislation that was enacted later that year. Ms. Warren and her co-authors claimed that “at least” 46% of personal bankruptcy filings in 2001 (the year from they collected the data) were the result of “medical causes,” and that this represented a 23-fold increase over 20 years.

Both conclusions are extremely suspect. First, the study provided an implausibly broad definition of “medical bankruptcy”—including any filer who reported uncontrolled gambling, drug or alcohol addiction, or the birth or adoption of a child.

Equally dubious, the authors classified a bankruptcy as having a “major medical cause” if the individual had accumulated more than $1,000 in out-of-pocket medical expenses (uncovered by insurance) over the course of two years prior to filing—regardless of income, and even if the debtor did not cite illness or injury among the reasons for bankruptcy….

In contrast to Ms. Warren’s studies, a battery of analysis, including research done by the Department of Justice’s Executive Office of the United States Trustee (which oversees the administration of bankruptcy cases), and by David Dranove and Michael Millenson of Northwestern University, concluded that fewer than 20% of bankruptcies are caused by health problems or medical expenses.

Last year Ms. Warren and her co-authors were back with an even more dramatic study, in the American Journal of Medicine, timed to promote President Obama’s health-care reform law. Drawing on 2007 filings, the authors concluded that 62% of bankruptcy filings were the result of medical issues and that the odds that a bankruptcy had a medical cause had doubled between just 2001 and 2007. This study was also flawed.

After Congress made it harder for people to skip out on their debts in 2005, the number of bankruptcy filings plummeted. In 2001, the year Ms. Warren used for the first study, there were 1,452,030 personal bankruptcy filings; in 2007 there were 822,590. Even if we are to accept the methodologies of the two studies for the sake of argument, there were 670,838 “medical bankruptcies” in 2001 and 510,828 medical bankruptcies in 2007—a drop of 160,000 per year. Yet Ms. Warren’s article nowhere acknowledges that the absolute number of bankruptcies and purported medical bankruptcies declined.

Similar criticisms of Warren’s methodologies and conclusions were made by writer Megan McArdle:^[7]

Does this persistent tendency to choose odd metrics that inflate the  case for some left wing cause matter?  If Warren worked at a think tank, you’d say, “Ah, well, that’s the genre.”  On the other hand, you’d also tend to regard her stuff with a rather beady eye.  It’s unlikely to  have been splashed across the headline of every newspaper in the United  States.  Her work gets so much attention because it comes from a Harvard professor.  And this isn’t Harvard caliber material–not even Harvard undergraduate.

So I think it matters on two levels. One, it matters how we evaluate the work–and I’ve been disappointed at how uncritically some people I  really respect have been willing to accept the 2001 and 2007 findings.   Not because I don’t think that there are medical bankruptcies in the  United States; I do!  I think medical bills are certainly the primary  cause of some of those filings, though I don’t know how many, and I’ve  also been writing about bankruptcy long enough to know that assigning  any one cause to the ultimate financial meltdown is, in many cases,  impossible.  (If you have nice consumer goods and no health insurance,  does a car accident count as a “medical bankruptcy” or a budgeting  deficiency?  If you lived right up to the edge of your income, was a job loss, or your spending pattern, to blame?  How you answer these  questions depends on a large number of prior value judgments that are  hotly contested in our society.)

It matters that we get this stuff right.  I am among the majority who  would like to see bankruptcies reduced in this country, and we’re not  going to be very effective at that if we run around thinking we can cure 2/3 of them by putting a national health care system in place, when in  reality a third or less have any strong causal relationship with medical bills.  Obviously, this was also held out as an argument for PPACA,  making an implicit promise to the American people which I believe to be  false.

But it also matters because a large part of Warren’s prominence comes  from the fact that she’s an academic.  If she came from . . . well, the  sort of think tank that publishes this sort of advocacy science . . .  she would have considerably less glamor, and power.

And perhaps it mattes most of all because this woman is now under consideration to head a powerful new agency.  If this is how she evaluates data, then isn’t that going to hamper her in making good policy?  If we’re going to have a consumer financial protection agency, I want one that has a keen eye to the empirical evidence on consumer welfare–not one that makes progressives most happy by reinforcing their prior beliefs.

Megan McArdle^[8] pointed out the weakness of making the causal connection on such a wide scale of medical bills and bankruptcy, since bankruptcy filers themselves did not identify medical expenses as the cause of the bankruptcies (emphasis mine):

Elizabeth Warren has another study out showing that medical expenses contribute to more than half of all bankruptcies–indeed, this time, it’s 70%, up from the 50% she found in 2001.

Now, it is possible that this is true. The fact that it seems to disagree with every other study I’ve ever read that is not authored by Elizabeth Warren, and also, the self-reports of the people in her study (only about a third of whom attribute their bankruptcy to a health problem) could just be a fluke. It doesn’t necessarily mean that it’s wrong.

Yet upon closer examination, it turns out that it is not just wrong, but actively, aggressively wrong. Warren and her co-authors have obscured important and obvious facts that call the integrity of the work into serious question.

The text itself raises a huge red flags. It’s hard to believe that more than half of people who have been pushed into bankruptcy by a medical issue don’t understand this fact. Perhaps they are not the brightest bulbs on the Christmas tree, but could it really be true that most people catapaulted into a financial crisis by their medical bills don’t even notice that health care expenses are their main problem?

McArdle elaborated on other allegedly misleading aspects of the study, and then followed up with:^[9]

She’s a professor at Harvard, and the head of the Congressional TARP oversight panel.  This conveys a certain responsibility to present data in the most illuminating way, not in the way that will induce journalists to say things that aren’t true.

The post 2005 increase in bankruptcies isn’t being driven by medical bankruptcies.  It’s simply rebounding from what every single analyst at the time, including Elizabeth Warren, agreed was an unsustainable drop.

A further challenge to Warren’s medical bankruptcy claims was published in The New England Journal of Medicine in March, 2018 ^[10].

Margot Sanger-Katz, writing for the New York Times, reports that Warren is responsible for the pervasive belief that medical bankruptcies are the cause of 62% of all bankruptcies in the United States. This number was challenged by Dobson, Finkelstein, et al. and revised to 4%.

Sanger-Katz writes ^[11]:

One of the reasons many people think medical bills cause so many bankruptcies is Elizabeth Warren, now a United States senator and possible Democratic presidential candidate. In 2005, she, along with David Himmelstein, Deborah Thorne and Steffie Woolhandler, published a paper in the journal Health Affairs documenting a memorable statistic: More than 40 percent of all bankruptcies in America were a result of medical problems, they wrote. In 2009, they updated that research with an even more startling number: Medical bills were responsible for more than 62 percent of all American bankruptcies.

. . . Last March, a team of economists from M.I.T., Northwestern University and the University of California, Santa Cruz, published another paper, making use of a California database of every hospitalization in the state. By examining the credit reports of all the hospitalized people to see who filed for bankruptcy afterward, the researchers concluded that medical shocks related to hospitalization could explain 4 percent of all bankruptcies.

Their paper was published in The New England Journal of Medicine under a bold title: “Myth and Measurement: The Case of Medical Bankruptcies.” The authors argued that their paper was the first to definitively demonstrate that medical shocks did cause some people to go bankrupt. It found that missed work caused by illness was often a bigger contributor to financial difficulties than medical bills themselves.

There is, of course, a huge difference between 4 percent and 62 percent.

Sanger-Katz reports that the methodology used by Warren and her co-authors was faulty.

Amy Finkelstein, a professor at M.I.T. and one of the authors, said the Warren paper’s approach is akin to trying to find out how to be successful in business by interviewing big technology entrepreneurs about how they got their start. Such a study, she said, might lead to the conclusion that you need to drop out of college to succeed, even though most college dropouts do not become billionaires.

“The original paper was extremely problematic precisely because it limited its analysis to the set of people who went bankrupt, and said how common really large medical bills are in that sample of people,” she said. “It says nothing about how common really large medical bills are in the nonbankrupt population.”

In 2006, Gail Heriot published a study^[12] that questioned the claims in and methodology of “Illness and Injury as Contributors to Bankruptcy” by David U. Himmelstein, Elizabeth Warren, Deborah Thorne, and Steffie Woolhandler.

Heriot writes:

In commenting to the media, lead author David Himmelstein stated, “Our study is frightening. . . . Too often private health insurance is an umbrella that just melts in the rain.”11 Co-author Elizabeth Warren stated in the Harvard Law School announcement of the study that “[a] broken health care finance system is bankrupting middle class America.”12 Their emphasis on health care financing encouraged journalists to believe that the study focused on bankruptcies that were indeed the result of a broken health care finance system.13 Their dramatic language suggested that the study had uncovered something on that topic that should frighten or shock the average American.

Should the media also share in the blame for the vast amount of misinformation that circulated about the study? Yes, of course—perhaps even a greater share of the blame. A careful reading of the study itself (and not just the press releases) would have gone a long way toward separating fact from fiction.

. . . . The truth is that the study is not about bankruptcies caused by medical bills. It is not even about bankruptcies caused by illness or injury. It is about bankruptcies that can—at least if you are willing to stretch things—be classified as medically related. Its central findings are that 54.5% of all bankruptcies have a “medical cause” and that 46.2% of all bankruptcies have a “major medical cause.”14 But “medical cause” and “major medical cause” are specially defined terms. Not everyone whose bankruptcy was classified as having a “medical cause” or a “major medical cause” would agree that illness and injury were involved at all. Some might even laugh at the notion.

For example, the authors state that their study classified bankruptcies in which the debtor cited “uncontrolled gambling,”15 “alcohol or drug addiction,” “death in family,” and the “birth/addition of new family member”16 as having a “medical cause” (but not necessarily a “major medical cause”).17 It is true there may be situations in which a researcher might legitimately want to classify those conditions as “medical,” but a study touted by one of its authors as showing that “[a] broken health care finance system is bankrupting middle class America” is not one of them.18 A father who has gambled away his family’s mortgage payment is not likely the victim of crushing medical bills and reforming the health care finance system will do nothing to solve his problem. Similarly, new parents who find that they can no longer afford their previous lifestyle now that one of them has to stay home with the baby will usually find the obstetrician’s bill the least of their problems. Babies are a financial hardship even when hospitals give them away free.

Heriot further contends that the significant difference in findings from earlier studies is rooted in these broad definitions of “medical cause” and “major medical cause.”

What accounts for the huge gap between the earlier studies and the present study?

. . . . The Fragile Middle Class authors found that a total of 19.3% of the sampled debtor households gave medical reasons for their bankruptcy.57 Included within that figure were 11.4% who cited illness or injury of either debtor, 1.8% who cited illness or injury of a family member, 5.7% who cited medical debts, 2.1% who cited insurance problems, and 2.1% who did not specify the nature of the relationship between their bankruptcy and their medical reasons.58

Unlike the present study’s authors, the authors in The Fragile Middle Class did not attempt to expand the count of medically related bankruptcies by including filings in which the debtors had specifically forgone the opportunity to cite illness or injury as one among many reasons for their bankruptcy. Their 19.3% finding is more directly comparable to the present study’s finding that 28.3% of all debtor households cite “illness or injury” as a significant cause of their bankruptcy.59 Even so, the discrepancy between the two figures is substantial. And while many factors might help explain this, one factor of great importance is the method by which the data was gathered.

. . . . There is nothing wrong with conducting a study to measure what proportion of bankruptcies can somehow be broadly classified as “medically related,” broadly defined. Such a study would have far fewer policy implications than a study of how many bankruptcies are primarily the result of medical bills or of illness or injury. But there is no reason why it should not be studied, provided that the authors make clear what the study is all about. This was not done in this case.

On April 20, 2005, President George W. Bush signed the Bankruptcy Abuse Prevention and Consumer Protection Act into law, despite the arguments generated from Illness and Injury as Contributors to Bankruptcy. Here’s hoping he did the right thing. No doubt it would have been helpful to have up-to-date, accurate empirical information about how often crushing medical bills are the primary cause of bankruptcy or even how often illness or injury is a major cause of bankruptcy. Unfortunately, reliable figures of that sort were unavailable. While Illness and Injury as Contributors to Bankruptcy was touted as a study addressing those issues, it did not.

References

1. ^ Michael Patrick Leahy, Warren Accused of 'Repeated Instances of Scientific Misconduct' Before Harvard Hire, Breitbart, Jun. 4, 2012
2. ^ Philip Shuchman, Review of As We Forgive Our Debtors: Bankruptcy and Consumer Credit in America, Rutgers Law Review, 1989
3. ^ Michael Patrick Leahy, The Academic Scandal Elizabeth Warren and Harvard Don't Want You to Know About, Breitbart, Jun. 11, 2012
4. ^ Michael Patrick Leahy, University of Texas Whitewash of Elizabeth Warren Scientific Misconduct Charge Entangles UVA, UC Presidents, Breitbart, Jun. 25, 2012
5. ^ George Mason University School of Law, Todd J. Zywicki bio page, George Mason University School of Law website, Jan. 19, 2013
6. ^ Todd Zywicki, In Elizabeth Warren We Trust?, Wall Street Journal, Sep. 30, 2010
7. ^ Megan McArdle, Considering Elizabeth Warren, the Scholar, The Atlantic, Jul. 22, 2010
8. ^ Megan McArdle, Elizabeth Warren and the Terrible, Horrible, No Good, Very Bad, Utterly Misleading Bankruptcy Study, The Atlantic, Jun. 24, 2009
9. ^ Megan McArdle, Why Elizabeth Warren's New Bankruptcy Study is So Bad, The Atlantic, Jun. 5, 2009
10. ^ Dobson, Finkelstein, et al., Myth and Measurement — The Case of Medical Bankruptcies, The New England Journal of Medicine, Mar. 22, 2018
11. ^ Margot Sanger-Katz, Elizabeth Warren and a Scholarly Debate Over Medical Bankruptcy That Won’t Go Away, The New York Times, Jun. 06, 2018
12. ^ Gail Heriot, Misdiagnosis: A Comment On 'Illness and Injuray as Contributors to Bankruptcy' And The Media Publicity Surrounding It, SSRN, 2006