Academic Research Controversies

In early June 2012, investigative reporter Michael Patrick Leahy at Breitbart.com[1] revealed a scathing critique of Warren published by highly regarded Rutgers Law Professor Philip Shuchman which was uncovered in the 1990-1991 edition of the Rutgers Law Review.[2]

The 60-page analysis concerned a book Warren co-authored, As We Forgive Our Debtors: Bankruptcy and Consumer Credit in America.  Shuchman wrote:

Most of their study replicates several earlier research publications. These are hardly mentioned. The writers make extravagant and false claims to originality and priority of research. There appear to be serious errors in their use of statistical bases which result in grossly mistaken functions and comparisons. Some of their conclusions cannot be obtained even from their flawed findings. The authors have made their raw data unavailable so that its accuracy cannot be independently checked. In my opinion, the authors have engaged in repeated instances of scientific misconduct. [emphasis added]

Professor Shuchman argued that Warren and her co-authors jumped to conclusions, proclaimed new findings which were not new, and most importantly, ignored or did not accurately reflect data.

On page 240 of the Rutgers article, Schuchman asserted that some of the finding reached reached conclusions which were  not supported by the data:

Burgeoning consumer credit and increased volatility, they say, are the two primary systemic factors that caused the “startling increases” in consumer bankruptcy in the 1980’s. But these increases are less than in other recent periods and are only about one-sixth more than the increases in business bankruptcies, which suggests that consumer credit is not the cause of the increase in personal bankruptcy filings. [footnotes omitted]

Professor Shuchman went even further (at pp. 243-244), and suggested that the data was presented in such a way as to preclude verification:

This book contains so much exaggeration, so many questionable ploys, and so many incorrect statements, that it would be well to check the accuracy of their raw data, as old as it is. But the authors arranged matters so that they could not provide access to the computer printouts by case, with the corresponding bankruptcy court file numbers, thus preventing any independent check of the raw data in the files from which they took their information. [footnotes omitted]

Leahy followed up with an examination of whether, as some had claimed, Warren and her co-authors were cleared of the charges made by Professor Shuchman:[3]

Twenty-two years later, Professor Shuchman’s charges of “scientific misconduct” against Elizabeth Warren and her co-authors remain publicly unanswered and unresolved. These unresolved charges associated with her first major book raise continue to raise questions that hang over Elizabeth Warren’s entire body of academic work.

The next three articles in this series will address the conduct of three institutions enmeshed in this scandal: The University of Texas, the National Science Foundation, and Harvard University.

Leahy looked into investigations conducted by The University of Texas, and concluded that the investigations were inadequate:[4]

The lack of serious scrutiny of Ms. Warren’s academic research has continued for the subsequent two decades. Questions about Ms. Warren’s empirical studies have not been fully explored, and specific policies she has promoted–in particular those that led in 2010 to the passage of the Dodd-Frank Act and its onerous Consumer Financial Protection Bureau–have been imposed upon the public.

Breitbart News is not alleging that Sullivan, Warren, and Westbrook engaged in scientific misconduct. We are, however, presenting evidence that suggests the 1991 investigation conducted by the University of Texas into the allegations brought by Philip Shuchman in his scathing sixty page review[5] of the book Sullivan, Warren, and Westbrook co-authored in 1989 was neither thorough nor exculpatory.

Warren’s research also has been criticized by law professor Todd Zywicki.[6] In September 2010, before Warren was a declared candidate for political office, Zywicki wrote[7] that Warren’s much publicized findings as to medical bankruptcies was suspect:

Consider Ms. Warren’s much-ballyhooed study on the alleged link among health problems, medical expenses and personal bankruptcy filings. Published in the February 2005 issue of Health Affairs, the report was timed to head off bipartisan bankruptcy legislation that was enacted later that year. Ms. Warren and her co-authors claimed that “at least” 46% of personal bankruptcy filings in 2001 (the year from they collected the data) were the result of “medical causes,” and that this represented a 23-fold increase over 20 years.

Both conclusions are extremely suspect. First, the study provided an implausibly broad definition of “medical bankruptcy”—including any filer who reported uncontrolled gambling, drug or alcohol addiction, or the birth or adoption of a child.

Equally dubious, the authors classified a bankruptcy as having a “major medical cause” if the individual had accumulated more than $1,000 in out-of-pocket medical expenses (uncovered by insurance) over the course of two years prior to filing—regardless of income, and even if the debtor did not cite illness or injury among the reasons for bankruptcy….

In contrast to Ms. Warren’s studies, a battery of analysis, including research done by the Department of Justice’s Executive Office of the United States Trustee (which oversees the administration of bankruptcy cases), and by David Dranove and Michael Millenson of Northwestern University, concluded that fewer than 20% of bankruptcies are caused by health problems or medical expenses.

Last year Ms. Warren and her co-authors were back with an even more dramatic study, in the American Journal of Medicine, timed to promote President Obama’s health-care reform law. Drawing on 2007 filings, the authors concluded that 62% of bankruptcy filings were the result of medical issues and that the odds that a bankruptcy had a medical cause had doubled between just 2001 and 2007. This study was also flawed.

After Congress made it harder for people to skip out on their debts in 2005, the number of bankruptcy filings plummeted. In 2001, the year Ms. Warren used for the first study, there were 1,452,030 personal bankruptcy filings; in 2007 there were 822,590. Even if we are to accept the methodologies of the two studies for the sake of argument, there were 670,838 “medical bankruptcies” in 2001 and 510,828 medical bankruptcies in 2007—a drop of 160,000 per year. Yet Ms. Warren’s article nowhere acknowledges that the absolute number of bankruptcies and purported medical bankruptcies declined.

Smilar criticisms of Warren’s methodologies and conclusions were made by writer Megan McArdle:[8]

Does this persistent tendency to choose odd metrics that inflate the  case for some left wing cause matter?  If Warren worked at a think tank, you’d say, “Ah, well, that’s the genre.”  On the other hand, you’d also tend to regard her stuff with a rather beady eye.  It’s unlikely to  have been splashed across the headline of every newspaper in the United  States.  Her work gets so much attention because it comes from a Harvard professor.  And this isn’t Harvard caliber material–not even Harvard undergraduate.

So I think it matters on two levels. One, it matters how we evaluate the work–and I’ve been disappointed at how uncritically some people I  really respect have been willing to accept the 2001 and 2007 findings.   Not because I don’t think that there are medical bankruptcies in the  United States; I do!  I think medical bills are certainly the primary  cause of some of those filings, though I don’t know how many, and I’ve  also been writing about bankruptcy long enough to know that assigning  any one cause to the ultimate financial meltdown is, in many cases,  impossible.  (If you have nice consumer goods and no health insurance,  does a car accident count as a “medical bankruptcy” or a budgeting  deficiency?  If you lived right up to the edge of your income, was a job loss, or your spending pattern, to blame?  How you answer these  questions depends on a large number of prior value judgments that are  hotly contested in our society.)

It matters that we get this stuff right.  I am among the majority who  would like to see bankruptcies reduced in this country, and we’re not  going to be very effective at that if we run around thinking we can cure 2/3 of them by putting a national health care system in place, when in  reality a third or less have any strong causal relationship with medical bills.  Obviously, this was also held out as an argument for PPACA,  making an implicit promise to the American people which I believe to be  false.

But it also matters because a large part of Warren’s prominence comes  from the fact that she’s an academic.  If she came from . . . well, the  sort of think tank that publishes this sort of advocacy science . . .  she would have considerably less glamor, and power.

And perhaps it mattes most of all because this woman is now under consideration to head a powerful new agency.  If this is how she evaluates data, then isn’t that going to hamper her in making good policy?  If we’re going to have a consumer financial protection agency, I want one that has a keen eye to the empirical evidence on consumer welfare–not one that makes progressives most happy by reinforcing their prior beliefs.

Megan McArdle[9] pointed out the weakness of making the causal connection on such a wide scale of medical bills and bankruptcy, since bankruptcy filers themselves did not identify medical expenses as the cause of the bankruptcies (emphasis mine):

Elizabeth Warren has another study out showing that medical expenses contribute to more than half of all bankruptcies–indeed, this time, it’s 70%, up from the 50% she found in 2001.

Now, it is possible that this is true. The fact that it seems to disagree with every other study I’ve ever read that is not authored by Elizabeth Warren, and also, the self-reports of the people in her study (only about a third of whom attribute their bankruptcy to a health problem) could just be a fluke. It doesn’t necessarily mean that it’s wrong.

Yet upon closer examination, it turns out that it is not just wrong, but actively, aggressively wrong. Warren and her co-authors have obscured important and obvious facts that call the integrity of the work into serious question.

The text itself raises a huge red flags. It’s hard to believe that more than half of people who have been pushed into bankruptcy by a medical issue don’t understand this fact. Perhaps they are not the brightest bulbs on the Christmas tree, but could it really be true that most people catapaulted into a financial crisis by their medical bills don’t even notice that health care expenses are their main problem?

McArdle elaborated on other allegedly misleading aspects of the study, and then followed up with:[10]

She’s a professor at Harvard, and the head of the Congressional TARP oversight panel.  This conveys a certain responsibility to present data in the most illuminating way, not in the way that will induce journalists to say things that aren’t true.

The post 2005 increase in bankruptcies isn’t being driven by medical bankruptcies.  It’s simply rebounding from what every single analyst at the time, including Elizabeth Warren, agreed was an unsustainable drop.

A further challenge to Warren’s medical bankruptcy claims was published in The New England Journal of Medicine in March, 2018 [11].

Margot Sanger-Katz, writing for the New York Times, reports that Warren is responsible for the pervasive belief that medical bankruptcies are the cause of 62% of all bankruptcies in the United States. This number was challenged by Dobson, Finkelstein, et al. and revised to 4%.

Singer-Katz writes [12]:

One of the reasons many people think medical bills cause so many bankruptcies is Elizabeth Warren, now a United States senator and possible Democratic presidential candidate. In 2005, she, along with David Himmelstein, Deborah Thorne and Steffie Woolhandler, published a paper in the journal Health Affairs documenting a memorable statistic: More than 40 percent of all bankruptcies in America were a result of medical problems, they wrote. In 2009, they updated that research with an even more startling number: Medical bills were responsible for more than 62 percent of all American bankruptcies.

. . . Last March, a team of economists from M.I.T., Northwestern University and the University of California, Santa Cruz, published another paper, making use of a California database of every hospitalization in the state. By examining the credit reports of all the hospitalized people to see who filed for bankruptcy afterward, the researchers concluded that medical shocks related to hospitalization could explain 4 percent of all bankruptcies.

Their paper was published in The New England Journal of Medicine under a bold title: “Myth and Measurement: The Case of Medical Bankruptcies.” The authors argued that their paper was the first to definitively demonstrate that medical shocks did cause some people to go bankrupt. It found that missed work caused by illness was often a bigger contributor to financial difficulties than medical bills themselves.

There is, of course, a huge difference between 4 percent and 62 percent.

Sanger-Katz reports that the methodology used by Warren and her co-authors was faulty.

Amy Finkelstein, a professor at M.I.T. and one of the authors, said the Warren paper’s approach is akin to trying to find out how to be successful in business by interviewing big technology entrepreneurs about how they got their start. Such a study, she said, might lead to the conclusion that you need to drop out of college to succeed, even though most college dropouts do not become billionaires.

“The original paper was extremely problematic precisely because it limited its analysis to the set of people who went bankrupt, and said how common really large medical bills are in that sample of people,” she said. “It says nothing about how common really large medical bills are in the nonbankrupt population.”

References

  1. ^ Michael Patrick Leahy, Warren Accused of 'Repeated Instances of Scientific Misconduct' Before Harvard HireBreitbart.com, Jun. 4, 2012
  2. ^ Philip Shuchman, Review of As We Forgive Our Debtors: Bankruptcy and Consumer Credit in AmericaRutgers Law Review, 1989
  3. ^ Michael Patrick Leahy, The Academic Scandal Elizabeth Warren and Harvard Don't Want You to Know AboutBreitbart.com, Jun. 11, 2012
  4. ^ Michael Patrick Leahy, University of Texas Whitewash of Elizabeth Warren Scientific Misconduct Charge Entangles UVA, UC PresidentsBreitbart.com, Jun. 25, 2012
  5. ^ Rutgers-Shuchman review,  Review of As We Forgive Our Debtors: Bankruptcy and Consumer Credit in AmericaScribd, Jun. 4, 2012
  6. ^ George Mason University School of Law, Todd J. Zywicki bio pageGeorge Mason University School of Law website, Jan. 19, 2013
  7. ^ Todd Zywicki, In Elizabeth Warren We Trust?Wall Street Journal, Sep. 30, 2010
  8. ^ Megan McArdle, Considering Elizabeth Warren, the ScholarThe Atlantic, Jul. 22, 2010
  9. ^ Megan McArdle, Elizabeth Warren and the Terrible, Horrible, No Good, Very Bad, Utterly Misleading Bankruptcy StudyThe Atlantic, Jun. 24, 2009
  10. ^ Megan McArdle, Why Elizabeth Warren's New Bankruptcy Study is So BadThe Atlantic, Jun. 5, 2009
  11. ^ Dobson, Finkelstein, et al., Myth and Measurement — The Case of Medical BankruptciesThe New England Journal of Medicine, Mar. 22, 2018
  12. ^ Margot Sanger-Katz, Elizabeth Warren and a Scholarly Debate Over Medical Bankruptcy That Won’t Go AwayThe New York Times, Jun. 06, 2018
Last Updated: August 6th, 2018