an unanticipated consequence of
Jack M. Balkin
Jack Balkin: jackbalkin at yahoo.com
Bruce Ackerman bruce.ackerman at yale.edu
Ian Ayres ian.ayres at yale.edu
Mary Dudziak mary.l.dudziak at emory.edu
Joey Fishkin joey.fishkin at gmail.com
Heather Gerken heather.gerken at yale.edu
Mark Graber mgraber at law.umaryland.edu
Stephen Griffin sgriffin at tulane.edu
Bernard Harcourt harcourt at uchicago.edu
Scott Horton shorto at law.columbia.edu
Andrew Koppelman akoppelman at law.northwestern.edu
Marty Lederman marty.lederman at comcast.net
Sanford Levinson slevinson at law.utexas.edu
David Luban david.luban at gmail.com
Gerard Magliocca gmaglioc at iupui.edu
Jason Mazzone mazzonej at illinois.edu
Linda McClain lmcclain at bu.edu
John Mikhail mikhail at law.georgetown.edu
Frank Pasquale pasquale.frank at gmail.com
Nate Persily npersily at gmail.com
Michael Stokes Paulsen michaelstokespaulsen at gmail.com
Deborah Pearlstein dpearlst at princeton.edu
Rick Pildes rick.pildes at nyu.edu
Alice Ristroph alice.ristroph at shu.edu
Brian Tamanaha btamanaha at wulaw.wustl.edu
Mark Tushnet mtushnet at law.harvard.edu
Adam Winkler winkler at ucla.edu
Why the "Million Dollar Law Degree" Study Fails (Final Post)
My first post criticizing Simkovic and McIntyre’s (S&M) study showed how they overstated the earnings premium from a law degree. My second post showed that the only way to make a useful risk assessment is on a per school basis, and that it is indeed risky to attend a bunch of law schools today. In this final post I will expose what I believe is the most profound flaw in their analysis—and I will apologize for one thing I did wrong.
Let me begin by repeating their core claim: S&M have identified “historic earnings norms” that reflect what people who attend law school today are likely to earn over the course of their careers. They claim that even law grads at the bottom quartile stand to earn “hundreds of thousands of dollars” above what they would have earned without the degree.
We must separate two issues: 1) do their “earnings premiums” accurately represent historic law grad earnings?; and 2) are these earnings likely to hold for the coming generation? For their claims to have merit, they must establish both #1 and #2.
HOW DOES ONE ESTABLISH “HISTORIC NORMS” WITH PREDICTIVE VALUE?
Though it is fundamental to their analysis, this question is not addressed anywhere in their paper. S&M just count earnings data from 1996-2011, and call the results a “historic norm.” To constitute a “historic norm,” however, the earnings during these 16 years must be representative of law grad earnings over time. In other words, if we were to take a 16-year slice of law grad earnings at any random period in the past century, we would come up with roughly the same earnings (inflation adjusted). But S&M did not actually look at any other years of law grad earnings. So they cannot possibly know that this 16-year earnings slice isrepresentative.
In response to this argument, S&M will likely say that they used the best data available. But that is non-responsive. If that’s the best data available, then the logically necessary conclusion is that they cannot determine whether this 16-year slice indeed represents a “historic norm.” The fact that they don’t have enough data to answer the question does not give them license to assume the answer.
The negative answer to #1 is enough to vitiate the predictive value of their study. In the absence of any empirical evidence (even rudimentary) that this 16-year slice is representative of lawyer earnings over time, then it has no bearing on what law grads will earn going forward because it has no predictive weight.
WILL EARNINGS OF THE COMING GENERATION RESEMBLE THE PAST?
S&M briefly address this question, noting the work of Bill Henderson, who argues that the legal profession is undergoing a structural shift that portends leaner times for lawyers going forward. S&M’s retort is that cycles come and go and there is no reason to think this time is different. Their response: “Predictions of structural change in the legal industry date back at least to the invention of the typewriter.”
Touche. But seriously, what are the chances that the earnings they found in their 1996-2011 slice are likely to continue? After all, S&M make a very strong claim: students who attend law school today are likely to obtain the lifetime earnings premiums they specify—and students who skip law school out of concern for the financial risks will make a foolish mistake.
To make a convincing case that the next generation of earnings will look like the past generation, S&M must delve into the abundant literature on the legal job market. But they don’t do this—at all. They close the issue with a conclusory assertion and a caveat:
The most sober interpretation of the recent decline in starting salaries and employment for recent law graduates is that it is part of a broad cyclical downturn following the shock of the financial crisis of 2007 to 2008 and the recession that followed. The historic data still offers the best, most objective indicator of value. That said, past performance does not guarantee future returns.
A quick word about each sentence: The first sentence dismisses the people (not sober) who believe a structural change is in the works without saying why they are wrong. The second sentence reveals, again, how heavily their argument depends on the claim that their earnings numbers represent “historic norms”—which, as I indicated above, they provide no evidence for. The third sentence misstates the issue: of course they cannot guarantee earnings results—what they must do is affirmatively show that the bounce back in the legal market (to match the circumstances in the 16 year slice) is likely to happen. And this they have not done.
WHY THE PAST GENERATION AND THE NEXT ARE LIKELY TO BE DIFFERENT
What makes the “historic norm” claim of S&M so problematic is that the legal market underwent a series of unique developments in the past generation. This was the age of the rise of Big Law. Twenty-five years ago a large corporate firm had a few hundred lawyers—today the largest firms have several thousand. Over the course of this period, compensation for top partners rose to unprecedented heights, firms hired many more young lawyers and paid them very high salaries. The bi-modal distribution of starting salaries—about 10% of law grads earn far above the bulk of grads with a large earnings gap separating the two—began to develop around 2000.
S&M’s study measures law grad earnings during this unique period—and as I argued in the first post their study did not count the effects of the recession in the legal market from 1992-1995 and the dismal fate of law grads from 2009 to the present. Their 16-year employment data slice thus centers around the best years of the best of times in the corporate law market. The emergence of the unusual bi-modal pay distribution is strong evidence that earning patterns during this period were anything but typical.
Thus, we have good reason to doubt that the 16-year slice is representative of “historic” law grad earnings. It can’t be, since the rise of Big Law was a new historical development. To claim that earnings figures taken from this period constitute representative “historic norms” is ahistorical.
As for the coming generation, there is a flood of recent studies, books, and articles saying that things are indeed different now. S&M dismisses Bill Henderson, but what about Richard Susskind (The End of Lawyers; Tomorrow’s Lawyers), Bruce MacEwen (Growth is Dead: Now What?), Stephen Harper (The Lawyer Bubble: A Profession in Crisis)? These are knowledgeable people.
The New Republic recently ran a lengthy feature article, “The Last Days of Big Law.” The chairman of a major law firm, Mayer Brown, said “I don’t think anybody who follows the profession would suggest that this is only a temporary situation.” A couple of months ago, another prominent law firm, Weil Gotschal, which was thought to have weathered the legal recession better than most firms, stunned the corporate law world by announcing a large layoff of associates. The Executive Partner explained the move: “We believe that this is not just a cycle, but that the supply-demand balance is out of whack across the industry.” I can multiply quotes like this.
S&M cannot dismiss managing partners at leading law firms who say the boom days are history, and are firing lawyers and cutting partner pay to adjust to the “new normal.” These are sober people. The suggestion that we are seeing “the last days of Big Law” is journalistic hype (none of the people mentioned above say this). And things will firm up when the current shake out has been completed. But at the end of this process Big Law will be leaner and more efficient. It is already happening: many corporate law firms now hire about half the number of new lawyers they once hired in 2008 and before, preferring instead to use cheaper and more flexible contract workers to handle excess work.
S&M are free to argue that this growing chorus of voices is wrong, but they need to make a case with more substance than “cycles happen.” In the end, here is their response: “It remains easy to tell stories about how various changes will eventually have this or that effect, and currently impossible to falsify, since we cannot measure the future.” Sure—but lots of people who make their living following the legal market are telling the same story—and it’s opposite the story S&M are telling. And S&M forget that the validity of their claims presupposes that their version of the future--that things will be the same as the past--is correct.
S&M are so convinced by their (unsupported) “historic norm” claim that they misunderstand what needs to happen in the job market going forward. They say:
The results suggest that—absent catastrophic and unprecedented changes exceeding changes already seen from 2008 to 2011 and uniquely affecting law graduates rather than the broader labor market—many college graduates who follow the critics’ advice and skip law school will forego a lucrative career and face higher long-term risks of financial hardship.
That’s wrong. It is not necessary for things to get worse to falsify the predictive validity of their projected lifetime earnings figures. To match the 16-year slice from which they derived their “historic norms,” the legal market must substantially improve going forward. Their earnings numbers were taken during the peak of Big Law—and if we do not get back to that, future law grad earnings will be lower. No one who follows the legal market, not even the optimists, says we are going back to the peak years of Big Law.
It is a mistake, moreover, to think that these developments are limited to the corporate law sector. What happens in this sector reverberates throughout the legal employment market. When grads from top law schools don’t land big firm jobs in NYC, DC, Chicago, and LA, they fan out to less desirable secondary markets like Indianapolis, St. Louis, Atlanta, etc., pushing out law grads from regional and local schools, who in turn are forced to seek less desirable employment (small firms or solo), or end up unemployed.
This brings me to another problematic assertion by S&M. About 40% of the law grads in their database do not practice law; they mention that a significant proportion of law students say they want to do other things with the degree, and point out that 10% of CEOs and 50% of Senators have law degrees. S&M suggest that we therefore need not worry about the many law grads who have not obtained full time jobs as lawyers in recent years (only 55% and 56%, respectively, had obtained long-term full time jobs as lawyers nine months after graduation in 2011 and 2012). But how many of those CEOs and Senators began their careers unemployed, or working as paralegals, or as compliance officers, or went back to their previous jobs as school teachers? Leaving law for something better is a world of difference from getting no employment boost from the JD to begin with. The latter is what many law grads struggle with today. The percentage of recent law grads who do not obtain initial employment as lawyers is unprecedented—and not by choice. And there is every reason to expect that their earnings will be lower than the earnings premiums S&M found in their study.
For S&M’s study to have predictive value, they must establish two points: 1) that their 16-year slice is sufficiently representative of law grad earnings over time to be considered a “historic norm”; and 2) that the next generation of law grad earnings will resemble the earnings period captured in their study. They did not make either argument—indeed they assumed away the first—and the weight of evidence is against them on both.
Now let me conclude with an apology. My first post in this series was less charitable toward Simkovic and MacIntyre than I should have been. I should have made the same criticisms with a less harsh edge. Given the tone of my comments, it may come as a surprise to many that a couple of months ago I was asked by a law journal to evaluate S&M’s draft for publication, and, despite my reservations, I gave it a positive recommendation because I thought it raised a useful new perspective on the issue of economic return on a law degree.
I have been debating these issues for over two years now and sometimes lose my equanimity. I believe getting it right is of utmost importance to young people thinking about a career in law. What knocked me off balance was this line in their article: “many college graduates who follow the critics’ advice and skip law school will forego a lucrative career and face higher long-term risks of financial hardship.” To say something that strong requires that they make a rock solid case for the correctness of their findings. I hope I have convincingly demonstrated in these three posts that their claims are anything but rock solid.
I have not read S&M study, but have read about it (and skimmed the actual paper). As a current contract attorney, graduate of 2011, I wanted to make several points:
1. Structural changes in big law were almost inevitable from the start. There does not seem to be a good reason to not outsource document review work to contract attorneys. It makes little sense for even large, corporate clients to pay big bucks for work that can be done at a much lower rate.
2. It is just as difficult to predict what will happen on the demand side in the legal market as on the supply side. I would think that fewer students will apply to law schools going forward, although by how much remains to be seen. *Given how hard law schools work at marketing themselves, it's doubtful that the number of law students will diminish greatly.
3. Some boost may be provided to the demand side of the legal market by changing national and state laws. This, of course, would require political will.
4. It seems to me that the legal industry, particularly in big law, has overreached. The prospect of getting paid $160k as a first year associate certainly is very attractive to prospective students. I was one of those who thought I'd land that kind of job (until the harsh reality set in not long after enrollment into law school). It seems to me, therefore, that one way to change this would be reduce starting salaries, and to accept reasonable pay cuts at the partner level. This might actually expand demand and may reduce supply.
To summarize, I think the legal industry has become unsustainable, much like the financial industry. And while it does not spell doom, the legal industry will have to rebalance itself to more sustainable and natural levels.
Is Brian being sandbagged at this Blog? Are insiders of the JD/Student Loan Institutional Complex in fear of their own futures if Brian is right? Or do they think they might make it in Big Law if he is right? S/M's forecasting may take a 16 year "norm" to prove accurate, a forecast without penalty but with the reward of continuing on with their present positions secure for 16 years. Frankly, I believe this piling on should be penalized. And where is the focus on the public interest for competent, affordable delivery of needed legal services that Big Law does not provide? Apparently the Complex is not prepared to adddress the public interest.
Maybe Brian is tilting at windmills that are just blowing air that apparently cannot resolve the Complex's problems, that may be structural.
Can we expect law schools to use the S/M JD economic forecast as a promotional tool to fill seats, especially at those below the top tiers? A college grad convinced by S/M to go to law school will be able to test the S/M forecast after 3 - or 4? - years of investment of much time and money. A JD student may learn during that time from his/her law school courses that reliance upon the S/M forecast may not be actionable if the forecast turns out to be wrong. against either the law school or S/M. Apparently there is no regulatory agency to protect the interests of the JD candidate/consumer. And there is no out via bankruptcy for student loans. And the JD student's investment loss, unlike an investment in securities, real estate, etc, cannot offset income to lower taxes. Perhaps law schools promoting S/M's forecast should disclose "side effects" in the manner of the FDA requirement for prescription drugs.
Follow the money involved with the JD/Student Loan Institutional Complex. While most law schools are not-for-profits, that doesn't mean that they can't make profits, being limited in how they use those profits, e.g., administration, faculty, facilities, etc.