Seeking Evidence: A Professor Looks for Empirical Proof to Improve Access to Justice

The Center for Court Innovation - Podcasts

In this New Thinking podcast, Harvard Law School Professor James Greiner talks with Aubrey Fox about why he launched the Access to Justice Lab, which has researched topics as varied as how to provide self-help materials to defendants involved in civil debt collection cases to the effectiveness of commonly used pretrial assessment interview tools in criminal court. Greiner also discusses what he sees as a strategy for improving the legal profession's openness to evidence-based thinking.

AUBREY FOX: So this is Aubrey Fox. I'm today's guest host for The New Thinking Podcast, which is the Center for Innovation's podcast where we interview legal experts and researchers. And I'm very fortunate to be joined today by Professor James Greiner, who is the William Henry Bloomberg Professor of Law at Harvard University. Does this mean that you're connected to our former Mayor Bloomberg?

PROFESSOR JAMES GREINER: Actually, I think that, if I have it right, the Chair was named after his father and he established it on his father's behalf. It's a rotating chair, so I have it for another few months before it rotates to someone else. But that is the connection.

FOX: So you're not dining with Mayor Bloomberg on a regular basis?

GREINER: I haven't had the pleasure yet. I'm sure it'll be terrific. I'm guessing he probably sets a good table, but I haven't had the pleasure yet.

FOX: Among many things, Jim, is the faculty director of something called the Access to Justice Lab, which is a really exciting new initiative that he started where he's trying to bring evidence-based research to the study of criminal justice and civil justice. And so we want to talk to Jim today about his work with the Access to Justice Lab and also some of his thinking about why it's important to introduce this evidence-based approach, and want to talk to him a little bit about some of the particular issues he's investigated.

Jim, maybe where we'll start is just to let you talk about how you got interested in this area of criminal justice and civil justice, and tell us a little bit about your background.

GREINER: Sure. So when I graduated from law school, I hadn't taken any statistics courses or any sort of quantitative courses in college or anything like that. I just wanted to be a regular practicing lawyer. So I did that for 3 years for the Justice Department and then 3 years for a private law firm, a law firm called Jenner & Block in DC and, at the time, I was litigating cases that involved some numbers. So, employment discrimination class actions, litigation about the 2000 decennial census, whether you could use statistical techniques to try to correct for over-counts and under-counts according to racial groups. I'd always been interested in social justice issues and criminal law issues.

When I moved over to the private law firm, I did a couple of pro-bono cases that we picked up. And basically, through the practice, redistricting was in the area I did, got interested in numbers and what numbers and evidence-based thinking could bring to the table in terms of social justice issues. And so after 6 years as a practicing attorney, I left the practice of law for 5 years to get a PhD in statistics.

And then when I got my PhD in Statistics I started researching in a bunch of areas, but the one I was having the most fun with, really enjoying and the one that I thought I could potentially do some good with was access to justice issues, more on the civil side than on the criminal side, but some on the criminal side also. So issues like, how do you make law work and be accessible for folks who can't afford to hire lawyers to interface with the legal system. And on the criminal side, how can we make existing institutions perform what we want them to perform better. How can we release more people who are not dangerous, how can we reduce the criminal footprint on communities and still maintain a system of law and order, etc.

So those are the big-picture issues that I found most interesting.

FOX: So, having a statistician as a professor of law. Is that an unusual development, or is that something that's been going on for decades and decades?

GREINER: No, I think there are only 2 of us still in the legal academy that have stats degrees-- stats PhDs. I know the other person, unsurprisingly. But having someone as a law professor who knows, as we say, "how to count", that's not all that unusual. It's just folks typically have economics degrees or political science degrees or something like that. But the hardcore statistics is what I really wanted to get invested in. And the reason for that was I pretty much knew when I was going into the statistics program that I was a litigator through and through. I was always going to be a litigator. I was going to care about courts and administrative agencies and the practice of law at a person by person level.

And so I didn't feel like I wanted to learn a lot about the big-picture political science theory and big-picture economic theory that those folks do. And so I just said let's go straight into the heavy numbers.

FOX: And so tell us a little bit about the Access to Justice Lab. What does it do, when did you get it started?

GREINER: We're talking in December of 2016. It's only been in existence since July of 2016, and it is funded by a generous grant from the Laura and John Arnold Foundation. The foundation itself cares about evidence-based thinking across a lot of different spheres, and it funded the lab. And basically the lab has 2 overall missions. One is to produce useful, rigorous evidence that would assist policy makers and judges and lawyers in what they do. Especially with respect, again, to access to justice related to folks who can't afford to hire lawyers to interface with the legal system for them.

And then the second overall purpose is to try to tear down the resistance to rigorous empirical evidence, especially via the randomized control trials, or randomized field experiment, that resistance that exists within the legal profession and within the judiciary.

I think that in law, especially with the practice of law, we are in roughly where the medical profession was in about 1938 or 1940. Basically, we are beginning to engage in the debate about whether our profession should be evidence-based in the way that medicine was engaging in a debate about whether drugs and medical devices should have evidence behind them before they are allowed to be sold to the public.

And there are many folks in law, on both the bench and the bar, who think that randomization in law is unethical or is unnecessary. Unethical because randomization takes away the professional judgment about who should get what, and unnecessary because we know that, as lawyers and judges, all of our professional judgements are perfect, or very, very good and, therefore, we don't really need to investigate them all that much. And so the Access to Justice Lab's second purpose is to try to suggest to folks that scientific-based thinking and evidence really can bring a lot to the table and it may end up overturning accepted truths within the bench and the bar.

FOX: I'm glad, in your comparison to the medical world, you didn't bring up leaches. So at least we're... We're 100 years ahead of where we could be.

GREINER: Right.

FOX: And, just to follow up for a moment, I kind of want to get your sense of the state of play on this because I understand that there may be resistance to RCTs, which are the most rigorous form of evidence-based analysis and require dividing a control group from an experimental group and giving some people something that you don't give the control group. But there is a history of doing research into criminological issues. So it's not like you're starting from square one. So I guess in this mix of some history and some resistance, where would you say we are at the moment?

GREINER: I'd say more resistance than history with respect to courts and judges and lawyers. So you're absolutely right that there is a reasonably well developed literature in the criminology field. And a good portion of it, not as much as many criminologists would like, but still a good portion of it backed up by randomized studies to try to figure out whether, say, afterschool programs prevent people from getting arrested, or whether certain types of treatment as a condition for probation are effective in preventing recidivism. But if you notice in the 2 examples that I gave you, one of them is sort of prior to the involvement, the intervention is prior to the involvement of the criminal justice system, and one of them is post sentencing, sort of after the involvement of what we call the lawyer-based or court-based criminal justice system.

And so there has been substantially less in that portion of the criminal justice system where the lawyers and the judges get involved to the point where, when a co-author and I try to catalog all of the randomized studies that have been done in United States law, criminal or civil, that involved randomization of a decision that would otherwise have been made by a judge or a lawyer, so it's how we defined randomized studies in law. We're taking a decision that otherwise would've been made by a judge and a lawyer and randomizing it to find out whether the decision is worthwhile or would be an effective style of treatment or something like that. We could only find about 50 in the history of the United States, based upon all areas of law, all sorts of settings.

So if you think about medicine starting from 1938 or 1940, the number would probably be uncountable, be tens of thousands or hundreds of thousands of randomized studies that have involved replacing a judgment that would've otherwise been made by a medical professional with randomization in order to figure out whether the treatment or the intervention works. Whereas we could find fewer than 1 per year in law. So again, I absolutely agree with you. Criminology has much more of a history. It's a much more developed field. But they're basically working in their rigorous evidence prior to the formal legal system or after it in the lifetime of a particular defendant or person or crime. And where we think the evidence really needs to be focused is, at least in terms of a new effort, we need to focus on where there isn't as much evidence right now, which is inside courts and law offices.

FOX: One of the reasons that I wanted to speak to you, Jim, was the Center for Court Innovation, while we're known more for our work on criminal court programs, we also have been developing a civil justice portfolio. And so I'm curious to hear you talk about some of the work you've done on issues, and you're continuing to do, like debt collection and the assignment of council in housing court cases. And I guess the framework I have for my question is, making the case for this kind of research may rest on finding kind of practical implications that flow from the research. So I'm wondering if you can give us some examples of research you're doing and what you see as the things that a practitioner would immediately identify as, well that's really interesting and useful.

GREINER: Absolutely. So I guess, two examples. And I don't want to go on for too long, so stop me if this turns into a monologue, but two examples with respect to financial distress work which you mentioned, the debt collection work that you mentioned, one of the premises of that work is that there is just never going to be enough money, there's just never going to be enough of a social commitment to provide a lawyer at state expense for every individual who can't afford to hire a lawyer who has a legal problem. Even if we were to restrict that set to people who are facing court adjudications that affect basic human needs by some definition, which is the ABA's proposed definition. It's called the Civil Gideon Project, this idea that we can provide lawyers at state expense. I just don't think the resources are ever going to be there. Some smart economists have tried to calculate that and it's in billions upon billions of dollars would be needed.

With that in mind, it's probably going to take a battery of lots of different things to address the access to justice problems that we're having in United States society, and one of those things, one of those many things that we need, is going to be self-help. How're you going to make complex subjects accessible, complex procedures and complex principles accessible to folks who can't afford to hire lawyers for them so they can try to address legal problems they have on their own?

So the financial distress work that I'm doing is an attempt to try to say, well, if what we need are self-help materials, and we need basically to educate people about how to solve their own problems, what sorts of principles can we draw from non-law fields that would help us make self-help materials more effective, or would we hypothesize that would make self-help materials more effective?

For instance, if we're trying to get people to a particular location at a particular time of day and on a particular date to do something they would rather not do because they decide it's good for them, how do we persuade people to do that?

Well, if you pose the question at that level of generality, you recognize that... Voting. Political scientists study how you get people to vote on a particular day at a particular time. Public health officials study how to get people out to take flu shots, or to get colonoscopies on a particular day. These are all studies that have been run.

But in law, our self-help materials don't actually use any of the lessons from any of those other fields about how to get people to a court date. We think that's bad. We think that if good lessons are learned from those fields, we should try them in law and see if they work.

Similarly, there's an entire field called adult education, or just education period, about how you communicate complex concepts to people who need to be able to use them. But in law, our self-help materials don't currently use the lessons from adult education. And you can see that because if you read the legal self-help materials, they're word-heavy. There's just lots and lots of text. Whereas adult education folks, or education folks generally are saying, no, no, no. You've got to use images. So, in our case, we're using cartoons.

And so, the idea there is to say, let's try to create the best possible self-help materials that, under the state of the art, we currently know how to create using the lessons from all these different fields. But then, let's don't declare victory there. Because maybe there's something about law, or maybe there's something about the way that we did the self-help materials that would make them not effective, or maybe there's something they're not understanding. Or maybe the best possible self-help materials are just not enough in a system that is so complex that self-help is just not an option.

So we go test them. We go put them into the field and we do a randomized experiment. In one randomized experiment, we might randomize people to either get self-help materials or an offer of representation from an attorney. Or in another experiment, we randomize whether people get self-help materials versus no further assistance. Our self-help materials versus existing self help materials which, in many cases, just don't exist.

So that's one example. I've got another one but, again, I don't want to go on and on.

FOX: I'll just cut in for a moment to say the research you're doing on self-help or pro se council is relevant to us because we run a program called Legal Hands, which essentially employs volunteers who are not lawyers. So it's a bit of a blended model. This is people who are physically talking to a community member, so it's not something that you go online to get self-help materials. But they're not lawyers. And it speaks to your general point which is, it may not be possible to have a lawyer for every person on every case of relevance to them.

GREINER: Absolutely. I think what your program is doing is mixing a partial service-based solution with a partial self-help so it might be a sort of guided self-help. Absolutely. Again, I think part of the problem, one of the reasons why I find this field so much fun to operate in but also so frustrating is that we just don't have any evidence to suggest, to guide us as to when we need a partial service-based solution, partial self-help, and what settings a total self-help solution is sufficient. Or maybe it's not the legal setting, maybe it's the type of person. Which people can take care of it on their own if guided appropriately either by self-help materials, or which ones are going to need some service along with it. So absolutely.

FOX: Let's hear your other example.

GREINER: Another example that comes to mind is the problem of triage which is basically, whenever there is more of a particular type of case, or another way to put it, more clients than you have the capacity to provide the highest level of assistance to. So in our particular study that we've been trying to get underway and still in the planning stages for, this setting is victims of domestic violence who come to a legal services provider and seeking assistance with obtaining civil protection orders from courts, and the legal services provider has the capacity to assist in terms of providing full representation to. Around 1/3 of the folks who come to it who are eligible for the help, income and asset eligible, and who are actively seeking civil protection orders.

Now that by the way, that number, 1/3, would go down substantially if the legal services provider did more outreach in the community. So we've looked in the court systems where this LSP is operating and in fact a substantial number of people are already in the court system seeking civil protection orders without council. And so you only see the tip of the iceberg, and you can't even handle the tip of the iceberg, in terms of providing full representation, the traditional attorney/client relationship to these folks, much less the whole iceberg. So what do you do?

And it turns out that in medicine, again, to use that analogy, there's been lots of study about how to make these triage decisions. It's both an ethical question and an operational question. Ethically, what do you want to do when you're in that sort of scare situation, and that comes up both in medicine in terms of something like when you don't have enough organs for transplant, it comes up when you have Hurricane Katrina knocking out power in a hospital and you don't have enough evacuation capacity to get all of your patients out of your hospital. That was the subject of a recent Radio Lab episode. So there's an ethical dimension to that question.

There's also an operational dimension. Suppose what you wanted to do was to say, we should use these resources to save the most lives. Or, in a civil protection order context, we should use these resources to try to get as many people as possible civil protection orders because that's what they're for and that's best for the community. How would you do that? How would you go about doing that? And one way you would go about thinking about it, the way that the military thinks about it when we're talking about battlefield casualties is, you say, well try to distinguish people into 3 groups. One is a group of people who are going to be able to succeed on their own. So in battlefield casualties, these are going to be people who are going to survive the battlefield even if not assisted immediately. You don't provide immediate assistance to them. In the legal services provider, people who are going to be able to get civil protection orders on their own, you don't offer full representation to them.

For people who are never going to be able to survive on the battlefield, or never going be able to get civil protection orders on their own, even if given the full representation, you don't give them full representation either, because it's not affecting the outcome.

The people you do give battlefield assistance to, or full representation to, are the folks that you can change their outcome. If you give assistance, they'll survive or they'll get the civil protection order. If you don't give assistance, they won't survive or they won't get the civil protection order.

So that's one way to think about it. But if you want to do that, you face the problem of, how do you go about distinguishing those people. Because a lot of study and a lot of different settings, again, emergency room setting is one example, turns out that's a very hard thing to do. To figure out who you can actually change the outcomes for. So we're pursuing a randomized experiment that's designed to try to add to our knowledge of how do you distinguish those folks. And most importantly, whether lawyers, given, exercising professional judgment, can in fact predict which people really would benefit from the help.

FOX: And I guess what you're talking about raises a big issue for me, which is, let's give a concrete example, in New York city they're talking about investing potentially $200 million more a year in hiring lawyers to represent people facing eviction in housing court. So there is actually more money now in areas of the law there hasn't been a lot of money in the past.

However, you're no doubt correct that there isn't enough money to cover every case at every moment. And so I guess having an RCT or a research study is great progress towards this habit of thinking in a kind of, I would use the term problem solving way. But how do we get to more fundamental change? What's the way to inculcate this way of thinking in a legal services organization in a more fundamental fashion?

GREINER: It's a really hard question and one that we've been scratching our heads about in the Access to Justice Lab about. Because that's our overall mission. One way of thinking about this is, if the Access to Justice Lab is going to be one of the few institutions around the country that's going to be doing randomized studies in the law and we're going to depend on those few institutions to produce evidence, we should stop. We're never going to be able to produce enough evidence. Even as a lab and even as an institution, 1 or 2 institutions are never going to be able to do it. Just the same way one center, Johns Hopkins can never produce enough randomized experiments in medicine to make progress in medicine possible.

So we have to start a movement. We have to get people to think this way and to demand this evidence. In terms of the strategies we're pursuing in the Access to Justice Lab, we're, first of all, going to try to produce studies and try to educate folks about, hey, this evidence from this study can be useful in your practice, it can get you to think about ways other than the way you were thinking before, or ask new questions about what you should be doing to be more effective.

We're in the process of producing a short course so that folks who, either want to do these studies as researchers or to form partnerships with researchers to get the study done, so these field operators will know more about what's required and how to do them. We will be doing persuasion in terms of just appearing at conferences and giving speeches and talking to anyone who will listen to us about why this is so critical and, quite frankly, we think that there's an ethical duty to do these studies.

Because if you think about it in the civil protection order context, our view is that if we triaged more effectively, more people would have civil protection orders than currently do with no additional injection of resources. When you think about it that way, there may be an ethical duty to do these studies, as opposed to, oh, well they're just curiosities. We think ultimately that we're going to need to persuade funders to demand evidence. Maybe not demand the evidence, show it to me right now or I'll cut your funding, rather say, if you don't have the evidence, I want you to, this is a funder speaking to a legal services provider, I want you to engage in an evidence gathering study. And sure, I'll fund your efforts, your services as long as you are gathering evidence about whether those actually work.

FOX: Well, this is fascinating work. And thank you so much for agreeing to speak with us. Again, I'm Aubrey Fox. I'm the guest host of the New Thinking Podcast for the Center for Court Innovation and joined today by Professor James Greiner who's a Professor of Law at Harvard University and the Faculty Director of the Access to Justice Lab. They have a great website that you can go online and take a look at some of the studies that they're participating in. Thank you so much, Jim.

GREINER: My pleasure, thanks for the opportunity.