How do AI jury models stack up?

AI is ever evolving, but for the time being, it is not proving to be accurate in predicting case outcomes. Credit to Nick Schweitzer at Trial by Data for running all of these tests on various AI models which show results are all over the place.

In this first image, you can see the win rate comparisons between AI and Big Data (with real people as jurors). I will add that I also ran a 4-panel mock trial on this case before we ran data and can attest to how strong the case was with jurors (though mocks are not statistically accurate):

Big Data win rate: 97-98%
AI win rates: 20-85%

‍

In this next image, you can see the awards. This case went to trial so we can tell how accurate data was compared to AI:

Big Data predicted value: $22.8M

AI predicted value: $1M-$20M (I believe Nick asked the $1M AI to “think harder” and then it produced $100M!).

Trial result: $25M

‍

We also have testing of some of the big named AI jury platforms which I won’t mention by name but which are not testing any better despite their claims of being accurate.

At the time being, AI is off by staggering amounts. It may not always be this way and I will stay on top of it but at this time, I cannot recommend AI testing of cases.

‍