Harvey,
the
legal
AI
company
whose
valuation
recently
hit
$11
billion,
recently
released
what
it
is
calling
the
Legal
Agent
Benchmark,
or
LAB
—
an
open-source
evaluation
framework
designed
to
measure
how
well
AI
agents
can
perform
extended,
real-world
legal
work
rather
than
the
discrete
reasoning
tasks
that
have
dominated
legal
AI
benchmarks
to
date.
Announced
May
6
in
a
post
by
Harvey
researchers
Niko
Grupen,
Gabe
Pereyra
(Harvey’s
cofounder),
and
Julio
Pereyra,
the
first
version
of
LAB
contains
more
than
1,200
tasks
spanning
24
legal
practice
areas,
graded
against
more
than
75,000
expert-written
rubric
criteria.
The
code
and
a
portion
of
the
dataset
are
available
on
GitHub.
“The
goal
of
LAB
is
to
provide
a
clear
picture
of
how
agents
can
be
deployed
to
support
legal
work
in
the
real
world,”
the
researchers
write.
“By
articulating
where
agents
can
do
all,
some,
or
none
of
a
task,
LAB
helps
law
firms
measure
the
ROI
of
AI
investments
and
where
such
investments
can
augment
their
teams’
work.”
Notably,
Harvey
is
launching
LAB
without
a
leaderboard.
The
company
says
it
will
work
with
research
partners
over
the
coming
weeks
to
produce
baseline
results
and
publish
standards
for
normalizing
submissions
before
any
rankings
appear.
“We’re
intentionally
launching
LAB
without
a
leaderboard
because
we
expect
the
dataset
to
evolve
over
time
and
we
want
to
work
with
the
community
to
ensure
results
are
clear
and
intuitive
in
how
they
convey
agent
performance,”
Harvey
says.
What
LAB
Tests
In
creating
LAB,
Harvey
says
that
existing
legal
AI
benchmarks
—
including
LegalBench,
CUAD,
LEXam,
and
Harvey’s
own
earlier
BigLaw
Bench
—
measure
short-horizon
reasoning,
such
as
ability
to
read
a
contract,
answer
a
question,
compare
cases,
or
analyze
an
argument.
LAB
is
meant
to
measure
something
closer
to
the
unit
of
work
that
actually
gets
delegated
inside
a
law
firm.
Each
LAB
task
is
structured
around
four
elements
that
mirror
an
associate’s
assignment:
-
An
instruction
written
as
a
partner-to-associate
request
—
short
(averaging
50
words)
and
framed
as
what’s
needed
rather
than
how
to
produce
it. -
An
environment
built
as
a
client
matter,
with
a
closed
universe
of
documents
that
the
agent
must
sort
through.
Materials
include
both
relevant
files
and
peripheral
ones
the
agent
has
to
learn
to
ignore. -
An
output
that
has
to
be
reviewable
legal
work
product,
not
just
an
answer. -
Verification
through
expert
rubrics
that
break
the
deliverable
into
atomic
pass/fail
criteria
covering
facts,
conclusions,
citations,
severity
ratings,
recommendations,
deadlines,
dollar
amounts,
and
formatting.
To
illustrate
the
structure,
Harvey
uses
a
fictional
corporate
M&A
example.
It
involves
a
$458
million
all-equity
acquisition
of
Crestview
Software
Solutions
in
which
the
agent
must
review
a
virtual
data
room
containing
eight
material
contracts
plus
adjacent
documents
such
as
a
10-K
and
a
deferred
compensation
plan,
identify
change-of-control
provisions
across
the
matter,
assess
deal
risk,
recommend
next
steps,
and
produce
a
draft
memorandum
for
the
deal
team
and
board.
The
rubric
for
that
single
task
contains
57
criteria
covering
nine
legal
issues
planted
across
the
materials.
LAB
uses
what
Harvey
calls
“all-pass”
grading,
meaning
that a
task
is
marked
complete
only
if
every
rubric
criterion
passes.
There
is
no
partial
credit.
The
rationale
is
that
a
deal
memo
that
catches
eight
of
10
material
risks
is
not
80%
useful.
One
missed
issue
could
blow
up
the
transaction
or
surface
as
a
problem
post-closing.
The
24
practice
areas
in
the
initial
release
span
transactional,
advisory,
regulatory
and
litigation
work.
Harvey
says
future
versions
will
expand
within
those
areas,
add
new
practices,
and
eventually
move
beyond
law
firms
to
in-house
legal
work
and
adjacent
professional
services
like
asset
management
and
banking.
Why
a
Benchmark?
Harvey’s
thesis
is
that
benchmarks
have
served
as
leading
indicators
of
capability
inflection
points
in
other
agentic
domains
—
most
visibly
in
software
engineering,
where
benchmarks
such
as
SWE-Bench
Verified
and
Terminal-Bench
2.0
tracked
the
shift
that
AI
researcher
Andrej
Karpathy
summarized
by
saying
coding
agents
“basically
didn’t
work
before
December
and
basically
work
since.”
Harvey
argues
that
similar
benchmarks
(GDPval,
OSWorld-Verified,
BrowseComp,
FinanceAgent,
and
others)
are
now
extending
legibility
to
knowledge
work,
web
research,
financial
analysis
and
professional
services.
Harvey
positions
LAB
as
the
legibility
layer
for
legal
agents.
The
use
case
Harvey
describes
for
law
firms
is
straightforward:
identify
the
workflows
where
agents
perform
well
enough
to
be
delegated
under
a
“review
pattern,”
identify
the
workflows
where
they
don’t
and
need
to
stay
heavily
human-in-the-loop,
and
make
deployment
and
ROI
decisions
accordingly.
For
most
firms,
that
may
matter
more
than
technical
details.
The
legal
industry
has
spent
two
years
cycling
through
vendor
demos
and
pilot
programs
without
a
shared
way
to
answer
the
question
every
managing
partner
and
innovation
lead
is
being
asked,
which
is
where,
specifically,
can
we
put
these
things
to
work?
A
credible,
public
benchmark,
particularly
one
structured
around
actual
deliverables
rather
than
multiple-choice
questions,
could
change
that
conversation.
Of
course,
it
could
also
complicate
it,
by
revealing
how
far
agents
still
are
from
autonomous
practice
in
many
areas.
Practical
Applications
of
LAB
To
my
mind,
a
few
practical
applications
of
LAB
jump
out:
-
For
law
firms,
LAB
offers
a
reference
point
for
vendor
evaluation.
A
firm
evaluating
competing
products
could,
in
theory,
ask
each
vendor
to
report
performance
on
specific
LAB
practice
areas
and
compare
results,
rather
than
rely
on
vendor
demos
and
case
studies. -
For
vendors,
LAB
offers
a
public
yardstick
for
claims
about
agent
capability.
Harvey
has
acknowledged
contributions
from
a
substantial
list
of
labs
and
companies
(including
Anthropic,
OpenAI,
Nvidia,
Google
DeepMind,
Mistral,
LangChain,
Fireworks,
Snorkel,
Mercor,
and
Stanford
LIFTLab),
which
suggests
the
major
frontier
labs
see
value
in
a
shared
evaluation
context
for
legal
agents. -
For
researchers,
LAB
provides
a
longer-horizon,
domain-specific
task
set
that
they
can
use
for
evaluation,
fine-tuning
and
post-training
work. -
For
legal
journalists
and
analysts,
LAB
could
provide
something
more
useful
than
vendor-supplied
claims
about
their
products
—a
way
of
actually
putting
those
claims
to
the
test.
The
Bottom
Line
It
is
worth
noting
that
LAB
is
a
benchmark
built
by
a
market
participant.
Harvey
is
a
dominant
and
well-funded
legal
AI
vendor,
and
the
company
has
not
been
shy
about
its
commercial
positioning.
The
tasks
and
definitions
of
“legal
work
product”
within
LAB
reflect
choices
about
what
good
legal
work
looks
like,
and
those
choices
were
made
by
Harvey’s
team
in
consultation
with
its
research
partners.
None
of
that
makes
the
benchmark
unreliable,
but
it
is
something
the
legal
community
needs
to
keep
in
mind
going
forward.
There
is
also
the
question
of
what
exactly
is
the
impact
of
“open
source”
in
this
context.
In
a
post
at
Alt-Counsel,
Houfu
Ang
argues
that
legal
open
source
is
not
really
a
community
but
rather
“a
federation
of
solo-author
archipelagos.”
He
points
specifically
to
projects
that
come
from
well-funded
vendors
such
as
Harvey,
whose
repositories
are
maintained
almost
exclusively
by
in-house
staff
in
what
the
Open
Source
Initiative
calls
“Open
Source
theatre.”
Virtually
none
of
these,
Ang
argues,
graduate
from
individual
showcase
to
sustained
codebase
with
outside
contributors.
Even
so,
LAB
is
the
most
ambitious
public
attempt
yet
to
measure
what
legal
AI
agents
can
actually
do
on
the
kind
of
work
law
firms
actually
delegate.
Whether
it
becomes
the
shared
yardstick
Harvey
wants
it
to
be
will
depend
on
how
the
leaderboard
rolls
out,
how
transparently
submissions
are
normalized,
and
how
much
room
the
project
leaves
for
outside
contributors
to
shape
what
gets
measured.