We
talk
a
lot
about
the
ethical
duty
of
lawyers
and
legal
professionals
to
understand
the
risks
and
benefits
of
relevant
technology.
But
when
it
comes
to
using
GenAI,
that
might
not
be
enough.
If
we
want
to
prevent
the
increasing
number
of
hallucinations
and
inaccurate
citations
that
are
bedeviling
lawyers
and
even
judges,
we
need
to
understand
how
and
why
GenAI
systems
fail.
That
was
the
point
of
a
recent
paper
by
a
group
of
scientists
and
engineers:
Dylan
Restrepo,
Nicholas
Restrepo,
Frank
Huo,
and
Neil
Johnson.
The
paper
carried
the
lengthy
title,
When
AI
Output
Trips
to
Bad
but
Nobody
Notices:
Legal
Implications
of
AI’s
Mistakes.
In
addition
to
their
own
calculations
and
analysis,
the
group
also
consulted
a
couple
of
lawyers:
Daniela
Restrepo
and
Jean
Paul
Roekaert.
I
can’t
vouch
for
the
mathematical
calculations
but
what
they
conclude
squares
with
my
own
experience.
The
Basic
Premise
The
group
concludes
at
the
outset
that
rather
than
a
random,
unpredictable
glitch,
a
physics-based
analysis
demonstrates
that
hallucination
is
a
“foreseeable
engineering
risk.”
Meaning,
of
course,
the
circumstances
generating
its
occurrence
can
be
at
least
a
little
predictable.
According
to
the
paper,
GenAI
systems
have
“a
deterministic
mechanism
at
its
core
that
can
cause
output
to
flip
from
reliable
to
fabricated
at
a
calculable
step.”
And
that
step
unfortunately
comes
when
the
lawyer’s
need
is
the
greatest.
The
group’s
analysis
starts
from
the
proposition
that
we
should
know
by
now:
GenAI
is
“a
probabilistic
text
generator
engineered
to
predict
the
next
most
plausible
token
in
a
sequence,
without
any
internal
concept
of
legal
truth.”
It
is
not,
argues
the
group,
a
database
of
verified
legal
authorities.
(The
group
focused
on
the
publicly
available
systems
and
not
on
the
closed
systems
that
claim
to
rely
on
verified
legal
authorities.)
What
This
Means
Because
it’s
predicting,
not
analyzing,
GenAI
does
well
when
faced
with
inquiries
about
valid
legal
principles,
logical-sounding
arguments,
undisputed
case
facts,
procedural
history,
and
the
like.
But
when
faced
with
something
novel
and
complex,
the
tool
is
pushed
“into
a
region
where
training
data
is
sparse.”
In
an
effort
to
please
and
respond,
it
is
then
prone
to,
well,
make
stuff
up.
The
paper
puts
it
this
way:
The
tool
is
therefore
most
prone
to
failure
exactly
when
the
lawyer’s
need
is
greatest:
on
a
difficult
point
of
law
with
sparse
precedent.
The
act
of
researching
an
unsettled
legal
issue
via
an
LLM
becomes
the
principal
trigger
for
the
tipping
instability.
These
are
important
points
since
lawyers
live
in
a
world
where
a
hallucination,
an
error,
can
have
devastating
consequences.
So,
as
we
have
discussed,
given
that
risk,
GenAI
outputs
must
be
checked
over
and
over,
often
mitigating
the
cost
savings
of
using
the
tools
in
the
first
place.
But
if
we
understand
why
the
errors
occur
and
more
importantly
when,
we
can
better
and
more
safely
use
the
tools.
A
Blessing…And
a
Curse
If
true,
then
the
group’s
findings
are
a
blessing
since
it
suggests
a
sliding
scale
of
verification:
less
where
the
output
focuses
on
well-known
information
and
much
more
when
it
strays
into
the
novel.
Saves
time
and
energy.
But
for
those
uninformed
of
this
predictability,
the
fact
that
failure
can
occur
at
a
certain
point
can
be
a
curse.
Why:
a
lawyer
with
a
legal
project
often
starts
with
undisputed
facts,
then
seeks
information
on
what
the
law
generally
is
with
respect
to
the
issues
at
hand.
And
then
goes
to
more
complex,
ambiguous
areas
thinking
it’s
okay.
The
example
given
in
the
paper
is
a
statute
of
limitations
question.
A
lawyer
starts
their
use
of
ChatGPT
by
plugging
in
undisputed
facts.
They
then
seek
the
general
law
with
respect
to
the
limitation
period.
All
well
and
good:
the
lawyer
gets
correct
responses
and
then,
in
the
words
of
the
paper,
“gains
confidence
in
the
tool.”
So,
the
lawyer
then
begins
asking
for
more
ambiguous
information
about
how
that
law
can
be
used
to
leverage
the
facts
or
to
develop
arguments.
If
the
lawyer
takes
all
the
outputs
and
prepares
a
brief
based
on
the
information
obtained,
they
(or
their
supervisor)
might
be
tempted
to
spot
check
the
first
few
paragraphs,
find
nothing
amiss
and,
when
pressed
for
time,
conclude
the
rest
of
the
outputs
are
also
fine
when
they
are
not.
So,
the
blessing
becomes
a
curse:
“AI’s
period
of
correct
output
increases
rather
than
decreases
the
risk
of
harm,
because
it
builds
the
user’s
trust
just
before
the
fabrication
appears.”
What
To
Do
So,
what
do
we
make
of
all
this?
Again,
I’m
no
scientist
but
I
do
know
from
experience
that
the
more
general
information
I
seek
from
GenAI,
the
more
prone
it
is
to
be
correct.
When
I
stray
into
more
ambiguous
areas
where
there
is
less
known
about
a
subject,
the
more
errors
I
tend
to
get.
For
example,
I
once
asked
for
information
about
a
well-known
painter.
I
got
great
information.
But
when
I
asked
about
another
painter
in
the
same
school
of
painting
who
was
relatively
obscure,
the
tool
just
made
up
a
name.
Or
when
I
asked
what’s
the
subway
stop
to
take
to
catch
the
Q70
bus
to
LaGuardia
Airport,
it
got
it
right.
When
I
asked
the
best
route
from
my
hotel
(which
involves
more
ambiguity),
it
sent
me
to
the
wrong
stop.
It
did
say
sorry
when
I
pointed
out
the
error
(after
some
argument).
The
point
being
for
lawyers
and
legal
professionals
is
to
understand
that
“AI
possesses
no
independent
legal
agency:
it
is
a
computational
tool.”
Granted,
it
is
a
computational
tool
with
which
you
can
converse
like
a
human.
It
reacts
in
human
ways.
It’s
tempting
to
anthropomorphize
it.
But
that’s
where
we
go
wrong.
Instead,
we
need
to
start
with
thinking
of
it
not
as
a
person
but
a
product
with
a
foreseeable
engineering
risk.
Like
a
sharp
knife
or
an
ATV.
A
risk
that
appears
to
materialize
when
faced
with
novelty
and
ambiguity.
But
it’s
that
novelty
and
ambiguity
that
creates
the
greatest
risk
of
hallucination,
according
to
the
paper.
For
lawyers,
that
means
if
you
are
going
to
use
this
sharp
knife,
you
better
know
how
and
in
what
circumstances.
You
need
to
know
how
to
do
that
safely.
The
paper
says
it
the
best:
“The
duty
of
technological
competence,
as
expressed
in
ABA
Model
Rule
1.1
and
its
state-level
counterparts,
must
evolve.
It
is
no
longer
sufficient
for
a
lawyer
to
know
how
to
operate
a
piece
of
software.
Competence
now
requires
a
practical
understanding
of
how
that
software
can
fail.”
That
it
is
clearly
right
about.
Want
to
use
GenAI?
Use
it
to
access
known
information
that
would
be
time
consuming
or
difficult
to
otherwise
get.
Ask
it
to
do
a
lot
of
things
where
accuracy
isn’t
that
important.
But
don’t
ask
novel
or
unsettled
legal
questions,
without
checking
and
double
checking
what
you
get
back.
Else
you
might
get
off
at
the
wrong
subway
stop.
Or
much
worse.
Stephen
Embry
is
a
lawyer,
speaker,
blogger,
and
writer.
He
publishes TechLaw
Crossroads,
a
blog
devoted
to
the
examination
of
the
tension
between
technology,
the
law,
and
the
practice
of
law.