NIST Face Recognition Vendor Test - Craig Watson, Image Group Manager
Summary
A slide presentation on NIST's Face Recognition Vendor Test (FRVT) by Craig Watson, Image Group Manager. The slides cover verification (1-to-1), identification (1-to-N) and investigation, followed by sections titled Twins, Ageing, Quality, Morphing and Demographic Effects. The demographic effects slide states that false positives are larger than false negatives, that false positives in females are generally greater than in males, and that false positives increase in the oldest and youngest age groups. It also states that for verification (1-to-1) false positive variations of 10 to 100 times increase are seen for Black, Asian and American Indian versus White.
Summary drafted by a model from the document's text below and checked by script against that text before publication. It is a navigation aid, not a reading of what the document proves. Where AI is used
Cite this document
“NIST Face Recognition Vendor Test - Craig Watson, Image Group Manager.”
Archived at Pandemic Darlings, https://pandemicdarlings.com/source-documents/2020-01-31-a26393-d212196-cwatson-nist-wshrws-ited-538e883171/
, SHA-256 92a20f35e25e250585baacae86d50c7c17cd1b1b5c823ba6b035ca489621f105.Full text
NIST FRVT
Craig Watson
Image Group Manager
craig.watson@nist.gov
1
Face Recognition Vendor Test
(FRVT)
1. FRVT 1:1 2. FRVT 1:N 3. FRVT Q
4. FRVT Morph
Core Biometric Search Automated Quality
Detection
Operation Performance Assessment
FRVT Demographic Effects
2
Verification of ID (1-to-1, Threshold)
Yes Correct
No False Negative
Above Threshold?
Yes False Positive
No Correct
https://en.wikipedia.org/wiki/EPassport_gates
CC BY 2.0. File:Heathrow Terminal 5 ePassport
gates.jpg
Created: 16 July 2010
3
Identification (1-to-N, Threshold)
SEARCH Yes Correct
aka
PROBE No False Negative
Images from NIST Staff
Above Threshold?
Yes False Positive
No Correct
GALLERY
aka
DATABASE
aka
ENROLLMENT
SET
Images from NIST Staff + synthetic
4
Investigation (Ranked List, Human Examiner)
Candidate Lists
SEARCH
aka
PROBE
Images from NIST Staff
GALLERY
aka
DATABASE Human
aka Examiner
ENROLLMENT
SET
Images from NIST Staff + synthetic
5
Face Recognition Vendor Test
Current Capabilities
6
Improvement since 2010
NEC NEC NEC
2010 FRVT 2013 FRVT 2018 FRVT
Rank 1 hit rate Rank 1 hit rate Rank 1 hit rate
~86.0% 95.9% 99.7%
2010 2013 2015 2017 2018
7
Face Recognition Vendor Test
Current Capabilities - Twins
Identical Fraternal
Images from U. Notre Dame Twins Day Database
8
Twins: A demographic with very high FPIR
Gallery: n=169 n=919 n=24 n=174 n=1066
• N = 640 000
portraits, and 17.0
• 104 images of ALG: YITU_5
Twin A
16.5
TVAL
Similarity Score
FPIR = 0.001
Probes: 16.0
• Second FPIR = 0.003
images of FPIR = 0.010
Twin A 15.5
FPIR = 0.030
• Images of
Twin B
15.0
AA AA AB AB AB
fraternal identical fraternal fraternal identical
SameSex SameSex DifferentSex SameSex SameSex
Images: Twins Day, Ohio, via Notre Dame 9
Algorithms: FRVT 1:N November 2018.
Gallery: Twin A; Probe: Twin A or B; Type of Twin
Face Recognition Vendor Test
Current Capabilities - Ageing
2002-08 2004-10 2010-05 2012 2013-08 2018-06 2019-03
Images from NIST Staff
10
as.factor(degrader)
●●●●● ●●● ● ● ● ● ●
Performance in
●
●
N=00640000
N=01600000
yitu_2
yitu_3
●●●●●
●●●●●
●● ● ●
●● ● ●
●
●
●
●
●
●
●
●
N~3 perspective: What
million
●
●
N=03000000
microsoft_4
N=06000000 idemia_4
●●●●●
microsoft_3
●● ● ● ● ●
N
●
● ●●●●
●●●●●
●
●●●●●
● ● ● ● ●
● ● ● ●
● ● ●
●
●
●
●
●
●
●
● ●
●
●
●
ΔT matters more?
●
●
N=12000000 microsoft_0
Years Lapsed (00,02]
idemia_3
● ●●●●
● ●●●●
●
● ●
● ●
●
●
●
●
● ●
●
●
●
●
●
1. Algorithm
as.factor(degrader)
●
●
Years Lapsed (02,04]
visionlabs_5
●●●● ●
ntechlab_4
Years Lapsed (04,06]
●● ● ● ● ● ●
● ● ●●●
● ● ●● ● ●
● ●
● ●
●
●
●
●
● ●
●
●
●
●
● ●
as.factor(degrader)
● Population size
2.N=00640000
●
yitu_0 ● ●●●● ● ● ● ● ● ● ● ●
N=01600000
●
●
Years Lapsed (06,08]
cogent_1
Years Lapsed (08,10]
cogent_0
●●●●●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
N=00640000
N=01600000
N=03000000
N=06000000
● Ageing
3.N=03000000
Algorithm
● Years Lapsed (10,12] ● N=12000000
●
visionlabs_4 ● ● ●● ● ● ● ● ● ● ● ● ●
● Years Lapsed (00,02] N=06000000
Years Lapsed●●● ●● ● ● ● ● ● ● ● ● ●
●
Years Lapsed (02,04]
(12,14]
neurotechnology_4 ● ● ●● ● ● ● ● ● ● ● ● ●
● ● ●● ● ● ● ● ● ● ● ● ●
●
●
Years Lapsed (04,06]
Years Lapsed (06,08] ● N=12000000
● ntechlab_3
Years Lapsed (14,18] ● Years Lapsed (08,10]
isystems_2 ● ● ●● ● ● ● ● ● ● ● ● ● ●
●
Years Lapsed (10,12]
Years Lapsed (12,14]
● Years Lapsed (00,02]
nec_0 ● ● ●● ● ● ● ● ● ● ● ● ●
●
● Years Lapsed (14,18]
ntechlab_0 ● ● ●● ● ● ● ● ● ● ● ● ●
Years Lapsed (02,04]
● ●cognitec_1
●●● ● ● ● ●
● ● ● ● ●
● ●
●
●
● ●
●
● ● ● ● ● ● Years Lapsed (04,06]
● ● ● ● ● ● ● ● ● ● ● ● ●
●
megvii_0
nec_1 ● ● ● ● ● ● ● ● ● ● ● ● ●
Years Lapsed (06,08]
idemia_0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● Years Lapsed (08,10]
rankone_2 ● ● ● ● ● ● ● ● ● ● ● ● ●
● Years Lapsed (10,12]
cognitec_0 ● ● ● ● ● ● ● ● ● ● ● ● ●
● ●● ●●
neurotechnology_3 ●
●
● ● ●
●
●
● ● ● ●
● ●
● ● ● ●
● ● ● ●
● Years Lapsed (12,14]
rankone_0 ● ● ● ● ● ● ● ● ● ● ● ● ●
● Years Lapsed (14,18]
−0.2 0.0 0.2 0.4
20% 0% False negative identification rate (FNIR) 40%
11
● ● ●● ● ● ● ● ● ● ● ● ●
Face Recognition Vendor Test
Quality
12
Identification (1-to-N, Threshold)
False Negative Rates
Mugshot – Standard Quality Images, Constrained
Webcam – Non-Standard, Less-Constrained
Wild – Non-Standard, Unconstrained
13
Face Recognition Standardization: New Work
Quality Scalar Standard ISO/IEC 24357
Approved 2019-08
ISO/IEC 29794-5 REV
Quality Vector Standard Approved 2019-07
Quality
standards
Next Generation Capture Subsystem ISO/IEC 24358
Standard Approved 2019-08
Quality Performance Testing Standard Baseline metrics included in
ISO/IEC 29794-1 Framework revision
SC37 BIOMETRICS 14
Face Recognition Vendor Test
Morphing
Source: NIST Staff
15
Morphing: An attack on
Morphed Image
ID credentials
• Modern face recognition algorithms will
authenticate both subjects against morph
on ID credential
+ • Multiple people can now use the same
fraudulent yet genuine document
16
Two-Image Differential
Single-Image Morph Morph Detection
Morph detection with additional live capture image
Detection (e.g. morph on passport + eGate webcam photo)
Morph detection with single image in isolation
(e.g. initial passport application) Morphed image is This image represents
contained in a a live capture during
passport an eGate border
crossing, say.
Goal: Determine whether image being Goal: Determine whether image on passport is
submitted is a morph or not. morphed by using the additional information
available in the live capture image.
17
Face Recognition Vendor Test
Demographic Effects
• False Positives larger than False Negatives
• Demographic effects vary depending on application, data, and algorithm
• Impact of differentials vary application
• Most accurate algorithms have lower demographic differentials
• False Positives increase in oldest and youngest age groups
• False Positives in Females generally greater than Males
• False Positives greater in East/West African and East Asian than Eastern European
• Verification 1-to-1: False Positive variations of 10 to 100 times increase for
Black/Asian/American Indian versus White
• Identification 1-to-Many: Some developers so False Positives are undetectable
18
NIST FRVT
Craig Watson
Image Group Manager
craig.watson@nist.gov
19
Verification (1-to-1)
Mugshot Data
Female Male
Algorithm
Asian
Black
American Indian
White
1e-04 => 1 in 10,000
1e-03 => 1 in 1,000
1e-02 => 1in 100
20
False Match Rate
Verification (1-to-1)
Applicant Data
Male Female
7 – East Asia 7
6 – South Asia
5 – East Africa
6
4 – The Caribbean 5
3 – West Africa 4
2 – Central America 3
1 – Eastern Europe 2
1
1 2 3 4 5 6 7 1 2 3 4 5 6 7
21
Verification (1-to-1)
Applicant Data
Male Female
7 – East Asia 7
6 – South Asia
5 – East Africa
6
4 – The Caribbean 5
3 – West Africa 4
2 – Central America 3
1 – Eastern Europe 2
1
1 2 3 4 5 6 7 1 2 3 4 5 6 7
22
File and source
- File
- 2020-01-31_a26393_d212196_cwatson-nist-wshrws-ited.pdf
- Size
- 1,169,494 bytes
- SHA-256
- 92a20f35e25e250585baacae86d50c7c17cd1b1b5c823ba6b035ca489621f105
- Original
- No public link identified.