DHS Put Identity Verification Vendors to the Test. The Results Raise Some Tough Questions.

Only one of seven commercial document validation systems evaluated by the U.S. Department of Homeland Security (DHS) met all three performance thresholds discussed in a recent Microblink Trust Talks webinar. For organizations relying on remote identity verification to prevent fraud and onboard legitimate customers, that finding raises an important question: How much confidence should businesses place in vendor-reported accuracy claims?

During the webinar, Arun Vemury, Senior Advisor for Biometrics and Identity Technologies at DHS, joined Tyler Brutsman, Manager of Sales Engineering at Microblink, to discuss findings from the agency’s Remote Identity Validation Evaluation Rally (RIVR). Their conversation explored how DHS evaluates commercial identity verification technologies, why sophisticated counterfeit documents remain difficult to detect, and what the results reveal about the industry’s progress. 

Their discussion highlighted an important reality: document verification technology has advanced considerably, but accuracy remains a complex problem, and independent testing is essential to understanding where systems perform well and where they fall short.

Why independent identity verification testing matters

The rapid expansion of remote identity verification during the COVID-19 pandemic created an urgent need for businesses and government agencies to establish trust without requiring people to appear in person.

As adoption accelerated, questions emerged about how well the underlying technologies actually performed.

Vemury explained that when DHS began examining available research, it found relatively little objective information that would allow organizations to compare systems consistently. Vendors used different evaluation methods, datasets and definitions of success, making meaningful comparisons difficult.

RIVR was designed to help close that information gap by providing independent, publicly accessible performance data.

The evaluation examined three areas of remote identity verification: document validation, facial matching between a selfie and an identity document, and liveness detection. The webinar focused primarily on document validation, which determines whether a photographed identity document is genuine or fraudulent.

Participation was voluntary, and vendors could remain anonymous in the published results. Seven commercial document validation systems were evaluated under standardized conditions, with participating systems identified by labels such as DVS1 through DVS7.

For businesses evaluating technology, this approach provides something that vendor marketing materials cannot: a common basis for understanding how different systems respond to the same testing challenges.

The challenge of detecting sophisticated counterfeit documents

One of the most revealing aspects of RIVR was the type of fraudulent documents used in the evaluation.

DHS collected photographs of 2,032 genuine state-issued identity documents from consenting volunteers. It also worked with the Homeland Security Investigations Fraudulent Document Laboratory and U.S. Customs and Border Protection’s Fraudulent Document Analysis Unit to assemble counterfeit documents for testing.

These were physical counterfeit credentials, not AI-generated document images.

According to Vemury, some were sophisticated enough to be difficult to distinguish from genuine documents even when examined in person. Yet producing them did not necessarily require an enormous investment.

“These aren’t things that cost thousands of dollars,” Vemury explained, noting that some cost approximately $50 to $100.

That distinction matters because physical counterfeit documents can reproduce visual and physical characteristics that make them challenging to detect through a smartphone camera.

As Vemury explained, identity documents were not originally designed to be authenticated using ordinary smartphone photographs. Important security features may require ultraviolet or infrared examination, while other features can be difficult to assess from a static image.

Remote verification systems must therefore make authenticity decisions using a more limited set of signals than an examiner with specialized equipment might have available.

For businesses, the implication is clear: detecting sophisticated document fraud requires more than recognizing whether an ID looks convincing.

Why accuracy cannot be reduced to a single number

The RIVR evaluation assessed document validation performance across three important measures.

System Error Rate (SER) measures how frequently a system fails to produce a usable decision. These failures can create problems for legitimate customers who cannot complete verification.

Document False Rejection Rate (DFRR) measures how frequently genuine identity documents are incorrectly rejected or cannot be successfully processed.

Document False Acceptance Rate (DFAR) measures how frequently fraudulent documents are incorrectly accepted as genuine.

Each metric captures a different operational concern. False acceptances expose organizations to fraud, while false rejections can prevent legitimate customers from accessing services. System errors introduce additional friction and uncertainty.

During the webinar, Brutsman emphasized the business consequences of these distinctions. A system that aggressively rejects suspicious documents might appear effective from a fraud prevention perspective, but that protection becomes less valuable if it also rejects too many genuine customers.

Vemury discussed the 10% performance thresholds referenced in NIST guidance, explaining that those figures reflected the state of the technology when earlier testing was conducted. He stressed that the longer-term objective should be to reduce error rates much further, ideally below 1%.

The RIVR results demonstrated how much room for improvement remains.

Of the seven document validation systems evaluated, only one met all three of the performance thresholds discussed during the webinar. Brutsman also noted that three of the seven systems had at least one performance measure approaching the equivalent of a coin flip.

These findings reinforce why buyers should examine multiple performance dimensions rather than relying on a single advertised accuracy figure.

They also demonstrate the value of independent benchmarks in identifying meaningful differences between commercial technologies.

Performance can change depending on the device and document

Another important finding was the degree to which verification performance varied across smartphones and document types.

For the document validation assessment, DHS used photographs captured with an iPhone 14, Google Pixel 7 and Samsung Galaxy S22.

The goal was not simply to determine whether a system could identify genuine and fraudulent documents. It was also to understand whether performance remained consistent when images came from different devices.

Vemury explained that some systems performed well with certain smartphones but encountered significantly greater difficulty with others. Results also varied according to the state that issued the identity document.

That variability introduces an important consideration for organizations serving large and diverse customer populations.

A verification system might demonstrate strong performance with one combination of device and document but produce less reliable results with another. Businesses need to understand whether their chosen technology performs consistently across the devices and credentials their customers actually use.

Vemury also raised an accessibility concern. Not every customer owns the latest smartphone, and lower-end or older devices may introduce additional image quality challenges.

For organizations operating at scale, performance consistency is therefore not merely a technical consideration. It can directly affect who can successfully complete an identity verification process.

Laboratory performance is only the beginning

Although independent testing provides valuable benchmarks, laboratory results should not be treated as guarantees of performance in every real-world environment.

RIVR used controlled image collection conditions to help establish a consistent basis for comparison. Actual customers, however, capture identity documents in far less predictable circumstances.

Someone might photograph an ID while sitting in a moving vehicle, standing outside in bright sunlight or using a phone in a dimly lit room. Motion blur, reflections, camera limitations and poor positioning can all affect the quality of the captured image.

Vemury noted that earlier DHS testing had identified substantial error rates even with carefully captured images. Introducing more difficult conditions would not necessarily improve those results.

This distinction is particularly important for businesses that rely on remote verification as a primary onboarding mechanism.

Independent evaluations can help organizations identify promising technologies and understand their limitations. But buyers should also evaluate how those technologies perform within their own customer journeys, including the devices, document types and capture environments they expect to encounter.

Brutsman emphasized the value of combining independent benchmarks with an organization’s own testing and due diligence.

The goal is not simply to find the vendor with the strongest headline result. It is to determine which technology can deliver the necessary balance of fraud prevention, reliability and customer experience for a particular use case.

Building a more transparent future for identity verification

The webinar also explored how DHS plans to continue evaluating identity technologies as both verification capabilities and fraud techniques evolve.

Vemury described RIVR as an ongoing effort, with future evaluations expected to incorporate updated smartphones, additional document types and increasingly sophisticated attacks.

Generative AI adds another dimension to that challenge. Although the document validation assessment discussed in the webinar focused on physical counterfeit credentials, Vemury acknowledged that AI-enabled attacks are becoming more sophisticated and will require continued attention.

The discussion also addressed mobile driver’s licenses and verifiable digital credentials, which introduce different approaches to establishing document authenticity.

Unlike a photograph of a physical driver’s license, a mobile driver’s license can incorporate cryptographic mechanisms that allow information to be verified digitally. These technologies could eventually change how organizations approach remote identity verification, although their adoption and implementation present additional considerations.

For now, the central lesson from RIVR is the importance of independent measurement.

Identity verification has become essential infrastructure for financial services, government programs, marketplaces and other digital businesses. As more organizations depend on these technologies, understanding their actual performance becomes increasingly important.

Independent testing helps make that possible by giving vendors clearer feedback, helping buyers make informed decisions and establishing measurable expectations for improvement.

For organizations evaluating identity verification providers, the takeaway is straightforward: ask how accuracy was measured, examine the trade-offs between fraud detection and legitimate customer acceptance, and look for independent evidence that supports vendor claims.

The strongest verification strategy begins with understanding not only what a technology can do, but also where its limitations lie.

To explore the findings in greater detail, visit the Maryland Test Facility for DHS remote identity validation evaluation results.