Fully Automated vs. Semi-Automated Verifications: Incorporating Humans in the Document Verification Process

Ivo Strandjev Director of Engineering

The Trade-Off Triangle: Accuracy, Speed, and Cost

There’s no such thing as a perfect fraud detection solution. Both automated models and humans are prone to errors. More time spent on a task generally leads to greater accuracy, but it can never completely eliminate mistakes. Therefore, any document verification system must balance three crucial factors:

  • Accuracy: The ability to correctly identify both genuine and fraudulent documents.
  • Speed: The time it takes to process a document.
  • Cost: The financial investment required for the verification process.

Identity document verification is unique in that fraudulent documents are relatively rare. However, the cost of missing a fraudulent document can be very high, depending on the use case. Conversely, rejecting genuine documents or causing delays for valid customers leads to friction and lost profit.

Fully Automated vs. Semi-Automated: The Pros and Cons

  • Fully automated systems are lightning-fast, often processing documents in seconds. While automated fraud detection technology has advanced, human experts remain the gold standard for accuracy. However, relying solely on experts is costly and difficult to scale.
  • Semi-automated systems (also known as “human-in-the-loop” systems) offer a middle ground. They leverage automation for speed and efficiency but strategically incorporate human review for complex or high-risk cases.

Measuring Performance: BPCER and APCER

Two key metrics evaluate document verification systems:

  • Bona Fide Presentation Classification Error Rate (BPCER): The rate at which genuine documents are incorrectly rejected (similar to “false rejection rate”).
  • Attack Presentation Classification Error Rate (APCER): The rate at which fraudulent documents are incorrectly accepted (similar to “false acceptance rate”).

Different parameters within the same use case (reviewing transactions) may prioritize one metric over another. For example, approving small transactions might favor a higher BPCER (accepting more genuine documents, even if a few fraudulent ones slip through) to minimize friction. Conversely, large transactions demand a lower APCER, focusing on security even if it means rejecting more genuine documents initially.

Balancing Assurance and Conversion with Human Review

Fully automated systems often allow configuration to adjust the balance between:

  • High assurance: Prioritizing a low APCER (minimal fraud acceptance) at the potential cost of a higher BPCER (more genuine document rejections).
  • High conversion: Prioritizing a low BPCER (accepting most genuine documents) at the risk of a higher APCER (more fraud acceptance).

However, even with adjustments, a very low APCER might result in an unacceptably high BPCER, impacting customer satisfaction.

Human-in-the-Loop Strategies

Semi-automated systems offer several ways to incorporate human expertise:

  1. High-Risk Review: Humans review only high-risk transactions (e.g., large amounts).
  2. Review of Suspected Fraud: Humans review all documents flagged as potentially fraudulent by the automated system.
  3. Three-Way Classification: The system categorizes documents as “definitely fraud,” “definitely genuine,” or “uncertain.” Humans review only the “uncertain” category. This allows independent control of APCER and BPCER, achieving both high assurance and high conversion.

Additional Benefits of Human Review

Beyond improving accuracy, human review can:

  • Validate Automated System Performance: By manually checking a random sample of decisions, businesses can verify the accuracy of the automated system on their specific data.
  • Assess Human Annotator Performance: The same random sampling technique can be used to evaluate the performance of human reviewers themselves.

In these cases, although human review is included, these are not technically human-in-the-loop processes.

Conclusion

Document verification requires a nuanced approach that balances speed, accuracy, and cost. Semi-automated systems, strategically incorporating human review, offer the flexibility to adapt to different risk tolerances and use cases, providing the best of both worlds: the efficiency of automation and the accuracy of human judgment.

June 20, 2024

Discover Our Solutions

Exploring our solutions is just a click away. Try our products or have a chat with one of our experts to delve deeper into what we offer.