We started this series discussing how the quest for a perfect model on the first try often results in delays and frustration. Instead, a better approach is to identify specific business problems or decisions that need to be made, define what a “better” outcome would be if automation or machine learning (ML) was applied, and then aim for incremental improvements.
As covered in our second article, one approach to incremental improvements is to consider the various ways you can handle discrete micro decisions. These are the small decisions you make about some aspect of your day-to-day workflow when you are designing and deploying automation and machine learning models. Generally, best practice is to use a combination of technology and humans in the loop (HITL) when handling these micro decisions.
When automating decisions—or applying ML to predict outcomes that are inputs to the decision making process—there are always exceptions and edge cases that emerge. The teams that seem to be the most successful in applying automation and ML recognize this reality and have a proactive plan to address these exceptions, often with the help of a trained human workforce.
In this article, we’ll explore how CloudFactory worked with three companies, each of which had a challenge with exception handling or a need to address edge cases. Our managed workforce was able to successfully address their challenges by assisting with the development of training data or by supporting key micro-decision points.
1. Correcting Labeling Errors in Automated Video Annotation
A leading athletic performance analysis company uses an advanced computer vision model to provide coaching tools and video analysis support for a variety of club, youth, high school, collegiate, and professional sports. The company’s model reviews video footage and tags and tracks player movement throughout game and practice recordings so that individual and team performance can be analyzed.
Working with CloudFactory, the company needed training data set creation as it expanded its product suite, but also needed validation and optimization of model performance as well as exception handling. CloudFactory’s managed workforce reviewed a percentage of all processed games and matched results to validate the model’s accuracy in tagging and tracking players—which was especially important when the model had a low confidence (the “squishy middle”) in its tagging. The inputs from our trained workforce teams helped to improve model performance and avoid possible AI drift due to a proliferation of edge cases or deficiencies in the training data set.
This human in the loop workforce provided real-time quality control to correct errors and handle exceptions where the model couldn’t establish tracking accurately, helping the company meet its 24-hour service level agreements with clients. Additionally, the corrected data and the exception handling results were fed back into the model as training data, so the model continued to learn and improve performance over time.
Because no model is static, having a reliable workforce partner available to help with quality control and exception handling who can also identify and report trending edge cases is a must for a robust and productive data pipeline. An additional benefit of having the same team support the entire model is the ability to efficiently acquire and apply appropriate business context and knowledge throughout the model’s lifecycle.
2. Applying Human Judgment to Sensitive Edge Cases
A business intelligence company applies machine learning algorithms to publicly available information, like social media networks, to conduct analyses of content trends.
CloudFactory’s managed workforce conducted real-time reviews and tagging of social media posts and articles for content usage in categories such as language, alcohol use, etc. to support the company’s efforts. The team also solved for edge cases that the machine learning algorithm was not able to handle.
CloudFactory’s data analyst teams accurately matched individuals to their social media profiles to scrape the relevant information needed to train its ML models. The CloudFactory team worked with the company to develop processes for managing and logging edge and corner cases so that they could be accurately categorized.
The types of cases that proved to be problematic differed for each project, but edge cases are so common that you should always be on the lookout for the phenomenon. It’s reminiscent of helpful examples from our recent LinkedIn Live guests:
- Dean Abbott’s “squishy middle” with truck invoice processing
- James Taylor’s human-in-the-loop task for disability claims
- Ian Barkin’s “fishy” money laundering indicators
3. Handling Exceptions in Real-Time for Automated Business Processes
Ibotta is a mobile shopping app that offers rebates to retail consumers. Their team built and deployed an optical character recognition (OCR) automation system to transcribe receipts submitted using their app. Accuracy is important, because the data transcribed from receipts determines if a retail consumer who used the app will get a rebate. And perhaps the most daunting challenge: Ibotta wanted to overcome OCR error rates for transcription in real-time.
As we’ve explored in this three-part series, exception processing is inevitable with automation, so it is prudent to plan for it from the very beginning. Near real-time exception handling is difficult to do in-house and the end-user experience is affected when it becomes their responsibility. Ibotta wanted a better option.
Ibotta purposefully created a tech stack with humans in the loop, because it knew that its proprietary OCR automation would not be able to process 100% of the receipts with 100% accuracy. It tapped CloudFactory to process exceptions, such as unreadable text on a receipt or questionable data (e.g., duplicate rebate submission). CloudFactory’s team also managed increases in volume of work as new brand partners chose Ibotta to offer rebates to their retail customers.
CloudFactory provided workforce support so Ibotta could create high-quality end user experiences. With the assistance of CloudFactory’s managed teams, Ibotta was able to route exceptions to people to apply their expertise and judgement. Doing this achieved faster turnaround times, higher accuracy, and structured data that could be used to improve its OCR automation system.
Successful Automation Requires Human-Computer Collaboration
Machine learning solutions are imperfect and an effective analytics solution must anticipate this fact. In our LinkedIn Live interviews with Dean Abbott, James Taylor, and Ian Barkin, all three identified instances where machine learning, decision management, and RPA systems need occasional human attention in real-time.
Identifying these requirements is not a weakness of the system, but rather a strength. If they are identified they can be addressed and built into the system.
Pursuing a solution that has complete reliance on a machine learning model to solve the entire problem 100% of the time is ultimately both less realistic and less cost-effective.
Acknowledging the reality of exception processing allows organizations to address the problem with a diverse tool kit, including around-the-clock access to training and managed external workforces. CloudFactory offers the best option in building that resilient collaboration between human and machine to maximize value and bring it sooner.
This is the final article in a three-part series.
Data Labeling ML Models AI & Machine Learning Exception Processing
