In 1910, it was preposterous to think that a manufacturer could produce and assemble automobiles quickly, with high quality, at scale. Henry Ford challenged that notion when he set out to mass-produce the Model T, resulting in the moving assembly line. The basic concept of that model - bringing the work to the workers - transformed the way products were mass-produced, creating a production model that is still in use by manufacturers.
Today, business leaders have the opportunity to apply similar efficiencies to data labeling for machine learning. Data operations are a lot like the assembly lines of yesterday’s industrial factories: data is the raw material, and teams have to get it through multiple processing and review steps to prepare it for machine learning (ML).
That’s why we see a growing number of data scientists, ML engineers, and AI project teams creating virtual data pipelines to clean, label, and structure the data that uses supervised learning to train machines.
For example, it takes millions of data points, labeled and structured accurately, to make it possible for self-driving vehicles to navigate city streets. Quality is critical because autonomous systems are only as good as the data used to train them: garbage in, garbage out.
Putting the [data] parts in place
A high-performing data pipeline requires a strategic combination of smart people, tools, and operations that can deliver high accuracy consistently across entire datasets. Here are four essential considerations for companies looking to apply this concept to their operations to accelerate quality data processing at scale.
- Combine technology and people – Think about the data pipeline as a tech stack with humans in the loop. People and machines are merged together in a workflow that directs tasks where they are best suited for high performance. People should be assigned tasks that require domain expertise, context, and adaptability. Machines can be given tasks that require repetition, measurement, and consistency.
Technology is also important in communication among workers. The right tools will give leaders direct contact with team members and the ability to view work quality and worker productivity. They will also allow workers to share insights with each other, which helps everyone on the team adjust almost seamlessly as tasks and business requirements evolve. - Use a trained, managed workforce – Dedicated, professionally managed data teams can deliver high accuracy because the same workers are on the same project over time. So, they get better at making decisions as their familiarity with the data increases and they gain a deeper understanding of how their work fits into the larger project. One recent study found managed teams delivered higher quality data labeling than crowdsourced workers.
When it comes to the management of workforces that label, annotate or enrich data, it’s best to have a closed feedback loop with a single point of contact working alongside the data team to facilitate direct communication. This person should be an expert in the data and business rules of the company, able to provide feedback, speed change requests, and train new team members. - Measure quality – Leaders need to define quality, identify how to measure it, and consider how important quality is across many tasks, today and into the future. In the data workforce industry, there are four primary methods we use to measure quality work:
- Consensus – When several people are assigned to do the same task, the correct answer is the one that comes back from the majority of workers.
- Gold standard – There is a correct answer assigned to a task, and quality is measured based on correct and incorrect tasks.
- Sample review – A random sample of completed tasks is selected, and a more experienced worker, such as a team lead or project manager, reviews the sample for accuracy.
- Intersection over union (IoU) - This is a consensus model often used in object detection within images. It combines people and automation to compare labeled data with the predicted bounding boxes from your model.
Managed workforce providers can and should typically use one or more of these methods to check the quality of their teams’ work. Be sure to partner with a workforce provider that is transparent about its quality metrics, productivity, results, and methods. - Design for agility – This is about more than getting workers to label data faster. It’s about anticipating use-case progression as teams design and validate their ML models. Teams should insist on the flexibility to add higher-level features that can advance their AI applications, even if those opportunities haven’t yet been surfaced.
Workforce services that charge by the hour, rather than by the task, are designed to support iteration in the work. Paying by the task can incentivize workers to complete tasks quickly, but without emphasis on quality. The best bet here is to look for options that get more cost-effective as the work scales.
Like Ford’s Model T challenge, AI demands an ambitious roadmap – one that combines short production cycles with a focus on quality and continuous improvement. It also requires agility when the tasks, process, or requirements change. Just like Ford, today’s business leaders are competing to design and launch the next big AI product. The winners will be those who strategically deploy people and technology along the way.
Data Labeling Workforce Strategy AI & Machine Learning Annotation QC
