Strong people skills; seamlessly work across disciplines
1. Build long-lasting relationships with external parties to guide the future direction of human evaluation
-Some experience with contract negotiation, security controls.
2. Strong technical aptitude, with experience and knowledge of training data generation for machine learning purposes.
-Required experience with training and evaluation data generation for text processing (NLP)
-Experience with a range of common data and file formats, general knowledge of AWS ecosystem
-Experience performing annotations and developing reference “golden” datasets
3. Oozes comfort and confidence with data, especially in the face of disparate data sets
-Experience handling and managing content with sensitive data containing PII (personally identifiable
4. Ability to break down complex tasks for consumption by a diverse, not immediately technical audience
-be a part of one of the coolest AI-based product
-Establish a human annotation process, combining the services of internal annotators with those of external annotation vendors;
-Act as the process flywheel to ensure evaluation tasks are prioritized, loaded, and worked on; drive evaluation quality and throughput;
-Define guidelines and user interfaces for human evaluation tasks in close collaboration with the Product team;
-In close collaboration with internal and external resources, monitor and report on the quality of the data generated through the human annotation process;
-Conduct vendor evaluations; provide ongoing recommendations to the Product team for the human evaluation vendor and platform of choice;
-Generate data sets for measurement and experimentation; source external data as required
Manage the human evaluation budget.
The mission for the Technical Program Manager, Data, is to own and drive project management relating to investments in size and quality of People Graph, including human evaluation at scale for measurement and model training. Example tasks include:
-Data set acquisition/generation
-Vendor selection and engagement
-Acting as the process flywheel, flagging issues and risks as they arise, reporting on status.