Machine Learning Evaluation Specialist

Machine Learning Evaluation Specialist (AI Training) About the Role AI is only as good as the challenges used to test it — and we need experts like you to design them. We're looking for machine learning specialists with deep domain knowledge to create the hard, research-level problems that state-of-the-art AI systems genuinely struggle with. Your work will directly shape how the next generation of AI models is evaluated and improved. This is a rare opportunity to apply your hard-earned expertise at the cutting edge of AI development. Organization : Alignerr Type : Hourly Contract Location : Remote Commitment : 10–40 hours/week What You'll Do Design complex, original ML problems rooted in your specialized research domain Craft evaluation tasks that expose the limits of today's most capable AI systems Draw from your own research experience to build challenges that go beyond standard ML pipelines Write clear problem statements, define evaluation criteria, and provide gold-standard solutions Review and assess AI-generated solutions for correctness, creativity, and methodological rigor Document problem difficulty, required domain knowledge, and expected failure modes Collaborate asynchronously with a global team of researchers and engineers Who You Are Graduate-level expertise (MS or PhD preferred) in a scientific or technical field that intersects with machine learning Strong working knowledge of ML methods — model selection, feature engineering, evaluation metrics, and pipeline design Deep familiarity with active research problems in your domain Ability to identify exactly where general ML knowledge breaks down and specialized insight becomes essential Experience publishing or conducting original research is highly valued Excellent written communication — you can articulate nuanced technical problems with precision and clarity Self-motivated and comfortable working independently on intellectually demanding tasks Example Domains We're looking for experts across a wide r