Senior Software Engineer – AI Evaluation

Senior Software Engineer – AI Evaluation About the Role What if your engineering skills could directly determine whether the world's most advanced AI systems are actually working? We're looking for Senior Software Engineers to design, build, and scale the evaluation infrastructure that measures AI performance — the critical layer between raw model output and real-world trust. This is high-impact, technically challenging work at the intersection of software engineering and AI research. You'll build the tools, pipelines, and frameworks that help leading research teams understand what their models can do, where they fail, and how to make them better. If you love building robust systems and care deeply about quality and measurement, this role puts you at the center of the AI revolution. Organization : Alignerr Type : Hourly Contract Location : Remote Commitment : 20–40 hours/week What You'll Do Design and build scalable evaluation pipelines and frameworks for assessing AI model performance across diverse tasks and domains Develop automated testing harnesses, scoring systems, and benchmarking tools for large language models and other AI systems Write clean, production-quality code to process, analyze, and visualize evaluation datasets at scale Create and maintain APIs, dashboards, and internal tools that enable research teams to run, track, and compare evaluations efficiently Collaborate with AI researchers and data scientists to translate evaluation methodologies into reliable, repeatable software Identify edge cases, failure modes, and reliability issues in AI outputs through systematic engineering approaches Optimize system performance, data processing speed, and infrastructure costs Contribute to the architecture and technical direction of the evaluation platform Write clear documentation and participate in code reviews to maintain high engineering standards Who You Are 5+ years of professional software engineering experience, with a track record of building and ship