Customer Facing Data Scientist (LLM)

Deepchecks’ LLM evaluation is a significant advancement in AI validation, built upon the foundations of our open-source ML testing package (1M downloads, 3K GitHub stars), and addressing concerns related both to quality and to risk.

Our solution specializes in evaluating LLM-based apps, enabling robust automatic scoring, version comparison, and auto-calculated metrics for properties like relevance and grounded-in-context.

Its standout feature is the ability to accurately estimate the expected annotation for LLM responses, surpassing common methods like GPT-as-a-judge. The solution is designed to enable both “coders” and “clickers” to iterate quickly while taking full control of both quality and compliance.

About the job

As a Customer Facing Data Scientist you will be at the core of our most important effort: how do we solve our customers’ LLM evaluation pains at scale. You will work with some of the world’s best AI groups and with our top notch product data science team. You will play a critical role in ensuring that our customers effectively use LLM Evaluation for testing and monitoring and have full ownership for identifying the best path of implementing the solution into our users’ LLM pipelines. Thus helping them to prevent hallucinations, create automatic evaluation flows, and enable continuous performance tracking and improvement. Being customer facing means that you will be working with our customers to understand their needs, help them customize the solution and investigate issues. This role requires expertise in data science and proficiency in Python. Communicating with customers means that you will need to present your ideas concisely, know how to find a way to work around customers/prospects’ objections and develop relationships. The ideal candidate holds a relevant degree and possesses hands-on experience managing LLM solutions in an operational production environment along with a passion to make our customers successful.

We’ll be going through a lot together, so we want your character and mindset to be a good fit for a fast-moving startup.

Required experience:

• B.Sc. in a quantitative field.
• At least 2 years of industry experience in AI/ML roles
• Exceptional problem-solving skills and the ability to troubleshoot complex technical issues.
• Strong collaboration and communication skills to work effectively with cross-functional teams.
• Customer-oriented mindset.

Advantage if experienced with:

• Major advantage – experience in a customer-facing position
• Developing LLM-based systems.
• MLOps or experience with ML/data engineering.
• Experience as a system engineer or technical project manager