Hybrid Senior AI Scientist, LLM

at Resaro

• Job title: Senior AI Scientist, LLM
• Job type: Full Time
• Experience level: Senior
• Salary: Not specified
• Degree requirement: Postgraduate Degree
• Tech skills: AWS, Cloud, Google Cloud Platform, Python
• Location requirements: Hybrid, Munich, Germany

About the role

• Senior AI Scientist at Resaro, developing AI evaluation frameworks for generative AI systems. Engaging in deep learning applications, mentoring, and enhancing model performance metrics.

Responsibilities

• Lead the design, implementation, and execution of robust frameworks to evaluate the performance of generative AI systems, including text and multi-modal models.
• Establish and refine metrics and benchmarks for model quality, including output fidelity, diversity, creativity, and bias detection.
• Perform technical AI evaluations, benchmarking and “red-team” tests on large language models, including assessing them for robustness in performance, embedded biases, vulnerability to jailbreaks and prompt injection attacks.
• Work with clients and more junior team members to design custom evaluation approaches using the latest scientific research that address the client’s needs.
• Work with the product management team to develop a suite of technical and analytical AI evaluation frameworks and tools that are backed by scientific research and methods.
• Lead the design and implementation of evaluation frameworks for Large Language Models (LLMs), including but not limited to GPT-based models, BERT, T5, and other state-of-the-art architectures.
• Define and refine metrics for evaluating model performance, such as perplexity, BLEU, ROUGE, accuracy, coherence, factual consistency, and bias detection.
• Lead efforts in curating and managing large, high-quality datasets for evaluating LLMs, ensuring data is representative, unbiased, and ethically sourced.
• Mentor junior data scientists, guiding them in best practices for LLM evaluation and the latest advancements in NLP.
• Stay up-to-date with the latest advancements in Natural Language Processing (NLP) and LLM evaluation, applying cutting-edge methods and tools to improve model performance.

Requirements

• Extensive experience as a data scientist training or deploying deep learning based natural language models/large language models in real-world contexts.
• About 5-8 years of working experience or a relevant postgraduate degree with 2+ years of working experience building and deploying LLMs.
• Strong experience in evaluating LLMs using metrics such as perplexity, BLEU, ROUGE, and human-centered evaluation techniques.
• Proven track record of managing and analyzing large, complex language datasets, including text preprocessing and tokenization.
• Excellent written and verbal communication skills, with the ability to clearly explain complex technical concepts to diverse audiences, including non-technical stakeholders.
• Solid programming skills in Python and experience building automated pipelines for continuous model evaluation.
• Passion and interest in applied research on the safe and responsible use of AI and with large language models.
• NICE TO HAVE: Published research in the field of generative AI or model evaluation.
• Hands-on experience with model explainability tools and methods.
• Familiarity with cloud-based platforms (e.g., AWS, GCP) for scalable model evaluation and deployment.

Benefits

• Health insurance
• Professional development