Data Analyst Specialist - Generative Ai Team Porto

Data Analyst Specialist - Generative Ai Team
Porto
Porto, Porto District, Portugal

We are seeking a highly skilled Data Scientist to join our team in Portugal. As a key member of our Gen
AI team, you will play a crucial role in evaluating, measuring, and assessing Agentic systems that aim to improve the wellbeing of all users.

Your primary responsibilities will include collaborating with product managers and engineers to define and measure metrics for user engagement, satisfaction, and chatbot effectiveness;



performing data cleansing, transformation, and quality analysis on product data;



assisting and designingrobust evaluation metrics to assess agentic AI performance;



crafting reusabletemplates and clear instructions to measure system behavior;



designing monitoring systems that track LLM behavior in production;



and creating and maintaining interactive dashboards using BI tools for
- time visualization of performance metrics and insights.

In this role, you will have the opportunity to work collaboratively in a team environment, communicate effectively across different departments, and explain complex technical concepts to
- technical stakeholders. You will also be part of a multidisciplinary team of product managers, engineers, and domain experts who are passionate about creating innovative AI applications that solve
- world problems.

We are looking for an exceptional individual with a strong background in data science and Generative AI or LLMs. Prior experience in metric design, data preparation, evaluation and testing, prompt engineering, LLM observability, and A/B testing is mandatory. If you are a motivated and talented Data Scientist looking to make a meaningful impact in the field of AI, we encourage you to apply for this exciting opportunity.

The ideal candidate will have a Bachelor's degree in Computer Science, Data Science, Machine Learning, Statistics, or a related field;



proficiency in Python for data handling and analysis;



advanced No
SQL and SQL skills for querying large datasets;



strong
- solving abilities, with a focus on experimental design and data analysis;



excellent verbal and written communication skills;



and fluency in English.

We offer a range of benefits and perks that support the health and
- being of our employees, including access to digital fitness programs and online wellness resources, additional fitness subsidy, flexible work arrangements, paid time off, parental leave, career growth opportunities, and a culture that values trust, flexibility, and integrity.

• Metric Design & Analysis:
  
  Collaborate with product managers and engineering teams to define and measure metrics for user engagement, satisfaction, and chatbot effectiveness;
  
  
  
  
• Data Preparation:
  
  Perform data cleansing, transformation, and quality analysis on product data;
  
  
  
  
• Evaluation and Testing:
  
  Assist and design robust evaluation metrics to assess agentic AI performance, combining automated evaluation (LLM-as-judge), adversarial testing,
  -
  -
  - loop evaluations, and custom behavioral metrics;
  
  
  
  
• Prompt Engineering:
  
  Craft reusable templates and clear instructions to measure system behavior, optimizing for challenging edge cases;
  
  
  
  
• LLM Observability:
  
  Design monitoring systems that track LLM behavior in production, capturing key metrics around information retrieval, hallucinations, and latency;
  
  
  
  
• A/B testing:
  
  Design, run, and analyze A/B tests to optimize chatbot interactions and user engagement;
  
  
  
  
• Dashboard
  Development:
  
  Create and maintain interactive dashboards using BI tools (e. G. , Tableau, Power BI, Looker, Metabase) for
  - time visualization of performance metrics and insights.

Main Requirements

• Bachelor's degree in Computer Science, Data Science, Machine Learning, Statistics, or a related field;
  
  
  
  
• Proficiency in Python for data handling and analysis;
  
  
  
  
• Advanced No
  SQL and SQL skills for querying large datasets;
  
  
  
  
• Strong
  - solving abilities, with a focus on experimental design and data analysis;
  
  
  
  
• Ability to workcollaboratively in a team environment and communicate effectively across different departments;
  
  
  
  
• Excellentverbal and written communication skills, with the ability to explain complex technical concepts to
  - technical stakeholders;
  
  
  
  
• Fluent in
  English.