Job
In this role, you'll be responsible for operational quality and QA monitoring, analyzing AI responses in real-time, and checking compliance with exact guidelines while logging issues in Jira or Linear.
This listing has been machine translated. Show original
Job technologies
Your role in the team
- You will own end-to-end overnight operational quality, prompt analysis, and QA monitoring across all live customer channels.
- Monitor live AI customer conversations and system alerts across chat, phone, and email channels in real time.
- Evaluate AI responses against stringent insurance guidelines and exact policy wording for compliance and accuracy.
- Inspect system prompts, rules, and underlying logs to diagnose root causes of model failures and edge cases.
- Run regression test suites across new prompt iterations and system version updates.
- Log detailed bug reports with reproducible context, evidence, and clear severity ratings in Jira or Linear.
- Deliver concise, actionable handover shift reports for the European morning team.
Our expectations of you
Qualifications
- AI-native professional comfortable reading system prompts and evaluating LLM outputs.
- Highly autonomous operator thriving in dedicated asynchronous ~21:00-06:00 UTC shifts.
- Strong written communicator who logs clear, evidence-backed bug reports.
- Regulated industry background (fintech, insurance, legal) is a strong plus.
- Professional German proficiency (C1+) is a significant advantage.
Experience
- 2+ years of QA or structured software/AI testing experience.
What we offer
- Extreme operational autonomy with high ownership over critical live systems.
- Zero corporate red tape: no steering committees, no endless sprints, same-day deployments.
- Direct impact working on cutting-edge production AI applications serving real customers.
- Flexible remote schedule ideally suited for daytime/afternoon hours in APAC and Americas time zones.
Topics you will work on
Job locations
About your employer
WhyBrilliant GmbH is an innovative HR tech company based in Berlin, focusing on AI-driven career and recruiting processes. The platform connects job seekers and companies through conversational assistance, providing support for both sides. Founded in 2026, the company operates on a success-based placement model.
- Startup
- Full Remote, Hybrid, Onsite
- Recruitment Agency
More jobs
These jobs might also interest you
Participation in security analyses and assessment of applications and IT systems. Support in the design and implementation of technical security measures (e.g., WAF, Load Balancing, Firewall, etc.). Conducting and supporting penetration tests,...
Operation, monitoring, and further development of the network infrastructure. Administration and configuration of firewalls, routers, and switches. Ensuring network availability, performance, and security. Participation in IT security concepts and...
You’ll be joining at a stage where we’re building and evolving our new CRM platform. That’s exactly what makes this role special: you won’t just develop features—you’ll actively help shape the technical foundation of a modern CRM ecosystem. Together...