Job
- Level
- Senior
- Location
- Munich, Ingolstadt
- Working Model
- Onsite
- Job Field
- Data, Test/QA
- Employment Type
- Full Time
- Contract Type
- Permanent employment
Job Summary
In this role, you will develop the AI evaluation engine for voice systems, conduct tests, evaluate quality, and optimize generated dialogues to make informed decisions and ensure software functionality is error-free.
Job Technologies
Your role in the team
- Build and own the AI evaluation engine: discrete (e.g., 1-6) and continuous (0-100%) scoring with confidence values and partial scores, efficient evaluation via embeddings and cosine similarity, with selective LLM-as-judge only where it earns its cost and latency.
- Define and maintain the quality metric set for an automotive assistant: hallucination rate, answer faithfulness and relevance, intent fulfillment, latency P95, BLEU / WER / CER - and defend their thresholds as release criteria.
- Develop generative test case and dialogue generation: LLM-based test case creation per domain (navigation, media, knowledge, car control) with configurable volume, complexity, scenario types and edge-case coverage; automatic generation of test descriptions, expected dialogue flows, success criteria and evaluation prompts.
- Make generated dialogue realistically human: hesitations, filler words, incomplete sentences and colloquial speech instead of rigid scripts; generative multi-turn conversations where follow-up turns are produced dynamically from the assistant's actual answer.
- Engineer test variability and reliability: randomised test parameters (language, voice characteristics, speaking rate, audio characteristics, dialects), complexity variants (short / nominal / complex) that preserve semantic intent, and automatic retry with clean separation of infrastructure failures from functional defects.
- Build prompt and model regression suites in LangFuse so prompt changes, model version upgrades and backend releases are caught before they reach vehicles - including version-over-version comparison and trend analysis.
- Contribute to audio simulation and multi-speaker testing: realistic background noise (engine, road, music, phone calls, sirens, horns, passenger conversation), mixed-language input, multi-speaker simulation from different seat positions with position-specific evaluation prompts, and comparison of synthetic against real native-speaker recordings.
- Cover conventional test engineering alongside the AI layer: unit, integration and system tests across domain, adapter and infrastructure layers (pytest, testcontainers), automated smoke and release tests, and the supplier speech-stack test suites.
- Support test execution in real vehicles and on the test rack (TISAX Level 3 environment), including device health checks, failover behaviour and triage of infrastructure versus assistant defects.
- Deliver reporting and analytics that non-engineers can act on: test statistics by batch and device, exports (PDF/CSV/Excel), version comparisons, performance trends and regression detection between software releases.
- Maintain traceability between test cases, requirements and defects, including versioned test cases with change history and diff views, and integration with the client's test specification and requirements management systems.
- Coordinate with the backend, frontend, AI, and hardware teams and with the client's departments; report quality status to management in a form that supports go/no-go decisions.
This text has been machine translated. Show original
Our expectations of you
Qualifications
- Practical command of LLM quality metrics - faithfulness, hallucination rate, relevance, plus at least one of BLEU / WER / CER - and the judgement to know what each does and does not prove.
- Prompt regression testing in practice: detecting quality drift across prompt edits and model version changes.
- Solid conventional test engineering: pytest, code coverage, test case design across positive, negative, boundary and regression cases; testcontainers or equivalent integration test tooling.
- CI/CD integration of automated test suites (Azure DevOps Pipelines, GitHub Actions) and comfort with Docker; working knowledge of REST APIs and SQL/PostgreSQL for test data and results.
- English C1, and willingness to work onsite in the Ingolstadt/Munich area, including hands-on test sessions in vehicles and on the test rack.
Experience
- 5+ years in test automation / quality engineering with strong Python, including ownership of a test framework rather than only writing cases in someone else's.
- Hands-on experience evaluating LLM or other non-deterministic systems: building evaluation pipelines, LLM-as-judge patterns, embedding/cosine-similarity scoring, and reasoning about scoring reliability and inter-rater agreement.
- Experience with LangChain / LangGraph or equivalent, and with an LLM observability tool (LangFuse, Phoenix, Langsmith or comparable) for tracing, prompt versioning and evaluation runs.
- Experience integrating ASR and TTS services (Azure Speech, Whisper, Cerence, Google TTS or equivalent), including streaming/real-time behaviour.
This text has been machine translated. Show original
Benefits
Work-Life-Integration
Topics You Will Work On
Job Locations
About Your Employer
Luxoft
As a leading IT service provider, Luxoft offers innovative technology solutions that help global enterprises achieve measurable business results. Services include strategic consulting, custom software development resources, and technical solution development. This allows companies to compete successfully by utilizing our expertise in industries such as financial services, automotive, communications, and healthcare & life sciences.
Description
- Company Type
- Digital Agency
- Working Model
- Hybrid, Onsite
- Industry
- Internet, IT, Telecommunication
Employer reviews
by devworkplaces.com
Total
(3 Reviews)4.5
Culture
4.4Career Growth
4.7Engineering
4.3Workingconditions
4.6
Test Engineer - AI Voice Systems
- Location
- Munich, Ingolstadt
- Working Model
- Onsite
- Diversity
- Open for all genders
