Quality and Test Manager Full Stack AI Systems

Monitoring

• Performance and E2E Monitoring, Response Time Analysis, Tracing
• Alerting on the given infrastructure
• Creation of technical and professional dashboards for monitoring based on evaluation
• Design and implementation of logging and audit mechanisms
• Ensuring Controlled AI Outputs and Traceability

QS-Management and Test Concepts

• Design, documentation, and implementation of measures to ensure high quality throughout the entire lifecycle.
• Documentation of established processes, measures, tools, and artifacts for quality assurance (e.g., test concept, test manual)
• Support in continuous quality assurance within the CI/CD process (e.g., automated smoke tests, nightly builds)
• Creation of reports and analyses to support decision-making
• Design and implementation of logging and audit mechanisms

Requirement-related QA / Test Data Generation

• Support in functional and technical requirements specification, documentation, and validation.
• Design and creation of test data (gold standard) for defining and verifying the quality standards of the conversation.
• Support in establishing editorial quality assurance (tone, comprehensibility, usability appropriateness)

Design, implementation, and execution of tests

• Support in the implementation of component testing (module and unit tests in React, Java, Python)
• Support in the implementation of interface tests (PACT, OpenAPI)
• Design and implementation of integration tests
• Design and implementation of automated end-to-end tests (Catchpoint)
• Design and implementation of automated UI tests
• Design, documentation, and execution of risk-based testing

AI-specific QA

• Design and implementation of ML- and LLM-specific tests (e.g., prompt stability, regression, response behavior)
• GenAI Pentesting, Prevention of Prompt Injection
• Simulation and validation of dialogic contexts (e.g., rule-based/generative transitions)
• Analysis and evaluation of generative responses (e.g., relevance, compliance)
• Validation and traceability of AI outputs including logging and audit trails

Usability, Accessibility, and Performance

• Conceptualization, execution, and evaluation of usability tests with target group(s)
• Preparation and support of usability and accessibility tests (BITV 2.0)
• Design, preparation, and support of load and performance tests
• Design and implementation of technical and business reporting dashboards as well as alerting (e.g., Grafana, Loki)

Handover to Line Operations

• Complete documentation of all artifacts and results
• Quality assurance of technical and professional delivery artifacts
• Development and handover of operational and maintenance concepts, including incident management and escalation procedures.
• Conducting workshops, training sessions, and handover meetings with the line operations team.
• Handover of CI/CD pipelines and deployment scripts, as well as monitoring and logging structures to the operations team.
• Preparation of a handover report with recommendations for action and documentation of open issues
• Detailed specifications suitable for tendering/procurement of operational services