Job
- Level
- Senior
- Job Field
- Software, Data
- Employment Type
- Full Time
- Contract Type
- Permanent employment
- Location
- Berlin
- Working Model
- Full Remote
Job Summary
In this role, you will develop agentic intelligence systems, evaluate agent performance, and design the necessary evaluation and ML models to efficiently optimize complex accounting workflows.
Job Technologies
Your role in the team
- Are you looking to have an impact on the daily life of millions of entrepreneurs in France (and tomorrow in Europe)? Are you looking for a work environment that values trust, proactivity, and autonomy? Are our Engineering principles aligned with your vision? Then Pennylane is the right place for you!
- By joining Core AI as an AI Engineer, you'll play a pivotal role in large projects, bringing your engineering and machine learning expertise to build the agentic intelligence, quality and models that our solution squads rely on.
- Build and maintain the agent harness - the core runtime that orchestrates LLM calls, tool use, context construction, memory and guardrails - and make it reliable in production.
- Own the evaluation stack: design eval pipelines, golden datasets, LLM-as-judge and human-eval workflows; track agent performance, regressions and failure modes, and turn production failures into systematic improvements.
- Perform prompt and behaviour engineering and retrieval/RAG at scale, defining what "good" looks like for agents completing complex accounting workflows end to end.
- Host and improve our own small models where it creates leverage: serving and inference of open-source LLMs, help fine-tuning and post-training (SFT / DPO / RL), and document-extraction models - for performance, cost and sovereignty.
- Work closely with Product teams and domain experts (accountants) so that fulfilling real user needs stays at the center of what we build.
This text has been machine translated. Show original
Our expectations of you
Qualifications
- Treat evaluation as a first-class discipline: golden datasets, LLM-as-judge, human eval, A/B testing, and measuring agent quality, regressions and edge cases.
- Have a good grasp of applied LLM/ML and AI infrastructure (model serving, vector databases, cost and latency).
- Have exposure to model fine-tuning and post-training (SFT, DPO, RL).
- Have a balanced blend of technical, business and product skills, communicate well (including with non-technical domain experts), and are fluent in English (French is not mandatory).
Experience
- Have 5-8 years of experience and are very strong in Python, with hands-on experience building LLM and agentic systems at scale in production - prompting, tool use, context construction, RAG, and handling failure, state and reliability (not just calling a model API).
- To thrive at Pennylane, you need: To speak English (levels are assessed and appreciated according to the department you're applying to), To be energized by an ever-shifting work environment, To be highly collaborative (within your team and with other stakeholders), To be sufficiently experienced to prioritize business-led actions in your day-to-day activity.
This text has been machine translated. Show original
What we offer
- Within one month: You'll learn everything about our company, our teams and our vision during the first onboarding week.
- You will become familiar with our stack and AI tooling (agent harness, evaluation tooling, model providers, MCP), and deliver a few small contributions that give you a concrete taste of our tools and processes.
- You will be given time to meet your future stakeholders and gain a deep understanding of our product and operations.
- Within 3 months: You'll be in charge of items in our roadmap, defining and prioritizing your tasks autonomously.
- You'll be comfortable with our technical stack (Python, agentic frameworks, evaluation tooling, model serving and AWS).
- You'll contribute to larger cross-team projects.
- Within 6 months: You'll proactively contribute to the team's roadmap.
- You'll work with engineers and data practitioners on improving our stack, AI platform and evaluation practices.
- You'll share your learnings and best practices within the team.
- And beyond - the team will keep growing with the company, which means: Opportunities to recruit and mentor new team members, Increased accountability in project leadership, Responsibilities to design and implement new processes, tools and best practices so your team works even more efficiently.
- Wherever you are based, you'll get 25 vacation days paid by Pennylane.
- You'll have a competitive compensation package.
- You'll get company shares to enjoy a piece of the success story you're building with us.
- You will have a budget to transform your home into a more comfortable workspace, as well as a monthly allowance to work from a coworking space whenever you feel like it.
- Through our partner Gymlib, you'll have access to 8,000 fitness spaces in Europe and more than 300 wellness activities.
- You'll have access to Busuu to perfect your English or your French.
- You'll get the latest Apple equipment.
- Depending on the team and the requirements of the position, you'll be able to work remotely from your country of residence, as long as it is in Europe and within a maximum time difference of two hours from the CET time zone.
- We regularly come together for company events such as Tech Days (which bring remote Pennylaners together every 2 months) and our annual company seminar.
This text has been machine translated. Show original
Benefits
Work-Life-Integration
Topics that you deal with on the job
Job Locations
This is your employer
Pennylane
As a French fintech startup, Pennylane offers an innovative cloud platform for accounting and financial management specifically designed for SMEs and tax advisors. The solution promotes real-time collaboration and integrates over 300 business applications.
Description
- Company Type
- Startup
- Working Model
- Full Remote, Hybrid, Onsite
- Industry
- Internet, IT, Telecommunication
AI Engineer
- Location
- Berlin
- Working Model
- Full Remote
- Diversity
- Open for all genders
- English Only
- English only required
