Master's thesis: Development and evaluation of iterative LLM-based code generation

Safe Intelligence — this forms the core brand of the Fraunhofer Institute for Cognitive Systems IKS . Connected cognitive systems drive innovation in many sectors, including mobility, healthcare, and automation in industry. Disruptive technologies such as artificial intelligence and quantum computing play a key role here. Fraunhofer IKS is conducting research to ensure that these applications are reliable and verifiably safe. We consider resilience and intelligence to be part of the same process.

What you will do

Current advances in generative AI are increasingly opening up new possibilities for automatic code generation. This is particularly relevant in industrial automation: PLCs form the backbone of many systems, and their programming in Structured Text (ST) is an established standard.
Although LLMs can already generate executable code from natural language requirements, the quality often falls short of expectations. An automated feedback loop that analyses generated code, converts errors into appropriate corrective feedback, and returns this to the LLM promises significant improvements. The aim of this master's thesis is to systematically evaluate the effectiveness of an LLM agent with such an automated feedback loop for the first time, using the example of generating ST code for PLCs.

Tasks:

• Systematic literature analysis on feedback mechanisms and quality metrics for LLM-based code generation.
• Identification of relevant metrics and tests (static/dynamic) that best reflect the code quality and functional safety of ST.
• Determination of the optimal format for transferring the metrics and test results to the LLM.
• Development of a prototype software module that automatically generates feedback in the defined format from the selected metrics and tests.
• Quantitative and qualitative evaluation of the feedback process by measuring the code improvements achieved through the feedback and comparing them with a baseline.

What you bring to the table

• Currently pursuing a degree in engineering, with a focus on informatics or AI
• Good knowledge in programming with python and on software development
• Knowledge on LLMs and langchain is a plus
• Analytical and structured mindset
• Good knowledge of the English and German language (B2).

What you can expect

• Participation in a dynamic team with innovative task areas and nice supervisors.
• Practical approach to your studies
• Excelent support for your thesis
• Collaboration on practical research projects.

We value and promote the diversity of our employees' skills and therefore welcome all applications - regardless of age, gender, nationality, ethnic and social origin, religion, ideology, disability, sexual orientation and identity. Severely disabled persons are given preference in the event of equal suitability.

With its focus on developing key technologies that are vital for the future and enabling the commercial utilization of this work by business and industry, Fraunhofer plays a central role in the innovation process. As a pioneer and catalyst for groundbreaking developments and scientific excellence, Fraunhofer helps shape society now and in the future.

Interested? Apply online now with your motivation letter, cv and a current transcript of records . We look forward to getting to know you!

Additional questions will be answered gladly by
René Beck per eMail: [email protected]
If you got further questions, please contact our colleagues from HR via eMail: [email protected]

Fraunhofer Institute for Cognitive Systems IKS

Requisition Number: 80650Application Deadline: