Ilmu Komputer & AI editorial
Open AccessOA2026
On the Prospects of Dynamic LLM Conversations in Software Development
A four-month longitudinal study on minimal interventions in developer-LLM interactions
Annemarie Wittig; Alina Mailach; Janet Siegmund; Norbert Siegmund· 2026· DOI 10.48550/arXiv.2608.30756
The core problem
Large language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively support their interactions during development activities. The quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. The authors' goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developers---be it harmful or beneficial. To this end, they conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) a *context*-aware group received intent-based conversation augmentation, (2) a *proactive* group received follow-up suggestions and tailored advice, and (3) a *control* group without intervention. The augmentations are deliberately minimal: (i) to reduce confounding factors and (ii) to isolate treatment effects.
Innovation
Analyzing interaction logs and user surveys revealed no major differences in interaction patterns across the three conditions. This indicates no detectable harmful effects in the measured outcomes when intervening in interactions. Moreover, the authors observed trends of increased satisfaction with the *proactive* treatment. The results suggest that even with minimal interventions, dynamic guidance mechanisms for developer-LLM interactions show observable effects. The absence of major differences in interaction patterns implies that the interventions did not disrupt the development workflow. The trend of increased satisfaction in the proactive group points to a potential benefit of follow-up suggestions and tailored advice, though the effect was not statistically strong enough to be conclusive in this study.
Large language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively support their interactions during development activities. The quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. The authors' goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developers---be it harmful or beneficial. To this end, they conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) a *context*-aware group received intent-based conversation augmentation, (2) a *proactive* group received follow-up suggestions and tailored advice, and (3) a *control* group without intervention. The augmentations are deliberately minimal: (i) to reduce confounding factors and (ii) to isolate treatment effects.
The study employed a four-month longitudinal design with third-semester computer science students engaged in a full-stack Web development project. Participants were assigned to three conditions:
Why it matters
The findings indicate that minimal interventions in developer-LLM interactions can produce observable effects, particularly in terms of user satisfaction. The lack of harmful effects suggests that such interventions are safe to implement. The trend toward increased satisfaction with proactive treatment implies that more severe augmentations may have the potential to substantially improve developer satisfaction. However, the study's limitations include the use of third-semester computer science students as participants, which may not generalize to professional developers. Additionally, the minimal nature of the interventions may have limited the detectable effect sizes. Future work should explore more substantial interventions and diverse developer populations. The results contribute to understanding how to effectively support developer-LLM interactions, highlighting the importance of dynamic guidance mechanisms. The study's longitudinal design over four months provides valuable insights into the stability of interaction patterns and satisfaction over time.
Who should read this
CS practitioners and researchers
Opening member content…