Collaborative AI tools are reshaping software development

Cory Hymel, Vice President of Innovation and Research at Crowdbotics, brings his extensive expertise in AI and software development to the forefront in this engaging Q&A. Hymel delves into the transformative impact of AI on the software development lifecycle, highlighting its influence on areas like code generation, requirements engineering, and context management. He explores the potential of multi-agent systems to address current gaps in off-the-shelf AI tools, emphasizing their ability to enhance performance and efficiency.

AI research: MSFT/GitHub/Crowdbotics study uncovers the potential of collaborative AI tools

From discussing the evolving roles of software engineers as AI becomes more capable to envisioning a future where AI democratizes software creation, Hymel provides a forward-looking perspective on AI's role in reshaping the industry. The discussion also touches on Crowdbotics’ joint research with Microsoft and GitHub, showcasing the tangible benefits of collaborative AI tools in improving developer productivity and task success rates.

ADM: In which areas of the software development lifecycle has AI already had the most significant impact?

Hymel: To date, code generation has been the hot topic not only in capital raises but also in research. Roughly 56% of the research publications in the past few years has been focused on code gen with the fast follow of 23% on maintenance which is tangential to code gen. Code generation in and of itself is mainly bisected into two main categories: full code gen such as Deven and code gen support such as GitHub Copilot. Empirically code gen support (i.e. code assistants) have had the largest impact on productivity however are also the most studied. 

ADM: How would you describe the importance of context in software development, and what are some common examples that illustrate why it's needed for successful development projects?

Hymel: Context is the most important thing in software development. That being said, context is a very wide concept that covers a lot of different areas. For instance, a developer may need context as to the current block of code they’re working on and how it may impact other areas. A designer needs user context to create proper experiences. Testers need context to know how outcomes should be reported. Product managers need context of the problem to accurately define requirements. Today that context is spread across a lot of individuals and is typically not shared efficiently. There are some knowledge management tools that look to capture that context such as Jira, Confluence, and other project management tools but it still relies on humans to consume that context and apply where they need it. AI is uniquely positioned to be a transformative technology by centralizing KM and making it not only more accessible but also extensible as it can automatically capture knowledge from various sources without manual intervention. You can find more information on how AI can act as KM in a whitepaper I recently published titled 'The AI-Native Software Development Lifecycle: A Theoretical and Practical New Methodology'.

ADM: What is requirements engineering in software development, and why does it play a vital role in the development process?

Hymel: Requirements engineering is the inflection point on when an idea starts to take shape into reality. It’s the first step where ideas are turned from abstract concepts into plans and requirements that software teams can execute against. This first point, the first dot on the timeline, is critical because it sets the tone and direction for all the subsequent work to come.  

ADM: Can you describe multi-agent systems and single-agent systems as they relate to software development?

Hymel: A single agent system is a platform, process, or experience that uses a single LLM model to drive it. A multi-agent system uses multiple models in unison. These could be multiple instances of the same model, for instance you could have multiple instances of GPT4 working together in a multi-agent setup. You could also have multiple, different models working together such as a GPT model and Claude working together to provide a singular experience for the end user. What’s so powerful about a multi-agent setup is that you’re able to customize each of the different models to be ‘experts’ at different tasks. In the software development life cycle, you have multiple different roles and responsibilities present therefore using a multi-agent setup is advantageous. What’s also important to note is that LLMs have finite amounts of ‘short term memory’ they can access at any given point in time. When trying to use a single-agent system you quickly reach this limit which can have negative performance effects. Therefore breaking out work into module streams in a multi-agent setup allows you to navigate complex tasks more successfully than a single-agent configuration.

ADM: What kind of impact have you seen multi-agent systems have on off-the-shelf AI tools for software development? Which gaps do they help fill?

Hymel: It’s a bit of a mixed bag today. Most ‘black box’ platforms such as ChatGPT, Devon, GitHub Copilot, and others will use multi-agents behind the scenes to create unique experiences for users. However today, there is little in the market of standalone, commercial products working together in a multi-agent configuration. The main reason they don’t today is due to integration immaturity between the products and more advanced reasons like consumption traceability for chargebacks. However, we will see more of this commercial, mulit-agent integration in the future due to the quick performance gains you get. We recently ran a joint study with GitHub and Microsoft to test a multi-agent configuration of commercially available products (Crowdbotics’ PRDAI and GitHub Copilot) and what we found was an improvement of GitHub Copilots code suggestion feature of 13.8% and improved developer task success rate of 24.5%.  This is without any retraining or any additional compute. You can find the full whitepaper here: [2410.22129] Improving Performance of Commercially Available AI Products in a Multi-Agent Configuration

ADM: As it stands today, how useful is AI when it comes to code generation, and where do you see it headed in the next 3-5 years?

Hymel: The answer is it depends. Today, junior or mid level developers are getting the most out of AI gen with senior developers getting very little. Which makes sense, the frontier models out there today were trained on publicly available data and most publicly available code isn’t incredibly complex. YES! I know about open source and all the complex great work that’s out there but when you think about the internet as a whole, most of the data isn’t custom enterprise grade code that most senior developers are working on and where they’d see the benefits from. Very soon, particularly before the 3 year mark, companies will begin finetuning code gen models on their own code with the models running on prem. These will offer some of the more interesting data points on how code generation will progress in the following years. 

ADM: Assuming that the bulk of code will be able to be written almost instantly with the help of AI in the coming years, how will the role of software engineering teams evolve?

Hymel: As AI becomes increasingly more capable and better at generating code, the role of software engineering teams will shift from creators to verifiers. While AI can generate code faster and more cost-effectively than humans, human developers will remain essential for validating the AI-generated outputs, making high-level design decisions, and guiding strategic direction. The business case for AI-driven code generation is compelling, given the significant cost difference between human software engineers and AI models, as well as the additional benefits such as uninterrupted work, contextual memory, and rapid mistake correction. Although AI models won't fully replace human engineers in the near term, their continuous improvement will make them increasingly effective, with human developers acting primarily as validators and input vectors rather than code creators.

ADM: With AI showing so much potential, what excites you most about where AI will take software development in the coming years?

Hymel: GitHub has a great company mission of “1 billion developers”. Meaning that anyone, anywhere can build software to solve some acute need they have, such as being able to program on a mobile device with their new GitHub Workspaces platform. This vision and world excites me a lot because it should be every human's right to solve problems with technology – it doesn’t matter if those are big problems or small problem, anyone should be able to easily deploy technology to make their lives e    easier. AI has the potential to reduce this barrier to entry and make the vision of “1 billion developers” a reality. 

About Cory Hymel

Cory Hymel is the Vice President of Innovation and Research at Crowdbotics. Cory is a passionate engineer, researcher, and futurist. From AI research on self-driving vehicles in the early 2000’s to founding multiple startups, his diverse experience has shaped his unique perspective on the future. He seeks to further advance the field of Human-Computer Interaction with a focus on AI augmentation in software development. He has held leadership roles at Prime Notion Technologies (Founder), Simble (Founder, acquired by Enventys Partners), and Gigster (acquired by Ionic Partners). Cory is regularly a guest speaker, pundit and panelist for technology events and programs worldwide.