As LLM shows its excellent capability at writing code and explaining code, developers are gradually feeling satisfied with products like GitHub Copilot, which releases them from coding line by line.

In this article, I want to discuss with you whether do we need code understanding as developer tools. Let’s begin with a paper from CMU researchers: Using an LLM to Help With Code Understanding.

Using an LLM to Help With Code Understanding

In this paper, researchers developed a Visual Studio Code plugin to provide code understanding functionalities. Let me illustrate what this tool could do.

Here is the user story:

1. A user is developing a 3D visualizer. He found the output was incorrect. Specifically, the bunny should stand on the chair instead of beneath it.

1. He located the bug to the lines from 26 to 30 as highlighted in the screenshot. However, he is not familiar with the Open3D library and computer graphics. He needs to ask others or the internet for more information.
2. Before he searched the internet, the CMU researchers said “Hold on, we just created a great code understanding tool. You can use this tool to understand your code and fix the bug in it.”
3. He said “OK, hope that works. Let me try.” He first selected the lines between 26 to 30, then clicked the “AI Explanation” button. The left panel is shown. In the top of the panel, there is an AI overview talking about what does this code snippet do. Umm, looks like the overview is not useful. He started to type in his question “How can I move bunny to sit upright?” Fortunately, the LLM respond with the explanation of the bug and showed the correct code.

Besides the general workflow we described above, the researchers also designed several functionalities to help people in more aspects, including:

• Click the “API” button to show the official documents inside the API call in the highlighted code.
• Click the “Concept” button to explain the concepts this code is manipulating.
• Click the “Usage” button to show an example code for the API.
• Click the “Embed” button to embed the explanation to the code as a comment.

With this tool, researchers established some experiments with 32 participants to investigate whether it can help people during their development. I don’t want to introduce too many details about the statistics in these experiments. Let’s directly take a look at the conclusions they obtained. And I will share my opintions.

Conclusion 1: There are statistically significant gains in task completion rate when using GILT, compared to a web search, but the degree of the benefit varies between students and professionals.

My opinion: Task completion rate is improved while task time consumption is not improved. This means if people have experience, they can always make it regardless of the tools they use. On the other side, beginners don’t know how to search the internet for a good solution, but the tool helps them avoid this weakness.

Conclusion 2: Overall, participants used Overview and Prompt-context most frequently. However, the way participants interact with GILT varied based on their learning styles and familiarity with other AI tools.