Developer AI assistants : promising but not ready
Since the emergence of generative AI and modern assistants, we’ve had more and more tools to help us in our work as developers. But is their cost justified by what they bring? And if so, how can you use them to increase the speed and quality of your code?
❓ If you’re a developer, data scientist, analyst or engineer, you’re probably wondering about the value of these tools and the evolution of your job in the context of their emergence. From my point of view, they enable a real improvement in our work and will become a standard in a few years’ time, but they still need to evolve before being adopted on a massive scale.
🎯 If you’re a decision-maker in the tech industry, it’s probably worth asking yourself whether you should invest in these technologies to boost the efficiency of your teams. To miss this opportunity is to miss out on a major competitive advantage. Even so, you’ll need to take into account the issues involved in securing your projects before investing.
In this article, I propose a few elements to help you in your reflection, based on the use of two solutions:
Github Copilot chat
JetBrains AI Assistant
At the end of the article, I’ve also included a few suggestions for possible developments that I think are important.
Up to 3 times faster with an assistant
After a while of testing, it seems clear to me that using an assistant saves time. I already had a hunch with GitHub Copilot, the intelligent peer programmer, which has since been confirmed by performance studies. For example, in a paper published in February 2023, a team from MIT demonstrated that using this tool reduced the development time of a javascript server by 55% [1].
🪐 With chats now integrated directly into the IDE, we’ve gone one step further:
- the assistant can be given more context by using open scripts
- suggestions can be integrated directly into the project
- logs can be analyzed in a context-sensitive way
- code can be refactored directly in response to a request
Okay, but how much time does that save?
It’s difficult to give a definitive answer to this question, but I’ve tried it out with a concrete case… I give you the details below, but the result is that I’ve divided development time by 3.
👨🏫 In this example, I want to create a user in my MongoDB database with the following fields: login and password. The original password, located in my .env file, has to be hashed with bcrypt before being stored in the database. But before that, I’d like to create randomly a local password to guarantee its solidity. I want to do all that using python as development language.
🕥 5 minutes of development with an assistant. Whether using GitHub Copilot Chat or JetBrains AI, I was able to get functional suggestions quickly, with few modifications to make. I was also able to integrate the suggestions in 1 click. All in all, the both tools enabled me to develop a complete, functional script in less than 5 minutes, with other benefits that we’ll see later.
🕙 15 min+ with a conventional method. Here are the steps and estimated time required:
- Find out how to connect to MongoDB with the python SDK (stackoverflow, documentation, other…) and sort the answers (3 minutes)
- Adapt the code (about 2 minutes)
- Find out how to use bcrypt (stackoverflow, documentation, other…) and sort answers (3 minutes)
- Adapt the code (about 2 minutes)
- Add lines of code to retrieve env var (2 minutes)
- Time lost in switching context and micro-errors in typing, copying or past (2 or 3 minutes)
Go even further with support for all stages of the development workflow
To ensure end-to-end development, IDEs have long offered extremely useful functionalities, such as debugging, syntax analysis, type consistency, and so on. AI assistants take this logic even further.
Clear error explanations
As we all know, debugging is an important part of a developer’s work. JetBrains AI allows you to start with the error, analyze its origin in context and suggest solutions. The developer’s workflow is thus streamlined.
Easier onboarding and reviews
JetBrains AI can also be used to explain commits, making it easier for reviewers to challenge their team’s code. This also makes it easier to understand the evolution of the code base, in case of absence for example.
Improving code quality
Many other actions are possible with JetBrais AI to improve code by using suggestions for refactoring, adding comments or testing.
Accelerate your skills development
In the example presented above, an assistant not only accelerated code development, but also provided an opportunity to learn. For example, during the conversation, the assistant mentioned the salting technique for hashing passwords to avoid dictionary or rainbow table attacks.
This is why GitHub Copilot chat offers a very interesting feature. After each exchange, the bot suggests a question. This allows you to dig deeper into a subject you wouldn’t know where to start.
A few precautions before embarking on your adventure
Of course, all this comes at a cost.
Direct. you have to take out a subscription that represents about $10 to $20/month, which can vary depending on the payment method, the type of subscription, etc.
Indirect. To function, these services need to analyze code and therefore transfer data from your project to the language model used to process the information. This means two things:
- your attack surface is enlarged
- your data can be accessed by a third-party service
For each service, security measures are presented to reassure users.
Sensitive files supposed to be inaccessible to assistants
When asked in chat, the JetBrains Ai assistant indicates that it only has access to files open in the IDE. Copilot indicates that it has access to open and recently consulted files.
I also asked whether the assistant had access to a sensitive file such as an .env file potentially containing secrets. The answer in both cases is the same: in principle, there is no access to these files.
⚠️ However, be careful, we can see that the .env file is displayed as a reference file in some cases by GitHub Copilot chat, for example.
JetBrains states in its terms and conditions that the data transmitted is not used to train models: “JetBrains does not work with LLM providers that use customer data to train models.”[2]
For GitHub Copilot, the data sent can be used to improve services. [3]. The documentation specifies that data is transmitted securely.
Limited parameterization of data use and monitoring still too weak
You can choose not to have GitHub Copilot keep your prompts and suggestions. It is also possible to request the deletion of prompts and suggestions associated with your account.
For JetBrains AI, there is an option to disable the plugin and another to print all prompts.
Suggestions for improving the AI assistant service
As a user, I see several avenues for improvement in these assistants that seem to me to be able to really finish making the difference and complete the start of new software development practices.
To date, I have yet to see a feature that would enable me to trace all the data exchanged by the assistant with the server. Nor have I seen a feature that lets you choose which files the wizard should ignore.
As a user, I’m looking for greater transparency
Add the use of an .assistantignore. The possibility of creating a file in the project listing all files that are forbidden to be read would be a real advance in terms of confidence in the service provided by the assistant. This would be the equivalent of the .gitignore used by the wizard to limit its file reading scope.
Add a “dump logs” or “monitor logs” function. To be really sure of having a complete view of the data I’m transmitting, I’d also like to be able to monitor the files or snippets sent in addition to the prompts. A bit like JetBrains’ “dump chat”, but for all the data exchanged. In other words, to be able to observe the history of all exchanges with the server, either continuously, or through access to the data history.
Authorization requests. The icing on the cake would be to be able to prevent sensitive data leaks, for example by setting the assistant to request permission to perform certain actions such as reading files or transmitting information, particularly when a file potentially contains credentials or sensitive data.
Extending assistance to architectural issues
The assistant’s scope of intervention could be extended to include architecture and code structuring issues. If the assistant had access to the file tree, it could be used for more complete refactoring, for advice on code organization and modularity, and even on software architecture as a whole.
Conclusion
Assistants are going to revolutionize the developer’s profession. But to exploit the full potential of these tools, it seems to me that we need to wait for certain functions to be added, particularly for those working on sensitive projects, in order to control data exchanges and guard against disasters.
