What are the limitations of using ChatGPT to write code?

Love them or hate them, large language models (LLM) are here to stay. After opening the Pandora’s Box of ChatGPT in late 2022, everyone from developers to grandmas began using the tool to get the answers they wanted – and fast. As with every other new technology, ChatGPT created a new set of security risks, impacting both organizations as malicious actors use it to write realistic phishing emails and developers as they seek to improve their delivery times.

If you’re looking to use ChatGPT for coding, you should understand its capabilities and limitations so that you make informed decisions about when and how to use it.

What is ChatGPT Coding Assistant?

While many developers use the free version GPT-3 version to answer coding questions, the paid version of OpenAI’s ChatGPT Coding Assistant leverages the advanced GPT-4 model provides features like:

Code completion: using predictive analytics to automatically suggest code snippets using its knowledge base to research syntax and relevant functions
Code generation: using natural language descriptions that describe functionality to generate code snippets for less experienced developers or alternate implementations
Code optimization: analyzing code examples or descriptions of inefficiencies to suggest improvements that improve application performance, like optimizing memory usage

OpenAI trained the ChatGPT neural network on large quantities of code examples so that it could learn and generate relevant code suggestions across various code semantics, syntactical patterns, and programming frameworks.

What are the benefits of using ChatGPT for coding?

Since ChatGPT can transform inputs written in normal human language into code-based outputs, the AI assistant enables developers to ship applications faster.

Close education gap

ChatGPT gives less experienced developers a way to ask technical questions without requiring them to use technical language. Less experienced developers can ask questions using everyday language, and ChatGPT answers with things like:

Unfinished code snippets for practicing new skills
Explanations of functions
Conversational answers so developers can ask follow-up, clarification questions

ChatGPT enables developers to learn at their own pace through dynamic interactions so they can understand how the code and functions work rather than simply copying and pasting the generated code snippets.

Faster time-to-market

ChatGPT’s code completion capability means that you can solve problems faster, reducing the time it takes to ship code. Instead of spending hours reading through search engine results, you can use ChatGPT to provide responses, enabling you to:

Analyze different completions
Understand the reasoning behind them
Compare multiple solutions
Observe patterns, coding conventions, and idiomatic approaches
Experiment with different ways to write code and solve problems

Fixing bugs faster

Developers can spend hours scouring the internet for how to resolve issues. Using ChatGPT, you can write a prompt that helps identify and fix the bug faster by:

Describing the issue, including any symptoms, error messages, or unexpected behavior
Pasting the code snippet into the prompt
Explaining how the code should work when fixed
Asking for guidance on how to fix the issues, including potential causes, debugging strategies, or specific code modifications

Improved code optimization

Code optimization improves your application’s performance and makes it easier to maintain. Using ChatGPT, you can break down code and analyze the smaller components for opportunities like:

Changing data types to reduce the number of conversions taking place
Simplifying code to make it easier to read, understand, and debug
Avoiding unnecessary computations by examining code and its dependencies
Reducing the number of unnecessary reads and writes
Finding the most efficient algorithms to reduce the time and resources used to complete tasks

What are ChatGPT’s challenges and limitations?

Using ChatGPT for coding offers various benefits, but you should be aware of some issues that can impact your application and organization.

Application security risks

The code snippets that ChatGPT generates can introduce security vulnerabilities into the application. Academic researchers studied ChatGPT outputs to determine whether the suggestions were secure code. The research consisted of:

21 programs in 5 different languages, C (3), C++ (11), Python (3), HTML (1), Java (3)
5 were initially correct
7 of the 16 incorrect versions were corrected using interaction with ChatGPT

The researchers found that they needed to ask follow-up questions about the code’s security, also noting that ChatGPT would correct for a vulnerability that the program intended to test while leaving others uncorrected.

Lack of context

Although ChatGPT appears to provide context-aware code snippets, it fails to consider an application’s business logic. When reviewing ChatGPT’s capabilities, the researchers found that, at the time of writing, it did not assume an adversarial model of execution, meaning that it lacked insight into ways that malicious actors could leverage business logic flaws to complete an attack.

Without appropriate penetration testing, the code can introduce security risks based on the attacker’s ability to use intended functionalities in unintended ways.

Intellectual property leakage

While OpenAI does not train its models on data from ChatGPT Team, ChatGPT Enterprise, or the API Platform, it does explain that it may use content generated by its services for individuals, unless the user opts out using the privacy portal.

Unless your organization uses the Enterprise paid model, any code you provide in a prompt may become part of the model training. Depending on how the model responds to future prompts, it can provide proprietary code in its answers, creating an intellectual property data leak.

Hallucinations

Hallucinations are outputs that appear plausible but, upon review, are incorrect. For example, ChatGPT may recommend non-existent packages or supply code that fails to work. These issues increase time-to-market, undermining a key benefit.

Code quality

Code quality refers to the average lines of code needed to perform a task, with fewer lines of code (LoC) indicating higher quality. In early 2024, researchers published a paper looking at ChatGPT’s and AI’s effectiveness. These researchers found that while ChatGPT performed better than Bard, Bing produced higher quality code than both the free version, ChatGPT 3.5, and the paid version ChatGPT-4.

Inconsistencies

ChatGPT relies heavily on the prompt’s quality to generate appropriate answers. The researchers investigating AI’s effectiveness for programming found that small changes within a GPT-4 prompt generated different code. While both functions worked as intended, the AI introduces randomness which can increase the time it takes to test functionalities and correct code.

Developer preferences

Everyone has their own coding preferences. For example, developers may choose to use:

A single or multiple ‘return’ when formatting code
“Default value with no else” or using a Boolean “if then else” to define values
Delimit with optional curly braces or not

ChatGPT may not always follow the format that you use when generating code, introducing readability issues, software bugs, or maintenance challenges.

Qwiet AI: AI for Secure Coding

With Qwiet AI, you can scan millions of lines of code in minutes for visibility into all components, dependencies, and reachable vulnerabilities. Our platform quickly returns accurate and detailed findings while significantly reducing false positives. With our Code Property Graph (CPG), you gain a holistic view of your code, including data flows across the application, to quickly determine reachability and risk impact.

At Qwiet AI, we provide more than just tools – we can deliver a fully managed application security service tailored specifically to your organization’s unique needs. Our deep expertise in application security combined with seamless CI/CD integration ensures you’re not just deploying a solution but establishing a robust partnership. We delve deep into the unique business logic that drives your application. Our specialized focus on these flaws ensures that every facet of your application’s logic is meticulously tested and secured.

Try Qwiet AI’s preZero platform for free to see how it can help you answer questions and mitigate risks.

About ShiftLeft

ShiftLeft empowers developers and AppSec teams to dramatically reduce risk by quickly finding and fixing the vulnerabilities most likely to reach their applications and ignoring reported vulnerabilities that pose little risk. Industry-leading accuracy allows developers to focus on security fixes that matter and improve code velocity while enabling AppSec engineers to shift security left.

A unified code security platform, ShiftLeft CORE scans for attack context across custom code, APIs, OSS, containers, internal microservices, and first-party business logic by combining results of the company’s and Intelligent Software Composition Analysis (SCA). Using its unique graph database that combines code attributes and analyzes actual attack paths based on real application architecture, ShiftLeft then provides detailed guidance on risk remediation within existing development workflows and tooling. Teams that use ShiftLeft ship more secure code, faster. Backed by SYN Ventures, Bain Capital Ventures, Blackstone, Mayfield, Thomvest Ventures, and SineWave Ventures, ShiftLeft is based in Santa Clara, California. For information, visit: www.shiftleft.io.