Skip to content
All posts

A Private GPT for Software Engineering: A Four-Level Analysis

Picture of Lenz Belzner By  Lenz Belzner  ·  8 minute read

Welcome to our exploration of AI in software engineering, focusing on a crucial aspect: privacy. Artificial intelligence (AI), especially large language models (LLMs) like GPT, is revolutionizing how we build and improve software. These powerful tools can write code, spot errors, and even suggest improvements, much like a seasoned programmer.

However, there's a catch: privacy. When you use these AI models, you often share sensitive data, like proprietary code or business-specific information. Imagine you're asking an AI to debug a piece of software that contains confidential company strategies. You wouldn't want this information to end up in the wrong hands, right?

To understand this better, let's dive into the "four levels of privacy" in using LLMs for software engineering. These levels range from using a third-party app with AI capabilities to setting up everything on your own servers, each with its own privacy implications. We'll explore these levels to help you figure out what suits your project best, keeping your sensitive data safe while benefiting from AI's power. Let's get started!

overview

Understanding Privacy Concerns with LLMs

When we talk about using AI in software engineering, especially about large language models, privacy isn't just a buzzword; it's a real concern. Why? Because these AI models, like the ones you've heard of - GPT, BERT, LLaMA, Mistral, and others - learn from vast amounts of data, and 'reason' based on the data you pass into them. Sometimes, this data can include sensitive information.

Let's break this down with a simple example. Imagine you're using an LLM to help write code for a new app your company is developing. You input some of your existing code into the AI to get suggestions for improvements. This code might contain unique techniques or secret algorithms that give your company a competitive edge. If the AI model is hosted on an external server, like in the case of a third-party service, there's a risk. Your unique code could potentially be accessed or stored outside your control. This is like giving a peek into your secret recipe to someone you don't fully know.

Another example is when you ask an AI to debug a program, and you feed it with error logs. These logs might contain confidential information like user data or system configurations. If these details are not adequately protected, they could be exposed to external parties, posing a privacy risk.

Every time you use an LLM in software engineering, think of it as sharing a piece of your project's confidentiality. The key is to understand where and how this information is being used and to make informed decisions about what level of privacy risk you're comfortable with.

The Four Levels of Privacy in AI Applications

In the world of AI and software engineering, not all privacy setups are created equal. Depending on how and where you use AI and LLMs, your privacy concerns can vary a lot. Think of it like having different types of locks on your data - some are simple, while others are like high-security vaults. Let’s explore these varying levels of privacy to see what fits best for your needs.

  1. Third-Party Apps Using Proprietary LLMs: You're using an app developed by someone else, which also uses an proprietary AI model like GPT, Claude, or Mistral Medium via an API. For example, you might use a third-party code editor that suggests code improvements. The privacy concern here is that your code goes through external servers, where you have less control over data security.

  2. App in Your VPC Azure Tenant Using Azure’s OpenAI API: You're still using external AI services (like Azure's AI), but within your own Virtual Private Cloud (VPC). Your code doesn’t travel as far, and you have more control.

  3. App in Your VPC Tenant Using Open Source Models (e.g., LLaMA or Mistral): Here, you're not just using your own VPC, but also open-source AI models. It means you can look inside the ‘AI engine’ and have more control over your data, making it a safer choice for sensitive projects.

  4. App and LLM On-Premise: Everything - both the application and the AI model - runs on your own premises. It’s like having all your secrets in a vault under your own house. The highest level of privacy and security, suitable for the most sensitive of projects where data cannot afford to be compromised.

Each level offers different degrees of privacy and control, and understanding these can help you make the best choice for your software engineering projects. The key is to balance your need for AI assistance with how much privacy your project requires.

Level 1: Third-Party Applications Using Proprietary LLMs

Level 1 is about balancing the convenience and advanced capabilities of using a third-party AI tool with the risks of sending your code out into a less controlled environment. It’s perfect for less sensitive projects where the ease of use and access to advanced AI outweighs the potential privacy concerns.

This setup means you're using a tool developed by another company, like an online code assistant, and this tool uses a powerful proprietary AI, like GPT, Claude, or Mistral Medium to help you write or improve your code. It's like asking a very smart, but external, consultant for advice. You type in your code or queries, and the AI, hosted on the third-party's servers, gives you suggestions or solutions.

level_1-1

Now, here's where you need to be a bit cautious. When you use these third-party apps, your code travels outside your company's walls. It’s like sharing your secret recipe with that consultant. The third-party app might store your data, and you don't have much control over what happens to it once it's out there. There's always a risk that sensitive information, like proprietary code or unique software techniques, could be exposed, either accidentally or through a security breach.

Let's shortly summarize potential risks and benefits of using a third party app to support your software engineering teams with AI:

  • Risks:

    • Data Exposure: Your code might contain confidential information which could be stored or processed outside your secure environment.
    • Dependency: Relying on an external service means you’re at the mercy of their security practices and uptime.

  • Benefits:

    • Ease of Use: These apps are usually user-friendly and require less technical setup.
    • Access to Advanced AI: You get to use state-of-the-art AI models without the need host and maintain them in-house.

Level 2: Applications in Your VPC Tenant Using Azure's OpenAI API

Level 2 offers a middle ground. You get more control and better privacy than using a third-party app, thanks to your own VPC. However, you’re still relying on your VPC's infrastructure and security, which can be a good thing if you trust your provider's systems but still want more control over your data. This level is great for projects where you need a balance between advanced AI capabilities and enhanced data security.

At this level, you're using AI tools like Azure's GPT, but within your own Virtual Private Cloud (VPC). You use Azure's version of the AI (like renting office equipment), but everything stays within your own private space in the cloud. Your data (the code you're working on) stays within your VPC. This means it's more secure than sending it out to a third-party, as in Level 1. Your VPC has its own strong security measures, which add an extra layer of protection to your data. It’s like having a lock on your office door and a guard at the building entrance.

One drawback of Azure's GPT is that it lags behind OpenAI. For instance, Azure currently only offers a slower, preview version of the GPT-4-Turbo-128k model, which is not recommended for production use. Additionally, the Assistants API is not yet available on Azure. Moreover, to use Azure's OpenAI API, one must apply and be selected.

level_2

Let's compare level 1 and level 2:

  • Level 1 Using a third party service means it's easy to access and use, but your sensitive information is more exposed. Also, maintainance cost is reduced, and access to the most powerful proprietary LLMs assures the best possible quality and AI support in your workflows.
  • Level 2 You have more control over your environment, and your data is safer, but you’re still using the building's (your VPC's) infrastructure and security. On the other hand, you may not have access to the most powerful LLMs, and you have to put in effort and have expertise in managing and running your VPC setup.

Level 3: Applications in Your VPC Tenant Using Open Source Models (e.g., LLaMA or Mistral)

Level 3 offers a high degree of control and privacy, as you're using open source AI models within your own virtual space. It's ideal for teams with the technical know-how to manage these models and the respective cloud infrastructure, and for projects where data security and customization are top priorities.

At this level, you use open source AI models, such as LLaMA or Mistral, in your own virtual private cloud You're not renting AI tools from a big provider like Azure. Instead, you're using AI models that are available for anyone to use, modify, and host in their own private cloud space. It’s like having custom-made software tailored for your specific needs and hosted in your own secure online space.

One possible downside is that the quality of an open source LLM's output may lack behind proprietary API offerings, at least at the time of writing.

level_3

Since you're using these open source models in your own VPC, you have a lot more control over your data. You know exactly how it works and can make sure it's really secure. If you have the expertise, and are able to put in the necessary effort, there may be less risk of your data being exposed or mishandled because it's all in your hands. However, with great power comes great responsibility - you're also responsible for keeping these models and your infrastructure secure and up-to-date, which can require more technical expertise and labor.

Let's contrast this approach using open source LLMs with using proprietary models like Azure's OpenAI:

  • Proprietary Models (like Azure's OpenAI): They're sophisticated and reliable, but you don't have much control over how they're built or how they handle your data.
  • Open Source Models in Your VPC: You have full control over how they work and handle your data, but it also means you need the skills and resources to build and maintain them.

Level 4: Applications and LLM On-Premise

Level 4 is for those who need the highest possible level of privacy and control, and who have the resources to support an in-house setup. It’s ideal for highly sensitive projects where any risk of data exposure cannot be tolerated. However, it requires a significant investment in both equipment and skilled personnel to manage the system effectively.

In this scenario, both your applications and the Large Language Models (LLMs) are running on your own computers and servers, right where you work - an "on-premise" solution. You're not using the cloud or renting space and tools from a big provider. Instead, everything is hosted and runs on your own physical machines.

level_4

By keeping everything in-house, you have the ultimate level of privacy. This setup means that your sensitive data, like proprietary code or confidential project details, never leaves your own controlled environment. It's extremely difficult for outsiders to access your information because it never travels across the internet or gets stored on someone else's server.

Let's have a look at advantages and challenges of the on-prem setup:

  • Advantages:

    • Maximum Privacy: Since everything is kept in-house, you have total control over your data.
    • Full Control: You can customize and tweak both the applications and the AI models to perfectly fit your specific needs.

  • Challenges:

    • Resource-Intensive: Setting up and maintaining your own servers and infrastructure requires significant resources, both in terms of hardware and technical expertise.
    • Upkeep and Updates: You are entirely responsible for keeping everything updated and running smoothly, which can be a demanding task. Note that this also includes infrastructure and security aspects, which require particular expertise and effort on your side.

Conclusion

Let's wrap up what we've covered about the different levels of privacy in using AI, particularly large language models (LLMs), in software engineering:

  1. Third-Party Applications Using Proprietary LLMs: Easy and convenient, but you have less control over your data. Great for non-sensitive tasks where ease of use is a priority. Also, proprietary LLMs can be leveraged, which may provide the highest quality for your software engineering workflows.

  2. Applications in Your VPC Tenant Using Azure's OpenAI API: You get more control and security than with third-party apps, making it a good middle ground for many projects. However, LLM offerings provided by your particular VPC host may not provide the highest quality generally available. Also, managing your VPC setup requires additional expertise and effort on your side.

  3. Applications in Your VPC Tenant Using Open Source Models: This setup offers even more control and privacy, ideal for projects that require specific customization and have a team capable of managing it. As for level 2, you have to ensure that the OS LLM provides sufficient quality for your needs, and that you have the resources in place to configure and manage the system.

  4. Applications and LLM On-Premise: Here, everything is kept in-house, offering the highest level of security. Best for highly sensitive projects where data privacy is non-negotiable, but it's also the most resource-intensive.

When it comes to balancing privacy and functionality in AI for software engineering, there's no one-size-fits-all solution. Each level offers a different mix of control, privacy, and ease of use. The right choice depends on the specific needs of your project and your capacity to manage the infrastructure. More control often means more responsibility, so weigh your options carefully to find the best fit for your project's requirements.