Test Data Generation Using Local API’s for Large Language Models

Introduction

In a previous post, I explored how the Large Language Model (LLM) ChatGPT, created by OpenAI, could be used to create custom test data generation tools quickly. I developed a persona generator with a simple user interface using ChatGPT; With a press of a button, the tool sent a prompt to the ChatGPT API, requesting a description of a random test persona, and displayed the response.

In a follow-up post, I discussed how LLMs similar to, but less capable than, ChatGPT could be run on a local machine, eliminating privacy concerns since sensitive data or questions never leave the local environment. Open-source LLMs are evolving rapidly and supporting tools are also being developed at a fast pace. In this blog post, I will share how I used a tool providing API access to a local LLM as a drop-in replacement for ChatGPT, keeping my data on my laptop.

The Test Data Generation Tool — Test Persona Generation

To help testers approach a product from different perspectives, it can be useful to work with test personas. These fictional characters represent various types of end-users and can uncover issues that may not be apparent when testing based on the testers’ own technical skill level. The tool I built is written in Python and utilizes the ChatGPT API by sending a prompt and displaying the response for the user. Check this post by Andreas Cederholm for some more inspiration on Test Personas!

GPT4All — API support

GPT4All is a tool that simplifies the installation process, provides a user-friendly interface and allows users to download different LLM models effortlessly. Recently, they introduced API support with a simple setting change, allowing users to access an API that mimics the ChatGPT API locally. To test this out, I used my persona generator and updated the ChatGPT endpoint to my local machine. I was delighted to find that it worked perfectly, and I received a response from my local LLM with a test persona! The prompt I used was the following:

“Create information about a randomly generated persona to be used for testing purpose. Provide Name, age, sex, bank account balance up to 1000€, preferred web-browser, technical knowledge level, backstory about the personas relationship with technology and define the technical skill level of that persona, a couple of test ideas of what can be important to test from the perspective of this persona”

The result that came back, as seen below, was not perfect but good enough to be useful. What the model seemingly had trouble with was to understand what I meant with test ideas from the persona perspective. Potentially the result can become better using a better prompt or by using a larger and more competent model.

‍

‍

The quality of the generated answers are a hit or miss. The model running on my office laptop is nowhere near as capable as ChatGPT3 (and even further from version 4). However, the answers are sufficiently useful with a little bit of patience and a little bit of prompt tweaking.

Why This Is a Big Deal and Potential Use Cases

So, why not use ChatGPT if it is faster and more capable? The main drawbacks are privacy concerns and potential costs. When using ChatGPT, data is sent to an external organisation with no control of how that data is used. Potentially costs may increase in the future or they might not scale depending on usage needs. By running an LLM locally, a person or an organisation can tailor the model to its specific requirements and keep potentially sensitive data within its own network.

Specific tools can be customized to suit the needs of the organisation’s members. An example could be a LLM trained on a business context with well described end-users and use cases. Based on this information, the LLM can generate test data and potentially accelerating the work of a tester. It could also help developers to better understand if their code aligns with the end-users’ needs and fulfills the intended value.

‍

‍

Below is an example of some experiments I did to see if I could get the models to consequently return test data in JSON format on the fly for automatic checking. As one can see in the screenshot below the model returns something that can be quite good (with the exception of the “vbnet” inclusion) but the biggest problem was consistency. Sometime the results was in correct JSON format and sometimes I got something completely different. This is however expected when running a limited LLM on a office laptop.

‍

‍

By running a larger model on a powerful server or utilizing the cloud the gap between the model used in this blog post and ChatGPT3 can be significantly reduced while retaining the benefits of control of data.

Summary

The API for ChatGPT opens up possibilities for using the API’s responses in various types of tools. However, a drawback is that requests for potentially sensitive data are sent to an external organisation, resulting in a loss of control over the data. Open-source tools like GPT4All with openly available LLM’s are evolving rapidly and now offer a drop-in replacements for the ChatGPT API endpoint. Tools that previously relied on ChatGPT for data can now connect to a local LLM. The downside is that the LLMs currently available for local use are not as advanced as ChatGPT3, but they may still be sufficient for specific use cases. The upside is that the data remains within the local network, ensuring data control and security.

The Code for the Test Persona Generator

Here is the Python code to see how the call is made. You need Python installed and potentially you need some additional dependencies. You need the GPT4All to have started the web server in the Application Settings tab, it typically starts the web server at localhost:4891.