Generative AI has been accused of being an overhyped toy or a black swan coming for our jobs. As a tester who has incorporated this technology into my daily routine, I put ChatGPT on trial to see whether it’s a useful tool for testers and developers or just a flashy pretender.

Some of the uses that come to mind are writing better unit tests, saving time on development or making more robust code. It can also be invaluable if you are working on a personal project or if you have test workforce restrictions.

But how trustworthy and helpful is it, really?

I put it to the test for several use cases:

• Test data generation (symbol sequences valid for tests)
• Code generation (machine executable commands valid for scripted testing/test automation)
• Test idea/case/strategy generation (human executable commands valid for testing activities)

Test Data

Defining test data is an important part of the testing process, particularly in unit tests where specific inputs are needed to validate code functionality. Yet it often involves monotonous work when you know exactly what to include or the data is iterative.

In such cases you can get the desired data by providing a prompt like, “I need a list of cities starting with the letters ‘XXX’” or “create a sequence of Unicode symbols containing a specific character set.” But how does it fare with more realistic testing tasks?

Simple Text Modifications, Replacement

Let’s consider a scenario where we need to incorporate an existing expression language into a new part of the system. Given the different usage context in this new location, the expression language syntax should be transformed before it can be used.

From a developer’s perspective, a “quick and dirty” approach to cover this functionality with unit test validation would be to use existing test data. I used examples from a product documentation website as the input. The idea for ChatGPT is to take the original expressions and modify them to serve as test data for our unit tests.

I asked ChatGPT to extract and transform the expressions from the product documentation website using the Webpilot plugin, and then transform each following the example below:

In the output, ChatGPT returned expected results for each expression so I can instantly put them in my tests:

As you can see, ChatGPT did its job well, aside from the fact that Boolean operators are case sensitive, so (True) should be replaced with (true).

This could have been done with some custom code or Excel, but we spared ourselves the monotonous job of copying a lot of data from a table, leaving room to deal with more complex work.

In more complex cases, with a couple more prompts you can make ChatGPT iterate over different types of data, other expressions or combinations of them.

Simulate a User

So we know it works when dabbling with formulas or attributes and creating data to use in tests. Now let’s take it one step further. Since we are testing syntax that could be generated by a user, let’s try to simulate the user — or at least mimic the user and then implement “user” ideas with expression syntax.

We are on shaky ground here. Simulating a user can be part of the testing process, but it’s insanely context-dependent. Reasonably applied, this approach could be beneficial for startups, solo developers, proof-of-concept projects or small companies