Using AI as a UX Research Tool to Audit Developer Content at Scale

It’s hard to sort through technical docs — particularly if you’re not a developer. Here at BigCommerce, we know our platform inside and out, and even our own team sometimes struggles to find the correct information in a reasonable amount of time. Which got me thinking: what if we used AI to help make it easier to get around our developer docs?

So I built a proof-of-concept: a research tool that uses Retrieval-Augmented Generation (RAG) with OpenAI’s vector store functionality, trained straight on our developer documentation. The goal? To help designers, PMs, and anyone else who isn’t up to their elbows in code get fast, accurate information about our platform.

The Problem: Docs Are Dense

We have a solid documentation portal — but it’s built for developers, by developers. For product designers (like me) or cross-functional team members trying to find our APIs, tech stack, or headless setup, it’s daunting.

You might know what you’re looking for (“How do I set up a multi-storefront with Catalyst?”), but you’re scrolling through five pages, reading half-complete examples, and still unsure.

The Idea: Feed the Docs to an AI

It really did cross my mind, though, while at the React Miami conference. DataStax was demonstrating their Astra DB vector database by rolling out SwiftieGPT — a model trained to be an encyclopedia of Taylor Swift. It was nice and somewhat unexpected, but something about it stuck with me: what if we did the same with something much more practical — like getting our team through the BigCommerce dev docs?

I started thinking: can we make our whole dev portal searchable with an intelligent assistant? One that understands context, quotes sources, and gives you real answers to real questions without having to sift through endless pages. That wondering led to this project.

Building the Assistant

Here’s how I did it:

1. Scraped the BigCommerce Developer Docs
   I stripped away all the extra fluff — navs, footers, etc. — and left only the good stuff behind. (a little Node.JS and Puppeteer magic)
2. Chunked and Embedded the Data
   To prepare the data for consumption by the vector database, I split it into small overlapping chunks (about 20% overlap) and embedded the content within OpenAI’s vector store. I made the size of each piece ~2KB, which kept the number of documents reasonable and searchable rank optimized within the 10,000-document limit. All the documents were uploaded via OpenAI APIs.
3. Created an OpenAI Assistant
   I created an OpenAI assistant telling it strictly to answer only from the data that was embedded. If it couldn’t find an answer, it should say so — no hallucinations.

All of this is powered by the tooling capabilities in the Playground at platform.openai.com. There, you can create your own assistant using the file search functionality and vector storage, without the need of a backend. It’s remarkably simple once you’ve prepared the content.

The model matters!

When I first tested out the assistant using GPT-3.5 and GPT-4, results were… disappointing. The models hallucinated wildly providing wrong answers or pulling up unrelated content. While the assistant was only programmed to use the file search, it had a tendency to go off script.

But as soon as OpenAI came out with GPT-4o, all this changed. All of a sudden the assistant began respecting file search limits, drawing proper content, and quoting true sources that I had embedded. Responses became accurate, relevant, and much more dependable.

This iteration of the model left me feeling as though I had something tangible, something greater than a demo. I felt secure using the output for UX research or documentation support.

What It Can Do

I ran a series of test queries to see how well it worked:

• “How do I create a table-like UI with BigDesign?”
  It gave a full breakdown: setup instructions, components to use, code examples — plus citations showing exactly which documentation files it pulled from.
• “What’s the difference between Stencil and Catalyst?”
  It summarized use cases, pros/cons, and implementation details for each of our available storefront technologies, all backed by links to the relevant docs.

Each time, it felt like talking to someone who really knows the platform — because it has been trained on the real docs.

Using It for UX Research

One of the most exciting parts of this assistant is how it can facilitate UX research — not just by pushing information to the forefront, but by helping us replicate real-user situations and look into the platform from many perspectives.

Role Querying the AI assistant

Because the assistant is context aware (thanks to the way OpenAI’s API handles requests and retrieval), you can frame questions according to the persona you’re creating for — whether that’s a developer who’s constructing APIs, a CEO evaluating headless commerce, or a merchant launching internationally.

Role-Specific Querying

Here are a few examples I tested:

• Executive persona:
  “I’m a CEO looking for an e-commerce platform that will suite the needs of my company. I’d like to integrate it to salesforce which is what we currently use. Is BigCommerce adequate for this? Why?”
  In response, the assistant explained how BigCommerce met the requirements by highlighting its strong API and webhook integrations with Salesforce, its flexible SaaS architecture, scalability, and extensive developer resources — all framed in a way that helps an executive assess whether the platform aligned with their business and technical goals.
• Merchant persona:
  “My business is based in Spain, but I serve multiple countries in Europe. Can BigCommerce help me customize experiences across different regions?”
  It responded with localization, multi-storefront, multi-currency, region-based content, translations, and compliance. Also provided insights for international expansion planning.
• Product team persona:
  “What are the advantages of using Catalyst over Stencil?”
  Both approaches were succinctly laid out by the assistant, highlighting flexibility, customization, developer experience, and speed of implementation — handy in weighing tradeoffs for a redesign.

Practical Applications for UX Research

Here’s how a tool like this can be put to work throughout the UX process:

• Early Discovery & Stakeholder Interviews
  Use it to simulate real user questions in advance of interviews. This informs more informed discovery discussions and uncovers how users already piece together information from docs.
• Persona Development
  Try out different information-seeking behaviors by persona (merchants, developers, partners) to discover more about their mental models, vocabulary, and expectations from our platform.
• Journey Mapping
  Use the assistant to quickly generate suggested steps, features, or documentation links related to different phases of a user journey. This is especially helpful in mapping flows for complex or niche use cases.
• Gap Analysis
  Identify points at which the assistant gives vague, incomplete, or very technical results. These are points at which our documentation is weak or the messaging is confusing.
• Content Prioritization
  Monitor which questions the assistant gives funny answers. These patterns dictate what content requires clarification, rewording, or should be prioritized in upcoming revisions.
• Onboarding Design
  Design more efficient onboarding flows based on what users will ask initially — and whether our docs already cover them in a clear and concise manner.

By building the assistant off our actual documentation, we can now see how well we communicate platform value — without necessarily having to rely solely on intuition or user testing. It’s like having a synthetic user that asks the questions that we want our customers to ask (and sometimes ones that we didn’t think of).

This enables faster, more confident design decisions — and tighter feedback loop between UX research, documentation strategy, and product design.

Where It Lives Now

Right now, I’m running the assistant through a Discord bot I built, just to avoid building a full frontend for this proof of concept. You can ask it questions in a Discord channel, and it’ll reply through OpenAI’s API.

The bot makes the tool feel lightweight and easy to test out for in-house teams. It’s fast, to the point, and doesn’t entail creating a whole new UI setup.

Why It Matters

For new developers, product managers, or designers coming up to speed on BigCommerce, this assistant could be saving hours of research and stumbling around. More significantly, it provides us with a means to test our documentation as it is queried with real-world questions. It points out gaps, inconsistencies, and areas of improvement.

This is not a convenience feature — it’s a view into how discoverable and usable our docs really are.

What’s Next

This project started as an experiment, but it opens up a lot of doors:

• UX research: What are the most commonly asked questions? Where are users getting stuck?
• Content strategy: Do we over-explain certain things? Neglect others?
• Documentation quality: Can we trust our docs to give clear answers without context?

I’m exploring how to roll this assistant out to a larger audience, and collaborating with our Developer Docs team to ensure it stays current with the best and latest information.

If you’re working on something similar or curious about how AI can support UX and product design, feel free to reach out. I’m happy to share what I’ve learned and help others explore how tools like this can enhance research, documentation, and platform understanding.