I’ve spent almost a decade now obsessed with the problem of truly enabling anyone—not just data teams—to explore and understand their business data. I still obsess over this as passionately as ever. It’s a much harder problem than I ever realized, but it’s just as important.

35 years ago, Microsoft Excel was first released. It was an incredible product that truly empowered business users to work with spreadsheets of data—and remains so today.

But when data gets larger than a spreadsheet, Excel no longer suffices, and there still aren’t any tools intuitive enough to take its place. Users are forced to learn databases, SQL, or complex Business Intelligence (BI) interfaces—or rely on data experts to do it for them.

Today, we’re excited to announce that we have—through thousands of design iterations, dozens of functioning prototypes, several hundred user tests, and countless hours of development—created an interface that truly enables the business user to work with data. We call the interface Visual SQL.

Reality and research

A little over a year ago, after a long customer roadshow for Chartio, I was forced to admit that we hadn’t yet solved the problem. We’d built an incredible product, created a profitable business with fantastic customers, and been rated as the most usable BI product on the market for years. But it was clear that we still weren’t usable enough.

I wrote a document summarizing the problems we were still facing in truly democratizing data, and it ended up being over 40 pages. I then had my product team run a series of user tests (thanks, usertesting.com!) on both our product and others in the self-serve BI space.

The tests confirmed that we were the most usable, but still not close to being intuitive. Filtering for users who claimed to be at least moderately competent at Excel, we found that only one in ten of them could use BI products. Users weren’t confident in what data they were exploring, what “measures” and “dimensions” (ie. rows and columns, groups and values) meant, and how the interface was going to map to their desired outcome in general. There was clearly a giant learning curve—and we needed to shrink it.

Problems and insights

We went back to basics, and—with a lot of effort—condensed our years of experience working with data and thousands of customers into the following two truths:

1. SQL is the most flexible and powerful way to query data, but it’s not good for exploration

I love SQL—so much so, in fact, that I wrote a whole SQL interface and tutorial to help people more easily get started learning it. SQL is an incredibly flexible language, and it’s THE language for working with data.

But SQL just isn’t going to be learned by a significant percentage of people. Even for people who do know SQL, it’s not a great way to explore. There are so many things to memorize that writing a SQL query frequently requires quite a few Google searches for that date format, or the right name of some function, or how to do window functions again. It’s also verbose, and prone to error. With straight SQL as your interface, you’ll never have the agility you need to truly explore your data.

2. BI is better for exploration, but still requires too many steps and is not nearly intuitive enough

Business Intelligence products have for years been promising to tackle this problem, but have some serious limitations. Most BI exploration interfaces are built to explore a single dataset or table at a time. This reduces complexity, but at a cost. It limits what the end user can do, and requires extra steps for data teams to build datasets.

Making these datasets also requires impossible foresight into what and how the end user might like to explore the data. It results in a lot of duplication in an ill-fated attempt to flatten a 3-D model into sets of 2-D tables mostly because of the limitation that the interface cannot handle relationships during exploration.

The result is that business users still have the data team as a bottleneck for a great percentage of what they might want to explore. While a noble attempt, these BI products will never allow truly democratized data exploration.

Insight – make SQL visual

With those truths in mind, we realized that to truly enable business users to work with dat