1. The DataFrame scale gap

When I started working on Polars, I was surprised how much DataFrame implementations differed from SQL and databases. SQL could run anywhere ¹. It could run embedded, on a client server model, or on full OLAP data warehouses.

Whereas for dataframes, the API was different per use case and performance was drastically lacking behind SQL solutions. Locally, pandas was dominant, and remotely/distributed, it was PySpark. For end-users, pandas was very easy to get up and running, but it seems to have ignored what databases have learned over decades, there was no query optimization, poor data type implementation, many needless materializations, it offloaded memory handling to NumPy, and a few other design decisions that led to poor scaling and inconsistent behavior. PySpark was much closer to databases, it follows the relational model, has optimization, a distributed query engine and scaled properly. However PySpark is written in Scala, requires the JVM to run locally, has very poor unpythonic UX (java backtraces for one), and is very sensitive to OOMs. It also was designed for commodity hardware a decade ago and row-based OLAP execution has proven to be suboptimal.

With Polars we want to unify those two worlds under one flexible DataFrame API, backed by high performant compute. Our initial (and achieved) goal was offering an alternative for pandas with a flexible API that does enable query optimization, and parallel streaming execution. Second we want to make running DataFrame code remotely easy. Just like SQL, a Polars LazyFrame is a description of a query, and it can be sent to a server to be executed remotely. This should be dead easy. In the cloud dominant era, you should not be limited to the boundaries of your laptop.

2. Run Polars everywhere

Our goal is to enable Scalable data processing with all the flexibility and expressiveness of Polars’ API.
We are working on two things; Polars Cloud and a completely novel Streaming Engine design. We will explain more about the streaming engine in later posts; Today we want to share what are building with Polars Cloud.

The features we will offer:

• Distributed Polars; one API for all high performant DataFrame needs;
• Serverless compute;
• Configurable hardware, both GPU and CPU;
• Diagonal scaling; scaling both horizontally and vertically;
• Bring your own cloud; AWS, Azure and GCP;
• On premise licensing;
• Fault tolerance;
• Data lineage;
• Observability;

It will be very seamless to spin up hardware and run Polars queries remotely, either in batch mode for production ETL jobs, or interactively doing data exploration. The rest of the post, we want to explore this through a few code examples.

3. A remote query.

It’s important for us that starting a remote query feels native and seamless for the end user. Running a query remotely will be available from within Polars’ native API.

Note that we are agnostic of where you call this code. You can start a remote query from a notebook on your machine, an Airflow dag, an AWS Lambda, your server etc. The compute needed for data-processing is often much higher than the compute needed for orchestration in Airflow or Prefect. By not constraining you to a platform where you need to run your queries, we give you the flexibility to embed Polars Cloud in any environment.

In the query below we start our first query.

import polars as pl
import polars_cloud as pc
from datetime import date

query = (pl.scan_parquet("s3://my-dataset/")
   .filter(pl.col("l_shipdate") <= date(19