Programmable sketchbooks

Success of spreadsheets

Making end-user programming powerful and ubiquitous is a central goal within our lab’s broader vision of making better tools for thought. Spreadsheets are a widely successful end-user programming tool, and in particular we are inspired by their support of gradual enrichment and programming in the moment.

Spreadsheets support gradual enrichment by allowing you to quickly put text and numbers in arbitrary boxes as if using a pen and graph paper, do basic math as if using a calculator, move things into neat rows and columns when you find yourself making comparisons, and add more formula-driven cells and labels if/when necessary as your model evolves—incrementally building dynamic complexity over time. You might eventually polish the formatting, clearly label input cells, and turn the model into a reusable piece of situated software. Each bit of effort to refine the model is a small investment made in a moment when the payoff is immediate.

When software allows adding dynamic behavior as part of gradual enrichment, it supports programming in the moment. Spreadsheets support dynamic behavior with live, reactive, declarative formulas you can place anywhere you would otherwise have data. They treat data and formulas equally: where they appear in the interface, how they are edited, and how they reference each other. This interchangeability supports programming in the moment by allowing you to keep working with the same tools, in the same mindset—thinking only about the specific thing you are modeling—even though you have switched to programming. They don’t force you to stop and switch to your programmer hat.

Through this lens, spreadsheets stand out as one of the more successful and prevalent instances of end-user programming. They give a broad range of users access to computational power while maintaining flexibility through the use of the familiar and adaptable grid—what Bonnie Nardi calls a visual formalism.

Bonnie Nardi’s classic book “A Small Matter of Programming” calls attention to the spreadsheet as a remarkably successful end-user programming environment and insightfully breaks down the factors that make it work.

Spreadsheets are immensely popular for uses that fit neatly within a grid of cells (and have been creatively misused enough to prove their flexibility), but they are not the end of the road for end-user-programmable media.

Despite the spreadsheet’s computational advantages, most people don’t reach for a spreadsheet the moment they want to think through a problem or communicate an idea.

Enduring value of pen and paper

Despite an overabundance of other tools, many people still turn first to pen and paper for ideating, note taking, sketching, and more.

There is something special that happens in the brain when making marks by hand, and a large body of research suggests it has various advantages, whether for learning and psychological development, retention of material, or activating more pathways in the brain.

What the hand does, the mind remembers.

We believe these attributes contribute to pen and paper’s staying power:

• Flexible & free-form
  While word processors constrain text to predefined lines, on paper you can write anywhere. You aren’t limited to writing text either—you can draw diagrams, sketch images, and doodle in the margins. You can combine these styles of expression freely, without stopping to change tools or select formatting or ask a system for permission.
• Immediate
  You can pick up a pad of paper and begin writing without hesitation, and without working through any system of menus or dialog boxes. This is essential for fluidly expressing and thinking through ideas—working at the speed of thought. This is also essential when integrated into a larger activity, like two people having a conversation over a pad of paper or a student taking notes alongside practicing an instrument.
• Intimate and tangible
  You can touch and feel a pen and paper, hold them, and move them around—there is nothing mediating your interaction. You don’t have to take care to avoid touching the wrong place and altering or losing your work, as is often the case with a digital tablet. You receive full tactile feedback even down to feeling the friction of your writing implement dragging across the paper’s surface. This results in a strong sense of direct manipulation and working with the material.
• Informal & “sketchy”
  The natural grain of digital media is to create precise and polished artifacts, but the fidelity of the tool you use should match the maturity of the idea you’re working with. Nascent ideas are tiny, weak, and fragile. They require attention, nimbleness, and open-ended thought to survive. When a tool pushes you into inappropriate levels of precision, it can slow you down and suppress creativity: you want to quickly throw a box on the screen but by the time you’ve chosen the drop-shadow depth and rounded-corner radius, that whiff of an idea has dissipated. Handwriting allows looseness, quickness, and incremental refinement—attributes hard to come by in the world of computers.
• Connected with human life
  Pen and paper are deeply entangled into our lives and shared cultures. Most people have been making marks on paper since infancy. Even if we don’t consider ourselves to be talented artists, we all have distinct and personally meaningful ways of expressing ourselves with ink. Simply put, it feels good to mark on paper.

Digital pen and paper, in the form of tablet/stylus interfaces, have extended the power of their traditional counterparts with sophisticated editing interactions like copy-paste, undo, and infinite zoom, while trying to keep some of paper’s unique charms.

Shortcomings of digital paper

We should be thoughtful about basic advantages of paper that we can lose when moving to a digital tablet/stylus. For example, you can:

• spread papers out over a table, or across a wall
• thumb through a stack of papers, re-sort them, put them in different folders
• pass papers to a colleague and not worry whether the contents have synced across the network yet
• cut a piece of paper, rearrange it, and glue it to another
• use different tools (straight edge, compass, highlighter) mixed and matched all on the same paper without the designers of those tools needing to know about each other
• operate pens with remarkably low latency
• work with sheets of paper without irrelevant pop-up notifications, running out of batteries, or system crashes
• purchase sheets of paper extremely cheaply

The Computer for the 21st Century by Mark Weiser in Scientific American’s September 1991 issue lays out some of these advantages in the defense of ubiquitous computing vs virtual reality.

We want a digital pen and paper that maintain as many of the above attributes as possible, but with all the benefits of a dynamic digital medium. The most important being programmability—describing new behaviors for a sketchbook to take on, automating and processing and simulating.

Research questions

What does it mean for a sketcher to program as they sketch? How might this new power aid them in their life and work? While we can’t foresee all the possible uses of a tool that doesn’t exist today, we can take inspiration from uses of traditional pen and paper to imagine a few use cases that hint at a larger set of possibilities:

• An event planner draws up a list to pair things that need to be done with people available to do them. After a bit of connecting lines and boxes, they add some behavior to highlight the remaining unmatched people/items to make them easier to see. Later on, they need to do this again for a different event, so they turn this first version into a template for easy re-use.
• A furniture designer sketches patterns for a table leg. After they decide they love a pattern that repeats a vine figure in a spiral down the leg, they automate the hand-drawn spiral pattern so they can try out different vine shapes instantly as they sketch. This helps them explore the design space quickly and stumble onto entirely new designs.
• An engineer on a circuit-design team is trying to illustrate to a colleague how their signal processing might be flawed. They roughly sketch what an incoming signal might look like, and then program in a model of what the circuit will do. The engineers take turns sketching on the page, trying out different signals and models, coming to a shared understanding of the problem.

These three examples share an interesting feature—they all illustrate programming in the moment while sketching. This is our ambitious vision for a programmable sketchbook.

However, this vision raises multiple layers of research questions at once:

1. What mental model or programming concepts are the right fit for interacting with hand-drawn marks?

   What features of a formula or programming system are important? Is a formal language even exposed?

2. How should an interface be designed for programming in the moment while inking?

   How do you interact with dynamic drawings? How do you see relationships between ink and the program? What affordances could make pen and tablet programming ergonomic? How do you edit a program “with pen in hand”?

3. What might you make with a programmable sketchbook?

   What are the most interesting use cases for this tool and how will it feel to interact both with the tool and with your creations? Which uses should the natural grain of this tool encourage?

A common research approach would be to start by designing a programming system from first principles. But because we believe the uses for this tool will be novel—and therefore unforeseeable—we don’t want the design to arbitrarily constrain the potential usage space.

Consequently these questions form a dependency stack in the reverse order: the use cases and aspirational feel of the tool determine the right design for the programming/editing interface, and in turn the way you interact with the models you build determines the right mental model and formula/programming system design.

If we want to start at the beginning of the dependency chain to explore use cases and the feel of a tool like this, we need to build a vertical slice through this whole stack to produce a working prototype we can actually play with.

For the bottom two layers we chose naive/incidental solutions—whatever was pragmatic, expedient, or best guess from our previous research experience in this area—so we could spend most of our time on the top layer.

Project goals

The essence of the Inkbase project was to experience that moment of making a sketch, enhancing it with dynamic functionality, and interacting with the result.

We hoped to unlock the next phase of this line of research by discovering interesting use cases, starting to understand the required ergonomics, and developing better intuition for how the tool should feel.

Prior art

While we haven’t seen work that directly achieves Inkbase’s goals, there has been a plethora of relevant work over the years, spanning from the 1960s to the present. A selection of projects we have taken ideas and inspiration from:

Sketchpad — Ivan Sutherland — 1963

(paper, video)

Sketchpad allowed the user to draw and manipulate dynamic models directly using a “light pen”. Even with the rudimentary hardware/software available at the time, it achieved a fluidity and level of direct manipulation in some ways unrivaled today. Creation of dynamic relationships/constraints directly with the pen is aspirational for Inkbase, though constraint solving is not currently implemented (explored in prior lab research).

ThingLab — Alan Borning — 1979

(video, tool, code, paper)

ThingLab is a constraint-based visual model builder, extending Ivan Sutherland’s ideas in Sketchpad. One of our favorite and most inspiring tools, it allows a model to be made by drawing it on the canvas with components that are easy to compose and also easy to create.

Viewpoint: Toward a Computer for Visual Thinkers — Scott Edward Kim — 1988

(paper, video)

Viewpoint is a demonstration that a simple text and graphic editor can be built by drawing it, and that this editor can be used to build itself. Inkbase takes inspiration from the goal that strictly everything is on the canvas, always visible, and that a wide range of functionality can be built using a small number of primitives.

INK-12 — Koile, Rubin, et al — 2010

(video, info, papers)

INK-12 is investigating how a combination of digital tools and freehand drawing can support teaching and learning mathematics in upper elementary school. Most interesting to us is the use of the computer to provide mental bookkeeping—helpful annotations and checking of the user’s math while still requiring the user to solve the problem themselves. More on this in our findings.

Drawing Dynamic Visualizations — Bret Victor — 2013

(video, notes)

DDV is a prototype system for making dynamic visualizations by drawing them. The user can draw and manipulate objects on the canvas and parameterize these historical actions to create imperative programs.
DDV is inspiring in its delivery of true “programming in the moment” while drawing.

Dynamicland — Bret Victor, et al — 2013

(site, notes from Omar Rizwan)

Dynamicland is a deeply inspiring new computational medium where people work together with real objects in the real world. Ordinary physical materials are brought to life by technology in the ceiling. Every scrap of paper has the capabilities of a full computer, while remaining a fully-functional scrap of paper. Dynamicland has influenced our thinking, particularly on programmable live objects and spatial relations.

Para — Jennifer Jacobs — 2014

(video, tool, code)

Para is a prototype digital illustration tool for making procedural artwork. Through creating and altering vector paths, artists can define iterative distributions and parametric constraints. Again, we are inspired by the creation and editing of dynamic relationships via direct manipulation.
Related: Dynamic Brushes

TickTock / Pronto — Dan Ingalls — 2016

(video: TickTock example, video: Pronto talk)

TickTock is just a tiny demo in Dan Ingalls’s Pronto talk where he brings a hand-drawn clock to life by manipulating the drawing and then combining the history of his actions with some pre-made widgets—programming by example. We find this an inspiring case of true programmable ink.

Object-Oriented Drawing — Haijun Xia — 2016

(paper, video)

OOD is a tool for drawing that gives the properties of canvas objects (e.g. color, stroke, etc.) spatial representations—moves them into the same workspace as the ink marks so they can be interacted with directly. We are inspired by the way some things that require code in Inkbase (e.g. linking the color of two objects) can be done via direct manipulation in OOD.
Related: DataInk, DataToon

Apparatus — Toby Schachman — 2016

(tool)

Apparatus is a unique and powerful hybrid graphics editor and programming environment for creating interactive visualizations. We love the approach of combining direct manipulation of the canvas with a declarative, reactive formula system.
Related: Cuttle is a more refined product version of these ideas.

Chalktalk — Ken Perlin — 2018

(paper, video, code)

Chalktalk enables a presenter to create and interact with animated digital sketches while presenting. Chalktalk mainly uses freehand drawing as a way to invoke and interact with preprogrammed objects. In comparison, Inkbase pushes toward private notebook use, working directly with ink as the primary material, and more accessible programmability/programming in the moment (see microworlds below for more thoughts on this).

---

Inkbase

To answer the research questions above, we built Inkbase, a native Swift application for the Apple iPad and Pencil, implementing a programmable sketchbook:

Below is a basic primer on Inkbase’s features followed by a sampling of things we made with it.

Inking and selecting

To support gradual enrichment, Inkbase is a sketchbook first—by default the pencil inks on the canvas and touch/drag does nothing:

Selection is a quasi-mode—while holding two fingers down on the canvas, touch an object to select or use the pencil to lasso a selection. Ink can be moved around the canvas by dragging it with a finger or pen while holding down an additional finger, or by just dragging objects that are already selected:

Objects and properties

Inked paths are the basic building blocks of everything you can do in Inkbase. We wanted them to be as open-ended and malleable as possible—they are live vector objects with dynamic properties. Selecting an object opens its property inspector:

All properties are live—including the original path as vector data—and can be inspected and edited:

Additional arbitrary properties can be added:

Dynamic behavior

Properties can have a value or can be an expression:

We wanted the reactivity of spreadsheets where the dependencies between properties are tracked by the system and values are automatically recalculated when necessary.

Inkbase uses a homegrown flavor of Lisp. In addition to being simple to implement (which allows us to easily interoperate between the core engine functions and end-user code), everything in Lisp is an expression (everything returns a value), which composes nicely with the idea of reactive properties. This is similar to spreadsheet formulas which always return a value into the cell.

Accessing properties is done with the ? function (with a shortcut button available in the Inspector for quick insertion of a property reference).

Property Lookup using (? object property) returns the value of the given property on the given object, and records the dependency in our reactive dataflow system. Now when that value changes, anything depending on it will be updated as well. This is analogous to referencing another cell in a spreadsheet formula using its name, e.g. =A6.

Inkbase objects always exist somewhere on the page. This makes accessing them by their position very natural.

We created a spatial query system where the user can ask for objects in a specific region, inside a specified path, or in a general direction (e.g. “to the right of…”):

The results of these queries are also reactive (refresh based on changes on the page), so they compose with the rest of the property system:

Thus we have programmable ink:

Lastly, our reactive system also allows for side-effects—p

Read More