Published on: 21 Feb 2023

Table of Contents

• Preface

• DAW GUIs are complicated
  • Performance problems
  • GUI library problems
  • Complicated logic
• My views on the Rust GUI landscape

• What actually makes a GUI “high performance”?
  • Rendering performance
  • Input handling performance
  • Update handling performance
• My views on the Rust GUI landscape – Part 2
  • The web-based toolkits
  • The immediate-mode toolkits
  • The Elm-based toolkits
• Potential plans moving forward

• Should I stick with Rust?

• Options using native Rust libraries
  • Vizia
  • Iced
  • Custom in-house solution
  • Non-Options
• Options using Rust bindings
  • GTK
• Options using C++
  • FLTK
  • JUCE
  • QT
• Development plan?

• Final thoughts

I would like to write about where my head’s been the past several months. If you’ve noticed that Meadowlark’s development has slowed down, this article explains why.

Essentially I’ve really underestimated how difficult it would be to develop the frontend/GUI of Meadowlark. Not just with how complicated a DAW’s GUI is, but also in finding a GUI library that is actually suitable for the task.

I want to use this blog post to do three things. First I want to highlight why this is such a hard problem. Second I want to share my thoughts on the current state of Rust’s GUI library ecosystem. And third I want to share some potential paths I can take for the future of Meadowlark.

I would like to hear any thoughts people may have (especially on the part on potential paths for this project). I am most active in my Discord server if you are interested in discussion.

Saying that Meadowlark’s frontend/GUI has unusual needs (both performance needs and just features in general) is an understatement. DAWs might just have one of the most complicated GUIs out of any piece of software out there.

Here are some complications I’ve come across, divided into three parts.

Performance problems

• DAWs have a lot of toolbars and panels (browser panel, timeline panel, piano roll panel, fx rack panel, mixer panel, audio editor panel, automation editor panel, settings panel, etc.).
• Some widgets like decibel meters and other visualizers are constantly being animated, meaning the GUI library needs to efficiently redraw the screen every frame.
  • In addition, visualizers can be expensive to render on the CPU (especially spectrograms/spectrometers). Ideally you should use custom shaders to render them on the GPU.
• Clips on the timeline are notoriously expensive to render. There needs to be some way to cache the contents of clips into a texture (Either directly or by making use of the GUI library’s “damage tracking” which I’ll get into later.)
  • Audio clips are the biggest culprit, because rendering waveforms requires the CPU to first do a linear search through the source material for peak values, and then render the waveform pixel-by-pixel (or even better use custom shaders to send commands to the GPU).
  • Automation clips can contain a bunch of bezier curves, which are slow to render.
  • Piano roll clips can contain lots of little rectangles in order to display a “minimap” of the MIDI notes inside of it.
  • On top of all this, clips can contain text labels which can also be expensive to render.
  • The fact that a timeline is zoom-able also makes it harder to cache the rendering of clips. If the timeline changed its zoom level, all visible clips pretty much have to redraw all of their contents.
• Piano rolls can also be expensive to render if there is a bunch of MIDI notes, especially if there are text labels on the notes.
• If the user clicks on a folder in a sample browser containing hundreds or even thousands of files, allocating a label widget for each file in the browser list will be very expensive. Something like the list factory in GTK is needed here.
• We want to reserve as much CPU as possible for the actual audio processing. Ideally the GUI shouldn’t take up more than one or two CPU threads.
• On some platforms, we also need to make sure there’s actually enough CPU left for 3rd-party plugins to render their GUIs.

GUI library problems

• DAWs have unconventional widgets and layouts. For example, not only does a mixer track contain custom widgets like knobs, sliders, and decibel meters, but all of those widgets are not positioned according to a traditional layout scheme like a list, grid, or a tree.
  • This is especially true if the DAW has a horizontal FX rack like I plan with Meadowlark.
• DAWs don’t follow traditional/recommended design standards or “human interface guidelines”. They have a bunch of custom styling to cram all of that information onto the screen (and to actually look like an audio application).
  • This custom styling and layout also makes it harder to support localization.
• Ideally I want to support loading user-generated themes.
• Preferably the DAW should let the user pop out panels into another window (or at least support a preset number of multi-window workspaces). Multi-window setups are harder to deal with in code.
• The timeline and piano roll are not simple “scroll areas”. They can be zoomed in and out, meaning there needs to be some kind of custom positioning and sizing logic for clips and MIDI notes.
• When the timeline is zoomed in very far and/or a clip is very long, the resulting width of the clip in pixels can be very, very long. This could cause issues if the UI library tries to render the whole thing (especially if the clip has expensive contents). So you need to make sure only the visible part of the clip is actually processed and rendered.
• In order to deal with really long audio clips, you need to stream the file from disk. This makes rendering their contents on the timeline much more complicated because now you are dealing with an async operation.
• Some widgets like knobs and sliders need to be able to listen to raw mouse input data as opposed to absolute mouse coordinates (or even better is the ability to use pointer locking). Otherwise if the user’s mouse hits the edge of a screen or moves outside the window while dragging one of these widgets, they will stop working.
  • This is especially true with a horizontal FX rack where there are bunch of knobs and sliders near the bottom of the screen.
• There needs to be some way to easily input the value of a knob/slider parameter using a keyboard. Ideally in some sort-of pop-up text input box.
• There needs to be extensive support for custom keyboard shortcuts. This includes shortcuts that could potentially conflict with accessibility features such as using the spacebar to start/stop the transport.
  • In addition, the UI library needs to support keyboard shortcuts even when a widget/window isn’t focused. For example pressing spacebar to start/stop the transport while inside a 3rd-party plugin window.
• There needs to be some way to access the raw window handles in order to host 3rd-party plugin GUIs.

Complicated logic

• There are three separate states that must be kept in sync: The state of the save file (which I call the “source state”), the state of the GUI, and the state of the backend. These have to be separate states because:
  • The source state, the backend, and the GUI sometimes want different units. For example, the start of an audio clip on the timeline may be in units of beats in the source state, units of samples in the backend, and units of pixels in the GUI. So units must somehow be efficiently converted from the source state the the backend/GUI state.
  • The backend process runs on a realtime thread, so you cannot just use simple mutexes to read the source state. Some other method must be used like lock-free message channels or garbage-collected clone-on-write smart pointers.
  • There are situations where the state of the GUI/backend can be different from the source state. For example when the user is in the process of dragging the position of a clip on the timeline, the clip needs to move in the GUI, but the change shouldn’t actually be committed into the source state or backend state until the user lets go of the mouse button. Otherwise it would cause expensive updates to happen every single frame the user is dragging the clip.
    • Another example is when the backend can’t find a plugin listed in the source state. In this case, both the backend and the GUI will have a “missing plugin” placeholder. But this shouldn’t cause the original state of the plugin to be overwritten.
• There can be a lot of drag-and-drop targets which can be complicated to implement. For example:
  • dragging samples/midi/automation from the browser onto the timeline
  • dragging samples/midi/automation from the browser onto a track header to add it to the timeline
  • dragging samples/midi/automation from the browser into an empty portion of the timeline to add a new track
  • dragging samples/presets from the browser onto a plugin in the horizontal FX rack
  • dragging a modulation source onto a parameter in the horizontal FX rack
  • dragging a plugin in the horizontal FX rack into a slot in a container device
  • dragging a plugin in the horizontal FX rack between two other plugins to reorder them
  • dragging a plugin from the horizontal FX rack onto a track header to move that plugin to that track
  • dragging a track header/mixer track into an area between two other tracks to reorder them
  • dragging a clip/multiple clips onto a differ