Our users run over a million apps on Pipedream: everything from internal tools to production workflows. We run production Pipedream systems on the platform itself: our email trigger, customer communication, error notifications, and more.

And like every app, things break. Third-party APIs go down, you accidentally ship a bug to a workflow, your app receives input it didn’t expect, and more. Failure is inevitable. So you need timely alerts, detailed logs, and automation of failure recovery.

We reflected on the work we’ve done internally, and our conversations with customers, and wanted to share practical advice for how to run and monitor critical apps on Pipedream.

Design your workflow

Workflows are apps, built in a series of linear steps. When we first design workflows, we consider two things:

1. When does the workflow run?
2. What do we do when that event happens?

In Pipedream, these map to steps:

1. The trigger step — Triggers define the type of event that runs your workflow. For example, HTTP triggers expose a URL where you can send any HTTP requests. Pipedream will run your workflow on each request. The Scheduler trigger runs your workflow on a schedule. Workflows can more than one trigger.
2. Actions and code — Actions and code steps drive the logic of your workflow. Anytime your workflow runs, Pipedream executes each step of your workflow in order. Actions are pre-built code steps that let you connect to any API without writing code. When you need more control than the default actions provide, code steps let you write any custom Node.js, Python, Go or Bash code.

Once you design your workflow, you should see what built-in triggers and actions you can use on Pipedream. We’ve developed and QA’d thousands of these integrations, so you don’t have to write code to use them — just fill in the required argument and go.

Pick the right trigger

Trigger steps, especially, can handle a lot of logic for you. When an API supports webhooks (like Google or GitHub), Pipedream will automatically subscribe to events from the source service. When it doesn’t, we poll API endpoints for new events on a schedule, triggering the workflow when they occur.

We also validate webhook signatures to authorize incoming requests, and dedupe duplicate events so your workflow doesn’t accidentally get triggered twice.

You can see the code for all of these integrations on our public GitHub repo, submit a pull request to fix an issue, or talk to our team if you need something we don’t have yet. You can even build your own integrations when you need custom logic.

Understand the fault tolerance of your event source

The Pipedream service itself, and your individual apps, will inevitably encounter errors. Many apps will retry webhook requests on errors. Stripe, for instance, provides detailed docs on exactly how it handles specific types of errors:

In live mode, Stripe attempts to deliver your webhooks for up to three days with an exponential back off. In the Events section of the Dashboard, you can view when the next retry will occur.

Find your app’s docs or reach out to their support team to ask if they can give you the same level of insight. Document this in your incident playbooks so you understand this behavior when things fail.

If you’re sending your own requests to Pipedream — from your app or from your users’ browsers — you should handle back off and retry in the same way. Where possible, retry all HTTP 5XX errors, and implement your own alerting / logging for these errors. If you’re seeing persistent unexpected errors, check Pipedream’s status page or reach out to our team.

Sanitize input: Add a filter step

Most trigger steps run on a broad set of events. For example, GitHub sends HTTP requests on all labeled and unlabeled events — when labels are added to issues and pull requests — but they don’t let you filter on events for specific labels. You get all or nothing.

At Pipedream, we want to run workflows on bug, app, triaged labels, and more. Since workflows are a sequence of linear steps, we can end the workflow early if we get an event for another label we don’t care about:

Use the Filter actions or end a workflow early in code to control when your workflow runs:

Add authentication and authorization

When you use an app-based trigger, Pipedream tries to validate the event using the guidance from the app provider. For example, many apps will pass a “signature” that encodes the data sent in the request along with a secret that you (or Pipedream) configures for the webhook. If you know the secret, you can decode the data and validate the signature, which helps confirm the event was sent from the app you expect and not some random server on the web. See the Stripe docs for an example.

When you run your own HTTP