Your guide to using webhooks

What are webhooks?

Put simply, webhooks are notifications that send event data to another application after the event occurs. Webhooks are a way for apps to communicate information through automation and user-defined HTTP callbacks – similar to SMS messages – typically triggered by an event such as a successful email delivery or bounce notification. For example, when you provide your banking app with your direct deposit information and phone number, and they send you an SMS message telling you that you’ve just received a deposit, a webhook is what makes it possible for your bank to send that text message with relevant updates.

What are webhooks used for?

Webooks are used to create notifications around specific events. Since webhooks are based on HTTP POST, they are easy to use. What’s more, webhook scripts can be written in just about any scripting language developers prefer, including curl, Ruby, Python, PHP, Java, C# and Go. Once the webhook’s data has been captured, it can be stored in a database and used to gauge the effectiveness of email campaigns or augment recipient profiles. You can also get more granular data than out of the box dashboard might provide by customizing the metrics the webhook returns.

How do webhooks work?

So, how does a webhook work?

When the event occurs, the source site makes an HTTP POST request to a URL the developer has configured to receive the webhook. Developers can configure them to cause events on one site to invoke behavior on another. Then, that webhook’s data can be sent to the configured URL as a webhook payload in JSON or XML format. Here’s the four steps you need:

Step one: Choose the desired data

The first decision developers make as they plan out their email tracking and response is exactly what data want returned. If the goal is to know when sent emails are bouncing, the user URL could perform a script on incoming POSTs to capture and save the bounced email address in a local database. The same script could be augmented to capture recipient name, subject or any other parameter provided by the webhook.

Let’s start by exploring the webhook POST request, which can be encoded as: application/x-www-form-urlencoded for most messages, and as multipart/form-data if there’s an attachment included with the message. The POST request method is designed to request that a web server accept the data enclosed in the request message’s body for storage.

Here’s an example of an HTTP POST made by Mailgun to a URI at Runescope. Note that the Content Type header is set as application/x-www-form-urlencoded:

The body of the message contains parameters stored as key-value pairs. (We’ll go into greater detail about the data that is posted in the section below.) Since the data is encoded, it will simply appear as gibberish. Here’s what the decoded body might typically look like:

Step two: Attach data to messages

When sending an email, some ESPs permit users to attach data to their messages by passing custom data to the API or SMTP endpoints. The data will be represented as a header within the email and is typically formatted in JSON. This custom data would then be included in any webhook events related to the email containing it. More than one header may be included and their values will be combined.

Example:

To add this header to a message:

Alternatively, you can easily delete an existing webhook:

Sometimes it’s helpful to categorize outgoing email traffic based on some criteria, perhaps separate sign-up emails from password recovery emails or from user comments. The ESP may permit tagging each outgoing message with a custom value. These values become tags that can recall or aggregate deliverability statistics. To attach a tag to a webhook message, supply one or more O:TAG to it.

Tagging Code Sample:

Step three: Setup the URL

In order to receive the data from a webhook, users must give their ESP a URL to deliver requests to. This means that they also need to set up the URL in their app, so it is accessible from the public web and different IP addresses (hence the need for security). The ESP’s webhooks will then POST data to the URL as application/x-www-form-urlencoded or multipart/form-data.

Step four: Create scripts to capture data

The final step is to add scripts to the URL that capture the data provided by the webhooks and process it in any way the developer sees fit. This is where you can get creative but also granular with the attributes you want to collect. The best way to look at this is with an example.

A use case for a webhook might be to track email bounces. For this you will need to capture an attachment and store the file locally. For example, the following code uses a combination of microframework for Python, Flask and Requests HTTP library. Here’s a quick Flask App to capture a file from a Bounce webhook, keep the native filename and store it locally on your webserver.

To see it in action, run your application and paste the URL (For example: http://yourdomainhere.com:100/webhook) into the Bounce webhook and click to “Test Webhook.” You’ll get a file named “message.mime” with:

These four steps will help you setup a complete webhook, but in order to benefit, you’ll need to trigger it and learn about event parameters.

Webhook events and parameters

As explained above, webhooks are triggered by specific events. In the email realm, these events include opens, clicks, unsubscribe requests, and other events related to email delivery. Here’s a complete breakdown of the Mailgun events that can trigger a webhook:

• Open: This event occurs every time a recipient opens a message. Open tracking is enabled by using the O:TRACKING or O:TRACKING-OPENS parameters when sending a message.

• Click: This event tracks every time a recipient clicks on links in an email message. Enable click tracking by using the O:TRACKING or O:TRACKING-CLICKS parameters when sending a message. As with opens, the appropriate CNAME records must be included in the user’s DNS.

• Unsubscribe: This event occurs when a recipient clicks on the “unsubscribe” link in a message. Spam Complaint: Not every ISP supports Feedback Loop (“FBL”) notifications for spam complaints, but developers should make sure that they get data from all of the ones that do.

• Bounce: An email message is said to “bounce” if it is rejected by the recipient SMTP server. These are often classified as hard or soft bounces as follows:

  • Hard bounces (permanent failure): Recipient is not found, and the recipient email server specifies the recipient does not exist. The app should stop attempting delivery to invalid recipients after one hard bounce.

  • Soft bounces (temporary failure): Email is not delivered because of a temporary issue, such as a full inbox. Apps can programmatically respond to soft bounces by reattempting a set number of times before removing the recipient address from the list.

• Failure: Failures consist of both hard bounces and soft bounces. Depending on its capabilities, an ESP may notify users through a webhook when a message is dropped (i.e., stop retries) for any of several reasons.

• Delivery: A successful delivery occurs when the recipient email server responds that it has accepted the message. Depending on the event, webhooks can deliver a variety of parameters to help identify and describe the message in question. This data can then be parsed via scripts for analysis. Common parameters include:

  • Event

  • Recipient

  • Sending Domain

  • Message Headers

  • Recipient Identifying Details: country, region, city, device, email client and OS

Depending on the ESP, other parameters may include custom variables, tags and campaign names and authentication or user IDs, among others. In addition, some events offer more detail, such as a URL clicked; a reason for/description of a negative type of event or special codes providing specific event details.

Securing webhooks

A receiving URL must be public. To keep contents secure, webhooks should include a signature timestamp and token to create a hashmap. A hashmap is just a way to store items with identifiers. The hashmap also uses an API key to verify that the data is coming from the developer’s ESP. Users should program their application to check this hashmap and compare it to that of the ESP, and then allow the POST to be made only if it matches.

To verify the webhook is originating from their ESP, developers should link the timestamp and token values, encode the resulting string with the HMAC algorithm (using the ESP’s supplied API Key as a key and SHA256 digest mode), and compare the resulting hexdigest to the signature. Also, users can cache the token value locally and refuse to honor any other requests with the same token. This will prevent hackers from using the token to repeat or misdirect actions.

Another level of security would be to check the timestamp to confirm that the POST attempt has been made within a certain timeframe.

Below is a Python code sample used to verify a webhook signature:

Creating webhooks with Mailgun

You can find complete information and code about creating and deleting different types of webhooks in our documentation, but we’ve included a few common examples below. Check them out!

You can use our API to create a new webhook:

Or update one:

Alternatively, you can easily delete an existing webhook:

Additionally, for information about your webhooks, you can get details about any webhook URL:

Of course, if we made examples of all the ways you can use our API to create and optimize your webhooks, we’d be here all day. The point is, you can do pretty much whatever you want (we promise we won’t tell your mom).

Mailgun makes it easy

While there are several methods for accessing the data generated by email delivery, including ESP dashboards and API calls, cut yourself some slack and sprinkle in a few webhooks to make things easy – they are the most flexible and efficient way to collect granular email message data.

Rather than pulling data from your ESP, opt to receive continuous push data related to email in real time. Mailgun makes webhooks easy which makes things easier on teams who use this real-time information to make decisions for current and future email campaigns across different web applications and web services. Cool, right?

You know what else is cool? More information on webhooks. For easy, step-by-step info on how to use webhooks with Mailgun, check out our documentation on webhooks. For more deep dives like this, subscribe to our newsletter, webhooks aren’t the only thing that can bring you granular data.