How we reduced our failed payment rate by 92%

Originally posted on Gousto Engineering & Data

Most Gousto customers choose to get their recipe boxes on a subscription basis: every week they choose their recipes from our menu, and we pick, pack and deliver them to their door. To give customers as much flexibility as possible, we let customers edit their box right up until it’s sent to one of our factories for picking.

This process works really well most of the time. But what happens when a customer’s payment fails?

It’s natural that a proportion of payments will fail — customers’ cards have changed, or there aren’t enough funds in the account. At Gousto we had a natural failed payment rate of around 2% for existing customers.

Until recently we only had two ways of handling a failed payment. First, we could send out the box anyway and put a debit on the customer’s account. Alternatively, we could just cancel the customer’s order.

Neither of these is a good solution. We never want to have to cancel a customer’s box, nor do we want to put a customer into ‘debt’. The majority of failed payments are because a customer has forgotten to update their card details and are still expecting to receive their order. I’m sure you can imagine how frustrating it might be to lose your food delivery for the week just because your debit card expired.

How did we reduce failed payments?

We decided to request a ‘dry run’ payment 24 hours before the actual payment is due — this is known as a pre-authorisation. Pre-auths allow merchants like Gousto to ‘reserve’ some amount of money against a customer’s payment method, without actually charging them. When we attempt to reserve these funds, we’ll receive a success or failure response from the customer’s bank. If unsuccessful, we can ask the customer to update their payment details.

As you can see, pre-auths allow us to catch a failed payment. We then email customers and give them 24 hours to update their card details in time for the final payment. This alone reduced our failed payment rate by 92%!

Many customers will receive a notification from their bank when the pre-auth is requested, which serves as an extra reminder of their upcoming box, with time for them to update their recipes before it’s dispatched. If the customer no longer wishes to receive their order, they also have the opportunity to cancel it through the app before it’s already paid for and dispatched. This reduces wasteful unintentional orders.

How do pre-auths actually work?

When a merchant pre-authorises funds on an account, it doesn’t have access to those funds yet; they’re ring-fenced for that merchant to use at some future point in time and the customer can’t use those funds for anything else.

If the merchant doesn’t end up charging the customer, the funds are released back to the customer by their bank after a period of time, typically five days. We built our system so that if a customer cancels a box that has been pre-authorised but not yet paid for, we would immediately void the pre-authorisation. This ensures we aren’t holding onto customers’ funds for any longer than is necessary.

How did we engineer this? Enter: Data bulkheading.

As I mentioned above, once a Gousto order is ‘cut-off’, it is no longer editable by the customer and is sent to one of our factories to be packed. This happens in batches of orders at a time, so it’s extremely important that this process is fault tolerant and speedy. To this end, many Gousto applications use a bulkhead architecture, and our shiny new pre-auth system is no exception.

The pre-auth process comprises a series of serverless functions, but we don’t want these functions to have to rely on a centralised data store to do their job. Instead, they have their own data store, which keeps a copy of all upcoming orders with payments due in the near future. Not only does this save additional functions from putting more load on a single database, but should there be a fault in another part of the system, the payments data store will be unaffected. Fun fact: the bulkhead pattern is named after the isolated sections of a ship’s hull. If the hull is pierced, only the damaged section fills with water, preventing the rest from flooding.

The pre-auth system is an ideal use case for serverless functions. Demand for these services is very spiky — when a batch of orders is ‘cut off’ for example, we need a large amount of concurrency to quickly process them all. The rest of the time, we don’t see much activity, so it doesn’t make sense to pay for running servers.

For our payment functions to maintain a storage bulkhead, they need to be kept up to date with the state of orders. We utilise AWS SNS and SQS for this or — in other words — pub’ sub’. When any application within Gousto makes a change to an order, we publish to a topic. One of our payment functions is subscribed to this topic, and if the change affects the price or payment of the order, this update is added to its queue. The change is then persisted to the payments data store, ready to be processed.

After testing pre-auths on a small group of customers for a few weeks, we saw very positive results, both financially and from the feedback we received from customers! We have now rolled this out to everyone who uses Gousto.