Papaya Pay · 2024 – 2026

Making automatic refunds safer

Principal engineer; designed the library, led the incident root cause ← all work

Stack: PHP, MongoDB, Cron, Sentry, PHPUnit

  • Three refund jobs moved onto one library, with the first tests that logic ever had
  • Guardrails for VIP users, uncleared payments, partial payments, and bills already paid out
  • Incident traced to two silent failures; about 75% of the over-refunded money recovered

I think automatic refunds are the scariest code in a bill-pay platform. If there’s a bug in bill payment, it’s loud: a biller doesn’t get paid and somebody calls about it. If there’s a bug in refunds, money goes to the wrong place quietly, and you usually find out from the finance team.

Three Jobs, One Library

By 2024 we had three scheduled jobs that refunded bills for different reasons: bills nobody acted on, bills a partner cancelled, and bills the user cancelled. Each one had its own copy of the refund rules, and they’d drifted. I pulled all three onto one shared library and wrote the first automated tests that logic ever had. One reviewer commented on the merge request, “Finally we have some tests of this critical flow-of-funds logic.”

The library has a few layers of guardrails, and each one is there because money had gone wrong that way before:

  • For VIP users, it sends an alert instead of refunding.
  • It won’t refund when a payment has been authorized but hasn’t cleared yet, since that money may still settle.
  • It won’t treat a partially paid bill like an unpaid one.
  • It won’t refund a bill if the money has already been paid out to the biller.

The Incident

In August 2026 we sent refunds on bills that had already been paid out, which is exactly what that last guardrail was supposed to prevent. When we dug in, it came down to two things that had been quietly failing for a while:

  1. The paid-out check relied on a job that aggregates clearings, and that job had been failing. When it failed, the check saw “no payout” and let the refund through. It had no way of knowing the data it was reading was bad.
  2. In one code path, a type cast turned a dollar amount into the wrong number of cents, which switched off a second failsafe that should have caught it.

The fixes themselves were small. The check now blocks the refund if the aggregation hasn’t run, there’s a separate alert when the aggregation fails, clearings only get marked done after they’ve actually been written, and the bad cast is gone. Working with our operations team, we got back about 75% of the money.

What I Took From It

The library had tests, but a test of the “already paid out” case on its own wouldn’t have caught this, because the problem was in the data the check depended on, not the check. The thing I keep coming back to is that a check that protects money needs to know the difference between “no” and “I don’t know,” and it should treat “I don’t know” as a no. Ours didn’t, and it took an incident and a bunch of recovered money to make that obvious.