Magento 2 – expired payment gateway API key detected too late

The shop stopped accepting card payments on Saturday night. Monitoring did not detect the problem because the site was working – only checkout was broken. Customers received a generic “Payment cannot be processed” error. Several hours of sales were lost before a report arrived Monday morning. Fix time: 45 minutes after the report.

Symptoms

• Checkout fails at every card payment attempt
• Bank transfer and cash on delivery work normally
• In var/log/payment_vendor.log: 401 Unauthorized - API key expired
• Problem started exactly at midnight

Cause

The payment gateway API key had an expiry date set in the operator’s panel. It expired at exactly 00:00. Nobody knew that API keys in this gateway had an expiry – they are usually permanent.

Solution

# Immediate: generate new key in the gateway panel
# Update in Magento configuration
bin/magento config:set payment/vendor_payment/api_key "new_api_key"
bin/magento cache:flush

# Verify
curl -X POST https://api.gateway.com/v1/test \
  -H "Authorization: Bearer new_api_key" \
  -H "Content-Type: application/json"
# Expected: {"status": "ok"}

<?php
// Prevention: monitor API key expiry
class ApiKeyExpiryCheck implements \Magento\Cron\Model\JobInterface
{
    public function execute(): void
    {
        $apiKey     = $this->config->getValue('payment/vendor_payment/api_key');
        $expiryDate = $this->config->getValue('payment/vendor_payment/api_key_expiry');

        if (!$expiryDate) return;

        $daysLeft = (strtotime($expiryDate) - time()) / 86400;

        if ($daysLeft <= 14) {
            $this->notifier->addNotice(
                'API key expiring soon',
                "Payment gateway API key expires in {$daysLeft} days. Renew at: https://panel.gateway.com"
            );
            $this->emailNotifier->send('admin@shop.com',
                "WARNING: Payment gateway API key expires in {$daysLeft} days"
            );
        }
    }
}

Takeaways

Monitoring that only checks site availability is not enough. You need monitoring that tests critical business paths: add to cart, checkout, payment. Synthetic monitoring (e.g. Checkly, DataDog Synthetics) simulates the full flow every 5 minutes and alerts immediately. External service API keys should be monitored for expiry dates – add this to a calendar or cron job.