We have received reports that some domains are not receiving email and that the Incoming log search is not responding as expected.
The team are investigating and we should be able to update you with a fix ETA withing a few minutes.
The team believes they have found the error and expect a fix should be in place by 16:30 UTC
Update:- we believe the fix to be in place and are monitoring progress
Update - the fix is proving only partially effective. Our engineers will be providing an resolution estimate shortly
Update - our engineers expect to be able to provide a fix estimation by 17:00 UTC
Update - mail delivery is improving but still requires progress monitoring as full functionality is yet to be fully restored
Update - full fix estimate in 1 hour (18:40 UTC)
Update - our engineers continue to investigate and monitor the situation regarding full efficacy of mail delivery which is currently intermittent.
Update - incremental restarting of cluster components is now under way. Full service should resume within 10 - 15 mins of this post
Update - we are continuing to monitor the situation but currently mail delivery seems to be operating as expected
Conclusion - mail delivery is functioning as expected. Investigation and synopsis results will be posted shortly.
The (overdue) synopsis.
Database ‘primary key’ index lookup failures were the culprit. For a handful of domains, as as email recipient couldn’t be cross-referenced accurately against domains and the domain’s users, emails were being put back in the ‘master’ delivery queue.
Needless to say the issue was fixed and emails were successfully delivered.
Kind regards
Michel.