Comodo ASG Offline Again - WHAT'S HAPPENING????

Comodo ASG is offline again, this appears to have been the case for the past 4 hours at least. Cannot even access the admin pages to see what is going on. Can anyone at Comodo tell us what’s going on?

It’s 11AM in this part of the world so none of our customers have been receiving emails this morning, not good.

Have since heard from Comodo Support. Service will be offline for another 4 hours, which pretty much means we’ll receive no emails for our entire business day in Asia/Australia. :frowning: Some heads need to roll…

The amount of outages in the last year has been significant, and they’ve raised their prices on top. It’s evening here, I just redirected my MX records for the moment, it’s extremely frustrating.

Yes agree, our subscriptions are not due until October so will be thinking seriously about switching to someone else such as MailGuard. Live and learn…

Hi Everyone

There was a problem in the Load Balancer. LB support is currently checking the problem. It will be alive soon. Will you update as soon as It is alive.

Sorry for this inconvenience.

Kind Regards,
Hasan
Product Manager

Really? So the load balancer has taken 7 hours to restart??

Still not working also admin login page.

Nearly 6pm Sydney Time (Australia), have already switched service providers (free 30 day trial, which I expect to pay for in 29 days to become my new service provider). I have to say, I’ve not been happy with Comodo Antispam Gateway. The Trend for micro businesses to use hosted antispam will continue… but I don’t think comodo is going to be able to take advantage of that,

Now 7PM and still no resolution, we have been unable to receive email through Comodo for almost 11 hours now. This is totally unacceptable for a service someone charges $ for. I bet it will be fixed around 10PM (8AM their time) when their staff arrive at the office in Clifton NJ and realise one of the cleaners kicked out the power lead to the datacentre last night.

It’s after 11am in western Europe and no sign of the system working - so we’ve had no incoming email this morning.

Very frustrating not to receive any notifications about downtime or an accurate timescale for a resolution.

I have had to update MX records to switch mail routes away from Comodo, now just need to switch to an alternative provider which delivers a reliable service.

I’m so angry !!! I’m losing so much money - I’ve started to look for an alternative.

http://community.spiceworks.com/cloud/anti-spam/reviews?source=product_taxons_59366

Hi Everyone,

We fixed the problem. Currently queued emails will be delivered to your mail servers. We are making sure this problem will not happen again.
Sorry for the inconvenience.

Kind Regards,
Hasan

Your company said it wouldn’t happen the last time in September 2014. I was silly enough to believe you then.

As I expected the system came back up around the same time people would have arrived in Comodo’s US office. This also happened last time. What this is telling me is you have limited 24X7 support processes in place, and no effective monitoring in place. We’re only a small company but even we can setup automated systems that will wake up key staff when an outage occurs. Why can’t Comodo do this, or do your customers outside of the US not matter to you?

Hi all,

You anger/disappointment/dissatisfaction with ASG is very much taken to heart here in Comodo HQ. The issues ASG suffered last week were obviously not intentional and we had erroneously believed that we had mitigated against the probabilities of the events that caused last week’s issues.

Does this mean that ASG will never have an outage again? That is not something that can be promised just as no AV vendor can guarantee 100% detection rates, no cloud-service provider/ISP can guarantee 100% uptime, no streaming TV provider can guarantee 100% availability, no vehicle manufacturer can guarantee 100% reliability. In the real world where we rely on imperfect technology and human beings there is no such thing as 100% but we do everything we possibly can to get as close to it as possible bearing in mind that ASG suffers from the same technology/human interactions as O365 (https://downdetector.com/status/office-365), Barracuda (https://blog.barracuda.com/2014/12/24/barracuda-email-security-service-outage/) and others. You may find Which cloud providers had the best uptime last year? | Network World interesting.

It is worth noting that since August 2012, ASG has suffered approximately 32 hours of total ‘downtime’. Working that out with some rough math - total time since measuring began = 35 months x 30 days (on average) per month x 24 hours per day = 25,200 hours. Divide 32 hours by 25,200 hours and we end up with (rounded) 0.00125% downtime and an uptime of 99.99875% which is better than any Exchange server, rack-mounted tin or competitor’s product I ever came across in my 15 years in the trenches.

We will be adding a ‘Delivery Delay’ alerting tool into Antispam Gateway with the upcoming August release so that should we not be able to deliver email, whether through a problem with ASG, a problem with your ISP or a problem with your mail-server, at least we will be able to tell you about it as it happens.

We will continue to try harder, test more, spread wider and deeper to avoid these circumstances from every happening again. We are truly sorry guys. PM me as needed/desired – I’m more than happy to connect with you directly and answer any further questions.

M.

To answer the individual comments - at the risk of seeming ornery :slight_smile:

@kdennis
“The amount of outages in the last year has been significant,” there have been 2 ‘outages’ this year totaling ~11 hours. There have also been 2 periods this year where some emails to some domains were delayed by up to 4 hours, the ASG platform did not go down during these ‘delayed delivery’ periods.

“…and they’ve raised their prices on top.” this is the first price increase for ASG in many years. ASG has evolved significantly in this period with add-ons such as editable email templates, administrator delegation, User and Group permissions, Users login location history and GeoIP restrictions, Relay Restrictions, the Audit log, LDAP importing of user accounts, auto-discovery of user’s mailboxes, highly configurable blacklist/whitelist rule creation, white- and blacklisting per user and the bolt-on mail-backup Archive feature to mention but a few.

@Ossie44
“Yes agree, our subscriptions are not due until October so will be thinking seriously about switching to someone else such as MailGuard. Live and learn…” while we would hate to lose you we would hate it even more if you are dissatisfied with the service. There are some reviews on various vendor’s products here > http://community.spiceworks.com/cloud/anti-spam/reviews .

“So the load balancer has taken 7 hours to restart??” it is a little more complicated than rebooting a load-balancer, there are many checks and balances that need to be put in place leading up to a reboot of anything within the ASG infrastructure (more info follows below)

“realise one of the cleaners kicked out the power lead to the datacentre last night.” - cleaners are not allowed into our datacentre, neither are the developers, nor the DBAs nor the DevOps guys. Only hardware/networking guys and gals are allowed in and only with just cause. Principle of Least Privilege and all that.

Furthermore, we were aware of the issue within minutes of it happening and activated the required support teams as needed. The customer-facing support guys were informed that there was an issue as the 1st line devs/devops/DBA/NOC/SOC teams were all brought into play to investigate. The problem was then escalated to the 2nd line devs/devops/DBA/NOC/SOC teams as the 1st line guys did not have the access rights required to get to this particular problem after they identified potential causes. Then some of the 3rd line guys had to be brought in to access those areas that the 2nd liners are not allowed to get to. While all this was happening the hardware guys got called out to the datacenter itself in case cable-switching or hardware replacements had to happen. This is why it can take some time getting the platform re-stabilized, not because someone in Clifton realized something was amiss.

“As I expected the system came back up around the same time people would have arrived in Comodo’s US office.” no-one in Comodo HQ, barring myself and 1 other, has any access to the ASG back-end/hardware/infrastructure, Clifton staff only have configuration access at a global level. Principle of Least Privilege again.

“…do your customers outside of the US not matter to you” ALL customers matter regardless of where they are or how small/large their organizations are.

“Your company said it wouldn’t happen the last time in September 2014. I was silly enough to believe you then.” the Sept. 2014 incident was completely unrelated to what happened last week.

@kjhan
“I have to say, I’ve not been happy with Comodo Antispam Gateway” may I ask what you have not been happy with? Do you have any support tickets etc. wherein you have requested assistance? Please contact me directly if you wish to discuss.

@AngryCustomer2 – Please contact me directly.

Best,

Michél