2 Mar 09:51:51
Details
2 Mar 09:51:51

After the work last night, load on the mail servers has been increasing this morning and making downloading and logging in slow again.

We are working on it and will post another update shortly.

Update
2 Mar 10:26:53

Customers may find that:

  • increasing the timeout (if using POP3) and
  • increasing the time between checking for new mail

will help problems with receiving email. These would be settings found in the account settings of your email program.

Update
2 Mar 12:20:58

In an attempts to bring the load down on the server and to allow users who have logged in to download messages we will be limiting the connections to the server. This will mean that logins will be blocked. The load should then decrease and we'll then let more users in.

Update
2 Mar 13:10:58

We are currently limiting POP3 logins until load has decreased. This will allow IMAP users to catch up. Once the load has leveled out and the IMAP users mailboxes have caught up, POP3 logins will be accepted.

Update
2 Mar 13:24:24

Restricting POP3 logins is helping the mail servers.

It will take a bit longer until the POP3 connections are accepted again though.

Update
2 Mar 14:42:01

Email is still sorting itself out and taking time, but it may not be until the end of the day that it will be fully usable again.

Update
2 Mar 16:10:39

Email is still the same at the moment - Most POP3 connections are being blocked to enable the servers to catch up. As a result IMAP (and webmail) connections will be slow and may time out.

We expect the load to lighten up as we get in to the evening.

Update
2 Mar 19:25:25

We still have staff working on email this evening.

Update
2 Mar 19:32:40

We've rebooted a network switch that is part of the mail service, and are working on reconnecting the file shares at the moment, this also involves restarting some of the servers.

Update
2 Mar 22:03:03

We've finished our work on the mail servers for this evening. The load on the mail servers is normal at the moment, but then it is out of hours for most customers.

We have some plans for the morning if the problems arise again.

Update
3 Mar 08:42:15

We will be doing some work in the datacentre this morning regarding email services. This involves physically moving some of the servers and assigning others to new switches.

Update
3 Mar 11:02:37

This is getting silly, we do understand. Please bear with us.

Update
3 Mar 13:00:00

Just to explain where we are. There is no simple obvious cause of the email issues. They are fine over night and not bad in the evening, but the change between "a tad slow" up until last week, and no "totally damn unusable" this week is far too sudden and makes now sense. So we have been looking at the cause. We are not ruling anything out. We think it an issue with access to the disk storage.

Right now we are reviewing all of the switch config and cables involved incase the slowness is down to some network problems that are somehow not showing on port stats.

We are also checking every thing we can, and have tried rebooting everything we can to try and address this.

We do appreciate how frustrating it is.

Update
3 Mar 13:04:53

There is some weird shit happening here - a simple move to eliminate one of the network switches involved, just in case, has taken hours not minutes and is again meaning rebooting all the mail stuff. We still have at least two more things to try if this is not helping.

Update
3 Mar 13:24:53

It looks like disk server access somehow, and we have been eliminiating some of the switch infrastructure. It does not seem to have helped a lot. So we are working on other possibilities.

Update
3 Mar 14:06:22

Email delivery does not appear to be anywhere near as badly affected, so setting email to forward a copy out, e.g. to a hotmail account (yuck), should work as a work around. The control pages on clueless allow this.

Update
3 Mar 14:15:14

We are still working on this. Still some things we can try. It is like tryign to find a needle in a haystack or something. Crazy.

Update
3 Mar 15:05:38

We are wondering if this is a case of load hitting a point where it all gets slow, and because it is slow, there is a back log, and because there is a back lo g there is lots of load. We can't see an actual fault or failure and attempts to find and eliminate possible causes have not worked. We are now making changes to try and make operations more efficient in any way we can. Longer term, very soon, we will have a very much faster disk server for this on order.

Update
3 Mar 15:24:32

For a lot of people mail has been working, just horribly slow. It is improving a little now, and we are not sure if this is just less usage or something we have done. We have another change to try shortly as we are working out where bottlenecks are happening.

Update
3 Mar 16:10:06

OK, we have made changes that seems to have made a significant improvement and things should start to catch up now.

Update
3 Mar 16:27:49

Just to add, you may want to restart your client. You may also have a delay to start with if you have a large inbox. But once that is over it should be OK again.

Update
3 Mar 17:30:55

FYI, email delivery is still catching up so still a bit slow as we start the evening.

Update
4 Mar 08:50:42

So far this morning email is very quick - 9am will be the clue though...

Update
4 Mar 11:23:08

Email is a bit slow still but working.

Closed
4 Mar 11:23:08

Short term we have made changes which effect the efficiency of mail handling and avoid some file locking time-outs which caused re-indexing of inboxes which caused load which caused file locking time-outs which caused re-indexing of inboxes which caused load which.... you get the idea.

Long term, a new disk server is on order which will be a lot faster.

Blame AAISP
Updated 4 Mar 11:23:08


 
Archive

Broadband Email Ethernet General Maidenhead Mobile VoIP

Links