Andrews & Arnold Incidents Feed

[PEW] Broadband: At Risk, Transit Planned Work

2012-05-16T11:08:08+01:00

One of our transit providers (ie, one of the links we have to the rest of the Internet) has planned work on Saturday morning.

Our link to them will be down for about 30 minutes between 0900 and 1200 on 14 April.

We don't expect any customer impact as we have multiple transit links.

[Minor] Maidenhead Data Centre: Intermittent Packet Loss affecting Maidenhead Hosted Services (Open)

2012-05-16T10:59:36+01:00

We are are seeing intermittent packet loss affecting services in Maidenhead - this will be most noticeable on VoIP calls where they may be short periods of silence during a call.

The loss is sporadic, it lasts a few seconds and there are long periods of time where the loss is not present, as yet there is no clear pattern to pinpoint the cause -a tricky one to track down!

We are investigating and working with the Datacentre to fix this.

[Minor] Email: Outgoing Email Problems (smtp.aa.net.uk)

2012-05-16T10:11:22+01:00

One of the servers that is part of our out going mail relay (smtp.aa.net.uk or smtp.aaisp.net.uk) has just suffered from a disk problem.

We've taken this server out of the pool, and will take 10 minutes for the DNS to update.

If you have problems sending email, please try again in 10 minutes. If still not working, then it maybe that your email program has remembered the old server and may need to be restarted.

Update 15 May 2012 13:33:07

The server is rebuilding it's RAID. We'll leave it out of service for the time being whilst we investigate.

Email in the queue is being processed, no email has been lost.

Email service should now be OK.

[Minor] Broadband: Packet loss on HOVE Exchange RED5-L-FAR RAS RAS (Open)

2012-05-16T09:58:50+01:00

Packet loss between 20:00 - 00:00 On all 21Cn lines that are on both the HOVE exchange and the RED5-L-FAR RAS.

We are reporting this to BT now.

Update 14 May 2012 10:56:57

BTO will investigate this tonight.

Update 16 May 2012 09:58:50

BT do not have any real update, so we have escalated the issue.

Update 16 May 2012 11:58:35

Looks like an MSAN facing port is over utalized and is causing the packet loss.
BT are contacting capacity management on this now.

[Minor] Broadband: Packet loss on the 21CN BATTERSEA Exchange (Open)

2012-05-16T09:58:34+01:00

Packet loss between 20:00 - 23:00 on all 21Cn lines connected to the BATTERSEA Exchange.

We are reportig to BT at the moment.

Update 14 May 2012 10:59:22

BTO will investigate this tonight

Update 16 May 2012 09:58:34

BT do not have any real update, so we have escalated the issue.

Update 16 May 2012 11:58:39

Looks like an MSAN facing port is over utalized and is causing the packet loss.
BT are contacting capacity management on this now.

[Minor] Broadband: Lines dropped: 100% 20CN-REGION-MILTONKEYNES and 100% 21CN-REGION-MQD

2012-05-15T09:12:25+01:00

Lines: 100% 20CN-REGION-MILTONKEYNES and 100% 21CN-REGION-MQD dropped at 2012-05-15 00:14:56
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 15 May 2012 09:12:25

This was as a result of a BT PEW

[Minor] Broadband: Lines dropped: 80% 20CN-REGION-MILTONKEYNES and 45% 21CN-REGION-MQD and 100% 20CN-BRAS-ERX20-MILTONKEYNES5 and 84% 20CN-BRAS-ESR6-MILTONKEYNES3

2012-05-15T09:12:11+01:00

Lines: 80% 20CN-REGION-MILTONKEYNES and 45% 21CN-REGION-MQD and 100% 20CN-BRAS-ERX20-MILTONKEYNES5 and 84% 20CN-BRAS-ESR6-MILTONKEYNES3 dropped at 2012-05-15 00:14:56
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 15 May 2012 09:12:11

This was as a result of a BT PEW

[Minor] Broadband: Lines dropped: 100% 20CN-REGION-EALING

2012-05-14T12:07:05+01:00

Lines: 100% 20CN-REGION-EALING dropped at 2012-05-12 02:01:43
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 14 May 2012 12:07:05

We have not been given an explaination for this outage.

[Minor] Broadband: Lines dropped: 100% 20CN-REGION-EALING

2012-05-14T12:06:56+01:00

Lines: 100% 20CN-REGION-EALING dropped at 2012-05-12 11:12:02
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 12 May 2012 11:14:08 Lines: 100% 20CN-REGION-EALING dropped again at 2012-05-12 11:12:02.

Update 14 May 2012 12:06:56

We have not been given an explaination for this outage.

[Minor] Broadband: Lines dropped: 100% 20CN-REGION-MANCHESTER and 100% 21CN-REGION-MR-DH

2012-05-12T10:51:58+01:00

Lines: 100% 20CN-REGION-MANCHESTER and 100% 21CN-REGION-MR-DH dropped at 2012-05-12 08:06:58
We have advised BT
This is likely to have affected multiple internet providers using BT

[MSO] Broadband: BT RADIUS Failure

2012-05-12T10:51:43+01:00

At 08:06:58 all 20CN and 21CN lines routed via Manchester dropped and at the same time we stopped seeing any platform RADIUS requests from BT.

This suggests a major routing issue within BT. It means that nobody can re-connect any broadband line.

We are trying to get an answer out of BT on this.

Update 12 May 2012 08:58:33

BT have a string of open issues covering Manchester area. No ETA yet.

Update 12 May 2012 09:08:05

We have confirmation that this is not just affecting us (thanks Thomas). BT are investigating.

Update 12 May 2012 09:42:39

Whilst BT are being a tad inconsistent in telling various ISPs what is happening, we have heard from another ISP that "Engineer is onsite at Manchester with a spare card. Changeout is about to commence with the assistance of vendor support. Next update will be at 10:20"

Update 12 May 2012 10:20:20

BT are not confirming the statement that a card is to be replaced, and in fact appear to be totally ignoring me now.

Update 12 May 2012 10:42:58

Looks to be working.

Manchester lines reconnecting

Platform RADIUS working

[Minor] Broadband: Lines dropped: 100% 20CN-REGION-EALING and 93% 21CN-REGION-21CN-BRAS-RED10-L-NWS and 94% 21CN-REGION-21CN-BRAS-RED11-L-NWS and 99% 21CN-REGION-L-NWS

2012-05-12T10:47:40+01:00

Lines: 100% 20CN-REGION-EALING and 93% 21CN-REGION-21CN-BRAS-RED10-L-NWS and 94% 21CN-REGION-21CN-BRAS-RED11-L-NWS and 99% 21CN-REGION-L-NWS dropped at 2012-05-12 02:04:16
We have advised BT
This is likely to have affected multiple internet providers using BT

[Minor] Broadband: Lines dropped: 100% 20CN-REGION-MANCHESTER

2012-05-12T10:47:03+01:00

Lines: 100% 20CN-REGION-MANCHESTER dropped at 2012-05-12 02:41:50
We have advised BT
This is likely to have affected multiple internet providers using BT

[MSO] Broadband: ADSL blip

2012-05-11T09:09:57+01:00

Lines on our "b.gormless" LNS dropped. We're investigating. Lines are coming back.

Update 10 May 2012 11:57:04

It looks like BT. Still investigating.

Update 10 May 2012 11:59:33

It looks like BT's link to b.gormless has gone down. We've taken b.gormless out of service.

Update 10 May 2012 12:00:24

BT fibre link is down - lines are moving to the backup LNS automatically.

Update 10 May 2012 12:35:49

BT have raised a fault and are investigating their end.
Update due by 13:15

Update 10 May 2012 13:00:50

Most lines swicthed very quickly to the alternative LNS automatically.

The status of lines on the control pages too some time to catch up on the RADIUS accounting, but this has now happened.

Graphs may have gaps which will be repaired on the over night status.

Update 10 May 2012 19:55:22

Although broadband lines look fine this link is still down, meaning we have lost resilience on links to BT.

We are still chasing.

Update 10 May 2012 20:11:47

BT are arranging an engineer to go to the Datacentre.

Update 10 May 2012 22:25:45

Still not heard anything more.
Chasing again.

Update 11 May 2012 09:09:57

Came back around 00:09.
It appears that this was a problem with a card BT side.
Need to confirm though.

[PEW] Broadband: 21CN Birmingham BT PEW

2012-05-10T09:20:05+01:00

BT have informed us that they have arranged planned engineering works in the birmingham area.

This will affect all 21CN lines on -BM-TH RASs.

Start: 03/04/12 00:00

Finish: 03/04/12 06:00

Update 10 May 2012 09:20:05

PEW finished as planned.

[PEW] Broadband: 21CN Colindale BT PEW

2012-05-10T09:19:58+01:00

BT have informed us that they have arranged planned engineering works in the colindale area.

This will affect all 21Cn lines on -L-NWS RASs.

Start: 12/04/12 00:00

Finish: 12/04/12 06:00

Update 10 May 2012 09:19:58

PEW finished as planned.

[PEW] Broadband: 21CN Stepney Green BT PEW

2012-05-10T09:19:51+01:00

BT have informed us that they have arranged planned engineering works in the stepney green area.

This will affect all 21Cn lines on -L-STE RASs.

Start: 11/04/12 00:00

Finish: 11/04/12 06:00

Update 28 Mar 2012 14:52:30

Update from BT, this is likely to affect our main BT link. As such this will probably cause a drop for all of our lines.

Update 10 May 2012 09:19:51

PEW finished as planned.

[PEW] Broadband: 21CN Faraday BT PEW

2012-05-10T09:19:43+01:00

BT have informed us that they have arranged planned engineering works in the faraday area.

This will affect all 21CN lines on -L-FAR RASs.

Start: 05/04/12 00:00

Finish: 05/04/12 06:00

Update 10 May 2012 09:19:43

PEW finished as planned.

[PEW] Broadband: 21CN Manchester BT PEW

2012-05-10T09:19:35+01:00

BT have informed us that they have arranged planned engineering works in the manchester area.

This will affect all 21CN lines on -MR-DH RASs.

Start: 04/04/12 00:00

Finish: 04/04/12 06:00

Update 10 May 2012 09:19:35

PEW finished as planned.

[Minor] Broadband: 21CN Packet loss on several GI-B RASs

2012-05-10T09:14:48+01:00

Packet loss between 5pm and 9pm on several GI-B RASs. Affected:

RED3-GI-B
RED4-GI-B
RED5-GI-B
RED6-GI-B
RED7-GI-B
RED8-GI-B

Update 2 May 2012 15:35:57

BT are investigating and will attempt to fix the issue today / overnight.
Will check for an update tomorrow.

Update 3 May 2012 09:42:00

BT had seen an issue but by the time they were investigating it the issue cleared. They will continue to investigate and chase this up.

Update 4 May 2012 10:15:53

Looks like packet loss is cleared, chasing BT for an update.

Update 4 May 2012 10:27:01

Traffic was moved to a different interface whilst BT was testing for the fault. Once testing was completed the traffic was moved back to this interface.

We did not see any loss last night, but since there is no overall resolution as of yet, we will need to keep an eye on the RASs.

We will check again Monday, If you do have problems on one of these RASs please do email support@aa.net.uk.

Update 10 May 2012 09:14:48

Looks like the issue is resolved, if you notice any problems please send us an email to support@aa.net.uk.

Thank you.

[Minor] Broadband: Lines dropped: 100% 20CN-REGION-ILFORD and 100% 21CN-REGION-L-STE

2012-05-10T09:11:32+01:00

Lines: 100% 20CN-REGION-ILFORD and 100% 21CN-REGION-L-STE dropped at 2012-05-10 01:09:47
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 10 May 2012 09:11:32

This was casued by BT planned works.

[Minor] Broadband: Lines dropped: 100% 21CN-REGION-MQD

2012-05-10T09:09:17+01:00

Lines: 100% 21CN-REGION-MQD dropped at 2012-05-07 18:05:43
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 10 May 2012 09:09:17

BT have been unable to give us a reason for this outage.

[Minor] Broadband: RADIUS accounting issues

2012-05-07T18:12:33+01:00

Once again, one of the LNSs is being tad odd with RADIUS accounting, and giving people free internet access. Obviously this has to be fixed :-)

Seriously though, this is an annoying bug in the LNS code, which we have finally tracked down. We are doing routine LNS rolling upgrades during the week which will resolve this. It is in fact related to the fact one LNS stopped showing graphs for a day recently!

The control pages allow you to pick a preferred time of night for LNS upgrade to re-connect your line (a few seconds of outage) as we know some people work (and play) odd hours. Please talk to support if you need any help on this.

Sorry for any concern this has caused. LNS code is tested before release on test systems but this bug was rather obscure and did not always happen, and then took rather a long time to happen, hence not being picked up until we rolled it out on the live LNSs. The fact we run multiple LNSs and gradually introduce changes also means this issue is not affecting everyone, and problems can be quickly averted if they were more serious.

An LNS switch or PPP kill on your line does not cause a re-sync, and is normally only a few seconds. If you find issues with these, please do contact support as this can depend very much on your make of router.

Thank you for your understanding.

Note, there are some router upgrades as well this week but these should be seamless.

[Minor] Broadband: Lines dropped: 66% 21CN-REGION-MQD and 80% 21CN-BRAS-RED1-MQD and 100% 21CN-BRAS-RED4-MQD and 100% 21CN-BRAS-RED7-MQD and 100% 21CN-BRAS-RED8-MQD

2012-05-07T18:10:39+01:00

Lines: 66% 21CN-REGION-MQD and 80% 21CN-BRAS-RED1-MQD and 100% 21CN-BRAS-RED4-MQD and 100% 21CN-BRAS-RED7-MQD and 100% 21CN-BRAS-RED8-MQD dropped at 2012-05-07 18:05:44
We have advised BT
This is likely to have affected multiple internet providers using BT

[MSO] Maidenhead Data Centre: Web & email storage problem

2012-05-04T19:16:08+01:00

There's a problem with web and email disk storage in Maidenhead.

We're investigating.

Update 4 May 2012 19:16:08

Now back. We're investigating the cause of this.

[MSO] Email: Mail and web servers not available

2012-05-04T19:15:52+01:00

Our main mail servers are not available due to an issue with the disk storage. This is being investigated now.

This is affecting our web sites too

Update 4 May 2012 19:13:34

Service restored - sorry for any inconvenience.

Update 4 May 2012 19:15:52

Now back. We're investigating the cause of this.

[Info] Broadband: Open DNS relays

2012-05-04T13:47:38+01:00

There are a number of customers with open DNS relays on their networks. Some of these appear to be being exploited as part of a DDOS with, we think, spoofed source DNS lookups.

As this is undoubtedly causing problems for whoever is being attacked, as well as causing excessive broadband usage for affected customers, we are going to attempt to make a list of these relays so that the affected customers can be identified.

You may see DNS lookup attempts from one of our IPs in your firewall logs.

[Minor] Email: "Sender verify failed" on outbound email

2012-05-04T13:16:38+01:00

Our outbound mail servers are occasionally refusing to send email with the error "Sender verify failed". This doesn't appear to be affecting many customers. We are investigating.

Update 1 May 2012 09:56:46

We're making changes to try and identify/resolve the underlying problem.

Still doesn't seem to be affecting many customers.

Update 2 May 2012 14:39:46

We've tracked this down to a peculiar DNS problem on one of our resolvers in the Maidenhead datacentre.

We will carry on monitoring this, but it should be 'resolved'.

[Minor] Email: Emergency Maintenance on 'D' POP3/IMAP server

2012-05-01T09:55:16+01:00

The 'D' email server will have one of its disks changed on Friday afternoon Monday at 5pm and will require a reboot. We expect this to take around 10 minutes.

Update 27 Apr 2012 14:36:25

This is being moved to Monday

Update 30 Apr 2012 17:05:46

This work is about to start.

Update 30 Apr 2012 17:11:54

The server is back online.

Update 30 Apr 2012 17:59:53

And is off again. Working on it.

Update 30 Apr 2012 18:16:13

After changing the disk the server was rebooted a second time and has failed to boot up. Datacentre staff have been tasked to investigate.

Update 30 Apr 2012 19:06:06

The Server is booting up now.

Update 30 Apr 2012 19:32:24

The server is rebuilding its RAID array, but is up and functioning. We're sorry for the inconvenience this caused.

Update 30 Apr 2012 22:59:20

Rebuild complete - it should be fine now.

[Minor] Broadband: RADIUS accounting issue

2012-04-27T17:47:44+01:00

It seems we have a slight RADIUS accounting issue which we are tracking down at present.

The result is (a) people not being charged for usage and (b) that you may get a ppp kill on the hour at some point in the next 24 hours.

It is not affecting everyone and looks specifically to one LNS at present. We are looking in to it.

Update 27 Apr 2012 17:47:44

We are slightly at a loss on thsi one - we have got lines back on the accounting and will be switching these to the "A" LNS over night. Sorry for any inconvenience.

[Minor] General: A&A Bracknell Office Powercut

2012-04-27T14:35:41+01:00

There was a power cut in the area affecting the local area around the Bracknell office.

Not customer affecting directly, but does mean that there will be problems contacting staff.

Update 27 Apr 2012 13:43:06

The lights are back on, computers booting up. Phones are back online...

[Minor] Broadband: Lines dropped: 86% 21CN-REGION-L-STE and 100% 21CN-BRAS-RED2-L-STE and 80% 21CN-BRAS-RED4-L-STE and 96% 21CN-BRAS-RED5-L-STE and 100% 21CN-BRAS-RED6-L-STE and 100% 21CN-BRAS-RED7-L-STE and 100% 21CN-B

2012-04-27T10:15:49+01:00

Lines: 86% 21CN-REGION-L-STE and 100% 21CN-BRAS-RED2-L-STE and 80% 21CN-BRAS-RED4-L-STE and 96% 21CN-BRAS-RED5-L-STE and 100% 21CN-BRAS-RED6-L-STE and 100% 21CN-BRAS-RED7-L-STE and 100% 21CN-BRAS-RED8-L-STE dropped at 2012-04-26 02:13:34
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 27 Apr 2012 10:15:49

BT have been unable to give us an explaination for this outage.

[Minor] Broadband: Lines dropped: 100% 21CN-REGION-L-STE

2012-04-27T10:15:44+01:00

Lines: 100% 21CN-REGION-L-STE dropped at 2012-04-26 02:13:35
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 27 Apr 2012 10:15:44

BT have been unable to give us an explaination for this outage.

[Minor] Broadband: Some customers not seeing CQM Graphs

2012-04-26T18:53:00+01:00

Customers on the 'B' LNS (B.Gormless), will be seeing white/blank graphs for their lines. This is being looked in to.

Update 25 Apr 2012 09:24:44

Whilst this is looked in to Customers can re-login to us and wll be connected to a different LNS, on which graphs will be shown.

Update 26 Apr 2012 18:53:00

We are upgrading LNSs over the next few days over night as normal maintence.

[Info] Broadband: Domain name management

2012-04-26T12:11:49+01:00

We have made changes to the domain control pages today.

1. The name servers are now shown as NS records in the zone rather than as a list at the top of the page. You can edit these at the top level even if we are not serving your zone as these are used to record the delegation details from the registry.

2. You can now add DS records to the zone. You can edit these at the top level even if we are not serving your zone as these are used to record the delegation details from the registry.

3. If the domain is a UK domain managed by nominet then you have new buttons. The "refresh" button will get the current contact details, NS, and DS records from the registry and update the domain details. The "send" button will send the details to the registry with immediate effect. This means customers can now change delegation NS and DS records as well as contact details on UK domains managed by nominet from the control pages.

Please ensure that your contact details are valid. As an individual you have the option to select "withheld" to not show these in the normal whois records, but you must have correct contact details otherwise you could lose your domain.

Any problems, please contact support for help.

[Minor] Email: 'D' POP3/IMAP Server High Load

2012-04-25T16:10:04+01:00

One of our POP3/IMAP servers is experiences high load, engineers are looking in to it, but some customers may be seeing error messages in their email programs.

[Info] Broadband: New 'Per-IP' Stats for Broadband Usage

2012-04-24T13:35:38+01:00

We now have per-IP usage stats available for broadband customers. You can view usage based on your IPv6 and legacy IPv4 addresses. The data is broken down in to daily and hourly totals for upload and download.

The usages is based on 'sampled' data, so will not be 100% accurate, but should give a good idea as to how your usage is spread across your machines.

The data is updated every minute or so.

You can find this from the Usage page on the Control Pages. We welcome feedback.

(Per-IP usage is not being collected for customers using L2TP, SIMs, Ethernet or the 'test LNS'.)

Update 30 Apr 2012 12:51:06

You can do reverse DNS lookups on the per IP stats now.

[Minor] Broadband: Possible routing issues

2012-04-24T07:08:44+01:00

It looks like something happened at around 00:20 this morning interrupting services to LINX.

This should not affect services as everything will simply take alternative routes.

We don't know what the issue is at this stage.

Update 24 Apr 2012 07:06:44 All sorted before 4am

[PEW] Maidenhead Data Centre: IPv6 changes (Open)

2012-04-21T13:57:19+01:00

Apologies for some minor distruption with IPv6 today in relation to services we run in the Maidenhead Data Centre. There have been a few minutes during today where routing has not been quite right for IPv6 for some of the services.

We have renumbered our office on to a separate /48 to improve native IPv6 routing. (A /48 is visible to more of the Internet).

Over the next few days or weeks we will be renumbering services in the data centre from 2001:8b0::/48 to 2001:8b0:1::/48 addresses. Thanks to a customer (Lawrence) for allowing us to change is allocation to free that up.

If you want to trust an IPv6 as being from A&A infrastructure you should be able to trust 2001:8b0::/47 as this includes the /48 was have in London and the /48 we have in Maidenhead and our offices.

At present there may be a handful of machines which are not technically A&A infrastructure within this space that will be renumbered this week as well. A&A infrastructure should have reverse DNS under aa.net.uk as well.

We aim to parallel run the old and new IPv6 during DNS changes so that there are no issues with any services. If anyone has any issues please contact support.

We are not planning any IPv4 changes.

Note, customers that have IPv6 on fibre from Maidenhead can now request a /48 if you need, routed via your existing /64 link.

[Minor] Maidenhead Data Centre: Routing issue

2012-04-21T12:06:35+01:00

There was a rather odd issue with routing in Maindenhead affecting several of our servers there. We think this may be an issue with a switch, but we have managed to get things working again. There may be some planned maintenance as a result later.

[Minor] General: Status Pages Unreachable

2012-04-19T11:03:36+01:00

This status page was offline this morning due to the datacentre in Amsterdam being unreachable.

The fault was due to a line card in London at a connectivity provider.

[Minor] Broadband: Lines dropped: 100% 21CN-REGION-L-STE

2012-04-18T14:54:44+01:00

Lines: 100% 21CN-REGION-L-STE dropped at 2012-03-14 08:06:42
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 14 Mar 2012 16:53:21

This happened again at 16:46 (Wednesday), Lines came back quickly.

We're contacting BT about it.

Update 14 Mar 2012 21:21:05

BT have confirmed that thay have an open incident regartding this, and BT 'Operate' will investigate.

Update 27 Mar 2012 10:19:27

Proactive are chasing for the cause and resolution; Should have an update soon.

[Info] Maidenhead Data Centre: IPv6 routing improvements

2012-04-18T14:51:28+01:00

We are pleased to confirm the hosting in the Maidenhead Data Centre now has full 1500 byte MTU native IPv6 transit in operation. This improves the MTU to our various servers, to customer servers, and to Ethernet transit customers. If you have any issues, please do contact support, but this should be completely seamless.

We may also be making some changes to the IPv6 addresses for A&A servers in Maidenhead in due course. We'll post more details when that is all decided. This should not have any impact as we will, of course, overlap assignments and make necessary changes to DNS records.