We try and ensure this page is updated as soon as possible with incidents as they happen. Our support number is 033 33 400999. Live discussion of issues as they happen is usually available on irc.
If you believe there is a major issue that is not shown here please text 01344400999 with a message starting MSO (this will wake staff if necessary).
We are pleased to confirm that the control pages now allow interactive BT fault diagnostics on all BT based lines (20CN and 21CN).
Until now this was only available on 20CN. We have now been able to enable this on 21CN lines and improve the reporting provided.
The disagnostics take you through simple questions and carry our line tests on the line to try and determine if there is a problem. Some of the wording can be confusing as it is mainly aimed at our use, but we have made the system available to customers directly.
The questions asked can often help eliminate the likely causes of problems and help us pin down the cause of the fault if there is one. The results also help BT resolve the issue when we report the fault in to them.
Simply select your line on the control pages and select the KBD diagnostics link.
If you do get crazy results, do let us know by email and we'll discuss with BT.
All lines on 20CN-BRAS-ESR10-FARADAY3 dropped at 2010-08-29 10:00:31 We have advised BT This is likely to have affected multiple internet providers using BT
Lines have recovered and we are chasing BT for an explanation for this outage. So far we have no news, but we will still tryto get the explanation.
This appears to have been resolved. We have had no word from BT reference the cause or the resolution, we will chase and try and get the information.
BT have advised us of engineering work planned to start end of May and finish in September which will affect customers who are connected to the following BRAS:
Milton Keynes: 24th MAY
Birmingham: 7th JUNE
Faraday: 14th JUNE
Stephney Green: 28th JUNE
Manchester: 12th July
Leeds: 26th JULY
Colindale 9th August
Southbank: 23rd AUGUST
Wolverhampton: 6th September
BT have advised that during the planned work customers can expect up to 3 hours downtime between midnight and 6am
Planned work notices will also be displayed on the control pages for the individual lines affected.
This work is needed to enable delivery of additional host link options.
Note that from previous experience, when this happens the outage is typically nearer one hour, but can be longer. Also, work on Stepney Green caused all customers to be affected not just that area. Also, last time work on Birmingham had a similar effect but apparently should not have. We'll monitor the situation carefully.
This is a Planned Engineer Work notice from BE regarding some network upgrades:
As some of you may be aware there is currently a project running to support PPPoA on the O2 network for the wholesale channel. This will allow customers to connect using the same encapsulation, VPI/VCI and MTU settings as employed by BT Wholesale, which will pave the way for seamless customer migrations.
We are now starting the deployment stage of the project and for this O2 will be making some changes to our MPLS configuration on the core network.
This work is being carried out on core switches on the O2 broadband network, which will affect exchanges in geographical regions. The following regions will be affected on the following dates (the inclusion of Manchester on 2 separate dates is correct):
08/09/2010:
Leicester
Oxford
Coventry
Christchurch
Time slot:
00:00 – 06:00
Tasks:
Configuring fast path redundancy on core switches.
This means that in the event of a core network link failure recovery via an alternative path will happen fast enough that PPP sessions will not drop and need to be re-established. This enhancement is being added to the existing PPPoE platform as well as the new PPPoA platform.
Service impact:
The configuration changes will disrupt traffic for a few seconds, this may be sufficient for sessions to drop and have to re-establish.
Further details regarding the PPPoA project will be announced when we approach the end of the project; we are working towards a go live date towards the end of October.
We are likely to update the primary of many of our core routers as well, but this should be pretty seamless.
Update
28 Aug 17:04:25
Moving lines A to B now.
Update
28 Aug 17:14:36
All lines moved over. As usual the graphs will be split for today but over night the archives will stitch them together. As usually each line had a simple ppp reconnect lasting typically a few seconds. Thank you for your patience.
Update
28 Aug 17:21:16
Core router updates also done.
Update
28 Aug 17:41:51
Some reports of the core router updates distrupting IPv6 for a minute. We are not sure why this happened.
At just before 17:00 today a minor routing announcement change was made. This was a very minor change and was not run as a proper planned work because it was so minor, or so we thought. Even so the change was not done in the middle of the day.
At 17:36 we were advised by MSO text that customers on Ethernet links were seeing problems. By 17:44 we had identified the cause of the problem and resolved it.
Technical details: The problem is in fact related to exhaustion of IPv4 space. We are involved in arrangements to swap IP blocks around between ISPs to free up some additional space, and this meant fragmenting one of our larger IP blocks. In principle this is simple, we announce the smaller blocks and drop the larger block announcement later. Unfortunately one of the smaller blocks is routed to Ethernet connected customers and was included in the list annonced at Telehouse in London.
The impact was that some of the Internet was not accessable to Ethernet customers, depending on routing. This included all of our ADSL lines and many services peering on LINX in London which could not see Ethernet lines. Unfortunately, as a partial failure it was not picked up by any of our automated monitoring which could still see Ethernet customers.
This was human error, and resolved within 8 minutes of being reported on a bank holiday weekend out of hours.
We will consider ways to manage this better - it is a trade off as we could have put more in automated control which would have stopped this specific error but could lead to more radical errors being possible with a lot less effort. We're reviewing procedures for checking changes like this before hand in future.
All lines on 21CN-BRAS-RED3-GI-B dropped at 2010-08-26 00:19:53 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 21CN-BRAS-RED2-L-WAT dropped at 2010-08-27 03:04:26 We have advised BT This is likely to have affected multiple internet providers using BT
We don't know why yet, but every BT 21CN line on L-WAT was off line for over 6 hours during the night. We'll update this when we get an explanation from BT.
This is likely to have affected many ISPs, not just us.
This is a Planned Engineer Work notice from BE regarding some network upgrades:
As some of you may be aware there is currently a project running to support PPPoA on the O2 network for the wholesale channel. This will allow customers to connect using the same encapsulation, VPI/VCI and MTU settings as employed by BT Wholesale, which will pave the way for seamless customer migrations.
We are now starting the deployment stage of the project and for this O2 will be making some changes to our MPLS configuration on the core network.
This work is being carried out on core switches on the O2 broadband network, which will affect exchanges in geographical regions. The following regions will be affected on the following dates (the inclusion of Manchester on 2 separate dates is correct):
09/09/2010:
Reading
London
Edinburgh
Glasgow
Preston
Bolton
Dundee
Aberdeen
York
Middlesbrough
Time slot:
00:00 – 06:00
Tasks:
Configuring fast path redundancy on core switches.
This means that in the event of a core network link failure recovery via an alternative path will happen fast enough that PPP sessions will not drop and need to be re-established. This enhancement is being added to the existing PPPoE platform as well as the new PPPoA platform.
Service impact:
The configuration changes will disrupt traffic for a few seconds, this may be sufficient for sessions to drop and have to re-establish.
Further details regarding the PPPoA project will be announced when we approach the end of the project; we are working towards a go live date towards the end of October.
All lines on 20CN-BRAS-ESR11-BIRMINGHAM6 dropped at 2010-08-24 04:57:45 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 21CN-BRAS-RED4-L-WAT dropped at 2010-08-23 16:42:04 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR10-KINGSTON5 dropped at 2010-08-21 00:09:14 We have advised BT This is likely to have affected multiple internet providers using BT
This is a Planned Engineer Work notice from BE regarding some network upgrades:
As some of you may be aware there is currently a project running to support PPPoA on the O2 network for the wholesale channel. This will allow customers to connect using the same encapsulation, VPI/VCI and MTU settings as employed by BT Wholesale, which will pave the way for seamless customer migrations.
We are now starting the deployment stage of the project and for this O2 will be making some changes to our MPLS configuration on the core network.
This work is being carried out on core switches on the O2 broadband network, which will affect exchanges in geographical regions. The following regions will be affected on the following dates (the inclusion of Manchester on 2 separate dates is correct):
13/09/2010:
Newcastle
Low Moor
Leeds
Batley
Warrington
Liverpool
Ellesmere Port
Manchester
Portadown
Belfast
Time slot:
00:00 – 06:00
Tasks:
Configuring fast path redundancy on core switches.
This means that in the event of a core network link failure recovery via an alternative path will happen fast enough that PPP sessions will not drop and need to be re-established. This enhancement is being added to the existing PPPoE platform as well as the new PPPoA platform.
Service impact:
The configuration changes will disrupt traffic for a few seconds, this may be sufficient for sessions to drop and have to re-establish.
Further details regarding the PPPoA project will be announced when we approach the end of the project; we are working towards a go live date towards the end of October.
BT have advised us of engineering work planned to start end of May and finish in September which will affect customers who are connected to the following BRAS:
Milton Keynes: 24th MAY
Birmingham: 7th JUNE
Faraday: 14th JUNE
Stephney Green: 28th JUNE
Manchester: 12th July
Leeds: 26th JULY
Colindale 9th August
Southbank: 23rd AUGUST
Wolverhampton: 6th September
BT have advised that during the planned work customers can expect up to 3 hours downtime between midnight and 6am
Planned work notices will also be displayed on the control pages for the individual lines affected.
This work is needed to enable delivery of additional host link options.
Note that from previous experience, when this happens the outage is typically nearer one hour, but can be longer. Also, work on Stepney Green caused all customers to be affected not just that area. Also, last time work on Birmingham had a similar effect but apparently should not have. We'll monitor the situation carefully.
This is a Planned Engineer Work notice from BE regarding some network upgrades:
As some of you may be aware there is currently a project running to support PPPoA on the O2 network for the wholesale channel. This will allow customers to connect using the same encapsulation, VPI/VCI and MTU settings as employed by BT Wholesale, which will pave the way for seamless customer migrations.
We are now starting the deployment stage of the project and for this O2 will be making some changes to our MPLS configuration on the core network.
This work is being carried out on core switches on the O2 broadband network, which will affect exchanges in geographical regions. The following regions will be affected on the following dates (the inclusion of Manchester on 2 separate dates is correct):
14/09/2010:
Manchester
Sheffield
Stoke
Birmingham
Northampton
Derby
Nottingham
Streetly
Tipton
Birmingham South
Time slot:
00:00 – 06:00
Tasks:
Configuring fast path redundancy on core switches.
This means that in the event of a core network link failure recovery via an alternative path will happen fast enough that PPP sessions will not drop and need to be re-established. This enhancement is being added to the existing PPPoE platform as well as the new PPPoA platform.
Service impact:
The configuration changes will disrupt traffic for a few seconds, this may be sufficient for sessions to drop and have to re-establish.
Further details regarding the PPPoA project will be announced when we approach the end of the project; we are working towards a go live date towards the end of October.
We will be moving 'Clueless' to new hardware in a different datacentre.
Clueless is our core database server, control page server, primary radius server and runs other tasks.
Customer Impact: During the move the control pages will unavailable. Other key services that Clueless performs will be handled by secondary/backup servers.
We do not expect broadband services to be affected by this work.
The work is being carried out on a week day as it will be easier and quicker for staff to respond to any problems that may arise.
Sorry, This has now been moved to Wednesday 18th August
Update
18 Aug 07:30:55
We expect to start this work before 8am, with the web control pages on clueless being taken off line. This is to stop updates to the database which is then moved to the new server.
Update
18 Aug 07:47:58
This work in in progress.
Update
18 Aug 08:58:41
This is still being worked on at the moment. Most things are moved across, some elements of the control pages are not yet working.
Update
18 Aug 09:26:56
Currently:
Line tests
Fault updates
Historical Graphs
RADIUS logs
are items on the new server which are not yet working. There may be others. -this is not affecting customer service, and they should be working again later this morning.
If you spot anything you think we may have missed, please do let us know on IRC.
Update
18 Aug 13:52:05
Current outstanding issues:
We're still working on the BT XML interface - meaning we have problems with XML communication to and from BT with regards to orders, faults and lines tests.
Historical CQM graphs are still in the process of being copied over
Update
18 Aug 16:27:12
Our BT XML systems are now operational, we are still in the progress of moving over old CQM graphs, this should be completed later this evening.
Thank you for bearing with us today.
Started
18 Aug 07:00:00 by AAISP Staff
Previously expected
18 Aug 10:00:00 (Last Estimated Resolution Time from AAISP)
Closed 18 Aug 19:51:33
Well done to all of the team working on this today - not many unexpected snags but the couple we did find were sorted quickly and services not interrupted. Adrian.
Sorry, someone is going to site now and will be a couple of minutes
Update
17 Aug 17:56:15
The call server did a silly when we made a minor change ready for work later in the week. As a result we have moved to new hardware. Someone is on site trying to work out WTF happened anyway, but that may wait until morning as we have managed to switch to the main call server on the new hardware now.
Update
17 Aug 18:35:37
As usual, with the new system for handling calls, any calls in progress and not recorded would have continued with no problems during the issues and would then not have been charged.
Because of the move of clueless, and due to some staff off sick and on leave, we expect to be busy Wednesday morning. We'll try and ensure calls, etc, are all handled as quickly as possible. However, some key support staff will be concentrating on babysitting the new server in the morning to ensure all services are running smoothly for everyone.
Having said that we do not anticipate any major issues.
Some problems with calls and registrations this morning. Some of the work preparing for the planned change at the weekend has had unexpected side effects and has been reversed.
The impact is on setting up calls. Calls in progress should not be affected and not billed either.
We will be doing more testing on the new call servers during this week.
We hope to be able to move over to the new call server pretty seamlessly, probably early one morning this week when we are happy that we have the new servers behaving exactly as we want them.
There is no change to configuration on customer equipment or IP addresses. This is simply a move to new hardware and setting up fallback hardware.
The legacy call server ("A" server, running asterisk) is not being touched.
This is a Planned Engineer Work notice from BE regarding some network upgrades:
As some of you may be aware there is currently a project running to support PPPoA on the O2 network for the wholesale channel. This will allow customers to connect using the same encapsulation, VPI/VCI and MTU settings as employed by BT Wholesale, which will pave the way for seamless customer migrations.
We are now starting the deployment stage of the project and for this O2 will be making some changes to our MPLS configuration on the core network.
This work is being carried out on core switches on the O2 broadband network, which will affect exchanges in geographical regions. The following regions will be affected on the following dates (the inclusion of Manchester on 2 separate dates is correct):
20/09/2010:
Dudley
Exeter
Plymouth
Maidstone
Cardiff
Chelmsford
Peterborough
Milton Keynes
Hemel
Slough
Time slot:
00:00 – 06:00
Tasks:
Configuring fast path redundancy on core switches.
This means that in the event of a core network link failure recovery via an alternative path will happen fast enough that PPP sessions will not drop and need to be re-established. This enhancement is being added to the existing PPPoE platform as well as the new PPPoA platform.
Service impact:
The configuration changes will disrupt traffic for a few seconds, this may be sufficient for sessions to drop and have to re-establish.
Further details regarding the PPPoA project will be announced when we approach the end of the project; we are working towards a go live date towards the end of October.
The problem is with one of our routers in Telehouse.
Closed 16 Aug 09:02:43
The problem appears to be an issue with one port on a router which meant automatic fallback was not working completely. Some traffic was not routing and some was. DSL lines did not drop. We're investigating the cause now.
Further investigation confirms the automatic fallback worked except for traffic to/from LINX which was black holed.
This is a Planned Engineer Work notice from BE regarding some network upgrades:
As some of you may be aware there is currently a project running to support PPPoA on the O2 network for the wholesale channel. This will allow customers to connect using the same encapsulation, VPI/VCI and MTU settings as employed by BT Wholesale, which will pave the way for seamless customer migrations.
We are now starting the deployment stage of the project and for this O2 will be making some changes to our MPLS configuration on the core network.
This work is being carried out on core switches on the O2 broadband network, which will affect exchanges in geographical regions. The following regions will be affected on the following dates (the inclusion of Manchester on 2 separate dates is correct):
21/09/2010:
Cambridge
Harlow
Crawley
Croydon
Swindon
Gloucester
Bristol
Tunbridge Wells
Fareham
Brighton
Southampton
Time slot:
00:00 – 06:00
Tasks:
Configuring fast path redundancy on core switches.
This means that in the event of a core network link failure recovery via an alternative path will happen fast enough that PPP sessions will not drop and need to be re-established. This enhancement is being added to the existing PPPoE platform as well as the new PPPoA platform.
Service impact:
The configuration changes will disrupt traffic for a few seconds, this may be sufficient for sessions to drop and have to re-establish.
Further details regarding the PPPoA project will be announced when we approach the end of the project; we are working towards a go live date towards the end of October.
To increase resilience and reliability of our VoIP service we will be moving C.Speechless on to new hardware.
The changeover should happen very quickly. We expect there to be just a few minutes where calls will not be able to be placed.
We'll update this post as the work is carried out
Further information: We have 2 VoIP servers, the A server and the C server. The C server is our new server. It's the C server that we will be moving. The new hardware is a pair of servers to enable fall-back etc.
We may leave this till the end of the month. Our analysis suggests that by this weekend all BRASs will have cleared anyway and then be fine for around 45 days more afterwards.
All lines on 21CN ST PETERS dropped at 2010-08-12 09:47:52 and 10:19:55
We have advised BT, this is likely to have affected multiple internet providers using BT.
BT IMT22697/10 and ALS 290440
BT advise that over 500 users are down, possibly 850 users. BT should be carrying out a remote switch over which they are hoping will fix the issue. Next update should be just after 11:00.
Vendor support are currently taking logs off MSAN to find route cause of fault and will be performing IPMB switch over to fix fault. Next update will be just after 12:30
All lines on 20CN-BRAS-ESR6-MILTONKEYNES3 dropped at 2010-08-12 05:54:23 We have advised BT This is likely to have affected multiple internet providers using BT
As customers will be aware we have seen a problem with 20CN BRAS resets over the last day or so. It seems BRAS connections resetting one after the other since yesterday morning. The effect for customers on the BRAS was that they lost service for a few seconds and reconnected. Most routers appear to reconnect in around 30 seconds.
Naturally we assumed it was a BT issue and have been hassling them. It looks like that was not the case, so we would like to apologise to BT for hassling them over this for the last day.
On checking our logs in more detail it looks like the issue is that our LNS has been running long enough for a sequence number to wrap around on the LNS/BRAS control messaging and it appears that we are mishandling this. It is the fact we have run for several months without even a software update that has finally highlighted this issue.
We will be doing an LNS s/w upgrade soon to address this. We'll not be logging these resets on the status pages for now.
All lines on 20CN-BRAS-ESR6-BIRMINGHAM5 dropped at 2010-08-12 06:09:51 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX11-MANCHESTER3 dropped at 2010-08-12 06:23:30 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX5-ILFORD3 dropped at 2010-08-12 06:25:25 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX8-BIRMINGHAM2 dropped at 2010-08-12 06:51:56 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR11-BIRMINGHAM6 dropped at 2010-08-12 07:10:15 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX4-READING2 dropped at 2010-08-12 07:13:02 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX12-EDINBURGH3 dropped at 2010-08-12 07:17:40 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX2-BIRMINGHAM2 dropped at 2010-08-12 07:19:57 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR1-FARADAY2 dropped at 2010-08-12 07:55:11 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR2-FARADAY2 dropped at 2010-08-12 08:04:44 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX6-MILTONKEYNES dropped at 2010-08-12 08:13:25 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX22-EDINBURGH4 dropped at 2010-08-12 08:17:22 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX13-BIRMINGHAM3 dropped at 2010-08-12 08:43:26 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR1-MANCHESTER5 dropped at 2010-08-12 08:44:32 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR2-BIRMINGHAM5 dropped at 2010-08-12 08:53:19 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX1-EDINBURGH2 dropped at 2010-08-12 08:58:48 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX1-READING2 dropped at 2010-08-12 09:04:20 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX13-ILFORD4 dropped at 2010-08-12 09:07:26 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR5-SHEFFIELD3 dropped at 2010-08-12 09:25:16 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR16-BIRMINGHAM6 dropped at 2010-08-12 09:48:13 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX6-KINGSTON2 dropped at 2010-08-12 10:07:03 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR5-FARADAY2 dropped at 2010-08-12 05:52:21 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 21CN-BRAS-RED1-L-STE dropped at 2010-08-12 05:48:05 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR11-KINGSTON5 dropped at 2010-08-12 05:29:28 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX4-KINGSTON2 dropped at 2010-08-12 05:22:29 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR6-EDINBURGH5 dropped at 2010-08-12 05:12:15 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX12-MILTONKEYNES2 dropped at 2010-08-12 05:11:13 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR6-KINGSTON4 dropped at 2010-08-12 04:49:58 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR13-BIRMINGHAM6 dropped at 2010-08-12 04:48:39 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX12-READING3 dropped at 2010-08-12 04:41:54 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX2-FARADAY dropped at 2010-08-12 04:38:57 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX19-SHEFFIELD4 dropped at 2010-08-12 04:35:59 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR3-EDINBURGH5 dropped at 2010-08-12 04:25:50 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR10-MILTONKEYNES4 dropped at 2010-08-12 04:24:00 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX8-MILTONKEYNES dropped at 2010-08-12 04:08:16 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR1-KINGSTON4 dropped at 2010-08-12 04:04:39 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR8-KINGSTON4 dropped at 2010-08-12 03:53:21 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX13-READING3 dropped at 2010-08-12 03:41:32 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX10-ILFORD4 dropped at 2010-08-12 03:27:08 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR9-BIRMINGHAM6 dropped at 2010-08-12 03:21:15 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR2-MANCHESTER5 dropped at 2010-08-12 03:07:19 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX6-READING2 dropped at 2010-08-12 02:34:00 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR3-READING4 dropped at 2010-08-12 02:29:07 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX7-FARADAY dropped at 2010-08-12 02:07:01 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX5-KINGSTON2 dropped at 2010-08-12 02:05:11 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR7-READING4 dropped at 2010-08-12 01:58:45 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX20-EDINBURGH4 dropped at 2010-08-12 01:35:44 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR6-ILFORD5 dropped at 2010-08-12 01:19:41 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR1-BIRMINGHAM5 dropped at 2010-08-12 00:43:19 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX7-KINGSTON2 dropped at 2010-08-11 23:39:14 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX2-ILFORD3 dropped at 2010-08-11 22:55:42 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR4-KINGSTON4 dropped at 2010-08-11 22:38:49 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX14-MILTONKEYNES2 dropped at 2010-08-11 22:33:20 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX3-BIRMINGHAM2 dropped at 2010-08-11 21:43:03 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR18-KINGSTON6 dropped at 2010-08-11 20:09:51 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX18-READING dropped at 2010-08-11 19:18:40 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX16-READING3 dropped at 2010-08-11 19:11:32 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR2-MILTONKEYNES3 dropped at 2010-08-11 19:06:11 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX12-BIRMINGHAM3 dropped at 2010-08-11 18:31:00 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR1-READING4 dropped at 2010-08-11 17:45:50 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX21-SHEFFIELD4 dropped at 2010-08-11 17:28:50 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX9-BIRMINGHAM3 dropped at 2010-08-11 16:49:10 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX7-MILTONKEYNES dropped at 2010-08-11 16:47:53 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX2-READING2 dropped at 2010-08-11 16:32:08 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR8-BIRMINGHAM5 dropped at 2010-08-11 16:30:37 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR12-BIRMINGHAM6 dropped at 2010-08-11 16:18:58 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX3-READING2 dropped at 2010-08-11 15:50:16 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX16-EDINBURGH3 dropped at 2010-08-11 15:15:39 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX16-MILTONKEYNES2 dropped at 2010-08-11 15:06:23 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR7-KINGSTON4 dropped at 2010-08-11 14:57:15 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX11-BIRMINGHAM3 dropped at 2010-08-11 14:20:22 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR6-READING4 dropped at 2010-08-11 12:54:34 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX10-READING3 dropped at 2010-08-11 12:21:12 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR4-READING4 dropped at 2010-08-11 11:57:39 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX4-BIRMINGHAM2 dropped at 2010-08-11 11:43:13 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX24-BIRMINGHAM4 dropped at 2010-08-11 11:24:10 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR7-MILTONKEYNES3 dropped at 2010-08-11 10:59:50 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR10-BIRMINGHAM6 dropped at 2010-08-11 09:21:45 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX9-READING3 dropped at 2010-08-11 09:11:49 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX16-BIRMINGHAM3 dropped at 2010-08-11 07:23:59 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX10-MILTONKEYNES2 dropped at 2010-08-11 06:54:27 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR9-MANCHESTER6 dropped at 2010-08-11 06:46:37 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR5-MILTONKEYNES3 dropped at 2010-08-11 05:06:50 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX19-BIRMINGHAM4 dropped at 2010-08-11 04:51:59 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX22-BIRMINGHAM4 dropped at 2010-08-11 03:05:35 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR14-BIRMINGHAM6 dropped at 2010-08-11 01:18:49 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX11-MILTONKEYNES2 dropped at 2010-08-10 23:17:03 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR5-KINGSTON4 dropped at 2010-08-10 22:54:54 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR5-READING4 dropped at 2010-08-10 22:33:54 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX15-MILTONKEYNES2 dropped at 2010-08-10 21:51:30 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX9-MILTONKEYNES2 dropped at 2010-08-10 21:13:44 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR8-MILTONKEYNES3 dropped at 2010-08-10 20:10:53 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR5-BIRMINGHAM5 dropped at 2010-08-10 09:06:00 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR2-READING4 dropped at 2010-08-10 09:01:04 We have advised BT This is likely to have affected multiple internet providers using BT
Almost all of these lines recovered with 20 minutes. We have not had any info on what happened to these lines, however we are still chasing BT and investigating.
All lines on 20CN-BRAS-ERX13-MILTONKEYNES2 dropped at 2010-08-08 21:44:20 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR2-EALING5 dropped at 2010-08-09 19:03:50 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR10-MANCHESTER6 dropped at 2010-08-08 02:40:38 We have advised BT This is likely to have affected multiple internet providers using BT
All 20CN lines on the ERX6-MANCHESTER2 are showing high levels of packet loss. This is affecting the whole RAS so it appears to be a BT core network issue.
BT are aware and are investigating at the moment. This will be affecting multiple ISPs that use BTs network.
BT are unable to raise an incident due to the lack of lines that have been reported with this issue. Therefore we have bulk reported all lines with this issue via the normal fault process.
Update
6 Aug 15:04:14
After reporting several individual faults, a member of BT picked up all the faults and sent them to BTO for investigation.
BTO have now raised an incident for this issue.
Update
6 Aug 15:47:57
Operational teams are currently carrying out remote diagnostics.
Update
9 Aug 10:45:52
BT resolved the issue at roughly half 4 last Friday.
BT have advised us of engineering work planned to start end of May and finish in September which will affect customers who are connected to the following BRAS:
Milton Keynes: 24th MAY
Birmingham: 7th JUNE
Faraday: 14th JUNE
Stephney Green: 28th JUNE
Manchester: 12th July
Leeds: 26th JULY
Colindale 9th August
Southbank: 23rd AUGUST
Wolverhampton: 6th September
BT have advised that during the planned work customers can expect up to 3 hours downtime between midnight and 6am
Planned work notices will also be displayed on the control pages for the individual lines affected.
This work is needed to enable delivery of additional host link options.
Note that from previous experience, when this happens the outage is typically nearer one hour, but can be longer. Also, work on Stepney Green caused all customers to be affected not just that area. Also, last time work on Birmingham had a similar effect but apparently should not have. We'll monitor the situation carefully.
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:12:19.
Update
28 Jul 14:14:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:13:48.
Update
28 Jul 14:15:11
Lines were dropping every minute or two since this started
Update
28 Jul 14:16:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:15:02.
Update
28 Jul 14:24:05
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:23:49.
Update
28 Jul 14:25:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:24:41.
Update
28 Jul 14:27:05
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:26:18.
Update
28 Jul 14:28:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:27:21.
Update
28 Jul 14:33:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:32:44.
Update
28 Jul 14:34:05
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:33:28.
Update
28 Jul 14:38:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:37:07.
Update
28 Jul 14:55:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:54:45.
Update
28 Jul 15:00:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 14:59:48.
Update
28 Jul 15:11:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 15:10:18.
Update
28 Jul 15:12:05
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 15:11:38.
Update
28 Jul 15:12:41
It is associated with IMT 21322-10 which associated with Colindale, BT believe the link between Colindale and Ealing has gone down, engineers have will be dispatched to both sites.
Update
28 Jul 15:20:07
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 15:19:18.
Update
28 Jul 15:21:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 15:20:28.
Update
28 Jul 15:22:07
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 15:21:56.
Update
28 Jul 15:24:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 15:23:26.
Update
28 Jul 15:25:06
All lines on 20CN-REGION-EALING dropped again at 2010-07-28 15:24:30.
This month's telephone bills included (for most people) two separate sets of itemised bills for each numbers. One is the old call server and one is the new call server.
No calls are duplicated so the total amount is right, but the two sections have different layout.
In some cases the time periods overlap as some people had incoming calls on one server and outgoing on another.
We have moved all call routing and logging to the new call server now, so next bills should be tidier.
We have also made a few tweaks today to the way the itemised bills are presented for the new call server. This change is in the PDF layout so you can re-load the PDF from the accounts system to see the newer format for the existing bill if you wish.
The main changes are for number formatting - replacing +44xxxx and 44xxxx UK numbers with the more conventional national number format. We have also added spaces to area codes to make the numbers more readable. In the case of calls which are forwarded on to another number, this is now showing where the call was forwarded on to as well, so as to make the bill cleaer.
There have been minor changes behind the scenes to finally fix the CLI issues we saw on mobiles when using prefixes (e.g. 9 for work number as a prefix). This is now sorted "properly" and the workaround we had in place removed. Any issues let me know on irc.
All lines on 20CN-BRAS-ESR5-READING4 dropped at 2010-07-31 13:46:51 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ESR4-KINGSTON4 dropped at 2010-07-28 21:32:37 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 20CN-BRAS-ERX10-SHEFFIELD2 dropped at 2010-07-26 03:39:17 We have advised BT This is likely to have affected multiple internet providers using BT
The was caused by an over running PEW, sadely though we ere not informed of the PEW in the first place. Bt did raise an incident at the time and had an engineer out very quickly to try and restore service as quick as possible.
We are quite impressed in the way that the BT proactive help desk kept us informed of what was going on without us having to chase. So thank you BT.
We have noticed that following the outage on the NOTTINGHAM LONGBOW Exchange ( http://status.aaisp.net.uk/?incident=345 ) it appears that all 21CN lines at the exchange have had their MTU capped at 1488 bytes. Packets larger than 1488 bytes are being dropped, this may affect Web traffic.
This is affecting multiple ISPs that use BT's equipment.
BT are aware of this and have an incident open at the moment.
This problem is still being investigated by BT; they are getting a list of affected circuits from ISPs at the moment.
Update
27 Jul 09:34:07
BT believe that they have resolved the issue. Thanks to a few of our customers who decided to test this, we can confirm that BT have rectified the issue.
27th July 2010: OFCOM have launched a new version of their code of practice on broadband speeds.
Whilst we agree with motives and principles and spirit of the code in general the latest version has a lot of details that seem badly drafted and have some serious issues. The main issue is that performance of broadband lines (in terms of speed) is to be measured by reference to the speeds of similar lines with the same ISP - thus making 1 in 10 customers below some new guaranteed minimum access speed. This is regardless of how good the ISP is, how fast their lines are or how consistent they are for specific line characteristics. We also feel the new code actively discourages pro-active monitoring of broadband line quality and speed (which we do a lot of).
In light of this, and along with some other small ISPs, we have chose to NOT sign the OFCOM code of practice on broadband speed.
Instead we are making what we feel are better commitments to our customers (see below) explaining how seriously we take the quality and speed of broadband lines.
The new code of practice does not really help customers tell which ISPs are better and which are worse. It would define 1 in 10 of our lines as problem lines whatever we did, and classify all of our best lines as problem lines even though they get maximum speeds. It is really badly worded.
We are seeing an increasing number of cases where customer equipment is used to make VoIP calls. This is typically where users have opened their VoIP system to the internet and had silly password (e.g. user name and password of 101).
Please be careful with passwords and firewalls. Support can help with firewalling rules, but the main problem seems to be silly passwords.
We can see EU's that 20CN lines on the ERX20-EDINBURGH4 and BALLATER exchange are currently affected by packet loss. This is likely to have affected multiple internet providers using BT.
BT are checking if this is related to another Edinburgh IMT and will update us.
There's a problem with our main call server at the moment. We're investigating, but it has resulted in phones dropping their registration and being unable to get back on, so no calls.
The replacement hardware for this server was installed today, which should address any stability issues once customers have been moved over.
How is it that hardware knows it is being replaced. The pair of servers for the call server to be moved were installed this afternoon ready to change over in the new few days and the call server decided to lock up and die... Somehow they know.
Started
21 Jul 18:19:54
Closed 21 Jul 18:31:55
Switching to new hardware any day now. This is new hardware with standby hardware. Sorry for the delay getting this resolved this evening.
One of our upstream carriers also experienced problems due to the same power outage in Telehouse North. This meant that incoming calls would have been broken for some numbers during the outage. This should be OK now.
If you are still having problems, please contact support.
All lines on 20CN-BRAS-ERX11-SHEFFIELD2 dropped at 2010-07-20 03:03:04 We have advised BT This is likely to have affected multiple internet providers using BT
It appears that all lines at the NOTTINGHAM LONGBOW exchange have been down since 02:30 this morning. This will affect multiple ISPs that use BT equipment at this exchange.
Planned work was due to be complete (PW133058 Tue 20 Jul 02:00-06:00)
BT were already aware of the outage and have raised an incident. They are currently working on a resolution.
We have a significant number of FTTC lines where BT have clamped the service at the BRAS to 1.75Mb/s for no good reason.
BT are currently rejecting faults repeatedly blaming "customer equipment misoperation". Given that the router on FTTC is BT equipment, as is the BRAS, it is hard to see how that could ever be true.
We are ugently pursuing this with BT. The ineptitude of BT's fault repair is an ongoing issue and this is (sadly) just a typical case of fault staff failing to actually read notes or even try to fix the fault.
Other ISPs seeing this. BT aware. Seems to be a software update (with no planned works notice) has gone wrong in BT.
Update
18 Jul 19:48:41
BT continue to reject fault reports blaming customer equipment, but we are persevering.
Update
18 Jul 20:01:29
We have escalated the issue to director level in BT now.
Update
18 Jul 21:35:04
We have rejected faults over 100 times now. We are told that BT plc (AKA BT Wholesale) are not the ones rejecting the faults back to us, but it is BT plc (AKA Openreach) that are doing it. Does not make a lot of sense. I am tempted to say that it is not Andrews & Arnold Ltd that keep rejecting the clear but Andrews & Arnold Ltd. BT plc have some strange ideas.
Anyway, there is a major outage raised in BT to try and address the actual issue and we await a resolution.
Update
19 Jul 08:05:35
Some lines are fixed as of 5am, and BT expect others to be fixed by 5pm.
They have, however, started blaming "internet congestion" on at least one fault, which again shows an incredible ineptitude.
Customers who have an 0800 number with us will be getting busy/engaged when called. We are looking in to this and hope to have it resolved shortly.
Last weekend we changed the last of the incoming call routes to go via the new call server. This had very few side effects but sadly a couple of customers needed to make slight config changes for the change of IP address.
This weekend we plan to change outgoing calls for people still using the old call server. This is mostly people using NAT or IAX. The change should really be seamless as the call still goes to the old call server. It is what happens once it gets there that we are changing - the call will be directed to the new call server.
Once completed it means all calls will use the new call server for all call routing decisions. The opens the way for newer features, and we hope to launch the new control pages for numbering soon.
The change also means that all call records and billing will be on the new server. As the bills are in arrears, the next bills will have old and new billing (separate pages) but from the bill after that we will only have new billing format.
The change also affects voicemail and call recordings which will use the new call server. The email format is slightly different on the new call server. Voicemails start after the outgoing message rather than including it, and are mono not stereo.
We are planning to retain the old call server for the time being to handle NAT and IAX, but both will be deprecated for any new customers. Eventually we may create special modules for SIP NAT and IAX handling rather than continuing to use asterisk, but this is much longer term.
We also have two new machines for the new call server which will be deployed soon. The change over should be seamless and the IP address for the new server is not changing. This will mean the new server has the power backup that it needs and hot standby.
Any problems with these changes, please do contact support.
This work needs a bit more of a re-think to allow for some complex cases (customers with blocks of numbers and calling line identify control within that block). We expect to do more testing during the week. Sorry for the delay.
Update
14 Jul 09:54:03
We are planning to do this work next weekend.
Started
18 Jul
Closed 18 Jul 08:48:00
Changes done and tested - any issues please contact support.
There was a temporary problem accessing Squirrelmail a few minutes ago. This has been resolved. We were moving the 'preference' files over to a new file server and missed a couple of the file permissions. We do apologise.
BT have advised us of engineering work planned to start end of May and finish in September which will affect customers who are connected to the following BRAS:
Milton Keynes: 24th MAY
Birmingham: 7th JUNE
Faraday: 14th JUNE
Stephney Green: 28th JUNE
Manchester: 12th July
Leeds: 26th JULY
Colindale 9th August
Southbank: 23rd AUGUST
Wolverhampton: 6th September
BT have advised that during the planned work customers can expect up to 3 hours downtime between midnight and 6am
Planned work notices will also be displayed on the control pages for the individual lines affected.
This work is needed to enable delivery of additional host link options.
Note that from previous experience, when this happens the outage is typically nearer one hour, but can be longer. Also, work on Stepney Green caused all customers to be affected not just that area. Also, last time work on Birmingham had a similar effect but apparently should not have. We'll monitor the situation carefully.
BT have raised an incident on this and the BT operate engineers are currently working on this, they do not currently have an ETA however we chase them again this afternoon.
Update
7 Jul 14:41:57
BT advise it is linked to ongoing Glasgow issue IMT 19036-10, BT technical conference calls are still ongoing trying to investigate the issue. They will call us with an update.
Update
8 Jul 11:21:44
BT closed the incident that was raised yesterday as they could not see the issue. However BT have had several calls today about the issue and they are in the process of raising new incidents for the different RASs that are affected.
BT have confirmed that some lines on 21CN-RED1-GW-TH, 21CN-RED3-GW-TH, 21CN-RED4-GW-TH, 21CN-RED5-GW-TH are affected.
Update
9 Jul 13:40:27
New IMT 19121-10 At technical conference call is currently in progress. next update is due for 14:45.
Update
9 Jul 16:03:07
this has been going on for quite some time now, without an major progress. Therefore we have reported individual faults with the hope of getting this progressed faster.
Update
9 Jul 16:09:09
Appears the fault has now been resolved.
We are now trying to find out what the cause of this was.
Update
9 Jul 16:19:47
Bt have said that the cause for this issue was a combination of things.
One of the causes was an MSPE VLAN bandwidth config error. There are other causes but BT are still investigating, we should have a full report in a couple days.
We can change terms and prices at any time and we update our web site with the changes.
We believe the terms are quite clear and very reasonable as they allow you to insist the previous terms (and price) apply if you want to cease a service because you are not happy about the change (within one month of the change).
We have today made minor changes to the wording of terms on the web site to combine the descriptions for changes to price and changes to terms in one place. This also clarifies what happens for any services that have a minimum term, which was not previously very clear. Basically, we can either agree to apply the old terms and price for the rest of the minimum term, or we can allow you to cease early with no penalty. This will typically depend on whether we are tied in to paying a supplier for a minimum term.
We have also clarified exactly when call prices for telephone services apply (at the time of the call).
Well, junk callers are calling honey pot numbers... I'll post any good ones we get.
Next step is making four million new numbers live and trapping junk callers to them too. If we can tie up enough of their operatives time we reduce the number of innocent victims of their criminal activity.
Public notice: If you call an A&A unallocated number your call may be recorded and posted on the internet for comedy purposes... As well as used as evidence in criminal and civil action against you if you are one of these junk callers.
ACR (anonymous call reject) normally provides a polite message, non-charged, and hangs up.
A new "nasty" ACR option now handles withheld and unavailable callers, answers the call, tells them they are a coward for not releasing their number and repeats until they hang up. It also sends a tone/DTMF digit 2 to tie up an operator for the typical junk caller.
You also have a new *113 code to record a customised ACR message (for standard and nasty mode). Note, no prompt on this - just a beep. Test by calling 141 and your own number.
Bear in mind people can genuining call wrong numbers and international callers are often calling as unavailable (hits the "nasty" mode). Young kids can hit keys on phones too, so do not be too rude.
We're interested to hear comments from customers.
Junk callers are becoming a real problem - we are getting an average of one every 90 seconds to our office DDI block alone now.
We are seeing increased peak and minimum latency on all lines on L-NWS and SL BRASs. L-NWS : Very high latency between 1pm - 5pm SL: Peaks of increase latency between 12pm - 8m
This is a congestion issue within the BT network so this wil be affecting other ISPs.
BT have raised an incident and are re-investigating.
Update
16 Jun 16:23:44
There is a planned capacity upgrade for the end of the week. We will check monday to see if the issue is resolved.
Update
8 Jul 10:24:01
The issue has finally been resolved by BT and they have changed the process so to be more proactive and hopefully stop this from happening again in the future. Please see http://status.aaisp.net.uk/?incident=330 for more details.
Started
11 Jun 15:35:32 by AAISP Pro Active Monitoring Systems
We have seen in the last week and a bit increased latency and some packet loss on some RASs caused by the congestion the World Cup has caused.
These are general congestion issues within the BT network and as such will be affecting all ISPs that use the same BT backhaul.
Areas and Descriptions:
All -DY RASs showing Increased Peak and minimum latency between 1pm - 4pm. All -WV RASs showing Increased Peak and minimum latency between 12pm - 4pm. All LS-BAS RASs showing Increased Peak and minimum latency between 12pm - 5pm. All -SL RASs showing Increased Peak and minimum latency between 12pm - 5pm. All L-NWS RASs showing Increased Peak and minimum latency between 12pm - 5pm.
BT are aware of these and are working on the problems.
If you think we have missed any areas then please contact support at support@aaisp.net.uk
There is a planned capacity upgrade for L-NWS and -SL for over the weekend. We are chasing the other problems.
Update
18 Jun 16:25:33
If you're not sure what RAS you are on, it will be displayed when you log in to the broadband control pages, and also on the CQM graphs that we produce for your lines.
Update
21 Jun 17:04:48
It appears that the capacity upgrade for -SL and L-NWS have not had any affect.
It had also been brought to our attention that GI-B RASs are showing packet loss during the day, we have emailed BT about this.
Currently we have not been able to contact BT via the phone, so we do not know the latest updates on these issues. We will chase again tomorrow and will hopefully have an update for you by the start of play tomorrow.
Update
25 Jun 11:04:39
Due to the severity of the congestion and the amount of time it has been going on for, we have escalated the isue within BT further and should hopefully have updates soon.
Update
29 Jun 22:08:24
This has now ben raised further and an incident has been raised to get this resolved.
Update
8 Jul 10:23:48
The issue has finally been resolved by BT and they have changed the process so to be more proactive and hopefully stop this from happening again in the future. Please see http://status.aaisp.net.uk/?incident=330 for more details.
We're also very pleased to say that the FireBrick FB6000 constant quality monitoring was key in showing BT the problem and confirming where BT had a working solution. The graphs were even used in the internal BT presentations on the problem and the solution.
At the moment it would appear that there is a problem with changing your line profile via clueless. We are unable to make the changes manually via our BE portal either. So there does appear to be a general issue at the moment.
All lines on 20CN-BRAS-ESR1-FARADAY2 dropped at 2010-05-31 17:41:58 We have advised BT This is likely to have affected multiple internet providers using BT
All lines on 21CN-BRAS-RED1-MR-DH dropped at 2010-06-08 00:29:24 We have advised BT This is likely to have affected multiple internet providers using BT
There are obviously cases where matters are totally beyond our control, e.g. if your favourite web site in the US is down we cannot really do much about that.
There are cases where matters are within our control such as our switches and equipment. In these cases we obviously work to fix such issues as quickly as possible.
There are however cases that are outside our direct control. These are where companies like BT have control over some crucial aspect of the service such as the copper pair and we have a contract with them. In these cases we will try and get the fault fixed, but we are at the mercy of the likes of BT and their policies and charges and so on.
Unlike BT, we do not consider fault repair to be an opportinuty to sell extra services (such as an investigation service). As such, any charges BT make to us in relation to fault repair we pass on at cost, making no profit. We also aim to minimise such charges.
Some BT policies have changed recently, and we are working on this now. It may have an impact on how quickly faults can be repaired. We hope to provide a full press release later today or tomorrow on the matter.
This is also causing tech staff to be busier than usual.
Update
6 Jul 19:43:16
We are sorry to report that over the last few days we have had some significant delays in getting faults progressed within BT. The issue is an unexpected change in the way BT handle faults at both the technical level and the policy level which has made it impossible to progress faults. Some of this was briefed to us but the full implications were not clear at the time so this came as a bit of a surprise.
We have had several discussions with BT during the day, and we are making progress on this. We expect to make a number of technical changes during the day on Wednesday that should allow faults to start progressing smoothly again.
The whole process of Special Faults Investigations has been an issue between BT and us (and many other ISPs) for some years, and while BT are working on new and better solutions (with SFI2) there are still some areas where we do not yet reach full agreement. However, we are continuing talks with BT on this to come up with longer term solutions.
We are expecting to be involved in trials of SFI2 over the next few weeks. SFI2 has a lot of changes to avoid conflicts and incorrect charges - these include new test equipment, better notes, and a clear sign off process where a working service is demonstrated to our customer at the end. We'll advise customers where this new process applies as and when we make use of the new service. We are hoping this will improve fault repair times as well.
Customers with blocks of numbers have had some slightly incorrect billing this month. We are working through these correcting them now. This means some credit notes and new invoices.
The issues are:-
(a) Calls before late June had not been billed
(b) Calls for 1st July have been billed on today (2nd) and the system would be billing every day as it happens at present, so this is being fixed.
OK, the correction means you will get a credit note for the £1/month charges for the blocks of numbers, and the call charges billed on 1st, and the call charges billed on 2nd. You then get a new invoice withe the £1/month line charges and call charges up to and including the 1st.
This will happen during the day as we have to go through and check and credit and re-bill each affected account manually.
Sorry for the confusion.
Update
2 Jul 09:49:24
I am pleased to say that most of the customers with blocks of 10 and 100 are using them for incoming calls only and so very few accounts have had to be adjusted.
We have done a lot of work investigating NAT and SIP.
We are maintaining some customers on the old server for now as the issues with NAT are just horrendous. I'll probably do a web page on the subject some time. It is mental.
However, we are progressing some of the back-end call routing which means some small changes. These changes were done for one carrier in one go some weeks ago with no apparent effects. We are now working on the other carrier and plan to make changes at the weekend. Given the lack of apparent issues with the first carrier, we do not expect any issues with the second carrier.
This will mean call bills will move to the new format for incoming calls. Which will help make the bills more consistent.
There was also a slightly un-planned 18 second outage around 15:30 today where calls would not be established but existing (non recorded) calls would have been fine. Apologies for that.
Some time over the weekend one of our trabnsit carriers has some brief maintenance for a few minutes. It should not affect any services, but is a risk.
There are many details of the mobile service which are gradually progressing. One of these is roaming.
We have had successful customer tests in Poland this morning. It is not perfect yet - as the phone can roam to a network which won't allow calls to be made, but one at least two of the networks that the customer could access the phone worked both ways (albeit with an odd CLI).
We don't have details of roaming prices yet, but it is a step closer.
All lines on the 21CN-BRAS-RED4-L-NWS RAS went down at 13:47:13. As of yet none have come back up. This will be affecting all ISPs that use BT. BT are aware and are investigating.
An incident has been raised. BT are still looking into the casue and should have an update soon.
Update
1 Jul 16:26:27
Ras was rebooted and BT have been systematically opening the ports. Some customers have come back up. Others should be up when the engineer opens their ports.
