Andrews & Arnold Incidents Feed

[PEW] Broadband: At risk - Router uprade

2012-01-28T17:57:54Z

There is a router restart expected this evening but it should have no impact on services.

Update 28 Jan 2012 17:57:54

All to plan, no blips

[Minor] Broadband: Lines dropped: 100% 21CN-BRAS-RED2-FK (Open)

2012-01-27T23:55:02Z

Lines: 100% 21CN-BRAS-RED2-FK dropped at 2012-01-27 23:54:07
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 27 Jan 2012 23:56:02 Lines: 100% 21CN-BRAS-RED2-FK dropped again at 2012-01-27 23:54:07.

[Info] Mobile: Data SIM roaming

2012-01-27T16:47:51Z

Thanks to those that have helped with testing the data SIM roaming. We expect at present that there will not be any extra billing for this.

In theory data roaming should work everywhere three have roaming as, unlike voice romaing, it is always sent back through the home network.

Unfortunately it seems that getting the billing systems in place over all of the links involved is proving a challenge, and so at present data SIM roaming is disabled on all SIMs.

Once we have the billing systems, and price lists, this will be offered as an option.

Thanks again for help testing.

[Minor] Broadband: Lines dropped: 100% 20CN-REGION-EDINBURGH (Open)

2012-01-26T17:24:02Z

Lines: 100% 20CN-REGION-EDINBURGH dropped at 2012-01-26 17:22:08
We have advised BT
This is likely to have affected multiple internet providers using BT

[Minor] Broadband: Lines dropped: 15% 20CN and 100% 20CN-REGION-EDINBURGH and 100% 21CN-REGION-FK (Open)

2012-01-26T17:23:02Z

Lines: 15% 20CN and 100% 20CN-REGION-EDINBURGH and 100% 21CN-REGION-FK dropped at 2012-01-26 17:21:37
We have advised BT
This is likely to have affected multiple internet providers using BT

[Minor] Broadband: Lines dropped: 85% 20CN-REGION-EDINBURGH and 89% 21CN-REGION-FK and 100% 20CN-BRAS-ERX16-EDINBURGH3 and 100% 20CN-BRAS-ERX6-EDINBURGH2 and 100% 20CN-BRAS-ESR4-EDINBURGH5 and 100% 21CN-BRAS-RED2-FK and (Open)

2012-01-26T17:22:02Z

Lines: 85% 20CN-REGION-EDINBURGH and 89% 21CN-REGION-FK and 100% 20CN-BRAS-ERX16-EDINBURGH3 and 100% 20CN-BRAS-ERX6-EDINBURGH2 and 100% 20CN-BRAS-ESR4-EDINBURGH5 and 100% 21CN-BRAS-RED2-FK and 100% 21CN-BRAS-RED3-FK dropped at 2012-01-26 17:21:37
We have advised BT
This is likely to have affected multiple internet providers using BT

[Info] General: 80Mb/s FTTC Trial - Initial Customer Online

2012-01-26T15:13:48Z

2012-01-12 13:35:00, updated 2012-01-26 15:13:14

It is impressive what you can do on a short bit of copper :-)

We have a number of customers on the FTTC 80Mb/s trial now. One has 79.7Mb/s and another has 57Mb/s. This is pretty impressive. The uplink is around 20Mb/s as well...

The trial is not going badly - the lines are syncing up at nice high rates, and the rates are correctly coming through to us.

There are a few minor teething troubles getting the profiles right in the middle somewhere - but this is a technical trial, so not at all surprising.

This part of the trial is expected to last until the end of the month, where the second phase of the trial starts. The second part is expected to be open to the whole country (where FTTC is currently available)

Update 26 Jan 2012 15:12:49

Here is a speed test result.

https://p.twimg.com/AkF64vvCIAAVMQF.png

-a number of speed tests were carried out, this was the highest download result, the results did vary though, and the upload isn't as high as we'd expect - should be nearer 20Mb/s.

The end user will be trying a new router to compare.

[MSO] Maidenhead Data Centre: Problem in Maidenhead - affecting VoIp/ethernet/email/etc

2012-01-26T15:10:49Z

We are seeing a major issue in Midenhead - high levels of transit packet loss that will be affecting VoIP, email, and Ethernet customers (as well out our offices).

Update 24 Jan 2012 21:47:11

Looking like some sort of denial of service attack.

Update 24 Jan 2012 22:34:04

Sorry for the delay posting more details. This affects our links directly and making it difficult. The problem appears to be a huge denial of service attack invovling tens of thousands of sessions and filling gigabit links.

We have identified the target and disconnected it, black holed the target and even tried to divert traffic but to no avail as yet.

We are still working on this.

Update 24 Jan 2012 23:03:38

Just an update to say that this is still being worked on...

Update 24 Jan 2012 23:23:30

Still working on this

Update 24 Jan 2012 23:28:57

The problem is now only affeting Ethernet customers on the same block as the address being DDOS'd. We are still working on the issue. Most other services will be working fine now.

Update 24 Jan 2012 23:57:55

Some side effects on other services from Maidenhead, but we are still working on narrowing down the issue.

Update 25 Jan 2012 00:01:48

DOS attacks are, thankfully, rare. This has to be the biggest we have seen.

We will, of course, be talking to the customer who is being DOSed to fine what could have provoked such a major attack. There is usually a reason.

Update 25 Jan 2012 13:20:48

The blackhole for the target machine was removed at 1pm today, however the traffic was still being sent and affected VoIP, email and Ethernet services.

The block is now in place again, and we'll continue to investigate.

[Minor] Broadband: Lines dropped: 100% 21CN-REGION-PE

2012-01-26T14:45:52Z

Lines: 100% 21CN-REGION-PE dropped at 2012-01-26 14:19:19
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 26 Jan 2012 14:21:02 Lines: 100% 21CN-REGION-PE dropped again at 2012-01-26 14:19:19.

Update 26 Jan 2012 14:24:30

Half the lines are now back.

Update 26 Jan 2012 14:45:52

All lines are now back.
Proactive are going to ask BT operate to investigate what caused the incident.

[MSO] Maidenhead Data Centre: Maidenhead datacentre problems again

2012-01-25T13:16:49Z

Similar to last night, access to servers and services in Maidenhead have high packet loss.

This will affect email, voip and Ethernet services.

Update to follow shortly.

Update 25 Jan 2012 13:13:19

Datacentre staff are working to blackhole the IP address that is the target of this attack.

Update 25 Jan 2012 13:14:43

The target IP address has been blackhole'd and service has been restored.

[PEW] Broadband: Overnight LNS switches

2012-01-25T07:15:36Z

We appreciate that doing an LNS switch in the evening is not ideal, and even though it should only be a few seconds blip in service it can cause inconvenience (especiall for those using NAT).

We have a major network upgrade going on, and we will be doing a number of moves between LNSs over the next two weeks as part of this.

However, the good news is that we believe the overnight LNS switchover scripts are ready to test. This means the LNS switch over is done during the night at a time to suit you. The control pages allow you to pick an hour during the night and we aim to follow your request on that.

However, this does, of course, need testing, so we plan to run this overnight tonight.

See more at http://aa.net.uk/kb-broadband-duallns.html

[PEW] Broadband: LNS switch this evening

2012-01-23T17:31:04Z

We have to do an urgent LNS switch over this evening. Apologies for any inconvenience.

Technical details: We have identified a latent issue in the current code which puts it at risk of problems if there was a failure of both of our RADIUS servers (somewhat unlikely).

Update 23 Jan 2012 17:31:04

The switch over went fine, and allowed testing of the new overnight scripts for this. We hope next time to run an LNS switch over night, and we may try this later in the week.

[MSO] General: Mail and web server issues

2012-01-22T08:33:27Z

It looks like we have a major issue with mail and web services.

This is being investigated now.

Update 22 Jan 2012 08:04:01

This looks like an issue with the main storage array used by web and email. Being worked on now.

Update 22 Jan 2012 08:31:56

Looks like we have it working again - web pages are fine - email being a bit sluggish catching up.

[PEW] General: At risk - router upgrades

2012-01-21T18:59:54Z

We are upgrading a few core routers this evening - no problems expected

[Minor] VoIP: Some Call Breakups

2012-01-20T16:26:54Z

One of our upstream carriers for VoIP is having a network problem and we have 60% packetloss to them.

This will cause VoIP problems for some calls and customers.

We expect this to be resolved quickly

Update 17 Jan 2012 15:00:19

Back to normal now

[Info] General: World IPv6 Launch - We're on the list.

2012-01-20T15:31:47Z

We're now listed on the World IPv6 Launch website:

http://www.worldipv6launch.org/participants/?q=2

(Along with 18 others at the moment!)

Like the day last year, this event is organised by the Internet Society but this time the idea to encourage networks and websites to turn on IPv6 and to keep it on.

Good news for IPv6! From our side there is nothing we need to do, change or enable on our network as we're already offering and supporting IPv6.

The date this happens is June 6th.

Update 20 Jan 2012 14:44:33

We have a page on our main website regarding the launch event now:

http://aa.net.uk/world-ipv6-launch.html

[Minor] Email: Some problems sending email to Hotmail/Live.com

2012-01-20T14:56:29Z

Hotmail have decided to black list one of our email server IP addresses. We've taken it off line for the moment.

customer who have had bounce back messages from hotmail/live.com should resend them in an hours time.

[PEW] Maidenhead Data Centre: Server Moving - Not Customer Affecting

2012-01-19T13:32:43Z

We'll be moving some servers around between racks in the Maidenhead datacentre on 19th January. These servers are mainly related to email services, but we are not expecting customers to notice anyting whilst the moves take place.

The servers moving will be:

tertiary-mx.co.uk (the backup email relay)
A couple of the Spam checking servers (there are many, so one being down at a time won't cause a problem)
One of the outgoing SMTP servers (which is offline at the moment)

[Minor] Broadband: BT Congestion at the CANONBURY exchange

2012-01-18T13:53:39Z

Since 2011-12-12 we have been seeing Packet loss all day on all BT CANONBURY exchange lines.

Update 17 Jan 2012 10:10:11

BT are looking into this at the moment.

Update 17 Jan 2012 10:31:56

BT have raised a work request regarding this exchange.

Update 17 Jan 2012 10:51:44

BT operate are looking into possible causes for this now.

Update 18 Jan 2012 13:53:39

Has been fixed

[Info] Broadband: Minor price adjustments

2012-01-17T18:14:27Z

From next month (Feb 2012) there are a few minor changes to prices.

FTTC and FTTP monthly prices are slightly reduced so that the VAT inclusive monthly price with two units is now £30

BE monthly prices are slightly reduced so that the VAT inclusive monthly price with two units is now £40

Premium and enhanced care options are increased to £10+VAT (£12).

Due to the increase in prices on premium and enhanced care, any customers wishing to cease these services next month will not be charged the usual £10 for stopping these services in an active line. Please email sales to have your invoice adjusted.

[Minor] Broadband: Lines dropped: 100% 21CN-BRAS-RED2-SL

2012-01-17T11:17:20Z

Lines: 100% 21CN-BRAS-RED2-SL dropped at 2012-01-11 10:46:16
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 17 Jan 2012 11:17:20

Bt have been unable to give us a reason for this outage.

[Minor] Broadband: Lines dropped: 100% 21CN-BRAS-RED4-L-FAR

2012-01-17T11:16:59Z

Lines: 100% 21CN-BRAS-RED4-L-FAR dropped at 2012-01-11 02:41:50
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 17 Jan 2012 11:16:59

BT have been unable to give us a reason for this outage.

[Minor] Broadband: 21CN Congestion at the GARFORTH Exchange (Open)

2012-01-17T11:09:54Z

We are seeing increased peak and minimum latency on all lines at the GARFORTH Exchange between 8pm till 10pm.

Update 17 Jan 2012 10:08:41

BT are investigating this at the moment.

Update 17 Jan 2012 11:09:54

BT Operate are investiagting.

Update 18 Jan 2012 13:54:29

BT think the problem is being caused by a traffic shaper their side. They have emailed the relevent department to get this sorted.

[Minor] Broadband: 21CN Congestion at the DEPTFORD Exchange (Open)

2012-01-17T10:52:00Z

We are seeing packet loss between 8pm - 11pm on all 21CN lines at the DEPTFORD exchange

Update 17 Jan 2012 10:08:58

BT are investigating this at the moment.

Update 17 Jan 2012 10:52:00 BT operate are looking into possible causes for this now.

[Minor] Broadband: 21CN Congestion on BRIXTON Exchange (Open)

2012-01-17T10:09:38Z

We are seeing packet loss between 8pm to 11pm on all 21CN lines at the BRIXTON exchange.

Update 17 Jan 2012 10:09:38

BT are looking into this at the moment.

[Minor] Broadband: 21CN congestion on the KINGSLAND GREEN Exchange (Open)

2012-01-17T10:09:19Z

We are seeing packet loss between 8pm to 11pm on 21CN lines at the KINGSLAND GREEN exchange

Update 17 Jan 2012 10:09:19 BT are investigating this at the moment.

[Minor] Email: Some Webmail Problems

2012-01-16T15:59:17Z

One of our webmail servers is off line at the moment, this will affect some of our email users from accessing it's web pages.

We expect service to resume shortly though and will update this post again.

Update 16 Jan 2012 15:44:36

This server (The G Mail server) is being rebooted.

[Minor] Broadband: Increased Latency on some Be Lines

2012-01-16T08:37:06Z

We're seeing increase latency since about 2pm on some Be lines.

It looks like a back-haul problem within the Be network, we've reported it to them.

The lines are showing an additional 50-70 miliseconds, and 1-3% packetloss.

Update 15 Jan 2012 16:01:14

Looking back to normal now...

http://p.twimg.com/AjNi6XrCIAEeOLk.png

Update 15 Jan 2012 22:24:39

The problem came back at about 16:45 and carried on until 22:00.

Lines are looking ok at the moment though!

Update 15 Jan 2012 22:27:49

This is due to maintenance within the Be network, it may continue during the night.

[MSO] Broadband: Friday 13th: Major issue with broadband service

2012-01-13T15:49:48Z

We're looking in to a network problem affecting broadband customers - More details to follow...

Lines are coming back and we arte trying to find what happened.

Update 13 Jan 2012 11:22:05

We have engineers onsite in telehouse looking in to this.

However, lines and routing are recovering now.

Update 13 Jan 2012 11:22:07

Looks like some network issue in our core rack in telehouse - engineers on site investigating now. Lines mostly back up now.

Update 13 Jan 2012 11:25:35

We are still somewhat at a loss as to what happened, but it has meant all lines being dropped of the live LNS. We have switched the the backup anyway.

However, and possibly unrelated, we are seeing issues with BT 20CN lines coming up - i.e. they are not. This was happening before this more major issue, and means 20CN lines are still down. We are chasing With BT.

Update 13 Jan 2012 11:35:49

20CN lines now on - looks like BT did a config change early which resulted in problems. This would have been a non issue if there was not also some sort of major blip which we have yet to identify.

[PEW] Broadband: At risk - new rack work (Open)

2012-01-13T14:24:40Z

We expect this to be very low risk but given the fun already today worth posting - we have staff working on the new rack in Telehouse, and this afternoon they are connecting up the existing rack.

We do not anticipate any issues.

[Minor] Broadband: Be line blip

2012-01-12T18:40:38Z

Looks like all Be lines went off for a couple of minutes - no idea why yet - looks to be back now.

Update: THN TFM 3 Power Failure affecting the interconnects at the carrier.

[Info] General: We're in the ISPReview Editors Pick :-)

2012-01-12T10:45:19Z

We get pretty good reviews on the ISPReview website, and we're often number 1 in their top 10 list too - They also list us in their recent 'Editors Pick - UK Best Broadband ISPs for 2012' article:

http://www.ispreview.co.uk/articles/12_Best_UK_Broadband_ISPs/

Thank you to all those who vote and review us!

[MSO] Broadband: Major blip in broadband service

2012-01-12T09:17:28Z

Seems a major problem affecting all broadband lines - service si restoring automatically, but taking several minutes to get people back on line.

Update 12 Jan 2012 01:39:17

Lines are coming back but much more slowly than we would expect

Update 12 Jan 2012 01:49:24

Both LNSs are seeing people connect constantly and then disconnect, which makes little sense. We are still investigating the issue now.

Update 12 Jan 2012 02:00:00

We have reset equipment and switched LNSs, which has affected lines that were not previously affected. We are trying to see a pattern here.

Update 12 Jan 2012 02:12:24

Logs confirm that at 2am. suddenly things started working a lot better and many more lines coming back. Still investigating what is going on here.

Update 12 Jan 2012 02:19:29

We think, whatever it was, has magically fixed itself. We are going to clear all sessions to one LNS now, so a blip again for some people.

Update 12 Jan 2012 02:33:27

We are not entirely sure of the cause of the original problem, but it is such a long time since we have had a major issue like this it appears that we currently have a problem with the speed with which our systems can recover.

It seems our RADIUS server is struggling and once overloaded it is too slow to handle the connections and so the connections timeout and re-try causing more load and more delay. The result is that while everyone is trying to connect, nobody can, and it basically took well over an hour for lines to connect.

More investigation to follow.

Update 12 Jan 2012 02:43:21

I am closing the mahor incident on this now - but we are investigating the cause and the slow reconnect.

[PEW] Broadband: LNS switches over night (Open)

2012-01-11T18:48:17Z

We should have the new links to BT any day now - and then we can start testing the two new LNSs. This will mean some more complicated arrangements with different lines being on different LNSs.

Whilst generally an LNS switch is a matter of a few seconds loss of PPP, some people do find it inconvenient (especially people using NAT!). So we are planning to move future LNS switches to around 1am. Obviously interested in feedback on this as it could affect gamers. It may be that 5:30am is better.

However, to do this we need to test new automated scripts that fully manage the LNS switch, including messages on irc, and pins on the graphs. This needs to be tested when we are around to monitor it. As such we do expect one or two more "normal" LNS switches in the evenings over the next few days. These should get a tweet and irc posting automatically before and after the switch.

Once we are happy with the automation we can schedule all future LNS switches (well, unless some emergency) to be at a less disruptive hour.

Thank you all for your patience on this.

Update 11 Jan 2012 18:47:59

The solution, of course (and which needs testing) it to let people set the preferred time of night on the control pages. The setting is now there per line.

Update 12 Jan 2012 03:12:21

We tried several things to fix the issues this morning, and that has meant an LNS switch has happend. We expect to do more testing later in the week.

Update 12 Jan 2012 15:20:12

Invetigations in to the problem early this morning have shown some possible causes. We have a new system ready and plan to do an LNS switch this evening. We expect this to fix the initial "crash" but we are not sure if it fixed the slow reconnects yet. We are still investigating.

Update 12 Jan 2012 17:38:55

LNS switch completed this evening - it has shown some interesting results in the data we have collected and will be analysed more. The changes we have made appear to have allowed a faster LNS switch with at least one line monitoring showing only 2 seconds lost packets. This is how it should be, of course.

[Info] General: Testing posting to Twitter & Facebook

2012-01-10T14:22:43Z

That's all folks, just a test

[PEW] Maidenhead Data Centre: UPS Work (No user impact) (Open)

2012-01-09T12:32:21Z

We have been informed by the Maidenhead datacentre that they have some planned work on their UPS systems.

They will be carrying out a replacement exercise on end-of-life capacitors in the UPS.

The unit will be taken off-line for the duration of the work however due to the configuration of the installation there will be no impact to the electrical supply or resilience to equipment

The work will be carried out on Monday 16th January, starting at 21:00 and should last 5 hours.

This work is factory advised and forms part of their robust Planned and Preventative maintenance regime.

[PEW] Broadband: LNS switch this weekend

2012-01-07T18:15:07Z

We are switching traffic between the main two LNSs this weekend. Still waiting for a date from BT on the new links for the two new LNSs to be deployed.

Update 5 Jan 2012 18:20:57

By popular demand we are planning to make a script that does the LNS switch which includes a pin (or two) on the graphs saying we did an LNS switch and even a tweet... We'll try this on Saturdays switch.

Update 7 Jan 2012 18:15:07

LNS switch finished later than expected.

1. the new scripts were not quite right, hence several tweets and messages

2. The LNS switch did not work as planned. A config error on our part tested something we had not tried on BT before, which is asking them to share the sessions over two LNSs. We tried several times to clear sessions but they kept sharing them evenly as we asked. We have corrected the config now and all sessions switched over.

[PEW] Mobile: Three maintenance - at risk period

2012-01-06T11:08:59Z

Three will be carrying out a restart of one of their SGSNs during this window, causing a ten minute interruption to its routing. This has the potential to affect customers in the south east of England if their traffic is passing through this SGSN.

[PEW] Broadband: BE Planned Work 10/01/12 (Open)

2012-01-05T17:15:38Z

Be are performing some planned work on 10/01/12 - between the hours of 00:00 to 08:00 on the following exchanges:

HALESOWEN, HEADINGTON, HEATH HAYES, HEDNESFORD, CMACO, ALDRIDGE, ASHFIELD, BEACON, BEARWOOD, BILSTON, BLACKHEATH, BURNTWOOD, BROADWELL, BROWNHILLS, BRIERLEY HILL, CALTHORPE, CANNOCK, CASTLE BROMWICH, CHELMSLEY WOOD, CHESLYN HAY, CRADLEY HEATH, DUDLEY, DRUIDS HEATH, EAST, EDGBASTON, ERDINGTON, FALLINGS PARK, FOUR OAKS, WELLINGBOROUGH, WEST WIGSTON, WESTON FAVELL, OADBY, RAUNDS, ROTHLEY, RUSHDEN, SHEPSHED, ST BARNABAS, STONEYGATE, SUTTON ELMS, THURNBY, KIDLINGTON, OAKLEY, OXFORD, SUMMERTOWN, WITNEY, ABINGDON, BICESTER, CARTERTON, COWLEY, DIDCOT, HARBORNE.

[PEW] Broadband: BE Planned Work 11/01/12 (Open)

2012-01-05T17:09:25Z

Be are performing some planned work on 11/01/12 - between the hours of 00:00 to 08:00 on the following exchanges:

HIGHBURY, KIDDERMINSTER, HORSELEY FIELDS, KINGS NORTON, KINGSWINFORD, LEAMORE, LICHFIELD, LYE, NORTHERN, PELSALL, PRIORY, RECTORY, RUBERY, SEDGLEY, SELLY OAK, SHELDON, SHIRLEY, SMETHWICK, SOLIHULL, SOUTH, SPRINGFIELD, STOURBRIDGE, STECHFORD, STONE CROSS, STREETLY, SUTTON COLDFIELD, TETTENHALL, TIPTON, VICTORIA, WOODGATE, WEDNESBURY, WEST BROMWICH, WILLENHALL, WALSALL, WOLVERHAMPTON, BLIDWORTH, BULWELL, CHALFONT DRIVE, EDWALTON, FAZELEY, GEDLING, ARNOLD, BASFORD, BEESTON, TAMWORTH, HUCKNALL, ILKESTON, KIMBERLEY, KIRKBY IN ASHFIELD, POLESWORTH, REDDITCH, RUGELEY, STAFFORD, BROMSGROVE, DROITWICH, HEADLESS CROSS, IPSLEY, HILLSIDE.

[PEW] Broadband: BE Planned Work 13/01/12 (Open)

2012-01-05T17:08:19Z

Be are performing some planned work on 13/01/12 - between the hours of 00:00 to 08:00 on the following exchanges:

ASHTON IN MAKERFIELD, KIDSGROVE, ASTLEY BRIDGE, ATHERTON, BACUP, BOLTON, BURNLEY, CARLISLE, DARWEN, DAUBHILL, FARNWORTH, FORMBY, HEYWOOD, HIGHER BOLTON, HINDLEY, HORWICH, LEIGH, LITTLEBOROUGH, ORRELL, PLATT BRIDGE, RAMSBOTTOM, ROCHDALE, ROSSENDALE, SHAW, SHEFFIELD, SHARROW, STAVELEY, WADSLEY BRIDGE, WOODHOUSE, WICKERSLEY, WOMBWELL, WORKSOP, LEEK, LONGTON, MILTON LEE, NANTWICH, NEWCASTLE, SANDBACH, STOKE ON TRENT, STONE, TRENTHAM, WILLASTON, ALSAGER, BIDDULPH, BLYTHE BRIDGE, BURSLEM, STOKE CITY, HASLINGTON, AINSDALE.

[PEW] Broadband: BE Planned Work 12/01/12 (Open)

2012-01-05T17:07:31Z

Be are performing some planned work on 12/01/12 - between the hours of 00:00 to 08:00 on the following exchanges:

BURTON ON TRENT, STOCKSBRIDGE, DERBY, ALFRETON, ALLESTREE PARK, ALVASTON, TRENTSIDE, WILLOWCROFT, WOLLATON, LANGLEY MILL, LEABROOKS, LONG EATON, MICKLEOVER, MANSFIELD, PINXTON, PEARTREE, RUDDINGTON, SANDIACRE, SHERWOOD, SUTTON IN ASHFIELD, SWADLINCOTE, HEMSWORTH, SOUTH ELMSALL, CONGLETON, ATTERCLIFFE, BALBY, BEAUCHIEF, BROOMHILL, BARNSLEY, CONISBROUGH, CHESTERFIELD, DONCASTER, DONCASTER NORTH, DRONFIELD, DINNINGTON, DARTON, ECCLESFIELD, GOLDTHORPE, HOLMEWOOD, HOYLAND, INTAKE, MEXBOROUGH, MOSBOROUGH, PENISTONE, RETFORD, ROTHERHAM, ROTHERHAM NORTH, RAWMARSH, CHELLASTON.

[PEW] Broadband: BE Planned Work 06/01/12 (Open)

2012-01-05T17:01:47Z

Be are performing some planned work on 06/01/12 - between the hours of 00:00 to 08:00 on the following exchanges:

BYFLEET, TIVERTON, BURGESS HILL, CHICHESTER, CRAWLEY, COSHAM, FAREHAM, GOSPORT, KEMPSTON, LUTON, NEWPORT PAGNELL, PENN, STEVENAGE, AVONMOUTH, HORLEY, HORNDEAN, HOVE, HAVANT, HAYLING ISLAND, HAYWARDS HEATH, KEMPTOWN, BEACONSFIELD, CHALFONT ST GILES, FLITWICK, HOLMER GREEN, WARMINSTER, BLUNSDON, BISHOPS CLEEVE, BISHOPSWORTH, BITTON, BARNWOOD, COMBE DOWN, CHURCHDOWN, CIRENCESTER, CLEVEDON, EASTON, STROUD, LLANDAFF, BLACKWATER, FARNBOROUGH, FLEET, IVER, LITTLEWICK GREEN, MARLOW, NEWBURY, THATCHAM, TWYFORD, WEST DRAYTON.

[PEW] Broadband: BE Planned Work 09/01/12 (Open)

2012-01-05T17:01:34Z

Be are performing some planned work on 09/01/12 - between the hours of 00:00 to 08:00 on the following exchanges:

KENILWORTH, OAKHAM, LEAMINGTON SPA, NUNEATON, RADFORD, RUGBY, STRATFORD ON AVON, TILE HILL, WALSGRAVE ON SOWE, WARWICK, BEDWORTH, BINLEY, COVENTRY, CHAPEL END, EARLSDON, EXHALL, FOLESHILL, BIRSTALL, BRAUNSTONE, CHAPEL BRAMPTON, COALVILLE, CORBY, DAVENTRY, DUSTON, EAST WIGSTON, EARL SHILTON, EVINGTON, GLENFIELD, GREAT OAKLEY, ATHERSTONE, AYLESTONE, BEAUMONT, BELGRAVE, HARDINGSTONE, HARROWDEN, HINCKLEY, KINGSTHORPE, KIRBY MUXLOE, KETTERING, LOUGHBOROUGH, LUTTERWORTH, MELTON MOWBRAY, MOULTON, MARKET HARBOROUGH, NARBOROUGH, NORTHAMPTON, KERESLEY.

[Minor] Email: Problems Sending to Hotmail

2012-01-05T08:38:02Z

We've have a few incidents this week where Hotmail have been blacklisting some of our IPs.

This has meant that some email has been bounced back.

We believe this is resolved, for now, so please do resend messages that have been bounced back,

Update 21 Dec 2011 11:10:03

We'll carry on monitoring the situation

[PEW] Mobile: Three maintenance - at risk period

2012-01-04T09:06:10Z

Three need to carry out emergency maintenance tomorrow morning (04/01/12) between 01:00-06:00

The mobile service should be considered at risk during this period. Sorry for the short notice!

[Info] Broadband: The future of LNS switches

2012-01-03T17:37:51Z

We currently operate two LNSs (the boxes that handle all of our broadband traffic). We have one live system and one as a hot standby backup. They are designe to handle a gigabit of internet traffic to customers, and as we saw when IOS5 was released they can even exceed that a little.

When we want to upgrade software, after testing on our test LNSs we load the new code in to the backup LNS. We then do an LNS switch - causing all lines to drop, one at a time, and reconnect a few seconds later to the new LNS.

During this month we are upgrading our core network and we now have four LNSs. The new ones are being tested now and once they are fully deployed we will operate with three live LNSs. These will share the load, but we are taking care to ensure a customer with line bonding is connected to just one LNS so that the bonding stays working. The 4th LNS is for upgrades.

What this means is that when we want to upgrade, we will upgrade the 4th LNS, and re-designate one of the other three as the new spare LNS. We will clear sessions from that LNS which will reconnect to the new LNS. This means an LNS switch will only affect a third of the customers. However, to upgrade the live LNSs to new code we have to do all three. It means an LNS switch will be in three distinct stages and may not even be done on the same day if this is just a routine upgrade.

However, during January (and possibly in to Februrary if BT drag their heels) we will need to do the existing LNS switches to move traffic to the right boxes while we actually deploy the new equipment and move BT links. This should be no more disruptive than a normal LNS switch, and we will try and schedule these for Saturday nights as usual.

Thank you all for your patience during this essential upgrade work.

[Minor] Broadband: Lines dropped: 100% 21CN-BRAS-RED4-LS-BAS

2012-01-03T15:34:38Z

Lines: 100% 21CN-BRAS-RED4-LS-BAS dropped at 2011-12-30 13:01:19
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 3 Jan 2012 15:34:38

This fault is linked to http://status.aaisp.net.uk/apost.cgi?incident=1324 but sadly BT have not been able to give us a reason for the outage.

[Minor] Broadband: Lines dropped: 56% 21CN-REGION-LS-BAS and 100% 21CN-BRAS-RED1-LS-BAS and 100% 21CN-BRAS-RED2-LS-BAS and 100% 21CN-BRAS-RED4-LS-BAS and 100% 21CN-BRAS-RED7-LS-BAS

2012-01-03T15:33:42Z

Lines: 56% 21CN-REGION-LS-BAS and 100% 21CN-BRAS-RED1-LS-BAS and 100% 21CN-BRAS-RED2-LS-BAS and 100% 21CN-BRAS-RED4-LS-BAS and 100% 21CN-BRAS-RED7-LS-BAS dropped at 2011-12-30 12:08:38
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 3 Jan 2012 15:33:42

BT have been unable to give us an explanation for this outage.

[Minor] Broadband: Lines dropped: 100% 21CN-BRAS-RED4-MQD

2012-01-03T15:32:26Z

Lines: 100% 21CN-BRAS-RED4-MQD dropped at 2011-12-27 23:01:20
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 27 Dec 2011 23:03:03 Lines: 100% 21CN-BRAS-RED4-MQD dropped again at 2011-12-27 23:01:20.

Update 3 Jan 2012 15:32:26

BT have been unable to give us an explanation for this outage.

[PEW] Broadband: LNS switch tonight (Open)

2012-01-03T10:28:11Z

One more LNS switch which should be after 5pm this evening.

This should finalise the work over Christmas.

We will de deploying new LNSs later this month we expect, and LNS switches like this will change how they work and be less impact than before once we have the new LNSs fully operational.

[Info] Broadband: Change to speed line on graphs

2012-01-01T13:17:49Z

The Constant Quality Monitoring (CQM) graphs for our broadband service includes a "speed line" which shows the line speed. This was set based on the BT BRAS rate which changed relatively infrequently. The speed line only shows the last 2 changes in speed.

The speed line is a useful reference, especially on the historical graph data.

We are now setting speeds based on the sync rate at connection where possible (21CN lines and BE lines). This can be several changes in speed on one graph losing the oldest speeds and causing confusion on the historical graphs.

We are making a change over the next few days which will make the speed line show on a per hour basis - showing the highest line in that hour. This will avoid losing historical data if there are many rate changes.

For most customers this will have no impact as the speed of your line will remain stable for a long period.

[PEW] Broadband: Network upgrade work December/January (Open)

2011-12-31T17:35:37Z

We may be carrying out a number of LNS switches during December and possibly in to January as part of preparations for network expanion and deployment of new LNSs.

We will aim to do LNS switches 5pm to 6pm on Satudays. We'll add details to this PEW and comment on irc as work progresses.

There may also be BGP router work, but this should be seamless and not affect any broadband sessions.

Update 23 Dec 2011 20:20:30

LNS restart just completed

Update 24 Dec 2011 10:50:42

There have been minor issues with graphs on BE lines since the LNS upgrade last night, and some BE lines failing. This has been worked around, and another LNS upgrade planned later today.

Update 24 Dec 2011 20:03:19

Another LNS switch now.

Update 28 Dec 2011 15:56:28

We expect to do another LNS switch this evening.

Update 31 Dec 2011 17:36:10

LNS update tonight - however we may switch back shortly as logs indicate the new code is not performing exactly as expected.

Update 31 Dec 2011 18:35:03

The unexpectec log output has now been analysed and is not a risk so a second LNS switch will not be needed tonight. We expect to do one tomorrow and probably Monday night as well.

Update 1 Jan 2012 17:09:04

LNS switch tonight and also core router upgrades which should be pretty seamless.

Update 2 Jan 2012 11:27:27

We expect one more LNS switch tonight, and that should be it for this week.