- ISP

How problems should be handled

nospam@example.com (Thomas) — Wed, 15 Apr 2009 12:31:30 +0200

You may wonder yourself what your ISP is doing or what his carrier is doing after a major outage. Let's use a real life example from Tuesday, April 14 2009. There was a fiber cut near Frankfurt, Germany.

First step:
You have to admit that a problem exists

Second step:
Inform your customer and identify the location of the problem. In our example it looks like:

The fiber break has been identified near motorway crossing BAB 5/67, 26.6 km from Lorsch in direction towards Frankfurt. The break is in a motorway crossing. Engineers are at the site for resolution.

Third step:
Give your customer more information.

The damage to the fibers has been caused due to drilling works at the site.Digging works will be started in 30 minutes. No ETR as yet.

Add some abbreviations in common use like ETR which means "Estimated Time of Repair". Looks more geeky

All other steps are simliar to the third step. Start fixing it and update your customer as much as possible.

Update 4

The digging works are still in progress to find the duct route. As this is a laborious task it takes some time. We will keep you updated and give an ETR as soon as possible.

Update 5

The digging works are still in progress. The duct route for the north side of the cable cut has been found.

Update 6

Our providers are still working on the issue. The splicing works will commence in another 2 hours. The approximate ETR to fix all live services is around 10:00 GMT. We will update you once the splicing works are started.

Important: Add GMT or Timezone information. Not every customer lives in the same timezone as you.

Update 7

Splicing works are still in progress. One of our sections from Frankfurt to Strasbourg is up restoring a lot of traffic. A lot of traffic yet remains impacted and we are working towards a speedy resolution .As of now due to the huge task ahead the ETR remains unchanged – 10 GMT

Update 8

A majority of the services have been restored as works are in progress. We are still waiting for the complete resolution and ETR remains unchanged.
We will keep you updated on the issue.

Update 9

Majority of services have now been restored. Splicing repairs still ongoing. Full restoration of services expected in an hour.

Update 10

Full restoration of services completed

Thats how it should be done and thats who interoute.com has handled it last night. Well done. It may look obvious but it's not the common way most ISP/Carrier keep their customer informed.

Early wake up call

nospam@example.com (Thomas) — Fri, 23 Dec 2005 07:38:00 +0100

I got an early wake up call at 6:15 o'clock yesterday morning. There was a major blackout at the IXEurope in Zurich (Switzerland). IXEurope 1 is one of the major Internet exchange point in Switzerland. A short abridgement from the official IXEurope statement:

IXEurope experienced a fault in service delivery on 22.12.2005 lasting from 05:58 to 06:30. This fault affected the power supply in our Zurich 1 Datacentre.
At 5:46 this morning (local time) the utility power supply failed across a large section of Zurich city including Hardstrasse and our DCs.
Zurich 2: - the UPS and backup generator operated as expected and Customer service was maintained
Zurich 1: - the generator failed to start and the UPS carried the load for 12 minutes until 5:58 and expired leaving all Zurich 1 Customers without power.
At 6:30 utility power was restored.

One of our new APC in Zurich also failed today but the impact wasn't that huge. Just the internal outgoing VOIP didn't work.
In days like this don't envy our network team. A stressful day.

Swinog 10

nospam@example.com (Thomas) — Thu, 21 Apr 2005 22:14:18 +0200

On wednesday I had the opportunity to participate in the Swinog 10 conference. It took place in the Altes Tramdepot in Bern (Switzerland).
It's a biannual meeting from network/system engineers to network/system engineers, ergo a good place to meet other ISP people. It was a chilling event. Good presentations and a lot of interesting discussions during the breaks and at the social event in the evening. I for myself had a very interesting discussion with Andre Oppermann from the FreeBSD project about a possible much faster routing implementation for FreeBSD.

Programm topics

Voice over IP by Cablecom
Urs Neidhardt, Ludwig Molnar (Cablecom)

VDSL & VoIP Deployment
Ramon Felder, Patton-Inalp

Criminal liability in relation to hash-links
Stephanie Müller (UNI ZH)

Service Provider Security Update
Nicolas Fischbach (COLT)

Traffic Engineering (TE)
Placi Flury (ETHZ)

SwissIX Update
Fredy Kuenzler (SwissIX)

Peering policy considerations
Andreas Diehl, TDC Switzerland

Slides and pcitures should be available soon.

Swiss"DoS"com a.k.a Swisscom ADSL blackout

nospam@example.com (Thomas) — Thu, 21 Apr 2005 16:01:59 +0200

Now, we know what the cause of the Swisscom ADSL blackout was. In a nutshell, they created an unintentional denial of service attack (DoS) on there own DSLAM

A quotation from the official Swisscom statement:

All services based on DSLAM technology were affected. The disruption was caused by a malfunction in the management network on the IPSS platform, whereby data packets were replicated. In turn, the routers and servers within the management network received the same data packets repeatedly, causing faults in the traffic control of the DSLAM network elements. The massive increase in the volume of management traffic triggered a “re-boot” of the DSLAM network, which actually caused the service disturbances you experienced.
…
The overall analysis and reconstruction of the interruption is not yet complete. Please be assured that Swisscom Fixnet Wholesale will take all steps necessary to insure that this situation cannot happen again.
…

ADSL blackout update

nospam@example.com (Thomas) — Sat, 16 Apr 2005 10:18:34 +0200

Swisscom closed the trouble ticket on Friday evening without giving any further information. Swisscom imposed gag order on all employees in this case!
The Swinog mailing list has a thread about theories and comments what could have been happened. You can read it here, here and here.

ADSL blackout

nospam@example.com (Thomas) — Thu, 14 Apr 2005 23:28:35 +0200

Today we had a national ADSL blackout from 10:00 to ~14:00. The Swisscom trouble ticket is not closed yet. But ADSL should work again for most users. Swisscom still has no clue what the cause of this blackout was. Additional information are expected for friday.
The situation was rather embarrassing for Swisscom because they overlooked this blackout. Carriers like SolNet pointed this problem out to Swisscom while they’ve lost adsl sessions.

Just for the records. You don’t see messages like this very often in Swisscom trouble tickets:

[Affected Locations:] Whole Switzerland

[Number of ADSL services:] All

[Work progress:] Still investigating in fault analysis

[Approximately expected repair time:] unknown

Another reason why I hate bluewin

nospam@example.com (Thomas) — Sun, 06 Feb 2005 22:17:48 +0100

Most people know that I’m definitly not a fan of bluewin. Neither from the job perspective nor from my private point of view.
Q: Why does bluewin excite me in such a way?
A: A view days ago I subscribed Teleclub. It’s very similiar to the Premiere pay tv in germany. Both companies share certain channels like the sport, series and crime channels. If you’ve a satellite dish then Teleclub give you almost full access to premiere programm. Almost!!
Of course my lovely SciFi channel is not in availabe. Neither with digital cable tv from Teleclub nor with the “special swiss” Premiere subscribtion from Teleclub (satellite dish).
Because this fsck** Bluewin has some exclusively rights for the “SciFi” and “13th Street” channel with their IPTV Streamit . This sucks. It only works with a bluewin adsl subscription and with Windows >=2000 because you need at least Internet Explorer 5.5.
And of course I’ve nothing from all those things.