AT&T Service In Bar Harbor, Maine Is A Steaming Pile of Fail (For Now)

Since around July 1st, my wife has been having issues with her iPhone 3GS receiving incoming calls. In the vast majority of cases, the phone just will not ring. From the caller’s perspective it rings and rings, and then eventually goes to voicemail, but the phone itself never acknowledges the incoming call in any way (no missed call indicator, no ring, no vibrate, nothing at all). We’ve tested it repeatedly with the phone sitting in front of us, with plenty of signal and no processes running in the background that might be interfering. We’ve tested both by calling the phone from a landline and from my iPhone 4. In all cases, the iPhone 3GS is able to place outgoing calls without issue.

She had also been complaining that the data on her phone was very slow, and often not working at all, and we noticed that her text messages were sporadically not working properly (often not coming in, or arriving hours late).

Any of this sound familiar? Well, we’ve now figured it out, and I’m going to tell you all about it. Read on for the sordid story, and even a little sliver of good news for us in Downeast Maine.

So, she complained about these issues for a while before I spent any time investigating. At first I assumed it was just user error. We are currently stuck with only EDGE data service up here, so the network being slow and timing out is nothing new. Plus, since EDGE can’t make calls and use data simultaneously, it is possible that the phone will block incoming calls occasionally while it is doing things like checking email or loading a web page. The missing text messages were odd, but she wasn’t able to give me any firm examples at first.

But then one day while we were hanging out in Bar Harbor with my parents visiting, I wasn’t able to call her. I knew roughly where she was, and that she should have plenty of service, and I called and called and called and just kept getting her voicemail. I must have called her 25 times over the course of two hours (we really needed her, as we were stranded waiting for her on a park bench)! There’s no way that she was using EDGE data for two hours solid while hanging out with my nephew and shopping in Bar Harbor! Either she had the ringer turned off or something was really not working. I’d also noticed during this time that the EDGE data on MY new iPhone 4 was not working properly either. While stuck on that park bench, I had plenty of time to try it out. I was completely unable to check my email, no web pages would open (even very simple text only ones on my own site), and I couldn’t get my voicemails to download. Regarding the Visual Voicemails, it was not that the Voicemail system knew it was broken (it was not showing the “Call Voicemail” button). It appeared to be working, but when you tried to play a voicemail, the little “spinner” would just spin and spin but the voicemail would never load or play.

Now, this happened just around the 4th of July weekend and Bar Harbor was absolutely overloaded with tourists, so I wrote it off to an overloaded AT&T network from the sudden rush of people.

But I still couldn’t get her to answer the phone. When she finally called me back, I had her check immediately. The ringer switch was turned on. The phone showed no missed calls, and though it did list my voicemails that I’d left, it wasn’t able to download or play them via the Visual Voicemail feature.

She came to meet us, and then I started investigating. I called to her phone from a variety of different phones that day and over the next few days. Occasionally, her phone would ring, but the vast majority of the time it would not ring at all or show any indication of an incoming call (just as described above). Each time, it would ring and ring (on the “calling” end) and then go to voicemail. Roughly 10% of the time, I could get it to ring, but even then, it would be very “slow”. By “slow” I mean that from the “calling end’s” perspective, you would hear 4-5 rings on the line before the phone itself would actually start ringing. This meant that those few times when the phone did actually ring, you’d only have 1-2 “rings” before it would go to voicemail. In testing, I did notice one odd thing though: if I rebooted the phone, it would almost always ring right away. However, if I then immediately hung up and tried it again, it would almost always stay silent.

At this point, I began to check my own phone. I noticed the same “slow ring” phenomena on my new iPhone 4, but in every test I performed during those days, it always eventually rang at least a couple of times before going to voicemail. So, at this point, I spent some time digging around online and found a few older Apple Discussion forum threads where people had reported similar issues. In most cases, it was just that they had left the ringer switch set to silent, but there were a few that matched the behavior of her 3GS. Those people said that they tried resetting the phone and then if that didn’t work, they returned it for new hardware. So, I set about trying some resetting of the phone over the next couple of weeks. I tried:

1. Resetting the phone normally and restoring from backup.
2. Resetting the phone normally and NOT restoring from backup.
3. Resetting the phone using a manually downloaded and selected copy of the current iOS 4 ISPW file, and not restoring from backup.

Nothing helped. The issue seemed to come and go from time to time, but it was pretty consistent. I was convinced that the phone was broken. We discussed it and decided that we’d take the phone down to the Portland Apple Store during a trip we had planned for the August 1st weekend.

In the meantime, I began to notice other issues with my own phone though. My visual voicemail had never really started working again. It would always list the new messages, but I could almost never get them to properly download. Sometimes I’d get the first few seconds, and then nothing, but more often they’d never download at all. Also, I started to notice that text messages seemed to occasionally not work, particularly longer ones. These turned out to be clues, but I didn’t realize it yet.

So I talked to a few friends who also had AT&T in the area. Turns out, ALL of them were having problems too. No one with an iPhone could get their Visual Voicemail to work properly at all, and EDGE data was either not working at all, or only working sporadically. Few people had noticed missed incoming calls to the degree I’d established with Jenn’s phone, but no one had really been testing either. I also spoke to two different people who had standard “feature phones” rather than iPhones who had noticed issues with missing incoming and outgoing text messages. Something was almost certainly going on with the network. And then, a few days later on July 25th, it happened to me. I got a voicemail, which I couldn’t listen to, but my phone had never rung (despite being in an area with full service the entire time). I tried (in vain) to listen to the voicemail via the Visual Voicemail system, and then set about testing my phone systematically.

That day, I called the phone 65 times over the course of 4 hours from both my landline and from another cell phone, and I recorded the details about each attempt. Of those 65 tries, it actually rang 8 times. In the interim, I had tried all the same steps as I tried previously with my wife’s phone (rebooting, resetting, etc). I made sure I wasn’t using any data applications by manually closing those apps via the multitasking bar (and by rebooting the phone). I tried with and without it connected to a WiFi hotspot. I tried with and without the 3G switch enabled.

So finally, I called and reported the issue to AT&T. The representative I finally got after a long hold time (and one disconnect) was quite kind and helpful. She had me turn off the phone, then she “reset” something in the network remotely, and then had me turn the phone back on. Once it was back up, she placed a test call to the phone. It seemed to hesitate, but it rang! I was happy, but skeptical. I asked if she wouldn’t mind trying again in a few seconds (since I did have 8 successful attempts in my tests, but rarely 2-3 in a row). She did. The next time she put me on hold to test it, the phone never rang. She came back a few minutes later and said she’d gotten my voicemail.

At this point she decided to elevate my issue to the “real” Technical Support team, and she called over to transfer me to this new representative. And we waited… And waited… And WAITED. In the end, I waited on hold for about 35 minutes. Luckily I just put the speaker phone on and went about my business in the house. Eventually, she got me through to the tech support rep and introduced me. I explained my story (again), and the new rep listened to my thorough explanation of all the things I’d tried to resolve the issue. I explained how I’d initially assumed that we had bad hardware, but then since it was happening on my iPhone too, this seemed unlikely. She agreed. And then she said, “Did all of this seem to start around July 1st?”

Up until then, I’d never really put a firm finger on the date. From my wife’s point of view, it started “around” the time when I’d installed iOS 4 on her phone (which was a day or two after the initial public release). But that wasn’t quite right… She actually started complaining about a week later. I told her yes. And she said “I thought so”. She explained that there was already an open support ticket in Hancock County for data problems. There wasn’t a resolution, yet, but that technicians were working on the issue. And then she explained something else…

On July 1st they’d started work on the local network for the upgrade to 3G service, which was being “lit up” around August 24th or 25th. She said she had no specific knowledge that this caused the issue, but that the timing made sense. And, of course, if you figure that they’re upgrading the network and the equipment on the local towers, that it would make sense that this work might temporarily reduce the available resources on the existing towers (as they reuse older equipment and take it out of service). Simultaneously, eight gazillion tourists showed up on the island for their summer vacations (Bar Harbor really starts “hopping” on the 4th of July weekend). It would make sense that a half crippled and partially upgraded network was just failing under the strain. However, she did give me some good news:

1. We were getting 3G on or around August 25th!
2. We were also getting two new towers in the area, which would go live sometime this fall.
3. It would make sense that the issues would resolve themselves when the 3G service goes live.

She provided me with a ticket number for the support issue, and noted on my account that I was suffering these effects. She recommended that I give it some time and then call customer service to request a credit on my bill for the time where I was unable to fully utilize my service. She did mention that she didn’t see anything in the ticket about the issues impacting voice service, only EDGE data, so that was odd, but she said that the issues were almost certainly related.

After I hung up, I figured it out. It was Visual Voicemail “causing” the incoming call issue all along.

The issue is that the iPhone uses cellular data to enable Visual Voicemail. I knew that you couldn’t use EDGE data and voice calls simultaneously, but I’d tested with the phone connected to Wifi, and tested it after a fresh restore with NO setup done on the phone at all (so there was no email configured, no apps installed, and certainly nothing running in the background). However, as soon as I “activated” the phone via iTunes, it would enable my Visual Voicemail.

Remember how I said all those messages would show up but never download? Well, those failed downloads were being done over EDGE. Voicemail is the one thing on the phone that will NOT use WiFi, even if you’re connected to an access point. Voicemail (and MMS) downloads ALWAYS happen via the cellular data network. It also explained the random missing text messages. Longer messages are automatically sent via MMS on the iPhone, and MMS is transported via the cellular data network. Since EDGE was randomly broken, you’d sporadically miss text messages as EDGE decided to crap out.

Prior to iOS 4, this wasn’t an issue. If the EDGE network was being a pain, and you couldn’t download a voicemail, as soon as you closed the Phone app it would stop “trying”. However, with iOS 4 and the new multitasking features, these failing voicemail download sessions could continue for quite some time in the background. The only way to get them to stop was to disable Cellular Data, which would turn off Visual Voicemail completely.

I tested with Cellular Data turned off, and the issue immediately and completely went away. The phone would now ring immediately (not delayed by 3-4 rings anymore), and would ring consistently. Of course, this means I couldn’t use my $30 per month data plan at all, which wasn’t too nice (especially since we had two phones so the price is $60 per month), but at least I could use my phone as a phone!

The last bit of good news is that the 3G upgrade should actually solve this issue. AT&T’s 3G service does allow simultaneous voice and data connections, so even if a voicemail gets “stuck” in the future due to backhaul issues (or something), it shouldn’t block my access to incoming phone calls, so long as my phone is connected via 3G. One would also assume that once the work is done on the network, that they will be back to full capacity (and the tourists start to leave within a week or two of August 25th anyway).

So… I discovered that AT&T is indeed a steaming pile of FAIL right now in Bar Harbor, Maine and most of Hancock county. But, they’re working on it. And I’m going to give them until the end of the month to get it straightened out.

After that, though, if it isn’t fixed I’m quitting, and they’re not going to get an ETF fee out of me when I do. Verizon will have the iPhone soon enough, I’m pretty darn sure.