iPhone 4 Benchmarking

I just realized that I now have 3 different generations of iPhones at my disposal, and decided it would be worth doing a little testing!

Read on for the tests and all the geeky number goodness!

The Contestants

Tonight, I’m testing three phones:

iPhone 3G 16GB
iPhone 3GS 16GB
iPhone 4 32GB

Both of the older generation iPhones have been updated to iOS 4 (the initial release), and the iPhone 4 is running the software it had out of the box. None of them have (ever) been jailbroken, and all three are loaded with a similar compliment of Apps (the 3GS has a slightly different set, but they are essentially identical aside from the 3GS missing a handful of games that my wife isn’t interested in playing).

Boot Up

Timing the length of time it takes to boot was more challenging that I’d hoped. There is some variability due to human error using the stopwatch, but I repeated each test three times to try to minimize the impact of that. I’m reporting the middle number for each phone. However, in no cases did the test results vary by more than 2-3 tenths of a second.

This is timed, as best as I could, from the instant of the release of the Sleep Button until the Lock Screen appears (with the animation finished). I rebooted each phone once before starting the test. I find that sometimes my phone will take longer to boot up when it is “acting up”, perhaps as a result of doing some sort of cleanup on boot, so I wanted to eliminate that from influencing the test.

iPhone 3G: 1:10.3
iPhone 3GS: 0:33.5
iPhone 4: 0:34.8

Now this is a very interesting result! The iPhone 3GS consistently beat the brand new iPhone 4 in the bootup test. It wasn’t by much (less than 2 seconds), but the win was repeated with every try (I even tested the 3GS four times just to make sure).

Perhaps it takes longer to check the additional Flash memory or RAM at boot? The iPhone 4 I’m testing has double the Flash of the 3GS model I’m testing against, and of course has 512MB of RAM instead of 256MB. Either way, for some reason, my 3GS boots just a hair quicker!

Navigon Loading

Next, I tested loading Navigon Mobile Navigator (North America), version 1.5.1. I rebooted each phone before starting. I timed loading Navigion from the moment I tapped the screen until the Navigon Home screen appeared (I was not actively navigating at the time, as that would be dangerous).

I repeated this test three times for each phone. However, I did not reboot in between each of the three tries. Instead, I launched Peggle and let it fully load. Then I exited to the home screen and launched Navigon from there (I did not use the Multitasking menu in any case). This produced some interesting results.

iPhone 3G Try 1: 0:19.0
iPhone 3G Try 2: 0:16.8
iPhone 3G Try 3: 0:16.8

iPhone 3GS Try 1: 0:07.8
iPhone 3GS Try 2: 0:04.5
iPhone 3GS Try 3: 0:04.4

iPhone 4 Try 1: 0:06.4
iPhone 4 Try 2: 0:03.4
iPhone 4 Try 3: 0:03.6

With each phone, the first launch of Navigon after a fresh reboot was slower than subsequent launches. After noticing this initially, I wanted to try to make the phones use up some of their available RAM, so decided to add in the Peggle launch in between and repeated the tests from scratch. The initial “round” of testing, with no Peggle launches in between produced similar numbers (though a few of the subsequent launches were even quicker).

Peggle

I wanted to test PopCap’s fantastic Peggle game next, but I haven’t really been able to finish the test. Peggle actually launches so quickly on my iPhone 4, that it is nearly impossible to time. I’m going to work further on collecting numbers and update this article as I collect more data.

Update: Well, I’m getting some Peggle numbers now. To try to make it more consistent, I’m recording from the initial “tap” to launch it, until the ball finishes bouncing across the screen in the opening animation (just as the ball exits from the display).

This time, I’m again testing three times and reporting the middle number. I’m rebooting between each try, and I am using Peggle version 1.1.

iPhone 3G: 0:18.5
iPhone 3G Relaunch (version 1.1): 16.4

iPhone 3GS: 0:11.2
iPhone 3GS Relaunch (before version 1.1): 0:09.7
iPhone 3GS Relaunch: Not able to test, instantly resumes.

iPhone 4: 0:10.3
iPhone 4 Relaunch: Not able to test, instantly resumes.

These results are also a bit interesting.

I chose to time until the ball animation bounces across the screen, after Peggle loads up, so that I could actually report an accurate number for the iPhone 4. This is because on the iPhone 4, Peggle does not ever even show the normal Peggle “loading” screen with the funny witticisms. Immediately after the Pop Cap logo, the Tap to Play button is visible! There is not even a second of loading of any kind. Therefore, it was difficult to test these directly unless I timed it until the ball finished it’s animation.

The iPhone 3GS does still show the loading screen, of course, but only for the briefest of seconds, as you can see from the results.

I was also able to test relaunching the application, somewhat. With Peggle version 1.1 and iOS 4 on the iPhone 4 and 3GS, the application effectively instantaneously relaunches to the “Tap to Play” screen (there is no PopCap logo and no loading screen at all). I’m not able to repeat my test conditions, though, because the ball never actually bounces across the screen. It resumes where it left off!

I was, however, able to test relaunching Peggle pre-1.1 using my wife’s 3GS (she hadn’t updated her copy of Peggle yet, and so I did this test before I updated it for the rest of the testing). I also included numbers for relaunching Peggle 1.1 on my old 3G phone. This shows quite clearly the benefit of the iOS 4 Saved State and Fast Application Switching APIs. It took the 3GS from a relaunch time of 9.7 seconds to truly instant-on.

Plants vs. Zombies

Again, I tested each phone using PvZ version 1.1 three times, rebooting between each try. Like with Peggle, PvZ 1.1 allows the iPhone 4 and 3GS to instantly resume relaunches using iOS 4′s Saved State APIs, so relaunching was not specifically tested for this game.

I timed using a stopwatch from the second the PvZ icon was tapped, until the “Tap to Start” button appeared.

iPhone 3G: 29.6
iPhone 3GS: 11.6
iPhone 4: 8.4

This game showed a fairly dramatic difference across the board. The iPhone 3G was actually well over 3 times slower than the new iPhone 4 at loading Plants vs Zombies.

Mobile Safari Web Browsing Test

Tech Report Front Page Load Time

I tested this by loading the front page of Tech Report and timing it with a stopwatch. I made sure all tabs were closed in Mobile Safari, and then rebooted each phone before I began the test. Then I loaded Yahoo.com (to give each phone an equal starting point) and then loaded the Tech Report home page from a bookmark. The time was recorded from the moment I tapped the bookmark until the blue bar finished and disappeared.

All tests were done using a WiFi connection (a Linksys WRT54GL 802.11g router), from the same spot in the house in the same sitting. The tests were all done with the phones in portrait mode, set down on the table. I repeated loading these two pages 4 times each, first Yahoo, then Tech Report, which was timed, then repeat. Mobile Safari was not closed in between. The results are listed in order, and in seconds:

iPhone 3G: 30.9, 17.6, 18.4, 22.1
iPhone 3GS: 9.8, 6.3, 5.8, 6.0
iPhone 4: 9.6, 6.2, 5.8, 5.9

As you can see, Mobile Safari does make good use of caching to speed up subsequent page loads. So the first load of each page took substantially longer than subsequent loads, despite loading Yahoo in between.

The iPhone 4 is not substantially faster than the iPhone 3GS in this test. I’d call it basically within the margin of error, with a slight edge going to the iPhone 4 (but only very slight). However, I can say that the iPhone 4 responds much better to having multiple tabs open in Mobile Safari simultaneously.

I did some informal testing on how many tabs you could open while keeping Mobile Safari responsive. I didn’t spend as much time making the test perfectly “fair” in every single way (rebooting before and making sure the tab order matched), because I figured that every person’s definition of responsive would be different anyway. Either way, I found that Mobile Safari remains absolutely responsive on the iPhone 4 all 8 possible tabs open with “heavy” tech enthusiast sites loaded (full Engadget home page, AnandTech, HardOCP, Tech Report, etc). It also seems to hold onto its cache of previous page loads, even when using multiple tabs and switching back and forth.

The iPhone 3GS doesn’t hold up quite so well, though still darn good compared to the 3G I’ve been used to dealing with daily. On the 3GS, I’d often see sluggishness after having 5-6 tabs open simultaneously, though this would depend heavily on which sites specifically were used. I didn’t play with the caching as much on the 3GS (my wife wanted her phone back), but it seemed to hit a limit at around the same point the sluggishness kicked in, which makes sense. I imagine this difference is due almost entirely to the additional 256MB of RAM in the new offering from Apple.

Of course, either of these results are a dramatic improvement over the iPhone 3G! Just 3-4 “heavy” sites open in tabs on that phone would bring Safari to a crawl, and I’d often experience crashes of Safari with more than 4 tabs open at a time.

Conclusions

These tests do pretty fairly illustrate the main differences I’m seeing in speed between the different iPhone models with regular daily usage. The iPhone 4 is marginally faster than the 3GS in most tasks, but not by much. There are a few instances where they are effectively tied (like the web browsing test and the bootup test), and there are a few tasks, especially loads of larger applications, where the iPhone 4 “feels” substantially faster. But both of these types of tasks are fairly rare, and Apple’s new flagship generally just feels “a little faster” than the 3GS. However, the difference shines in that the iPhone 4 will basically “take” anything you throw at it without slowing down. When operating conditions are not “perfect”, the iPhone can handle things where the 3GS might stutter (and the 3G would melt-down).

I’m talking things like listening to a large video podcast using the iPod app in the background while running Navigon in the foreground, and multitasking back and forth to Beejive IM. The iPhone 4 handles these scenarios as though nothing else is being done on the phone at all, while the 3GS can feel sluggish at times, hesitate and stutter for moments, and seems to crash a bit more often when doing “heavy” multitasking.

Frankly, it has to be primarily the difference in RAM capacity. The iPhone 4 just isn’t hitting RAM limits as often at all. This would make sense because app developers have been targeting a 256MB phone all along, and now it has 512MB to play with! If you can remember how it felt to use Windows XP with 512MB of RAM back when that was a “standard” amount, and then to switch to an “enthusiast class” machine with 1GB of RAM or more? That’s just how this difference feels.

Otherwise, the raw speed improvements feel like something akin to upgrading from one CPU to a newer one in the same family with a higher clock speed (or maybe a newer die spin). This is a far cry from the dramatic improvements between the iPhone 3G and the 3GS (and certainly the jump I made from the 3G to the 4 directly), which felt more like going from an Atom CPU to a Core 2 Quad! But the improvement is tangible and measurable.