so you want to review an Android device...

[Tim Murray]

Okay, so you're a reviewer, you want to know what an Android device is doing. You think, oh, I'll use random tool X that runs on the device and see what it says about the device's clockspeeds at various points in time.

No, stop. Just stop. You're wasting your time, you'll be wasting my time when I read your review, and nobody will be happy.

Instead, use the tool I do when I'm working on Android performance: developer.android.com/tools/help/systrace.html

"What's systrace," you ask? Systrace is basically a cool wrapper around both Linux ftrace (kernel-level tracing events) and a user-mode component called atrace that allow you to get a TOTALLY SWEET timeline view of everything that is happening on the system. Okay, maybe not everything, but we keep adding more tracepoints so we can figure things out (and eventually those make their way to you).

So get Android Studio and use it. Or better yet, check out aosp_master and use it from external/chromium-trace, because that is probably a newer version than what it is in Android Studio.

Once you have it, plug in your Android device. Open a terminal, and try

./systrace.py sched freq idle input -b 8192 -t 10 -o hats.html

Hit enter, tap around on the screen for ten seconds or so, then when it says that bats.html has been written, open that file up. Whoa, it's everything that was running on every core (sched), the clockspeed of every core (freq), the cstate of every core (idle), and markers for when it received input events (input)! You can figure a ton of stuff out by looking at these traces in the right environments, and they mean way more than looking at CPU frequencies in isolation (as CPU frequencies don't matter particularly much because if that CPU is deep idling, then it's not drawing that much power). Why is this better than something that just samples the clockspeeds? Because it's not sampling. Every time something is scheduled on a core, or the clockspeed changes, or the idle state changes, the kernel writes an event to the log. You won't miss anything.

If you want more excessive detail, workq is a good one, as it will tell you what the kernel worker threads are doing, although that one requires root.

If you're really snazzy, you can figure out how to kick off systrace, disconnect the device, then write the log out later just to make sure that there are no thermal impacts from charging or that the governor settings didn't change while the device was charging or things like that.

Also look at governor settings at /sys/devices/system/cpu (especially the cpufreq subdirectory). Read up on the interactive governor in the kernel documentation. Look at how the governors are configured in the init.rc files on the device (that's generally where they're set up, FYI). If you're on 5X/6P, open /dev/cpuset and marvel at the brave new world of devices that actually try to explicitly support big.LITTLE.

Pretty much everything you need to figure out how an Android device is there and doesn't require source. Source will help (and you should look at kernel source if you're interested at all in architectural quirks), but systrace and the various sysfs settings make almost everything very, very obvious.

love,
yr pal that wants more interesting mobile device reviews

 

[Nebuchadnezzar]

Also look at governor settings at /sys/devices/system/cpu (especially the cpufreq subdirectory). Read up on the interactive governor in the kernel documentation. Look at how the governors are configured in the init.rc files on the device (that's generally where they're set up, FYI). If you're on 5X/6P, open /dev/cpuset and marvel at the brave new world of devices that actually try to explicitly support big.LITTLE.

That's nice and dandy for Nexus devices but basically every other vendor does their own stuff. Interactive documentation is useless because every SoC vendor actually goes in to change it, and if you really want some in-depth coverage you do need source code to go in to see what they did. init.rc is usually read protected so unless you have root that also is not a reliable way to see anything. Vendors deploy their own user-space power management which for example alter governor configurables depending on context and scenario, again that's all closed source most of the time (Yes you can go ahead and decompile the .jar's within the ROM and try to figure it out).

I mean systrace is very nice for developers or if you really want to look at stuff such as maybe GPU driver time, but one has to keep in mind that we still need to write something that is actually presentable to the reader. In that regard, is it really that more useful than for example just logging the time-in-states like for example I do here? I also feel much less worried about whatever overhead tracing might have and I can do it over prolonged periods all without losing information.

 

[orangpelupa]

but doesnt that make the review cannot be compared to other review that commonly use something like geekbench or 3dmark?

 

[BRiT]

but doesnt that make the review cannot be compared to other review that commonly use something like geekbench or 3dmark?

Why would you prefer something that is obviously inaccurate to something that is precise?

You might as well review them using a randomly generated number.

 

[orangpelupa]

isn't comparability also important? for example, because reviewer keep using the same tool, I can easily see that the new Snapdragon 600 family have slower GPU than the old 801. I also can see my 2-3 years old Xperia CPU still faster than new 100-200 dollar phones. This allows me to wait-and-see for newer phones to get more "leap".

 

[Ailuros]

All a reviewer needs for that is to use a representative amount of past devices/SoCs to give you that comparability, albeit it doesn't take a crystal ball really to acknowledge that a former high end phone will still hold its own against mainstream or low end smartphones appearing today.

Other than that the above proposed methodology isn't for everyone either, since its rather for those few reviewers that really know what they're doing and respect a wee bit more their work, than any random bloke on the internet that thinks he's an online "journalist" or Lord help a "reviewer".

 

[zed]

Does anyone really care about phone CPU/GPU speed?, OK I dont play games (though most best selling games are simple stuff on the phone so I assume this aint too much of an issue even here) but I've never owned a single phone where I've gone 'I wish opening/running/closing this app was quicker'.
OK I've never owned an el cheapo smartphone, but CPU speed has always been a nonissue to me, battery life OTOH now thats important
Not knocking what you're doing Tim btw

 

[Ailuros]

Assume a smartphone does well in synthetic benchmarks (because somebody bothered to optimize for it for obvious reasons....), but when you open not so popular application N you get annoying stutters f.e. Obviously in such a case it's not the hw that is at fault but the underlying sw, but the catch in the given case is that chances are high that the "let's run Geekbench & 3dmark"-reviewer won't catch such a shortcoming, while with Tim's scenario chances are by a lot higher that they will.

Reviews should NOT be about sterile performance numbers, but a combination of factors. Since power is the most important factor in a mobile device reviewers for mobile devices should concentrate more on sustainable performance or else a fine balance between performance and power consumption. If power wouldn't be a headache you could run a 4GHz core i7 in your smartphone today and charge it only once a week.

In that regard no I don't care about CPU or GPU performance per se, but I do care how stable, how fast and with how much power each device can finish the tasks thrown at it. And for that synthetic benchmarks will not give you the right answer; Tim's suggestion will.

Last but not least: a geek can finally understand how big.LITTLE really works, see its advantages and its shortcomings. Finding shortcomings and pointing them out is in the longrun for the benefit of everyone since it'll force manufacturers to optimize far more carefully then just for synthetic applications.