Revisiting Linux Part 1: A Look at Ubuntu 8.04
by Ryan Smith on August 26, 2009 12:00 AM EST- Posted in
- Linux
CPU Benchmarks
We’ll start our short look at Ubuntu’s performance with our CPU intensive benchmarks. Up first is SuperPi, a single-threaded pi-calculating benchmark. Here we time how long it takes to calculate Pi to 1 million digits.
We ran this test several times more than usual just to make sure we weren’t seeing any kind of error. The Linux version of SuperPi really is about 30% faster than the Vista version. Keep this in mind, this will be an important point later.
Meanwhile the situation for LAME is inverted. Vista outscores the Linux version by nearly 20%.
Using the cross-platform X264-based Handbreak for our video encoding test, Vista once again pulls ahead of Linux.
Once more Vista is ahead by a large margin.
From what we can tell, there’s little-if-any innate performance advantage to Vista or Linux in these benchmarks. Our working theory is that the performance difference comes down to the compiler used. Many Linux applications are compiled with GCC, while for Windows it’s either the Visual Studio compiler, or Intel’s own compiler (which is also available for Linux). There’s also a matter of compiler settings, as we saw in our quick breakout of Firefox benchmarks.
Meanwhile SuperPi uses a lot of hand-rolled code, although we’re still not sure why it’s outperforming Vista on Linux by as much as it is.
To shed a little more light on this idea of compiler performance, we have a few benchmarks of Windows application performance under Ubuntu through Wine.
Here we see a most amazing thing: Ubuntu is outperforming Windows at running Windows applications! As we’ve removed the influence of compilers the Photoshop results are particularly interesting. From what we can tell it’s normally as fast under Linux as it is Vista, however there seems to be a short gap of low-CPU usage when running it under Vista that doesn’t occur when running it under Ubuntu. As a result Ubuntu finishes a few seconds earlier.
There are a number of conditional cases that mean that applications running under Wine don’t always match or beat Windows performance, but in our tests there’s no performance hit to using Wine to run Windows applications.
These results also lend a great deal of support to the idea that there’s a significant difference in performance between the two operating systems due to their compilers. This goes particularly for the LAME benchmark, where the performance gap melts away under Wine. This is something we’re going to have to look in to in the future.
Facebook
Twitter
RSS
195 Comments
View All Comments
ParadigmComplex - Wednesday, August 26, 2009 - link
I concur - while most of the article is quite good, Ryan really seemed to have missed quite a bit here. His analysis of it seemed rather limited if not misleading.Not everything *has* to be a package - I have various scripts strewn around, along with Firefox 3.6a1 and a bunch of other things without having them organized properly as .deb's with APT. The packaging system is convenient if you want to use it, but it is not required.
Additionally, Ryan made it seem as though everything must be installed through Synaptic or Add/Remove and that there where no stand-alone installers along the lines of Windows' .msi files. It's quite easy on Ubuntu to download a .deb file and double-click on it. In fact, it's much simpler then Windows' .msi files - there's no questions or hitting next. You just give it your password and it takes care of everything else.
The one area I agree with Ryan is that there needs to be an standardized, easy, GUI fashion to add a repository (both the address and key) to APT. I have no problems with doing things like >>/etc/apt/sources.list, but I could see where others may. I suspect this could be done through a .deb, but I've never seen it done that way.
Ryan Smith - Wednesday, August 26, 2009 - link
Something I've been fishing for here and have not yet seen much of are requests for benchmarks. Part 2 is going to be 9.04 (no multi-Linux comparisons at this point, maybe later) and I'd like to know what you guys would like to see with respect to performance.We'll have a new i7 rig for 9.04, so I'll be taking a look at a few system level things (e.g. startup time) along side a look at what's new between 8.04 and 9.04. I'll also be taking a quick look at some compiler stuff and GPU-related items.
Beyond that the board is open. Are there specific performance areas or applications that you guys would like to see(no laptops, please)? We're open to suggestions, so here's your chance to help us build a testing suite for future Linux articles.
cyriene - Monday, August 31, 2009 - link
I'd like to see differences between PPD in World Community Grid between various Windows and Linux distros.I never really see AT talk about WCG or other distributed computing, but I figure if I'm gonna OC the crap out of my cpu, I might as well put it to good use.
Eeqmcsq - Thursday, August 27, 2009 - link
Cross platform testing is pretty difficult, considering there are a multitude of different apps to accomplish the same task, some faster, some slower. And then there's the compiler optimizations for the same cross platform app as you mentioned in the article. However, I understand that from an end user's perspective, it's all about doing a "task". So just to throw a few ideas out there involving cross platform apps so that it's a bit more comparable...- Image or video conversion using GIMP or vlc.
- Spreadsheet calculations using the Open Office Calc app.
- Performance tests through VMware.
- How about something java related? Java compiling, a java pi calculator app, or some other java single/multi threaded test app.
- Perl or python script tests.
- FTP transfer tests.
- 802.11 b/g/whatever wireless transfer tests.
- Hard drive tests, AHCI. (I read bad things about AMD's AHCI drivers, and that Windows AHCI drivers were OK. What about in Ubuntu?)
- Linux software RAID vs "motherboard RAID", which is usually only available to Windows.
- Linux fat32/NTFS format time/read/write tests vs Windows
- Wasn't there some thread scheduling issues with AMD Cool and Quiet and Windows that dropped AMD's performance? What about in Linux?
While I'm brainstorming, here's a few tests that's more about functionality and features than performance:
- bluetooth connectivity, ip over bluetooth, etc
- printing, detecting local/network printers
- connected accessories, such as ipods, flash drives, or cameras through usb or firewire
- detecting computers on the local network (Places -> Network)
- multi channel audio, multi monitor video
Just for fun:
- Find a Windows virus/trojan/whatever that deletes files, unleash it in Ubuntu through Wine, see how much damage it does.
Veerappan - Thursday, August 27, 2009 - link
I know you've said in previous comments that using Phoronix Test Suite for benchmarking different OSes (e.g. Ubuntu vs Vista) won't work because PTS doesn't install in Windows, but you could probably use a list of the available tests/suites in PTS as a place to get ideas for commonly available programs in Windows/OSX/Linux.I'm pretty sure that Unigine's Tropics/Sanctuary demos/benchmarks are available in Windows, so those could bench OpenGL/graphics.
Maybe either UT2004 or some version of Quake or Doom 3 would work as gaming benchmarks. It's all going to be OpenGL stuff, but it's better than nothing. You could also do WoW in Wine, or Eve under Wine to test some game compatibility/performance.
Once you get VDPAU working, I'd love to see CPU usage comparisons between windows/linux for media playback of H.264 videos. And also, I guess, a test without VDPAU/VAAPI working. Too bad for ATI that XvBA isn't supported yet... might be worth mentioning that in the article.
You also might want to search around for any available OpenCL demos which exist. Nvidia's newest Linux driver supports OpenCL, so that might give you a common platform/API for programs to test.
I've often felt that DVD Shrink runs faster in Wine than in Windows, so the time to run a DVD rip would be nice, but might have legal implications.
Some sort of multitasking benchmark would be nice, but I'm not sure how you'd do it. Yeah, I can see a way of writing a shell script to automatically launch multiple benchmarks simultaneously (and time them all), but the windows side is a little tougher to me (some sort of batch script might work). Web Browsing + File Copy + Transcoding a video (or similar).
Ooh... Encryption performance benchmarks might be nice. Either a test of how many PGP/GPG signs per second, or copying data between a normal disk partition, and a TrueCrypt partition. The TrueCrypt file copy test would be interesting to me, and would cover both encryption performance and some disk I/O.
One last suggestion: Folding@Home benchmarks. F@H is available at least in CPU-driven form in Windows/Linux, and there's F@H benchmark methodologies already developed by other sites (e.g. techreport.com's CPU articles).
Well, that's enough for now. Take or leave the suggestions as you see fit.
haplo602 - Thursday, August 27, 2009 - link
you are out of luck here ... linux does not compare to windows because they are both different architectures. you already did what you could in the article.especialy in a binary distribution like Ubuntu, compilation speed tests are meaningless (but Gentoo folks would kiss your feet for that).
boot up times are also not usefull. the init scripts and even init mechanisms are different from distro to distro.
compression/filesystem benchmarks are half way usable. on windows you only have NTFS these days. on linux there are like 20 different filesystems that you can use (ext3/4, reiser, jfs and xfs are the most used. also quite many distros offer lvm/evms backends or software raid.
I do not think there is much benchmarking you can do that will help in linux vs windows, even ubuntu vs windows because the same benchmars will differ between ubuntu versions.
the only usable types are wine+game vs windows+game, native linux game vs same windows game (mostly limited to unreal and quake angines), some povray/blender tests and application comparisons (like you did with the firefox javascript speed).
GeorgeH - Wednesday, August 26, 2009 - link
Not really a benchmark per se, but I'd be curious to know how the stereotypes of Windows being bloated and Ubuntu being slim and efficient translate to power consumption. Load and idle would be nice, but if at all possible I’d be much more curious to see a comparison of task energy draw, i.e. not so much how long it takes them to finish various tasks, but how much energy they need to finish them.In know that’d be a very difficult test to perform for what will probably be a boring and indeterminate result, but you asked. :)
ioannis - Wednesday, August 26, 2009 - link
is there some kind of cross platform test that can be done to test memory usage? Maybe Firefox on both platforms? not sure.By "no laptops", I presume you mean, no battery tests (therefore power and as a consequence, heat). That would have been nice though. Maybe for those looking for a 'quiet' setup.
but yes, definitely GPU (including video acceleration) and the GCC vs Visual Studio vs Intel compiler arena (along with some technical explanation why there are such huge differences)
ParadigmComplex - Wednesday, August 26, 2009 - link
If you can, find games that are reported to work well under WINE and benchmark those against those running natively in Windows. It'd be interesting to see how the various differences between the two systems, and WINE itself, could effect benchmarks.Fox5 - Wednesday, August 26, 2009 - link
Number 1 use of Ubuntu is probably going to be for netbooks/low end desktops for people who just wanna browse the net.In that case, the browsing experience (including flash) should be investigated.
Boot up time is important.
Performance with differing memory amounts would be nice to see (say 256MB, 512MB, 1GB, and 2GB or higher). Scaling across cpus would be nice.
Ubuntu as a programming environment versus windows would be good to see, including available IDEs and compiler and runtime performance.
Ubuntu as a media server/HTPC would be good to see. Personally, I have my windows box using DAAP shares since Ubuntu interfaces with it much better than Samba. And as an HTPC, XBMC and Boxee are nice, cross-platform apps.
Finally, how Ubuntu compares for more specific applications. For instance, scientific computing, audio editing, video editing, and image manipulation. Can it (with the addition of apps found through it's add/remove programs app) function well-enough in a variety of areas to be an all-around replacement for OSX or Windows?
Speedwise, how do GIMP and Photoshop compare doing similar tasks? Is there anything even on par with Windows Movie Maker?
What's Linux like game wise? Do flash games take a noticeable performance hit? Are the smattering of id software games and quake/super mario bros/tetris/etc clones any good? How does it handle some of the more popular WINE games?