Jump to content

Change Mode

The Woes of Benchmarking and my Vista Vs Windows 7 Experience


Recommended Posts

Hey everyone. With the advent of Windows 7 on the horizon I wanted to try and explore some claims about Windows 7 and the performance gains it might have to offer. When Microsoft released this little tidbit to various news sites I wanted to try and get some real information on which system was faster. Vista or Windows 7. Well after putting over 14 hours of work into this little project of mine I think it's time to post my results. Since there was no information I could find out I had to conduct some experiments of my own. These are the results.

 

Testing Objective and Methodology

The objective was to find out which was faster overall in gaming and a few non gaming benchmarks including my overall experience with each system. I made sure to adhere to strict guidelines and get the most consistent test results each time.

 

Each game was run with FRAPS at 180 second intervals 3 times in a row. After each game was tested I took the average fps and averaged them out. I also included a few synthetic tests just for the sake of it. Including PC Pitstop's Results of course.

 

I ran each game at the native resolution of 1440x900. Also each game excluding Crysis Warhead had 4X MSAA and 8X AF. I tested each game with the highest playable settings also.

 

Test Setup

Motherboard: Asus M3A78-T 790GX AM2+ Socket

CPU: AMD Phenom X4 920 at 3.06 GHZ

GPU: XFX 8800GTS 512 at stock clocks

RAM: 2 Gigs of OCZ SOE PC 5400 DDR Memory

PSU: Xclio Great Power Supply 550W

HDD: Seagate Barracua 7200RPM 160 Gig IDE Drive (Primary Drive Partitioned)

Maxtor 40 Gig IDE drive (Primary Purpose: Secondary Backup)

Seagate Barracuda 500 Gig SATA 32MB cache

Cooler: Gigabyte G Power Lite

Case: Antec Nine Hundred

 

Drivers: Vista: Xtreme G 185.81 Drivers

Windows Vista: Nvidia GeForce 190.38 BETA Drivers

 

Note: If you ask why I didn't test Windows 7 with the Xtreme G Drivers I was unsure if Xtreme G's Windows 7 drivers had the supposed enhancements for Windows 7's WDDM1.1 So I opted for the stock reference drivers.

 

Windows Vista Service Pack 2 Vs Windows 7 RC Performance Results

 

Synthetic Benchmarks

3DMark 06 Vista: 10812 Windows 7:10837

Posted Image Posted Image

 

 

Super Pi: Vista: 22.754s Windows 7: 22.597s

Posted Image

Posted Image

 

 

PC Pitstop results: Vista: 9435 Windows 7: 9090 Vista Results Windows 7 Results

 

Gaming Benchmarks

Crysis Warhead: Vista: 28.58 fps Windows 7: 27.12 fps

Call of Duty World at War: Vista: 42.57 fps Windows 7: 51.74 fps

Team Fortress 2: Vista 113.094 Windows 7: 93.938

 

Note: TF2 was tested on a server with a max of 32 players. The maps used were Granary for Vista and Payload Race for Windows 7. So take this one with a grain of salt. Several repetitions of trying to reproduce a very consistent result proved difficult so I attempted to get as consistent as I could with this one.

 

 

 

Conclusions and Overall Personal Experience:

After several sleepless nights and tons of caffeine to try and get a conclusive result I finally finished. :overclocking: Overall each system trades blows. Interestingly there was a few instances when Windows 7 was actually up to ten fps on average. I am still investigating this and will get back to you as soon as I can. Moving on though, Windows 7 has a new feel and is very snappy. But in terms of smoothness and responsiveness I prefer vista. Aesthetically Windows 7 is pleasing but the new task bar is something I am still coming to terms with. I am used to have the show desktop button on the left not upon the right. Not to mention I like to adjust the start menu to my specifications. My preference goes to Vista on this. Overall I thoroughly enjoyed this little experiment of mine and hope that people enjoy what I have posted. Thanks.

Edited by ShadowPredator
Link to post
Share on other sites

that was a very good read, and very cool you took the time to test it out and show us. one thing i noticed in your tests, Windows Vista in the Pc Pit Test had the internet score but Windows 7 did not. i noticed personally Windows 7 seems to optimize the internet connection a little better, especially for the tests.

 

overall i think you did a great job and was interesting to read, and i thank you for your time putting it together, nice job :tup:

Link to post
Share on other sites

I am used to have the show desktop button on the left not upon the right.

Have you tried clicking on that button and dragging it to the position you want?

Link to post
Share on other sites

Have you tried clicking on that button and dragging it to the position you want?

 

I actually did try that. Didn't really work. For one reason or another I went into customization and tried messing around with the settings. Heck I even manually click, held and tried to drag the show desktop button onto the left side. To no avail it did not work. Don't get me wrong, the Windows 7 toolbar is very intuitive but my preference is to the Vista start menu. Edited by ShadowPredator
Link to post
Share on other sites

Nicely written comparison report and results.

 

Since I've never used Vista, I will not comment on the different, thought Windows 7 does please my eye, and looking at your results, up to 10 fps on avg, now that's something interesting..

 

As for show desktop, well could always use the old traditional method of Windows key + D. Actually I'm not entirely sure what exactly it is, but in Windows 7, at the end of taskbar (towards the right), after the icons, there is a button there which sometimes pointing at it, shows the desktop, and take the cursor away from it comes back to the page you were on, you could try clicking on it..

 

Thanks for the review :):tup:

Link to post
Share on other sites

Nicely written comparison report and results.

 

Since I've never used Vista, I will not comment on the different, thought Windows 7 does please my eye, and looking at your results, up to 10 fps on avg, now that's something interesting..

 

As for show desktop, well could always use the old traditional method of Windows key + D. Actually I'm not entirely sure what exactly it is, but in Windows 7, at the end of taskbar (towards the right), after the icons, there is a button there which sometimes pointing at it, shows the desktop, and take the cursor away from it comes back to the page you were on, you could try clicking on it..

 

Thanks for the review :):tup:

 

Remember that was just one level where Windows 7 was ten frames faster. I still am unable to isolate what happened. I tested and retested. The funny thing is Vista is ten frames faster in the first test level but Windows 7 ends up being faster in the other one. Ultimately I cant tell what that anomaly is.
Link to post
Share on other sites

No worries.. Oh and just to let you know, the button after the clock on the taskbar in windows 7 is a show desktop button, click it and everything minimized, and click it again everything comes back to how it was..

Link to post
Share on other sites

Boot time are an unshakable 45 seconds for me. That's on about 6 varying installations and using everything from Segate 7200 rpm, raptor 10,000 rpm and SSD raided drives. It's 45 seconds no matter what I do.

Link to post
Share on other sites

Very nice, I wish we could see more ppl doing some real tests in the pit, I really didnt think there would be much difference, I dont even know if theres a diff between the 2 apart from 7 having a bit of a face lift and a few new features but the rest looks an acts like vista and even a few benchmarks report win7 rc as vista ultimate come to think of it i dont think ive ever seen it say any different, Maybe the final version of win7 will perform better? who can say, also one other thing i will mention is ive noticed theres a bit of a performance decrease after installing sp2 for vista mostly in gaming, and somthing that does worry me is the perfrmance difference between the basic version and ultimate version? I thought the only difference was ment to be that ultimate had more fetures and and Aero gui maybe im wrong does anyone know?

Link to post
Share on other sites

Very nice, I wish we could see more ppl doing some real tests in the pit, I really didnt think there would be much difference, I dont even know if theres a diff between the 2 apart from 7 having a bit of a face lift and a few new features but the rest looks an acts like vista and even a few benchmarks report win7 rc as vista ultimate come to think of it i dont think ive ever seen it say any different, Maybe the final version of win7 will perform better? who can say, also one other thing i will mention is ive noticed theres a bit of a performance decrease after installing sp2 for vista mostly in gaming, and somthing that does worry me is the perfrmance difference between the basic version and ultimate version? I thought the only difference was ment to be that ultimate had more fetures and and Aero gui maybe im wrong does anyone know?

 

All in all I appreciate the warm reception I received from this. Windows 7 is bound to get faster. How fast depends on Microsoft and the driver teams. It could be like Vista Post SP1. Getting faster and faster. One of Vista's biggest problems was the drivers. Other than that not much else to say.
Link to post
Share on other sites

somthing that does worry me is the perfrmance difference between the basic version and ultimate version? I thought the only difference was ment to be that ultimate had more fetures and and Aero gui maybe im wrong does anyone know?

 

 

The only possible performance benefit of Basic over Ultimate would be the bit of extra free ram from less services, other than that they are the same.

Link to post
Share on other sites

The only possible performance benefit of Basic over Ultimate would be the bit of extra free ram from less services, other than that they are the same.

 

You would think that basic had the advantage but i explained that wrong i mean ive been running vista ultimate for ages and its been perfect super fast never crashes or hangs, after 2 weeks of constantly running it would still be quick and responsive, and now ive been running basic for a few days and ive had nothing but hell regular crashes running slow every thing has to hang for a few seconds wen i start a new task, updates keep causing problems, i even re-installed yesterday and still the same yet if i stick my other hd with ultimate theres no probs at all. Its like they are two different os's all together apart from them looking almost the same. It just dont fiqure.

 

Oh and sorry shadow for invading your thread

Link to post
Share on other sites

install w7 on an old Pentium 2 machine or even a netbook and the performance difference is more than obvious. w7 is way more efficient without losing function. On higher end computers, its completely moot experience cause as long as you have enough hardware resources for the OS and anything else, they wont interfere and fight for resources, therefore performance is not very noticable except only a few benchmark numbers which never mean squat anyway.

Link to post
Share on other sites

Any discussion is welcome Got the flavor. I am actually contemplating of wiping again and doing another performance comparison of Windows XP SP3 to end the age old debate of which is faster. I admitted I tried going back to XP a while ago and all I could say is I hated it. The responsiveness of it versus vista was not there and I missed the snappiness of super-fetch to say the least. For three days I was all into XP and then boom I went right back to Vista.

Link to post
Share on other sites

You can't end that age old debate of which is faster cause your results are completely linear. Your specific hardware, your specific tests..etc. The resulsts must be repeatable on any hardware,driver,application configuration. Then you have the problem of tests vs peoples general usage, which the latter will form their opinion, Thats why the debate of OS vs OS will always exist.

 

In my previous post, thats why i said to test the OS on the slowest hardware configuration you can, Not only does it show with obvious certaintity which OS is more efficient counting its functionalities, The performance tests will more clearly show which OS can perform when resources are tight.

 

As far as pure gaming performance, thats based soley on Direct x version to kernal and your video/audio drivers as they rely on the OS kernal differences. Thats why games will vary so much between OS's, each game runs different depending on OS Kernal, audio and video driver coding. No OS or driver will show an absolute gain for all video games, but rather as a combined result of many games and average those. plus you can only test certain DX versions per OS. a few games, DX10 is faster than some DX9 on both xp and vista but you cant compare dx10 performance on XP so using games is not a very good comparison for OS performance.

Link to post
Share on other sites

Meh, a properly configured XP SP3 installation should fly no problems.

 

I probably don't know what the hype of about your super-fetch is because I run off of a SSD that screams when I am loading apps, and I have all that prefetch stuff disabled.

 

One thing I hope win7 will be able to do is play classic games without major work arounds.

 

I tried playing D2 on a vista machine...It wouldn't even launch.

Same thing with a bunch of classic sims games (for a family friend's sister)

 

Anyways, good luck with your tests, hope you find somethin.

Link to post
Share on other sites

You can't end that age old debate of which is faster cause your results are completely linear. Your specific hardware, your specific tests..etc. The resulsts must be repeatable on any hardware,driver,application configuration. Then you have the problem of tests vs peoples general usage, which the latter will form their opinion, Thats why the debate of OS vs OS will always exist.

 

In my previous post, thats why i said to test the OS on the slowest hardware configuration you can, Not only does it show with obvious certaintity which OS is more efficient counting its functionalities, The performance tests will more clearly show which OS can perform when resources are tight.

 

As far as pure gaming performance, thats based soley on Direct x version to kernal and your video/audio drivers as they rely on the OS kernal differences. Thats why games will vary so much between OS's, each game runs different depending on OS Kernal, audio and video driver coding. No OS or driver will show an absolute gain for all video games, but rather as a combined result of many games and average those. plus you can only test certain DX versions per OS. a few games, DX10 is faster than some DX9 on both xp and vista but you cant compare dx10 performance on XP so using games is not a very good comparison for OS performance.

 

Whether or not games is a very good comparison of OS performance it was not my intended desire to see. I set out to test from a gamers standpoint what Windows 7 has to offer in terms of performance over Vista. General usage and everyday applications were irrelevant to what I set out to to. My intent was not to try and see which operating system is better. I was reading about how Windows 7 was going to be faster and much better. Microsoft released a press release stating the top reasons to get Windows 7. Well one of them stated was gaming. Now, I know as much as the next person that we should all take press releases with a grain of salt. But I had a strong interest in which operating system is going to be faster in terms of gaming.

 

You are right, each kernel is different. Drivers vary from configuration to configuration and I understand that. I made sure to use the latest and most stable drivers I could at the time for each operating system. If I didn't state that I do apologize. I must digress, Direct X 10.0 overall only offers improved graphical quality. Not an increase in performance over Directx X9. The only iterations I have seen is in Assassin's Creed and Far Cry 2. Each was an instance of Direct X10.1 not Direct X10.0 Which ATI was a winner in the former of the two. These are the only instances when I have seen Direct X10.1 outrun Direct X9.

 

If I had the resources to scrounge together a netbook or install upon a Pentium 2, then yes I would be able to tell the difference. But that was not my intent here. It was to help illustrate which operating system was faster in the all intent purpose of gaming and a few synthetic benchmarks. And I have to disagree that even on higher end hardware the differences will be moot. I will disagree and say that even on higher end hardware people can notice this. Each user is dependent, but the majority of users will not notice. One such as myself will most likely notice the responsiveness, snappiness, overall boot time.

 

Synthetic benchmarks may mean squat to you but the truth is there are people out there and circles that say benchmarks do matter. The boys here in the pit are a prime example of when benchmarks matter. It may be a proverbial whose is bigger and can go faster, but it still matters. In overall real world performance and general usage then it might mean squat.

 

I was trying to speak in terms of which is faster in performance in gaming. Testing on the slowest hardware configuration is not a possibility. My testing methodology was to gain the most consistent result over several games using FRAPS which records real world testing. I tested three times repeatedly using the same level to try and get the most consistent result, afterward which I averaged the results and compared them.

 

Moving on, as far as I know the direct X version terms of Direct X9 should be uniform across the board with Vista and XP. There are possibly different versions, but I made sure with Windows 7 and Vista to manually update to the last one for each. The consistency was my greatest challenge and possibly the weakest link in my original post, but I persevered and made sure I adhered to my rules.

 

Yes, my testing may be linear but it is consistent nonetheless. I set guidelines and adhered to specific rules that I laid out to achieve my stated objective. Which I frankly did, with an attempt at the most consistent results I could attain to the greatest of my ability. I appreciate your posting and speaking up. If you wish to test several things such as which is faster and is better for overall general usage I invite you to test and post your thoughts.

 

Summary: My intent was to see which is faster in gaming. I in my personal view find that gaming is a good way to measure overall system performance. It is a hobby of mine which I enjoy greatly. My testing was not to see which is better in everyday applications but gaming. Anyone that wishes to contribute please I invite you to do so and welcome the input.

 

 

Meh, a properly configured XP SP3 installation should fly no problems.

 

I probably don't know what the hype of about your super-fetch is because I run off of a SSD that screams when I am loading apps, and I have all that prefetch stuff disabled.

 

One thing I hope win7 will be able to do is play classic games without major work arounds.

 

I tried playing D2 on a vista machine...It wouldn't even launch.

Same thing with a bunch of classic sims games (for a family friend's sister)

 

Anyways, good luck with your tests, hope you find somethin.

 

I possibly didn't state it correctly, but when I was using Windows XP for some reason or another it did not feel as responsive. Its not the fact of how Supefetch is hyped or anything, but for Vista the ram caching is much more aggressive in Vista, which in turn helps my computer feel much more responsive. Back when I used XP religiously, I always viewed my RAM as a resource rather than a cache. When moving to Vista it was very difficult to try and see all this memory being used by Vista, but got used to it.

 

Don't get me wrong, an XP SP3 installation tweaked and adjusted does fly. Oh believe me, I know. That thing can scream, but overall in terms of responsiveness and snappiness Vista for me feels much more responsive. Could just be me but I don't know.

Edited by ShadowPredator
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...