F@H a good benchmark?

[sharder8]

Hey guy's . . . remember a couple weeks ago when I mentioned one of my customers wanted to compare benchmarks between my system and his . . .

I provided him my benchmarks using the programs he wanted, then I then requested he run F@H to show his real computing power. Today I got this email from him and responded with my answer. Anyone care to add to my response, feel free.

His email:

what do you mean If I join a team what kind of benchmarks do they have. OK I joined techimo team #111 now where do I go to get benchmarks because I'm not going to run this f@h software because its not a benchmark. My techimo username is nomadpro22 and I cant find anything on techimo that relates to benching so what exactly do I do?????

My response:

F@H not a benchmark????

It will stress your CPU and GPU to the max and show what kind of power it actually is able to produce. Plus, the work it accomplishes while running is a good cause (Cancer research).

I find it a better benchmark than most of the other ones out there and it's easy to see where your system lies in relation to thousands of other computers out there. If you don't beleive me, ask and find out the varience in CPU/GPU points between the different CPU's and GPU's. HDD speed can be over-come and made up for by the CPU, GPU, and/or memory in many of the tests. Most of the benchmarks don't give much weight to HDD for that reason. However, F@H will stress the CPU or GPU to the max and also tax the memory.


Harder

Harder

[nemowolf]

I would say that it is not a classic benchmark but a Real World benchmark. Most typical benchmarks run for a certain duration and give you an evaluated score. F@H instead requires weeks, months, or longer to give you an idea of how you compare against another machine. Because it introduces variables, the different WU's and GPU vs CPU units, it allows for a more advanced level of benchmarking. No benchmark i have seen allows you to basically aggregate the scores of both your CPU and GPU to find your total overall performance level against another pair.

Anyone who has run 3dMark knows about what score they can expect based on other GPU/CPU combo's but factors such as memory, HD, FSB, Intel/AMD, etc come into play and that is shown in a F@H general score.

[sharder8]

I would say that it is not a classic benchmark but a Real World benchmark. Most typical benchmarks run for a certain duration and give you an evaluated score. F@H instead requires weeks, months, or longer to give you an idea of how you compare against another machine. Because it introduces variables, the different WU's and GPU vs CPU units, it allows for a more advanced level of benchmarking. No benchmark i have seen allows you to basically aggregate the scores of both your CPU and GPU to find your total overall performance level against another pair.

Anyone who has run 3dMark knows about what score they can expect based on other GPU/CPU combo's but factors such as memory, HD, FSB, Intel/AMD, etc come into play and that is shown in a F@H general score.

Thank you Nemo . . .

That's what I'm trying to get nomad to understand.


Harder

[sharder8]

Come on guys . . . need some help here!!


Harder

[RobRich]

If you want to compare raw processor performance, then yeah, F@H could be a decent casual real-world test - technical aspects like memory throughput*, SIMD optimization**, etc. aside.

Back when I was a hardware analyst, I remembering using the RC5-64 DC project as part of a processor comparison review. Not because it was a superior technical choice for general benchmarking, but more because it was a real-world performance metric with which many people could better relate than compared to SPECint, SPECfp, and similar technical benchmarks.

*F@H is probably not extremely bandwidth intensive.
**DC clients oftentimes have different optimization paths - SIMD and otherwise - for different processors.

[Sky King]

I think it depends on what you are trying to benchmark. I don't think FAH accurately represents, in any way, real-world usage. I don't think it provides much more than a raw CPU throughput benchmark. We simply don't know enough about the way the developers have managed memory access, caching, etc. They make optimization choices using various tradeoffs and objectives that are different from what real-world, or benchmark programs use. The mantra in DC (distributed computing) is maximum CPU cycles per minimum impact on the user.

(Remember that as power folders that often dedicate iron to folding, this is not the norm and not the design audience for distributed computing.)

Put another way, the engineering objective in distributed computing software is to maximize the use of idle CPU clock cycles while minimizing the impact on everything else. Since CPU prioritization is the only throttling mechanism used to mitigate folding's impact on the user, the developers will try to minimize things like network access, disk usage, etc. because these are not as easy to prioritize as CPU cycles.

In other words the tool you are using for benchmarking is a tool specifically designed from the ground up to minimize its stress on the system, not maximize it.

I have a suspicion that, when all done, the comparisons between your folding performance and his will be roughly the same as the comparison of your CPU clock, number of cores, and CPU type, especially if you inlcude hyper-threading versus true multiple cores in your comparison.

I have 8x 2.67 GHz i7 cores running. When I first started folding with just multiple instances of the Windows console on it, each core individually produced almost exactly the PPH I was getting in my single core 2.8 GHz Celeron. I get much better PPD by using the linux folder under VMs, but that is a different animal all together.

My point is I am not sure that FAH will produce much different comparison between your two machines than could have been calculated by pen and paper.

[RobRich]

I have a suspicion that, when all done, the comparisons between your folding performance and his will be roughly the same as the comparison of your CPU clock, number of cores, and CPU type, especially if you inlcude hyper-threading versus true multiple cores in your comparison.

^Pretty much.

The performance data being generated is relevant enough for a generic processor comparison, especially if comparing similar processors executing the same software code path under the same or at least similar operating system, but it is not discernibly viable enough for a detailed system performance analysis.

[sharder8]

Hey guys, you're missing the point.

I'm not a gamer and I look at R@H and F@H to compare my computers. Nomad is a gamer and that's the benchmarks he looks at. I'm looking at raw power in order to make the comparrison.

I don't care how fancy or expensive a computer is . . . only at what kind of raw power it has.

Harder

[RobRich]

I'm looking at raw power.

Harder

Therein is the problem. You can compare performance from a casual standpoint, but comparison from a technical standpoint could be convoluted.

Case in point, as I mentioned earlier, some DC clients use different code paths for different processors. Say your have an Intel Core 2 Quad, yet your friend has an AMD Phenom II. Now say the DC client detects the processor type, then uses different code paths to execute the same workload. One code path might be highly optimized for the specific processor, while the other is baseline generic coding compiled for broad support. In such a scenario, the performance comparison produces relatively invalid results, again from a technical performance comparison standpoint. You determined the performance difference for a certain software application, but did you truly measure the raw hardware performance difference between each system?

You also need to know the type of workload being processed. For example, is the workload memory bandwidth dependent? Say you both have the same processor, yet one has DDR2-667 while the other has DDR2-1066. One would perhaps think the system with DDR2-1066 would be faster for the specific application, though in actuality it might offer little to no real world benefit assuming the workload already does not even saturate the DDR2-677.

BTYW, for clarity's sake, I have little to no idea how F@H develops its core code. I would not be surprised if specific code paths and/or optimizations are applied to specific processor families, though I simply do not know enough about the project's development process.

[Flying JJ]

Would something like this help? It shows what diffrent cpu's and gpu's are getting on diffrent projects ( WU's ). It also has the amount of overclocking associated with each. A little hard to navigate, but if you try hard enough, you get alot of data.

fahinfo.org Folding @ Home PPD Database

[nunyadam]

or maybe this chart fahinfo.org Folding @ Home PPD Database

[KarmaKiller]

Although I love my folding, I gotta agree with Sky and RR.
I think it would be extremely difficult to compare PC's from just folding. My main catch is while you have the CPU crunching in the background, most of us still use our PC's for other tasks at the same time. So if I'm watching a HD movie, that will probably take up more processing power then say you, who might only browse the web.
So I guess my point is if the PC's did nothing but fold, for a set amount of time (which again I think is another catch) then maybe you could get a close comparison.
Another thought I'm having problems digesting is the different WU's could vary on different processors.

But I understand what your saying Harder completely. I really do. I just don't think it would show all the much difference unless your talking single core vs multi cored chip.

IMO, the best way to compare systems is PPD per Watt. If your system can put out say 5k PPD while using only 350watts, and his puts out say 6500 PPD using 650 watts, then in my mind the higher wattage system is the loser.
*shrugs* I'll quit rambling now..

[sharder8]

Oh well . . . the guy thinks F@H is a waste of time and that's it.

Tried to get a new member on the team . . . if only for a short time . . . and he thinks it's a waste. (This thread didn't help!)


Harder

[KarmaKiller]

EDIT:
Totally read that wrong.. Maybe I should wake up before I start posting..
Sorry.