Non "cheating" shaders results.

S

Senile

New member
These were the original ShaderMark 1.7 results for the 5800U, 5900U & 9700 Pro.
http://www.3dvelocity.com/reviews/5900u/5900u_4.htm

The reviewer got in touch with ToMMTi-Systems because of suspicions & they sent him replacement shaders that would defeat any "optimisations".
New results:

http://www.3dvelocity.com/reviews/5900u/5900u_16.htm

"Run using ToMMTi-Systems revised PS2.0 shader code. These new shaders will be made available at http://www.tommti-systems.com from June 25th. In the words of author Thomas Bruckschlegel these 2.0 and 2.0 partial precision shaders deliver the same image as the old ones, but they have some shuffled instructions, which makes them invisible for current driver optimisations. Unlike in the review, these were run using DX8 Mip Filter Reflections due to some apparent problems with DX9 Mip Filter Reflections as reported by ToMMTi. "

I think the new 5800U results will give Doomtrooper & Hellbinder something to laugh at. :) ;)
 
I'm all about Nvidia getting the sandpaper condom, but i'm having some difficulty believing that these scores are legit. the difference is mindboggeling. Is this a joke?
 
Uhm, wow.

Uhm, wow.

Man, I can't even work up a mild anger at 'em anymore for stuff like this...I've just come to expect it from them. :(

"nVidia: Truly redefining the graphics industry in ways no one has before!"

:bleh: :(
 
Is this the part where I get to say I told you so? ;)

I called those ShaderMark scores as being unrepresentative as soon as I saw them, looks like I was right.

A couple of points about the new scores though:

a. Where are the 5900 Ultra scores? I guess they no longer have their review sample?

b. Take a look at the difference in scores for the 5800 using full and partial precision - There basically isn't any. So it looks like all those people saying that partial precision is necessary for the GeForceFX line to perform well were wrong, it really makes no difference (in Pixel Shader 2.0 operations at least).
 
It does seems that way doesn't it hanners? So I gotta ask.. how the hell are they making them so much faster with hacks then? what are they doing. If it's not lower precision that is fixing them then what is?
 
Hanners said:

b. Take a look at the difference in scores for the 5800 using full and partial precision - There basically isn't any. So it looks like all those people saying that partial precision is necessary for the GeForceFX line to perform well were wrong, it really makes no difference (in Pixel Shader 2.0 operations at least).
Click to expand...

Oorrr there is no full precision :D
 
Ady said:
It does seems that way doesn't it hanners? So I gotta ask.. how the hell are they making them so much faster with hacks then? what are they doing. If it's not lower precision that is fixing them then what is?
Click to expand...

They are probably either totally replacing all the ShaderMark shaders with their own rewritten shaders, and/or lowering the precision down to FX12 (thus using integer precision, and not floating point at all, which is totally outside of the DirectX 9 spec).
 
To be fair the radeon results do take a dive as well so they better have a good explonation as well...
 
Pojo said:
To be fair the radeon results do take a dive as well so they better have a good explonation as well...
Click to expand...

Not to defend ATi, but IIRC, ATi already did after the 3dmark2k3 debacle.. ATi optimized their shader routines by executing the instructions in a more efficient way. All games and benchmarks benefit from these kind of optimizations. Such optimizations are perfectly valid and acceptable, just read John Carmack's or Tim Sweeney's comments on that.

Clipping OTOH (what nVidia did) is not valid optimization (its plain cheating), because these kind of hacks only work in specific programs that use predefined paths.
 
Last edited:
Pojo said:
To be fair the radeon results do take a dive as well so they better have a good explonation as well...
Click to expand...

In true ATi fanboy style, I'm going to put the difference in ATi scores down to this:

Unlike in the review, these were run using DX8 Mip Filter Reflections due to some apparent problems with DX9 Mip Filter Reflections as reported by ToMMTi.
Click to expand...

:p


Seriously though, it would be nice to hear an explanation for this from Catalyst Maker as well. Are they reordering shaders again for this benchmark, or worse? :bleh:
 
tech9 this has been talked to death in every forum with end result being that any sort of optimizing on case specific benchmarks is not ok. I understand what ATI did and its nothing compared to the blatant cheats Nvidia did but it still should not be in there. Thats why ATI will remove these tweaks from their future driver version at least in regard to 3DMark2K3.

If ati or nvidia have tweaks they like to use on benchmarks they need to give them to the benchmark developer and let them decide if its a fair optimization thats applicable to real world games. If they choose to include it into the benchmark then fine but you cant just replace or tweak code and not tell anyone.

Anyways I hope the lower Ati results are in fact just because of some bug but something tells me that its another optimization again. Its most likely not as bad as nvidia but only time will tell.
 
Pojo said:
tech9 this has been talked to death in every forum with end result being that any sort of optimizing on case specific benchmarks is not ok.
Click to expand...

Re-ordering shader instructions so that the GPU executes them more effeciently IS valid optimization; heck all operations benefit from this, its not limited to a single benchmark and/or program.

John Carmack's response to the 3dmark2k3 debacle:

Rewriting shaders behind an application's back in a way that changes the output under non-controlled circumstances is absolutely, positively wrong and indefensible.

Rewriting a shader so that it does exactly the same thing, but in a more efficient way, is generally acceptable compiler optimization

Source: http://slashdot.org/comments.pl?sid=65617&cid=6051216
Click to expand...

Tim Sweeney's response to the 3dmark2k3 debacle:

Pixel shaders are functions, taking textures, constants, and texture coordinates as inputs, and producing colors as outputs. Computer scientists have this notion of extensional equality that says, if you have two functions, and they return the same results for all combinations of parameters, then they represent the same function -- even if their implementaions differ, for example in instruction usage or performance.

Therefore, any code optimization performed on a function that does not change the resulting value of the function for any argument, is uncontroversially considered a valid optimization.

Source: http://www.beyond3d.com/forum/viewtopic.php?t=6041
Click to expand...

End of story.

NOFI, but I know for a fact that John Carmack and Tim Sweeney actually know what they are talking about, unlike those wannabe 3l1T3 hAX0r kids on the forums that you are referring to :rolleyes:
 
Last edited:
Please stop your nonsense tech9 the shader shifting is great for game engines when its a general tweak that helps multiple titles. What was done in 3DMark2003 was special case shader optimizations that do no effect anything but that last test in that benchmark. It only and I repeat only applies to this test and never gets used anywhere else. The purpose of a benchmark is to test how a card runs a specific set of task. The work load should be the same for all cards. Just because ATI decided to tweak the benchmark for r300 cores doesn't mean they will take the same effort to do it in games. The benchmark is not there to see how good ati and nvidia are at tweaking the code for best case scenarios. Its there to test a specific predefined work load which should be as close to neutral as they see fit.

Normal games should have such optimizations. It would better still if they were game engine tweak that benefited multiple titles. Benchmarking is a complete different issue and bottom line is that no one but the creators should make any tweaks or substitutions. It doesn't matter if the same optimizations are welcome in games. It also makes no difference if the output result is the same.

Here is a simple example that illustrates my point:

1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1=20

1*20=20

Same result but very different work loads. If the benchmark was made to test the speed of addition then using multiplication completely negates the benchmarks results. We are not testing how well multiplications works even if its a valid substitute in any other circumstances.

I don't know how such a simple thing can be so hard to understand. Of course there are other issues regarding benchmarking such as api's used, hardware limitations and many others. This however does not change the principal or the goal behind any benchmark.

Thanks for letting me know that Beyond3D forum is full of "wannabe 3l1T3 hAX0r kids" but I can make up my own mind as to whos more in the know on the issue.
 
Brent over at hardocp forums just saw the new shadermark results, here is what he had to say!

"Holy crapola

wtf is nvidia thinking?

i will not tolerate cheating in this manner"

Now if only Kyle would make a statement like that and blister it on his front page, these tactics would get the coverage they deserve!
 
Hanners said:
a. Where are the 5900 Ultra scores? I guess they no longer have their review sample?
Click to expand...

As far as I can guess the 5900U went back even before the results could be analysed. :(
 
HA HA

just posted this thread over at madonion

you can just guess what was said

"its obviously broken for nvidia cards theyve done something!!"
 
quote:
--------------------------------------------------------------------------------
Unlike in the review, these were run using DX8 Mip Filter Reflections due to some apparent problems with DX9 Mip Filter Reflections as reported by ToMMTi.
--------------------------------------------------------------------------------

I think ATI is doing something shady too (pun intended :) ) unless switching to "DX8 Mip Filter" causes a 50% drop in performance. Can someone with the technical knowledge explain this? Or better yet can someone from ATI comment?
 
Pojo,

Good example, but it should be more like this

Benchmark workload
1+2+3+4+5+6+7+8+9 = 45

nvida method
2*4*6 = 48

ati method
1+3+5+7+9+2+4+6+8 = 45

The latter re-orders because let's say my hardware likes to do all odd calculations first, then evens. A perfectly legit optimization because I'm following the standard. Both are add operations, I have not removed any code, I have not added any code. I simply re-ordered it.

The former rips out the old code and replaces it. Not an optimization but a blatant cheat. It's not even the same operation, completely removed code, and replaced it with custom code because let's say this hardware does proprietary multiplies, but can't do standard addition and can't handle odd numbers. And the result is close, but not even the exact same result.

I know, I know... what hardware can add but not multiply and vice versa, but I'm building off your example.

What is making everyone upset is that if the nvidia method was like the ati method, you would still not get much, if any, performance out of the FX line. They have to resort to their 'appication specific optimization' methods because the GPU performance of the FX line is really that bad.

And this is why they really want the industry to adopt Cg because it's doing precisely what I've shown above and as evidenced in benchmarks and games now designed 'the way it's meant to be played'.

BUT, the deal is we have STANDARDS for a reason. We can see throughout all these exposed cheats that ATI has been faithful to standards all along. We can see that all they have done is valid optimizations for every architecture. No matter how much websites try to expose something that is not there, ATI is ALWAYS (yes, even with the quack issue that was fixed and the performance was still there) within tolerance. That cannot be said for nvidia, especially since the FX line has been introduced.
 
You must log in or register to reply here.
Back
Top