Massive on Cheating
I think it's a good idea to jump right into this topic first, since it's so high on everyone's list lately. (Ed note: For good reason)
I will not even speculate about what is going on behind closed doors at Massive - or even if there are
closed doors at Massive. However, after the entire 3DMark/nVidia fiasco (which I refuse to go into here), many people are "concerned" about what kind of stance Massive is taking on benchmark optimizations - or, more bluntly, benchmark cheats.
When directly asked on the AquaMark3 forums by user rms what their position was on benchmark cheating, Massive responded with this statement
, which I will quote below:
Statement of Massive Development regarding AquaMark3 / IHV cooperation & cheating attempts
As a developer of interactive high-end 3D gaming applications and benchmarks we obviously have to collect in-depth Know-How about a specific hardware from it's manufacturer.
This type of cooperation is common practice for all major developers. It serves the sole purpose to utilise the latest technologies in the correct way and thus giving the customer the best gaming and performance experience possible.
You may decide to support a specific hardware platform or a specific hardware feature by a game application. However this not an option for any serious benchmarking application.
While developing AquaMark3 we take precautions in a way so that all IHV's (Independent Hardware Vendors) with a vital interest in AquaMark3 are treated equally. So there is no reason for those vendors to conduct any cheating attempts on AquaMark3.
Once a benchmark is released there are obviously multiple possibilities to cheat its results (both for any IHV and any user). We took precautions to prevent and detect cheating attempts in this post release phase and during the lifetime of AquaMark3. Unfortunately we can not prevent this in all cases by the application itself.
The most powerful anti cheating tool will be the AquaMark Result Comparator (ARC), the online database and the AquaMark3 forum. We will immediately inform all customers about our evaluation of the latest cheating attempts (if they should happen).
As a developer of high end 3D gaming applications we will of course be very specific and accurate when we have to judge if a specific optimization is of general interest or if it only targets AquaMark3 in a questionable way.
We are always open to discuss our policy with interested users in this forum.
Alexander Jorias / Managing Director Massive Development
Ingo Frick / Technical Director Massive Development
A bit naive perhaps regarding the part about IHVs being treated equally and there being no reason to cheat - some IHVs aren't interested in a fair comparison. It seems though that Massive isn't hiding behind any preconceived notions that IHVs are completley trustworthy. Hopefully, they stand behind that statement and try and actually protect the validity of their product. We can only wait and see.
Also from the AquaMark3 forums in a less official post, Alexander Jorias, Managing Director Massive Development, had this to say
when asked by user g__day "Will ATi and NVidia cards execute exactly the same shader code?
The code path are not optimized in a vendor-specific way. The capabilites of the [graphics] card are considered when the engine determines which set of shaders is used for a spefic material / effect. So if two cards report the same capability flags for a given situation exactly the same code / [Vertex Shader] / [Pixel Shader] path are used.
... we are comparing apples vs apples.
... and then this quote, also from Alexander Jorias, in another thread
originated by g__day entitled "How widely is DX9 precision hint used in AQ3?"
should be noted as well:
the precision hint is used wherever more precision does not add visual quality. in case the visual quality is need the hint is not given. this simple rule applies to all am3 pixelshader 2.0
You might have noticed in other AquaMark3 reviews that the GeforceFX, which has been shown by many reviews to have a weak DX9 Pixel-Shading engine, compares well to the strong Pixel-Shader power of the Radeon 9800 Pro in AquaMark3. This is not a conspiracy between Massive and nVidia as some people would have you believe, the reason why the GeForceFX performs relatvily well is legitimate and easily understood.
What AquaMark3 does is determine which shaders can be executed in DX9 low precision without affecting their final output, regardless of what card is in the system. Since the nV3x supports low precision hints, it is able to take advantage of the performance benefit offered there. The R3x0 on the otherhand converts whatever precision it is asked to do to 24bit high precision, even if asked to do low precision calculations. Therefore, any performance benefit that might have been gained from low precision shaders is negated.
Initially, I was against partial precision hints. However, after thinking on the subject I can now say that it actually makes sense. An analogy would be using a small truck to carry a small load and a large truck to carry a large load. Using a large truck to carry a small load is a waste of resources.
Of course, throwing away half your load just to fit it in a small truck is not very smart either...