"We have now established that nVidia's Detonator FX drivers contain certain detection mechanisms that cause an artificially high score when using 3DMark03. We have just published a patch 330 for 3DMark03 that defeats the detection mechanisms in the drivers and provides correct results." this is how Futuremark introduce their latest patch for their popular 3DMark03 software.
Saratoga, Calif.-based Futuremark issued a statement claiming that nVidia tweaked software needed to run its new GeForce FX 5900 chip, in order to distort performance in Futuremark's 3DMark 03 testing application. Futuremark is one of the leaders in software and services for performing PC benchmark tests.
According to Futuremark when the patch was applied a drop of as much as 24.1 per cent was observed in certain nVidia products, while competition products performance-drop stayed within the margin of error of 3 per cent.
An nVidia spokesperson said Since nVidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer. We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad.
ATI meanwhile are not too keen to capitalize on another, possible, embarassment for their rivals. ATI's Dave Baumann, partly in response to nVidia's statement said:
Despite still being a full Futuremark Beta member, ATI did not make it out of the report entirely unscathed either. There is a performance difference of about 8 per cent in Game Test 4, that accounts for about a 2 per cent difference in the final 3DMark03 score, between the new and old versions, indicating that although not visually different something was occurring on this particular test.
In order to add further validity to the importance of benchmarking tests ATI's Christ Evenden stated:
The 1.9 per cent performance gain comes from optimization of the two DX9 shaders (water and sky) in Game Test 4 . We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically and functionally identical to Futuremark's and there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality and as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, and so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST.
What does all this mean however? Most knowledgeable gamers do not buy a card based, solely, on the score achieved in a benchmarking test. It is reliability, performance and value for money which usually determine which card is bought. Benchmark tests are mainly utilized in order to claim the performance crown, giving the bearer increased prestige and an advantage created by the publicity. All that cheating on tests will achieve is to render such tests useless and to create consumers who do not trust the manufacturers. Unfortunately, this time, the reputation of the biggest graphics chip manufacturers has been questioned and there is little chance that they will make it out of it without some doubt remaining in the publics mind. Doubt which can only be cleared by the release of solid, reliable and high quality products.