Following the very suggestive statement by Futuremark regarding nVidia's drivers on "steroids", the company have now issued a revised statement which uses very careful language in order to explain what exactly happened. Both Futuremark and nVidia have also
issued a joint statement which attempts to bridge the gap created, first by nVidia's withdrawal from Futuremark's beta member tests and then by the announcement regarding cheating drivers. It is hoped that this barrage of statements will be able to reverse the damage to the reputations of both companies that the initial Futuremark announcement may have caused.
Although stories of bold press releases which are later toned down do create an impression of conspiracy, see Roswell, the truth may be a bit simpler than that.
Futuremark mention the possibility of a change in the way their test measures performance and suggest the possibility that they may have been too hasty with the announcement. They do however, stick to their guns, as far as the update they released is concerned. This fact alone seems strange since claiming that the update was needed so that 3DMark03 continued to produce comparable results, still suggests that nVidia were not really playing fair.
If the problem was caused by a difference in philosophy as Futuremark suggest then nVidia's claim that they offer true optimized gaming performance rather than just power for the sake of power, could begin to hold water. Either way Futuremark depend heavily on the existence of at least two strong competitors in any hardware field and that means that they need nVidia. The only way they can keep nVidia and at the same time convince the public that they offer an efective and relevant performance tool is to sort this out as soon as possible, even if that means revising their whole approach to benchmarking.
The Futuremark Statement
For the first time in 6 months, as a result of Futuremark's White Paper on May 23rd, 2003, Futuremark and NVIDIA have had detailed discussions regarding NVIDIA GPUs and Futuremark's 3DMark03 benchmark. Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now
states that NVIDIA's driver design is an application specific optimization and not a cheat .
The world of 3D Graphics has changed dramatically with the latest generation of highly programmable GPUs. Much like the world of CPUs, each GPU has a different architecture and a unique optimal code path. For example, Futuremark's PCMark2002 has different CPU test compilations for AMD's AthlonXP and Intel's Pentium4 CPUs. 3DMark03 is designed as an un-optimized DirectX test and it provides performance comparisons accordingly. It does not contain manufacturer specific optimized code paths. Because all modifications that change the workload in 3DMark03 are forbidden, we were obliged to update the product to eliminate the effect of optimizations identified in different drivers so that 3DMark03 continued to produce comparable results.
However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks.
nVidia's Statement
NVIDIA works closely with developers to optimize games for GeForceFX. These optimizations (including shader optimizations) are the result of the co-development process. This is the approach NVIDIA would have preferred also for 3DMark03.
Joint Statement
Both NVIDIA and Futuremark want to define clear rules with the industry about how benchmarks should be developed and how they should be used. We believe that common rules will prevent these types of unfortunate situations moving forward