No announcement yet.

Update to GPU video benchmark charts

  • Filter
  • Time
  • Show
Clear All
new posts

  • Update to GPU video benchmark charts

    We are planning on making some changes to the GPU benchmark charts over the next couple of days to improve their accuracy but with a consequence of (temporally) reducing how comprehensive they are.

    As background.
    The charts are made up of average results for each video card type. When we originally made the charts public we used results from PerformanceTest V7. For popular GPUs types we had up to 10,000 samples to average. For rare GPUs maybe only a few samples to make up an average.

    When PerformanceTest V8 was released (Oct 2012) we didn't have a huge number of benchmark results collected. Only about 3000 submissions at launch. For comparison, for PerformanceTest V7 we had around 600,000 submissions over preceding few years.

    We want the charts to have a comprehensive list of video cards, so on the PT8 release date we decided to combine the results from PerformanceTest V7 and PerformanceTest V8 to make up the charts. The plan was that over time the PerformanceTest V8 result will start to dominate and displace the V7 results. The method for doing this will be to take the final average V7 result for each CPU and use this as 1 sample for each type of GPU. Then as more V8 samples arrive this single V7 result will be averaged with all the V8 results. Effectively diluting the V7 result into insignificance over time.

    The impact of this can be seen in the charts, the sample count (which you can see via mouse over on the charts) will reflect the V8 sample count (+1 for the single V7 sample).

    Although we made a deliberate effort to make the PT7 and PT8 3D results comparable, it is an impossible job in the end. PT8 uses DirectX11, Direct compute, Higher resolutions, etc.. So the results for every video card is never going to be the same as for PT7 which didn't know about any of this.

    The Consequences (5 months on)
    The plan worked, mostly. Since the release of PT8 we have collected around 65,000 baseline results with new submissions arriving at the rate of about 20 per hour, 24 hours a day.

    For popular GPUs the charts are totally dominated by the PT8 results. However for some of the very rare and very old video cards we still don't have any PT8 results. Meaning that the charts are now really a mix of PT7 and PT8 results. With rare cards displaying the PT7 result and popular cards displaying the PT8 result. Which isn't really desirable as the accuracy of the benchmarks for old rare cards is now pretty poor (see the comments on comparability above).

    As extreme example of this can be seen with the now rare, Radeon X1800 CrossFire Edition and Radeon X1900 CrossFire Edition cards. These cards did better in PerformanceTest V7 than in V8. So we have results like this,

    Radeon X1800 CrossFire Edition, 1066 3DMark, 1 sample, made up of only PT7 results.
    Radeon X1800 GTO, 237 3DMark, 6 samples, made up of mostly PT8 results. (the PT7 result was ~700)

    These cards should be ranked much closer together.

    What's changing
    We are going to remove all the PT7 results from the video charts.

    Older PT7 results will be filed away on this page as a static list never to be updated again,

    This means many of the older and rarer cards are going to drop off the main charts. So the charts will be less comprehensive than they are now, but more accurate. Our estimate is that we'll have around 800 different types of video cards in the list once the update in done in the next couple of days (down from 1900). This is probably only around 40% of all the video card ever released for Windows based PCs however. As time goes on we should get more submissions and the list should grow in size again. There are some cards that will probably never make a re-appearance however like the classic S3 ProSavage, RIVA TNT & Rage Fury.

  • #2
    Nice cleanup. On a somewhat unrelated note, I would argue the classification can be a bit misleading because you only need a score of 750 to be in the high end group. So an aging card like the Geforce 9600 GT is considered high end for example, but it will really struggle with any modern title. It may not even play a few games on low settings. Meanwhile, cards in the 300 range are considered "mid to high end", which I think is very generous, since it's impossible to buy anything this slow and weak these days. I would leave the high end only to those with more than 2500 points. Mid to high end between 1300 and 2500, low to mid end 600 to 1300... and I think most people would consider anything under 600 as pretty much low end. Anything under 100 as ancient history.


    • #3
      You are correct. The cards at bottom end of the high end chart aren't very high end. It turned out like this because the list would be too long otherwise. So we split it into 4 separate charts. The top 25% of cards get called high end, regardless of it this matches popular perception. So people might consider only the top 5% of cards to be high end, but this would make for a rather small chart. If you were in a poorer country it might be 20%.

      Still people should have enough common sense to realize that the cards at the top of the chart are the best.