Synchroniszation issues with SLI and CrossFire

[ChrisRay]

I thought I'd go ahead here and display UT3 on 9800GX2 cards from 1 GPU to 4 GPUs in latency. To give you guys a better idea of whats going on and what increasing the the amount of AFR is doing. I took only 100 frames in this case and
used the UT3 Deimos Map with 16xAA/16xAF



1 GPU

16xAA/16xAF 1680x1050

AVG 34 FPS



Frame, Time (ms)
    1,     0.000
    2,    27.074
    3,    56.307
    4,    87.132
    5,   116.179
    6,   145.131
    7,   169.784
    8,   204.288
    9,   233.159
   10,   262.923
   11,   292.473
   12,   322.507
   13,   352.017
   14,   381.586
   15,   411.064
   16,   441.138
   17,   467.791
   18,   499.963
   19,   531.042
   20,   559.656
   21,   589.864
   22,   620.279
   23,   650.192
   24,   680.713
   25,   710.491
   26,   740.934
   27,   766.226
   28,   801.618
   29,   831.503
   30,   861.913
   31,   891.762
   32,   921.623
   33,   949.040
   34,   980.577
   35,  1009.604
   36,  1039.672
   37,  1069.171
   38,  1098.195
   39,  1127.361
   40,  1156.043
   41,  1183.926
   42,  1216.107
   43,  1244.913
   44,  1274.907
   45,  1303.598
   46,  1332.216
   47,  1360.964
   48,  1389.595
   49,  1417.806
   50,  1446.821
   51,  1474.548
   52,  1493.034
   53,  1512.845
   54,  1533.250
   55,  1553.810
   56,  1574.784
   57,  1594.828
   58,  1615.219
   59,  1631.470
   60,  1654.473
   61,  1671.612
   62,  1693.719
   63,  1713.170
   64,  1732.179
   65,  1752.094
   66,  1770.919
   67,  1790.117
   68,  1808.289
   69,  1828.003
   70,  1846.868
   71,  1872.386
   72,  1891.147
   73,  1909.331
   74,  1927.426
   75,  1946.287
   76,  1960.520
   77,  1982.130
   78,  2000.099
   79,  2018.241
   80,  2035.320
   81,  2053.132
   82,  2070.958
   83,  2088.296
   84,  2102.113
   85,  2122.730
   86,  2139.770
   87,  2156.561
   88,  2173.863
   89,  2191.406
   90,  2207.880
   91,  2232.666
   92,  2250.956
   93,  2275.716
   94,  2292.520
   95,  2310.522
   96,  2327.861
   97,  2345.959
   98,  2363.320
   99,  2381.670
  100,  2399.386

2 way AFR

800GX2 2 way AFR

16xAA/16xAF 55 FPS



Frame, Time (ms)
    1,     0.000
    2,    23.251
    3,    37.420
    4,    58.645
    5,    77.613
    6,    92.450
    7,   114.450
    8,   133.021
    9,   151.792
   10,   167.241
   11,   187.880
   12,   201.923
   13,   223.636
   14,   237.733
   15,   254.805
   16,   276.932
   17,   293.148
   18,   313.911
   19,   334.683
   20,   354.056
   21,   374.022
   22,   392.897
   23,   411.962
   24,   426.175
   25,   443.987
   26,   467.272
   27,   486.385
   28,   506.120
   29,   525.663
   30,   543.978
   31,   562.958
   32,   582.167
   33,   595.963
   34,   617.535
   35,   634.010
   36,   653.931
   37,   667.928
   38,   689.534
   39,   708.618
   40,   722.425
   41,   744.731
   42,   763.371
   43,   782.292
   44,   799.396
   45,   818.075
   46,   831.492
   47,   853.070
   48,   866.957
   49,   882.967
   50,   904.478
   51,   918.501
   52,   934.723
   53,   956.465
   54,   969.880
   55,   986.265
   56,  1003.028
   57,  1026.244
   58,  1046.465
   59,  1063.674
   60,  1081.266
   61,  1094.882
   62,  1110.541
   63,  1131.252
   64,  1148.886
   65,  1166.458
   66,  1184.516
   67,  1197.354
   68,  1218.291
   69,  1233.764
   70,  1247.010
   71,  1267.064
   72,  1284.777
   73,  1297.860
   74,  1318.415
   75,  1336.000
   76,  1353.179
   77,  1369.781
   78,  1387.855
   79,  1404.872
   80,  1421.063
   81,  1437.393
   82,  1454.253
   83,  1466.459
   84,  1484.802
   85,  1500.703
   86,  1513.392
   87,  1530.615
   88,  1546.260
   89,  1561.870
   90,  1573.811
   91,  1592.431
   92,  1607.990
   93,  1623.619
   94,  1639.689
   95,  1652.883
   96,  1669.158
   97,  1684.951
   98,  1696.627
   99,  1713.890
  100,  1729.086

3 way AFR

16xAA/16xAF 1680x1050

3 Way AFR 9800GX2

66 FPS


Frame, Time (ms)
    1,     0.000
    2,    14.809
    3,    36.531
    4,    55.247
    5,    74.779
    6,    92.073
    7,   110.632
    8,   129.111
    9,   148.151
   10,   167.451
   11,   186.127
   12,   205.026
   13,   223.613
   14,   243.313
   15,   261.835
   16,   280.535
   17,   299.554
   18,   313.060
   19,   334.218
   20,   352.834
   21,   367.311
   22,   388.577
   23,   408.769
   24,   427.858
   25,   446.820
   26,   465.476
   27,   484.188
   28,   503.180
   29,   521.546
   30,   539.992
   31,   559.189
   32,   577.431
   33,   595.868
   34,   614.422
   35,   628.173
   36,   649.512
   37,   668.124
   38,   686.428
   39,   705.360
   40,   724.033
   41,   742.462
   42,   755.776
   43,   778.337
   44,   796.825
   45,   815.875
   46,   834.967
   47,   853.475
   48,   871.421
   49,   884.958
   50,   906.506
   51,   940.048
   52,   959.902
   53,   980.526
   54,  1000.534
   55,  1019.928
   56,  1034.381
   57,  1050.408
   58,  1071.971
   59,  1089.898
   60,  1108.124
   61,  1125.227
   62,  1142.622
   63,  1160.161
   64,  1173.424
   65,  1194.519
   66,  1211.533
   67,  1228.452
   68,  1240.837
   69,  1261.165
   70,  1278.234
   71,  1295.251
   72,  1308.040
   73,  1327.572
   74,  1344.743
   75,  1361.489
   76,  1378.773
   77,  1395.295
   78,  1411.521
   79,  1427.697
   80,  1441.494
   81,  1458.748
   82,  1474.552
   83,  1490.709
   84,  1506.731
   85,  1523.046
   86,  1538.790
   87,  1555.225
   88,  1571.991
   89,  1583.962
   90,  1598.445
   91,  1616.243
   92,  1631.849
   93,  1647.193
   94,  1662.706
   95,  1678.221
   96,  1693.423
   97,  1708.451
   98,  1722.795
   99,  1737.503
  100,  1752.506

4 way AFR

16xAA/16xAF 1680x1050

76 FPS

9800GX2 4 way AFR.



Frame, Time (ms)
    1,     0.000
    2,    17.615
    3,    38.653
    4,    56.662
    5,    71.880
    6,    87.698
    7,   109.398
    8,   123.345
    9,   139.849
   10,   156.585
   11,   173.332
   12,   194.883
   13,   213.259
   14,   226.749
   15,   243.810
   16,   266.096
   17,   284.397
   18,   302.953
   19,   321.641
   20,   340.009
   21,   353.346
   22,   375.211
   23,   393.943
   24,   411.867
   25,   425.231
   26,   447.935
   27,   466.789
   28,   482.066
   29,   496.718
   30,   512.705
   31,   533.635
   32,   551.839
   33,   569.545
   34,   587.134
   35,   599.860
   36,   616.424
   37,   636.653
   38,   654.724
   39,   673.927
   40,   694.781
   41,   712.353
   42,   729.735
   43,   742.743
   44,   764.688
   45,   777.433
   46,   793.458
   47,   808.783
   48,   829.520
   49,   847.586
   50,   865.191
   51,   877.281
   52,   893.493
   53,   913.825
   54,   931.091
   55,   948.839
   56,   966.513
   57,   984.168
   58,  1001.494
   59,  1014.018
   60,  1029.752
   61,  1050.008
   62,  1067.161
   63,  1084.260
   64,  1101.465
   65,  1118.299
   66,  1134.620
   67,  1151.449
   68,  1163.192
   69,  1178.216
   70,  1193.166
   71,  1211.795
   72,  1227.911
   73,  1244.068
   74,  1255.835
   75,  1270.442
   76,  1288.367
   77,  1303.010
   78,  1319.538
   79,  1334.829
   80,  1350.347
   81,  1369.091
   82,  1386.848
   83,  1402.680
   84,  1418.351
   85,  1429.626
   86,  1444.093
   87,  1461.741
   88,  1477.116
   89,  1492.554
   90,  1507.896
   91,  1519.476
   92,  1536.565
   93,  1551.513
   94,  1566.242
   95,  1580.087
   96,  1593.914
   97,  1608.435
   98,  1622.754
   99,  1636.617
  100,  1650.884

 

[Albuquerque]

Well shit, call me confused, but the 4-way SLI seems to have a pretty even frame render distribution -- am I seeing things? The latency is a different story, 70ms is a bit much. But in reality, sub-100ms latency over an internet connection is typically considered "very good" last I checked. That means, IMO, except in the case of LAN parties I don't see it being a huge detriment.

Thanks for posting up some real data Chris

 

[ChrisRay]

Yup. The worse case scenerio seems to be 3way as far as an unven bump goes. And even then i dont consider it a monsterous problem.

 

[Albuquerque]

I fiddled with my own crossfire setup this weekend, trying to figure out how to make it worse. Having VSYNC disabled is the easiest way, along with games that don't put enough stress on the video card.

One perfect example was HalfLife 2 EP2 with no AA or AF at 1680x1050 and vsync disabled. It was very apparent. But after turning up the AA and AF and enabling vsync, it was basically invisible. I don't recall if there's a way to enable triple buffering on these...

 

[ChrisRay]

Yes. Does Crossfire have a visual load balancer indicator like SLI? You can generally get an idea for how well GPU load balancing is occuring. Nvidia's load balancing is relatively dynamic. ((Though as I said through profile modifications you can try to change the way the dynamic mechanism is predicted)). So if your massively CPU limited in HL2. The balancing will change. And "should" not be too apparent. The biggest problems are when the load balancer fails to make a prediction and balancing goes off for 1 or 2 frames. This is what usually causes the micro stutter on SLI.

As I mentioned before though. This shouldnt be too common but some games are a little more erratic than others. I havent played the Need for Speed game here. But its probably having to repredict the balancing that is needed if theres a large stuttering that is happening regularly. But the more consistent the GPU load. The less of a chance that this will happen.

 

[Enforcer]

I thought I'd go ahead here and display UT3 on 9800GX2 cards from 1 GPU to 4 GPUs in latency. To give you guys a better idea of whats going on and what increasing the the amount of AFR is doing. I took only 100 frames in this case and
used the UT3 Deimos Map with 16xAA/16xAF

Those numbers are not readable as is. Here are some stats:

1 GPU
Average difference between adjacent frame times: 2.48838383838384 ms
Average frame time: 24.2362222222222 ms
Jitter in % : 10.2672100278988

2 GPU
Average difference between adjacent frame times: 3.80591919191921 ms
Average frame time: 17.4655151515152 ms
Jitter in % : 21.7910503005635

3 GPU
Average difference between adjacent frame times: 2.35152525252526 ms
Average frame time: 17.7020808080808 ms
Jitter in % : 13.2838917527245

4 GPU
Average difference between adjacent frame times: 2.78473737373737 ms
Average frame time: 16.675595959596 ms
Jitter in % : 16.6994773709116

 

[ChrisRay]

The data comes from UT3 and the Deimos map. Its accessible to anyone. If you want a perfect comparison you can check it and post it yourself. AFR will never have 100% identical frame distribution.

 

[Chris123234]

Well shit, call me confused, but the 4-way SLI seems to have a pretty even frame render distribution -- am I seeing things? The latency is a different story, 70ms is a bit much. But in reality, sub-100ms latency over an internet connection is typically considered "very good" last I checked. That means, IMO, except in the case of LAN parties I don't see it being a huge detriment.

Thanks for posting up some real data Chris

There is a huge difference between input lag and connection lag, especially when dealing with twitch shooters. The screen lag on my HDTV drives me insane when I play shooters.

 

[Albuquerque]

There is a huge difference between input lag and connection lag, especially when dealing with twitch shooters. The screen lag on my HDTV drives me insane when I play shooters.

Either one is going to induce targetting error, and if ~15ms is the average "jitter" then I'm not entirely convinced it is as evil as some make it out to be. A 100ms ping time is going to cause you more aiming / shooting problems than a 20ms stutter.

I'm not saying you can't see it and it doesn't bother you, but I still don't see this as some sort of make vs break situation. But I'm sure I'm wrong, and I would need to be a twitch FPS purist to know the difference.

Suffice it to say that I won't ever know the difference if that's the case.

 

[Arun]

I don't think Chris123234 was talking about stutter, just input lag in general. And indeed he's right: if your ping time is 100ms and your 'keyboard to visible on screen' latency is 20ms, then you'll see your movements after 20ms and *predicted* enemy movements too. If you have 20ms ping time and 100ms 'keyboard to visible on screen' latency, even your own movements are going to lag quite a lot. And don't even get me started if both are >100ms!

 

[Albuquerque]

Ok, so I need help understanding the engine vs display lag. I thought that's what we were just talking about?

How does this work, and how do you measure it?

 

[apoppin]

Are you guys paying attention to the micro stutter discussions going on at atf video? nHancers creator believes that a simple driver fix with a 2ms delay added to the first GPU will synch it!

i will look it up for you after work if you like. I am sorry to butt-in if you are already aware of what they concluded over the last few days. If not it is important to consider, i think

 

[digitalwanderer]

I'd be interested in a link to that one Apoppin.

 

[apoppin]

I'd be interested in a link to that one Apoppin.

my pleasure, dw

http://forums.anandtech.com/message...RDFRM=&STARTPAGE=13&FTVAR_FORUMVIEWTMP=Linear

Well, the answer is so easy I never figured that it would be a problem...

You do that by adding a one-time delay of 2 ms to the OTHER card. Then both cards are aligned again.

This post above post is by Grestorn. It is all a good discussion so you might want to start with the first page. It is a "tame" as the 'other' place gets.

http://forums.anandtech.com/message...ORDFRM=&STARTPAGE=1&FTVAR_FORUMVIEWTMP=Linear


... i guess i should have changed my username ... but what is done is done; i lurked here forever and you guys still do the best benchmarking and IQ comparisons [my new specialty, btw - i hope you can help me later figure it out better]

i can cut "aggressive" down by 90% but that's it .. after that i turn spineless and collapse into a pile of incomprehensible "polite" after that

 

[Betanumerical]

I have a 2 GPU solution and use AFR, I have never seen what you mean by 'stutter' the only real framedrop I have ever seen is when the framerate drops significantly when going underwater in TF2 for some reason.

 

[Chris123234]

I don't think Chris123234 was talking about stutter, just input lag in general. And indeed he's right: if your ping time is 100ms and your 'keyboard to visible on screen' latency is 20ms, then you'll see your movements after 20ms and *predicted* enemy movements too. If you have 20ms ping time and 100ms 'keyboard to visible on screen' latency, even your own movements are going to lag quite a lot. And don't even get me started if both are >100ms!

Here is an exaggerated example:

Ever play a shooter where you shoot a guy in the head, and a second later he dies due to lag? There is no display lag there, but the connection lag was present.

Now try running around and playing with a second long display lag and no connection lag.

While the display lag is obviously not a second long, even the slight display lag in my plasma TV, when used as a monitor, drives me crazy even in just the desktop environment, much less games.

For a console example, may Rock Band without the input delay for HD TV's and watch yourself fail miserably in 10 seconds.

 

[digitalwanderer]

Thanks for the links Appopin, I think I remember you now. You're a regular over at Ananananananda Tech aren't you? And an anti-Rollo one at that too if I recall..

Am I right? (Sorry, I got a shit memory but names stick. It either means you were a buddy or an antagonist, but i'm almost never sure. )

 

[apoppin]

Thanks for the links Appopin, I think I remember you now. You're a regular over at Ananananananda Tech aren't you? And an anti-Rollo one at that too if I recall..

Am I right? (Sorry, I got a shit memory but names stick. It either means you were a buddy or an antagonist, but i'm almost never sure. )

you are welcome . . . and you have a good memory

i hate to admit it now .. it's called "infamy"
- and you would be thinking of my mother - Mary Poppins
=D

Yes i may have been "AnandTech's apoppin" but no more; i learned and escaped out of their Klein's bottle. --[it is easy when you realize there are more than 4 dimensions and you were never really "trapped"]
--and if i post another "emoticon" i will puke!

yep, i did expose AEG viral, well over 2 years ago [my career start was "investigative journalism" - in 1972 in Dublin] .. but things have changed. Rollo has become my friend and he is more transparent now although old habits die hard and he will always be "Mr nVidia" - you would think it is "his" company. But then you realize he has bought into the nVidia Vision completely .. and i am trying hard NOT to be converted; but i am currently switching out my 8800GTX and my own HD2900 Crossfire; back-and forth for a IQ comparison .. and i am impressed with the single-GPU; and i have no more hope for r700 either against GT200 .. so .. uh ...

where are the white paper discussions?

Benchmarking and IQ comparisons are my new passion .. not industry analysis and speculation although i will keep on reading what is being written - even by supposed "pros" - after all even i need a good laugh, now and again at the silly predictions they make; and no one EVER calls them on it afterward [think inq]

I have a 2 GPU solution and use AFR, I have never seen what you mean by 'stutter' the only real framedrop I have ever seen is when the framerate drops significantly when going underwater in TF2 for some reason.

read the thread i lniked to - if you can stand it; there are some good people in there [ignore me].

MS should be apparent with AFR; however with CrossFire, if you use Xfire AA you no longer use that AFR rendering and Micro Stutter goes away

Frankly, my own conclusion is that some of us are really *bugged* by it - like some people with "input" lag and triple buffering issues; others never notice it at all!

i believe that PC graphics is all compromise .. the willingness to "suspend disbelief" and some of us pick at one issue more than another because of the way we notice things - or perhaps our set up minimizes what other really notice.

For me, a second GPU allows me to turn up the AA at 16x10 or 16x12 .. i don't like 30" displays and have a 16x10 20" Samsung and a 19" CRT [well and a 14x9 and .. ]; i always prefer the 2nd GPU, if it scales

Frankly, i used to be an "audiophile tweak" with an overpriced Tube preamp Harmon Kardon Citation modified by Mark Levinson himself, you can NITPICK anything and i also see "Micro stutter" in a single GPU [very occasionally]
- but it does not bother me as i am a GAMER .. not a HW elitist as some also are; i know a guy that wears headphones because he cannot STAND the whisper of his Ultra. My own crossfire can scream and i can blow dry my hair at the same time, for all i care, as long as it looks and plays OK

 

[Betanumerical]

read the thread i lniked to - if you can stand it; there are some good people in there [ignore me].

MS should be apparent with AFR; however with CrossFire, if you use Xfire AA you no longer use that AFR rendering and Micro Stutter goes away

Frankly, my own conclusion is that some of us are really *bugged* by it - like some people with "input" lag and triple buffering issues; others never notice it at all!

i believe that PC graphics is all compromise .. the willingness to "suspend disbelief" and some of us pick at one issue more than another because of the way we notice things - or perhaps our set up minimizes what other really notice.

For me, a second GPU allows me to turn up the AA at 16x10 or 16x12 .. i don't like 30" displays and have a 16x10 20" Samsung and a 19" CRT [well and a 14x9 and .. ]; i always prefer the 2nd GPU, if it scales

Frankly, i used to be an "audiophile tweak" with an overpriced Tube preamp Harmon Kardon Citation modified by Mark Levinson himself, you can NITPICK anything and i also see "Micro stutter" in a single GPU [very occasionally]
- but it does not bother me as i am a GAMER .. not a HW elitist as some also are; i know a guy that wears headphones because he cannot STAND the whisper of his Ultra. My own crossfire can scream and i can blow dry my hair at the same time, for all i care, as long as it looks and plays OK

I'm reading it now, but the odd thing isn't that I don't notice a difference with my SLI turned on compared to it turned off (for input lag), the weird thing is that i'm really quite sensitive to framedrops and low framerate and yet I didn't notice a thing when changing from single GPU to dual GPU when playing first person games (TF2, UT3, etc).

 

[apoppin]

I'm reading it now, but the odd thing isn't that I don't notice a difference with my SLI turned on compared to it turned off (for input lag), the weird thing is that i'm really quite sensitive to framedrops and low framerate and yet I didn't notice a thing when changing from single GPU to dual GPU when playing first person games (TF2, UT3, etc).

Make sure that you "strain" the graphics system ... if you are getting 50 FPS it is not very noticeable

it is a momentary "stutter" and rather different from input lag [although most people really don't see it either]