Network packet loss due to audio usage, Win7

[Comark Corp]

Hello,

I've come across a rather strange problem when testing under Windows 7. I am running all standard tests except USB, tape, and printer. I have set the network test set to fail if packet losses exceed .1%. When running the tests, the systems will fail after approx. 20 minutes with excessive network packet loss. There are two NICs in the system, and I have set Burnintest to test all NICs with each NIC pointing to a different server. If I remove just the sound tests, there is 0% network packet loss even after running for 12 hours. This same failure mode was observed on three different platforms using all different motherboards - one a 945GME, one a 965GME and the last a Q45 chipset. They all have Realtek audio chipsets, but the chips are different between them (ALC888, ALC655, etc.) At this point I was thinking that the driver might be to blame, as all these have the latest driver versions from Realtek, but there are no previously released drivers available to try. So, I instead disabled the onboard audio and installed a PCI audio card using a Crystal Media chipset using completely different drivers. The failure persists. I then removed the audio test from Burnintest, and instead played a .mp3 file with Windows media player. The failure persists. In fact, even when media player is *paused* and not playing the .mp3 file (but the file is locked), Burnintest STILL fails with packet loss. Tested both 5.3 build 1036 and 6.0 build 1017. I'm beginning to think this is a problem with Windows 7. DRM or something? Note that If I only test the MIDI function with Burnintest that the failure still occurs.

Hoping you guys have some ideas, I'm pretty much out of them.

Thanks.

Jay

[Ian (PassMark)]

Jay,

I have not heard of this problem and I don't know. It sounds like a chipset hardware issue when under load. I had a look at some of the related Intel specification updates, e.g. http://www.intel.com/Assets/PDF/specupdate/316274.pdf, and note that there are some issues related to DMA Transfer completion for DMA sensitive devices in some scenarios. Depending on your setup, maybe it is related to this.

Also, http://www.intel.com/Assets/PDF/specupdate/313057.pdf, refers to
"Intel ICH8 Integrated LAN DMA Error" and "ICH8 GbE Packet Buffer Writing Error". Less likely I suspect.

I would look at updating your motherboard BIOS.

I would test XP on the same system. The specification updates do refer to specific Vista (and hence probably Win7) issues, but none seemed related.

It would be interesting to remove some load and only run the Network and audio tests to see if the problem still occurs. Also I don't know if both of your NICs are on the motherboard and you get errors on both NICs.

Regards,
Ian

[Comark Corp]

Jay,

I have not heard of this problem and I don't know. It sounds like a chipset hardware issue when under load. I had a look at some of the related Intel specification updates, e.g. http://www.intel.com/Assets/PDF/specupdate/316274.pdf, and note that there are some issues related to DMA Transfer completion for DMA sensitive devices in some scenarios. Depending on your setup, maybe it is related to this.

Also, http://www.intel.com/Assets/PDF/specupdate/313057.pdf, refers to
"Intel ICH8 Integrated LAN DMA Error" and "ICH8 GbE Packet Buffer Writing Error". Less likely I suspect.

I would look at updating your motherboard BIOS.

I would test XP on the same system. The specification updates do refer to specific Vista (and hence probably Win7) issues, but none seemed related.

It would be interesting to remove some load and only run the Network and audio tests to see if the problem still occurs. Also I don't know if both of your NICs are on the motherboard and you get errors on both NICs.

Regards,
Ian

Hi Ian,

Thanks for your reply.
If I was only seeing the problem on a particular motherboard, it wouldn't bother me so much. BUT every motherboard I've tested so far (with different chipsets) exhibits the failure. I would think this would eliminate errata on particular chipsets and the likelihood of it being a BIOS problem.

None of the three motherboards I've tested has this anomaly when running Windows XP. They all pass testing. One of the motherboards uses the Q45 chipset and ICH10 to which neither of those specification updates would apply - it is also using core 2 quad CPU which I would think would eliminate the CPU being overloaded. All of these motherboards have dual Intel GB PCI express NICs built into the MB and the packet loss occurs on both.

I've tried setting the duty cycle to 25% for all tests and the problem persists. I've also tried testing just one LAN port, and it still fails. I'll do some more experiments by removing tests other than audio and LAN, and see what happens. Not that that would be a solution, however...

What is the strangest part as I mentioned before: remove the audio test and it passes, BUT just having an MP3 file file locked in Windows media player BUT NOT PLAYING - just paused; and it fails. Weird.

Jay

[Comark Corp]

Hi Ian,

Thanks for your reply.
If I was only seeing the problem on a particular motherboard, it wouldn't bother me so much. BUT every motherboard I've tested so far (with different chipsets) exhibits the failure. I would think this would eliminate errata on particular chipsets and the likelihood of it being a BIOS problem.

None of the three motherboards I've tested has this anomaly when running Windows XP. They all pass testing. One of the motherboards uses the Q45 chipset and ICH10 to which neither of those specification updates would apply - it is also using core 2 quad CPU which I would think would eliminate the CPU being overloaded. All of these motherboards have dual Intel GB PCI express NICs built into the MB and the packet loss occurs on both.

I've tried setting the duty cycle to 25% for all tests and the problem persists. I've also tried testing just one LAN port, and it still fails. I'll do some more experiments by removing tests other than audio and LAN, and see what happens. Not that that would be a solution, however...

What is the strangest part as I mentioned before: remove the audio test and it passes, BUT just having an MP3 file file locked in Windows media player BUT NOT PLAYING - just paused; and it fails. Weird.

Jay

I found that if I remove the CPU and 3D tests, the test will then pass. Since that's really not a solution for us, I did some more testing and found that if I use the Windows 7 troubleshooting utility, and run the diagnostics in Windows XP service pack 3 compatibility mode, all the tests pass! There are still some lost packets, but they are well under the .1% loss that I have set in the config.

[Ian (PassMark)]

It is hard to say, but we had an interesting problem a few weeks back, where CPU SIMD errors were reported only when the audio test was running. This occurred under XP and not Vista (unknown in Win7). In this case it turned out that the XP audio device driver was changing some SIMD 'compatability' features at the kernal level to overcome a 'limitation' of the XP driver. This effected some SIMD operations if being processed by the CPU at the same time as certain audio driver operations. The audio vendor had removed this limitation in the Vista driver.

While this does not answer your specific problem, I do wonder if it could be something related (but not the same), in terms of different drivers setting different CPU settings, but another operation occurring at the same time on the CPU that requires different setting - i.e. some sort of driver incompatability ('limitation'). The CPU and 3D tests are the 2 tests exercising SIMD. I don't know how this would impact the network test (but could probably come up some theories).

To fully solve the problem, I think you would need to identify the relevant common drivers between all your platforms and then contact the drivers vendors.

Edit 1: It could also be a Win7 operating system issue.

Regards,
Ian

[Comark Corp]

It is hard to say, but we had an interesting problem a few weeks back, where CPU SIMD errors were reported only when the audio test was running. This occurred under XP and not Vista (unknown in Win7). In this case it turned out that the XP audio device driver was changing some SIMD 'compatability' features at the kernal level to overcome a 'limitation' of the XP driver. This effected some SIMD operations if being processed by the CPU at the same time as certain audio driver operations. The audio vendor had removed this limitation in the Vista driver.

While this does not answer your specific problem, I do wonder if it could be something related (but not the same), in terms of different drivers setting different CPU settings, but another operation occurring at the same time on the CPU that requires different setting - i.e. some sort of driver incompatability ('limitation'). The CPU and 3D tests are the 2 tests exercising SIMD. I don't know how this would impact the network test (but could probably come up some theories).

To fully solve the problem, I think you would need to identify the relevant common drivers between all your platforms and then contact the drivers vendors.

Edit 1: It could also be a Win7 operating system issue.

Regards,
Ian

The only common drivers seem to be the Intel NIC and graphics drivers. I disabled the onboard Intel NICs and instead added 2 Realtek NIC boards. Same failure.

I then disabled the on-board Intel graphics controller and installed a Nvidia card and driver. Same failure.

I agree that it could be a Win7 problem, but I'd rest easier if other people were seeing this bug. Dumb question perhaps, but are you saying that you've tested similar configurations and not seen anything?
I've emailed my test.bitcfg as I do not see a way to attach a file. Could you see if this works for you after changing the network settings to reflect your own servers?

Thanks again.

[Ian (PassMark)]

I re-tested on 2 different Windows 7 systems with your configuration(*) and could not reproduce the problem you reported.

(*) Whilst it may not be relevant, our hardware is newer hardware and does not have a Parallel or Serial port, so I excluded these tests. I don't believe this would have impacted the results.

Out of interest, is your hardware more than a couple of years old?

Regards,
Ian

[Comark Corp]

I re-tested on 2 different Windows 7 systems with your configuration(*) and could not reproduce the problem you reported.

(*) Whilst it may not be relevant, our hardware is newer hardware and does not have a Parallel or Serial port, so I excluded these tests. I don't believe this would have impacted the results.

Out of interest, is your hardware more than a couple of years old?

Regards,
Ian

Hi Ian,

The three motherboards I have tried this on so far are:
http://www.commell.com.tw/Product/SBC/LV-67A.HTM
http://www.commell.com.tw/Product/SBC/FS-97C.HTM
http://us.msi.com/index.php?func=pro...8&prod_no=1267

Being in the industrial field, many of our customers require serial and parallel ports as well as long-life hardware. Also, the first one is using the Q45 chipset and is quite new.

Just in case, I removed the serial and parallel port tests, but the failure persists.

[Ian (PassMark)]

I don't know what the problem is.

I guess you have tried anything I would suggest, but maybe you could try different destination addresses, a longer timeout, different cables, removing/changing any intermediate networking gear, updating audio and network device drivers, getting the latest Windows patches, testing the Network ports one at a time (just to see if you get the same result), increasing the ratio value (if you think it may be appropriate) ...

Regards,
Ian

[Comark Corp]

I don't know what the problem is.

I guess you have tried anything I would suggest, but maybe you could try different destination addresses, a longer timeout, different cables, removing/changing any intermediate networking gear, updating audio and network device drivers, getting the latest Windows patches, testing the Network ports one at a time (just to see if you get the same result), increasing the ratio value (if you think it may be appropriate) ...

Regards,
Ian

Hello Ian,

Finally narrowed this down some more using a packet sniffer. We have a SolidWorks server on our network which is broadcasting UDP packets. This appears to screw up the network tests on machines that are running Windows 7, but only Windows 7 (not XP or 2000)
At this point we added a router and put the machines running BurnInTest behind it, isolating the Solidworks server from the machines running BurnInTest. All the network tests now pass. Our software guy created a small executable and script which simulates the SolidWorks server so we could see if it would break the test again. It does. I don't know if you folks would be interested in this, but I've emailed the .zip (could not find a way to attach the file) with the two needed files to in case you would like to investigate further. This could be more of a problem with Windows 7 than anything else.

Thanks.

[Ian (PassMark)]

Thanks. I sent you a direct email regarding this. I will test it when I get the file.

[Comark Corp]

Thanks. I sent you a direct email regarding this. I will test it when I get the file.

Hello Ian,

I was wondering if you had had a chance to see if the program causes network issues with Windows 7 boxes yet? I'd just like to see if you folks get the same results we do - Windows XP systems are unaffected while Windows 7 boxes are. If you are seeing the same result as us, we'd be very curious to see if you think this could be a bug with Win7...

Thanks.

[Ian (PassMark)]

Sorry for the delay in responding.

Do you mean, does the UDP broadcast program you sent us cause timeout on the BurnInTest standard network test? If so, yes it does.

I can confirm that a constant UDP broadcast on the network (like from your server for license validation) causes the standard network test to have timeouts on the Windows 7 systems we have tested and not the Windows XP systems we have tested. We have not yet tested the different Operating systems on the same hardware, and will set up a system with both Operating systems and test this. At this stage it would seem likely based on your testing and our testing to date that UDP and ICMP handling of load may be different under Windows 7, when compared to XP. I'll post the results of the test from the same hardware (probably won't be until Tuesday though).

[Ian (PassMark)]

I can confirm that a constant UDP broadcast on the network (like from your server for license validation) causes the standard network test to have timeouts on the Windows 7 systems we have tested and not the Windows XP systems we have tested. We have now confimed this when using the 2 operating systems on identical hardware.

I did some investigation with a software protocol analyzer and found that when the timeout occurs, the test packet is sent correctly, but the reply is not received (in Windows). It would seem plausible that this could be a Windows 7 ICMP/UDP broadcast load handling issue.

[Comark Corp]

I can confirm that a constant UDP broadcast on the network (like from your server for license validation) causes the standard network test to have timeouts on the Windows 7 systems we have tested and not the Windows XP systems we have tested. We have now confimed this when using the 2 operating systems on identical hardware.

I did some investigation with a software protocol analyzer and found that when the timeout occurs, the test packet is sent correctly, but the reply is not received (in Windows). It would seem plausible that this could be a Windows 7 ICMP/UDP broadcast load handling issue.

Hi Ian,

Well, I'm glad to see that we get the same results. Did you also happen to notice that if the sound tests are dropped, the Win 7 boxes will pass? (Note that I did not test with the program I sent you, but rather on the same network where the Solid Works server is located (No router in between.)

Thanks for all your help.