Investigations

The case of high amount of Broadcast traffic

I was looking into a different kind of problem the other day when I had to do a network dump on one of our servers.

As I was watching the network dump scrolling down the my screen I kept noticing all the DHCP Requests that was flying by my screen.
Something wasn’t right.
 
I contacted our networking department and they had already a case on that subnet with high amounts of Broadcasts. 
 
I decided to take the first DHCP Request and see what was happening.
I filtered the requests on the EthernetAddress and started looking for a pattern.
 
Image
 
After studying the DHCP Request I found that the client requesting didn’t get any answers. And it kept sending request pretty often. 
After looking at the time stamp it sends out an Request and then after 2 seconds, it sends a new one doubling the number of seconds to time out. 
It is sending out in the following patteren: 0 – 2 – 4 – 8  and last 16 seconds before it started the same procedure again. 
 
Since the EthernetAddress didn’t get any address on our network, no DHCP on our Server scopes, 
I had to get our networking to find the port the MAC was sitting on. 
After they directed me to the correct networking port I could log on to the attached server and check it’s hardware.
 
When I was studying our Windows 2008 R2 server I couldn’t find any mac address with the getmac command on the server.
 
What could it be?
 
After looking into the packet abit more I saw that it had a VendocClassIdentifier called: brcmftsk.
brcmtfsk
 
Tried to google it but it didn’t return to many good results.
 
In the middle of the lunch break, as I was discussing the matter with an college, he tipped me of with an article[1] regarding Broadcom and DHCP.
 
 
The author of the article has exactly the same problem as me, but he solved it years before I knew it was a problem 🙂
 
As it turns out, all our Windows servers are requesting an address for its iSCSI adapter on our network.
This was the default configuration on the iSCSI Adapter and according to the article it had to be turned of manually.
 
Since we didn’t use iSCSI or DHCP on the server segment we didn’t notice any depletion in the IP Scope or disruption on the iSCSI service.
But after having 200+ servers in the same segment requesting DHCP request for both of their iSCSI adapters, the amount of Broadcasts was questionable.
 
But how do you turn configure the iSCSI on 200+ servers? It is not manually at least that I know.
 
The best way of doing it is, doing it properly, install the DroadCom Managed Applications Control Suite and configure it there. 
But not all our servers has this suite installed and that would require extra downtime and planing. 
 
So our quick and dirty workaround for this issue was to disable the driver on our servers.
 
The driver service for the iSCSI Adapter is named BXOIS in the registry and is a Kernel loaded driver.
We decided that we should just disable this service and then the driver will be disabled.[3] 
As we have an Active Directory environment, we added this to the default configuration for all our Windows Servers.
 
After we applied the Group Policy we could se the registry had changed properly.
 
PS C:\> reg query  HKLM\System\CurrentControlSet\Services\BXOIS
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\BXOIS
    Start    REG_DWORD    0x4
 
After rebooting our server the following iSCSI devices was listed like this in Devmgmt.msc
 
DeviceDriver disabeld
 
The warnings sign says the following on the driver when you open it:
A driver (service) fir this device has been disabled. An alternate driver may be providing this functionality. (Code 32).
 
After we deployed this to all our servers we saw that the Broadcasts on our network dramaticly dropped.
 
Sources:
 
 
 
 
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s