The case of high amount of Broadcast traffic

I was looking into a different kind of problem the other day when I had to do a network dump on one of our servers.

As I was watching the network dump scrolling down the my screen I kept noticing all the DHCP Requests that was flying by my screen.
Something wasn’t right.
I contacted our networking department and they had already a case on that subnet with high amounts of Broadcasts. 
I decided to take the first DHCP Request and see what was happening.
I filtered the requests on the EthernetAddress and started looking for a pattern.
After studying the DHCP Request I found that the client requesting didn’t get any answers. And it kept sending request pretty often. 
After looking at the time stamp it sends out an Request and then after 2 seconds, it sends a new one doubling the number of seconds to time out. 
It is sending out in the following patteren: 0 – 2 – 4 – 8  and last 16 seconds before it started the same procedure again. 
Since the EthernetAddress didn’t get any address on our network, no DHCP on our Server scopes, 
I had to get our networking to find the port the MAC was sitting on. 
After they directed me to the correct networking port I could log on to the attached server and check it’s hardware.
When I was studying our Windows 2008 R2 server I couldn’t find any mac address with the getmac command on the server.
What could it be?
After looking into the packet abit more I saw that it had a VendocClassIdentifier called: brcmftsk.
Tried to google it but it didn’t return to many good results.
In the middle of the lunch break, as I was discussing the matter with an college, he tipped me of with an article[1] regarding Broadcom and DHCP.
The author of the article has exactly the same problem as me, but he solved it years before I knew it was a problem 🙂
As it turns out, all our Windows servers are requesting an address for its iSCSI adapter on our network.
This was the default configuration on the iSCSI Adapter and according to the article it had to be turned of manually.
Since we didn’t use iSCSI or DHCP on the server segment we didn’t notice any depletion in the IP Scope or disruption on the iSCSI service.
But after having 200+ servers in the same segment requesting DHCP request for both of their iSCSI adapters, the amount of Broadcasts was questionable.
But how do you turn configure the iSCSI on 200+ servers? It is not manually at least that I know.
The best way of doing it is, doing it properly, install the DroadCom Managed Applications Control Suite and configure it there. 
But not all our servers has this suite installed and that would require extra downtime and planing. 
So our quick and dirty workaround for this issue was to disable the driver on our servers.
The driver service for the iSCSI Adapter is named BXOIS in the registry and is a Kernel loaded driver.
We decided that we should just disable this service and then the driver will be disabled.[3] 
As we have an Active Directory environment, we added this to the default configuration for all our Windows Servers.
After we applied the Group Policy we could se the registry had changed properly.
PS C:\> reg query  HKLM\System\CurrentControlSet\Services\BXOIS
    Start    REG_DWORD    0x4
After rebooting our server the following iSCSI devices was listed like this in Devmgmt.msc
DeviceDriver disabeld
The warnings sign says the following on the driver when you open it:
A driver (service) fir this device has been disabled. An alternate driver may be providing this functionality. (Code 32).
After we deployed this to all our servers we saw that the Broadcasts on our network dramaticly dropped.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s