Hi,
I have had an SR open with VMware support for a month now but we are no closer to a fix. I was hoping someone has experienced a similar problems and may have the solutions.
I am seeing dropped received packets for all my Windows VM but none of the RHEL VM under ESXtop. Netstat indicates issues too
The issue appears for Windows VM regardless whether it is 2003,2008 or 2012. Most of NIC are using vmxnet3 but I am seeing the issue regardless of NIC driver in use. I am seeing this at physically separated sites but only in those that are used for replication.
The VM are connected to a VDS which makes use of 10 Gbps connectivity and are located on two different clusters which make use of same storage backend (IBM N-Series 6240).
I have come to the following conclussions:
Site A:
-It is not vDS related as I moved VM to a vSS and packets still drop
-It is not pNIC related as my two clusters ( One HP DL380 and other DL560 )use different hardware and different type of pNIC. Therefore it is not a driver issue and I tried upgrading the driver
-Although no sign of contention, I evacuated hosts of all VM except one to rule out it out. I still noticed the dropped packets.
-I moved VM to local storage just in case there is something storage related but I still saw dropped packets
Site B:
-Same behaviour and these hosts have completely different hardware (IBM hosts) so not hardware or driver related.
Site C:
-I have a couple of DL380 hosts that make up a cluster and this one does not have any issues. It uses same make of storage backend but different model. All networking is configured the same and VM's are rolled from same template.
The thing that Site A and B have in common is that they use same type of backend storage. These are used for replication so we can use SRM.
Some things done in an attempt to fix it:
-Ensure that ESXi was patched to 5.1 U2. This did not make any difference
-Tested adjusting the buffers for the guest OS nic as per VMware's advice. This did not make a difference
-VMware asked to create vanilla windows VM. No software or joined to domain. Problem still exist.
One thing that made the problem disappear is altering the guest OS MTU to 9000. I just tried this out of desperation and VMware has not given me an explanation as to why this would be.
We never had issues with MTU 1500 in guest OS and prefer not to change this. We also ran a packet trace on one of the VM and noticed that the VM actually creates packages that are bigger than 1500 bytes even though the MTU is set to 1500.