We have a relatively simple network setup where we have a VM running on a HP DL380 G6 host with VMware ESXi 5.5 (build 2068190) installed, that is communicating with several physical servers connected on the same switch (Cisco 2960X stack).
The NIC teaming mode has been reduced to basic level (ESXi: originating port id + 1 vmnic uplink, Switch: regular trunk port) to pinpoint the root cause of the issue.
The host has multiple networkcards:
~ # esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:02:00.00 bnx2 Up 1000Mbps Full d8:d3:85:ad:4b:7c 1500 Broadcom Corporation NC382i
Integrated Multi Port PCI Express Gigabit Server Adapter
vmnic1 0000:02:00.01 bnx2 Up 1000Mbps Full d8:d3:85:ad:4b:7e 1500 Broadcom Corporation NC382i
Integrated Multi Port PCI Express Gigabit Server Adapter
vmnic2 0000:03:00.00 bnx2 Down 0Mbps Half d8:d3:85:ad:4b:80 1500 Broadcom Corporation NC382i
Integrated Multi Port PCI Express Gigabit Server Adapter
vmnic3 0000:03:00.01 bnx2 Up 1000Mbps Full d8:d3:85:ad:4b:82 1500 Broadcom Corporation NC382i
Integrated Multi Port PCI Express Gigabit Server Adapter
vmnic4 0000:0f:00.00 e1000e Down 0Mbps Full 00:15:17:aa:1b:7d 1500 Intel Corporation 82571EB Gigabit
Ethernet Controller (Copper)
vmnic5 0000:0f:00.01 e1000e Up 1000Mbps Full 00:15:17:aa:1b:7c 1500 Intel Corporation 82571EB Gigabit
Ethernet Controller (Copper)
vmnic6 0000:10:00.00 e1000e Down 0Mbps Half 00:15:17:aa:1b:7f 1500 Intel Corporation 82571EB Gigabit
Ethernet Controller (Copper)
vmnic7 0000:10:00.01 e1000e Down 0Mbps Half 00:15:17:aa:1b:7e 1500 Intel Corporation 82571EB Gigabit
Ethernet Controller (Copper)
vmnic8 0000:0a:00.00 sfc Up 10000Mbps Full 74:46:a0:72:7e:ac 1500 Solarflare SFC9020 [Solarstorm]
vmnic9 0000:0a:00.01 sfc Up 10000Mbps Full 74:46:a0:72:7e:ad 1500 Solarflare SFC9020 [Solarstorm]
~ # esxcfg-vswitch -l
vSwitch1 1536 5 128 1500 vmnic8,vmnic9
PortGroup Name VLAN ID Used Ports Uplinks
Production 4095 0 vmnic8,vmnic9
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch2 1536 11 128 1500 vmnic5
PortGroup Name VLAN ID Used Ports Uplinks
BE001-Public 999 2 vmnic5
We encountered some strange network behavior when the VM (.13) was communication with a host (.1) over vSwitch2 using a uplink nic with the e1000e driver.
As verification we used ICMP packets, which gave us an error the packet was changed on the way. Tcpdumps on the different hosts show that:
- the ICMP Request is arriving correctly at the destination host.
- the ICMP Responds is correctly sent at the destination host
- the ICMP Responds has been modified when arrived on the VM
Ping cmd and output on the VM (.13)
Tcpdump on the VM (.13) - initiator of the ping:
Tcpdump on the destination host (.1) - responder to the ping:
Tcpdump on another VM on the same vSwitch using promiscuous mode - just a witness:
However this does not prove anything, we see perfect communication when switching the uplink port for vSwitch2 to vmnic3, which is using the bnx2 driver.
Next step would be to trace the communication on the egress side of the switch to see if the packets are correctly sent to the VMware host, but a switch should not interfere on that level.
Leaving me with the idea that the network card / e1000e driver is corrupting the inbound packets on this VMware host, however it's listed in the VMware HCL...
Has anyone seen this behavior before with the e1000e driver or Intel 82571EB network card?