I've been using VMXNET3 for my network device and recently implemented SR-IOV. I've noticed a huge drop in performance.
Host is SuperMicro X9DRH-7F, 2x E5-2667v2, nic is x520-DA1 connected to my switch via DAC.
I don't know what my max output is across the physical network as I don't have any other 10Gb clients. iperf and ping show good 1Gb performance.
Using iperf between guests on the same host with VMXNET3 I get a little over 6Gbps.
Switching to SR-IOV, iperf from FreeBSD server dropped as low as 3Gbps to my windows VM and 4.8Gbps to my Ubuntu VM.
Doing a loopback test with iperf I get 40+Gbps on the FreeBSD (FreeNAS) VM, 9.5Gbps on the windows VM and 25Gbps on the Ubuntu VM.
I've tried TSO and LRO, a number of different combinations of sysctl tuneables and the changes are very little if anything at all. Only switching back to VMXNET3 gets me back to 6Gbps.
I would appreciate some help narrowing this issue down. I'm hoping to achieve 9+Gbps, or even more between guests on the same host.