VMQ issues with NIC Teaming

hi

apologies if long 1 thought more information can provide better.

we have designed , built new hyper-v environment client, utilising windows server r2 / system centre 2012 r2 since putting production, seeing problems virtual machine queues. these manifest either high latency inside virtual machines (we’re talking 200 – 400 msec round trip times), packet loss or complete connectivity loss vms. not vms affected problem manifest on hosts. aware of these issues having cropped in past broadcom nics.

i'll give little bit of background problem...

frist, environment based entirely on dell hardware (equallogic storage, powerconnect switching and pe r720 vm hosts). environment based on server 2012 , decision taken bring speed r2. due number of quite compelling reasons, surrounding reliability. core virtualisation infrastructure consists of four vm hosts in hyper-v cluster.

prior redesign, each vm host had 12 nics installed:

• quad port on-board broadcom 5720 daughter card: 2 nics assigned host management team whilst other 2 nics in same adapter formed live migration / cluster heartbeat team, vm switch connected 2 vnics exposed management os. latest drivers , firmware installed. converged fabric team here configured in lacp address hash (min queues mode), each nic having same 2 processor cores assigned. management team identically configured.

• two additional intel i350 quad port nics: 4 nics teamed production vm switch uplink , 4 iscsi mpio. latest drivers , firmware. vm switch team spans both physical nics provide level of nic level fault tolerance, whilst remaining 4 nics iscsi mpio balanced across 2 nics same reasons.

the initial driver upgrading once again seeing issues vmq in old design converged fabric design. 2 vnics in management os each of these networks tagged specific vlans (that accessible same designated nics in each of vm hosts).

in setup, similar issue being experienced our present issue. once again, converged fabric vnics in host os on occasion, either lose connectivity or exhibit high round trip times , packet loss. seemed correlate significant increase in bandwidth through converged fabric, such when initiating live migration , affect both vnics connectivity. cause packet loss / connectivity loss both live migration , cluster heartbeat vnics in turn trigger sorts of horrid goings on in cluster. if disabled vmq on physical adapters , team multiplex adapter, problem went away. disabling vmq don’t want resort to.

so…. decision refresh environment 2012 r2 across board (which driven other factors , not issue alone) accelerated.

in new environment, replaced quad port broadcom 5720 daughter cards in hosts new intel i350 qp daughter cards keep nics identical across board. cluster heartbeat / live migration networks use smb multichannel configuration, utilising the same 2 nics in old design in 2 isolated untagged port vlans. part of re-design working (live migrations complete faster hasten add!!)

however…. same vmq issues we witnessed previously have arisen on production vm switch used uplink virtual machines on each host outside world.

the production vm switch configured follows:

• same configuration original infrastructure: 4 intel 1gbe i350 nics, 2 of in 1 physical quad port nic, whilst other 2 in identical nic, directly below it. remaining 2 ports each card function iscsi mpio interfaces san. did try , achieve nic level fault tolerance. latest firmware , drivers have been installed hardware (including nics) fresh latest dell server updates dvd (v14.10).

• in each host, above 4 vm switch nics formed switch independent, dynamic team (sum of queues mode), each physical nic has rss disabled , vmq enabled and team multiplex adapter has rss disabled vmq enabled. secondly, each nic configured use single processor core vmq. sum of queues team, cores not overlap , host processors have hyper threading enabled, cores (not logical execution units) assigned rss or vmq. configuration of vm switch nics looks follows when running get-netadaptervmq on hosts:

name                           interfacedescription              enabled basevmqprocessor maxprocessors numberofreceive
                                                                                                        queues
----                           --------------------              ------- ---------------- ------------- ---------------
vm_switch_eth01                intel(r) gigabit 4p i350-t a...#8 true    0:10             1             7
vm_switch_eth03                intel(r) gigabit 4p i350-t a...#7 true    0:14             1             7
vm_switch_eth02                intel(r) gigabit 4p i350-t ada... true    0:12             1             7
vm_switch_eth04                intel(r) gigabit 4p i350-t a...#2 true    0:16             1             7
production vm switch           microsoft network adapter mult... true    0:0                            28

load hardly issue on these nics , single core seems have sufficed in old design, carried forward new.

the loss of connectivity / high latency (200 – 400 msec before) seems arise when vm moved via live migration host host. if setup constant ping test candidate vm , move host, 5 dropped pings @ point remaining memory pages / cpu state transferred, followed dramatic increase in latency once vm , running on destination host. seems though destination host struggling allocate vm nic queue. can move vm , forth between hosts , problem may or may not occur again. intermittent. there lengthy pause in vm network connectivity during live migration process however, longer have seen in past (usually ping or 2 lost, seeing 5 or more before vm nework connectivity restored on destination host, being enough cause disruption workload).

if disable vmq entirely on vm nics , vm switch team multiplex adapter on 1 of hosts test, things behave expected. migration completes within the time of a standard tcp timeout.

vmq looks working, if run get-netadaptervmqqueue on 1 of hosts, can see queues being allocated vm nics accordingly. can see vm nics appearing in hyper-v manager “vmq active”.

it goes without saying don’t want disable vmq, given nature of our clients business, cannot afford these issues crop up. if can’t find resolution here, left no choice ironically, see less issues vmq disabled compared being enabled.

i hope enough information go on , if need more, please let me know. here appreciated.

i have gone on configuration again , again , appears have been configured correctly, struggling one.

many thanks

matt

can list nics in server in simple table format:

card/port, team name, teaming mode, vmq state, rss state, usage/purpose, basevmq proc, maxvmq proc

also check mslbfoprovider, ndis, ndislmplattform event logs eventual error messages

---

gleb.

Windows Server  >  Hyper-V