I have had a couple of experiences now with ESX4 and ESX4i data-center moves. These moves have included the complete shutdown of the entire infrastructure, move, and restart.
There has developed a pattern within vSphere that we have had to right procedures for, but I am concerned it may be a systemic issue as it has occurred in different data-centers, configurations, platforms, and versions. The only correlation so far is vSphere.
On to the details: I prepare to move the data-center by shutting down all VMs. I down the hosts in descending order (10, 9, 8, etc.). Stop storage, package, and ship. Once the new site is racked, cabled, and powered, I turn up storage and validate. I then power on the hosts in ascending order (1, 2, 3, etc.). I allow each host to fully boot before starting the next.
At this point I attempt to validate the hosts are "online" but connecting a physical computer into the management network. I can ping the router and storage management interfaces. I am unable to ping any of the hosts. I am unable to access the web interface to download the client. On hosts where remote ssh is enabled, I am unable to connect.
At this point, each vSphere host is a black hole. It behaves like the firewall is blocking all traffic or all services are down.
I open the "unsupported" console and ping from the hosts to the router successfully. I can confirm that networking is correct.
I then issue "vim-cmd vmsvc/getallvms" and sometimes a list is returned and other time a 514 error (vmdb failure).
I work my way through the hosts until I locate either Active Directory or DNS VMs registered to the host. I then issue "vim-cmd vmsvc/power.on " and watch the VM successfully load.
I modify the laptop network and can successfully ping and use the VM services.
Next, I reboot all of the other hosts. I am now able to ping them, use the web interface, and ssh. The only difference being that Domain Controller or DNS service was available when they came up.
So, is there a known issue that vSphere must have DNS services available to start successfully? What is causing this black hole effect if no DNS is available? Any ideas as to the root cause of this black hole?