More SSLv3 (Poodle vulnerability) woes as this time NetApp VSC 5.0 is broken!
So my vCenter 5.5 update 2a got updated to update 3e without much of a problem but SRM and VSC are busted now. Great. Virtual Storage Console sort of works but the backup jobs tab hasn’t got any entries and you cannot re-create them due to the following errors:
Unable to connect to Virtual Storage Console server. Please make sure that the Virtual Storage Console server is running.
To cut this long story short – if you have vCenter Server 5.5 update 2 you will have issues if you patch your hosts to the latest available patch level for ESXi. VMware disabled SSLv3 (POODLE and all that..) in update 3b for ESXi meaning if your vCenter Server is running update 2 you won’t be able to connect until the vCenter is patched to update 3b as well. Running ESXi hosts on update 3b and having vCenter Server on update 2 is normally a perfectly valid configuration but because SSLv3 got disabled as part of this process the connectivity is broken.
Example error messages in vpxd log file when patched host in being added to vCenter Server include:
2016-02-26T15:47:12.729Z [07448 error 'HttpConnectionPool-000001'] [ConnectComplete] Connect failed to <cs p:0000000017ddfdf0, TCP:spn-esx-04.alex.com:443>; cnx: (null), error: class Vmacore::Ssl::SSLException(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)
2016-02-26T15:47:12.729Z [07400 error 'httphttpUtil' opID=30D532A7-00000053-90] [HttpUtil::ExecuteRequest] Error in sending request - SSL Exception: error:140000DB:SSL routines:SSL routines:short read
2016-02-26T15:47:12.730Z [07400 error 'vpxdvpxdHostAccess' opID=30D532A7-00000053-90] [VpxdHostAccess::Connect] Failed to discover version: vim.fault.HttpFault
Last week I have installed VMware vSphere 5.5 on my test host and today was the time to get the NetApp Virtual Storage Console 5.0 going so I could take advantage of Rapid Cloning and other good stuff that VSC 5.0 includes.
Installation was straight forward (recommended read – Virtual Storage Console 5.0 for VMware® vSphere® – Installation and Administration Guide) and next logical step was to add my Storage Systems so I could provision datastores etc. From within VSC section in vSphere Web Client I was trying to add new Storage System just to be presented with the following:
“Unable to add storage systems due to insufficient privileges. You do not have sufficient permission to perform this action on: the root object. Contact your administrator to add the following mission privileges: Add, Modify, and Skip storage systems”
So here we are, lovely Thursday morning at work and requirement for new VM comes up – I’m thinking not a big deal since I have deployed thousands of VMs before but there is a catch this time (there always is!) All of my Windows Server templates are virtual machine HW version 8 and I need to deploy one server to ESXi 4.1 host – great! ESXi 4.1 uses HW version 7 at the most so HW version 8 will not work – if you attempt to add HW version 8 to the inventory on ESXi 4.1 host you will be met by the following outcome:
VM adds fine and without any errors but its grayed out and with invalid status. Not much you can do here apart from removing it from the inventory.
Closer look at what’s happening (or not happening as a matter of fact):
“Perf Charts service experienced an internal error. Message: Report application initialization is not completed successfully. Retry in 60 seconds.”
Now, this error has been around for as long as I can remember. There are many causes of it but I will try to cover the one I have experienced (and solved)
Let’s get to it.
In vCenter 4.x this has never been an issue and charts stopped working since I have upgraded my vCenter to version 5.0 Update 2. Generally you look at log files for vCenter (stats.log is what we’re after) to determine the root cause. Location of stats.log depends on version of Windows and its as follows: