View previous topic :: View next topic |
Author |
Message |
spiffywiffy n00b
Joined: 08 May 2007 Posts: 21
|
Posted: Fri Jan 20, 2012 4:23 pm Post subject: [SOLVED] Gentoo & ESXi 5.0-Problems with newest vmware-t |
|
|
Hello:
I get the impression that Gentoo is not widely used in the Enterprise and particularly not widely used as a Guest OS for vSphere/ESXi in the Enterprise. But, my hope is that someone out there is in our boat and can possibly help with this strange issue we are having. Or perhaps someone can help in some way anyway.
Our environment:
We currently run 4 ESXi hosts on Dell M610/M1000e blade hardware with shared EqualLogic storage. All are running the latest build of ESXi 5.0.
We are running about 40-50 Hardened Gentoo Linux VM's. Mixture of 32 and 64-bit kernels. Kernel versions ranging from 2.6.29-hardened to 3.1.5-hardened.
The story:
We used to run ESXi 4.0. We never had any issues compiling and running official VMWare Tools in Gentoo, which would gracefully handle restarts and shutdowns (amongst other features it obviously provides) from the vSphere client no problem.
Since migrating our VM's to ESXi 5.0 and subsequently upgrading out of date VMWare Tools to the latest version (8.6.0.6868 (build-446312)), we are experiencing a strange issue. In order to gracefully restart or shutdown a Gentoo guest from the vSphere client, we must issue the restart or shutdown _twice_ from the vSphere client. Nothing happens at all the first time. On the second attempt, the shutdown or restart process begins normally. If anyone is familiar with the vSphere workings related to HA and Update Manager for example, you can see why this would be a problem in automating tasks.
At this point I should note, that yes, we have tested open-vm-tools. They work fine and do not exhibit the above issue. However, not using VMWare distributed tools limits our ability to manage the automated updating of tools for example via VMWare Update Manager.
Also, we DID test officially supported distros (like CentOS), with the same version of tools we are installing in Gentoo. They do NOT experience the issue noted above.
We also did not experience the issue on VM's migrated from ESXi 4.0 to 5.0 before we updated tools to newest version.
So, this appears to be Gentoo and newest VMWare tools related. Not sure which is really at fault as logs do not help. The command to shutdown or restart just appears to go into a blackhole until the 2nd attempt.
It is not the end of the world, it is just annoying and inconvenient. VMWare support will not help us because Gentoo is not "officially" supported as a guest OS. This is fine - we are well aware that Gentoo is not officially supported, however, we have always found ways to make it work well for us until now. Our use of hardened Gentoo stems back to pre-VM days as a corporate policy and will likely not change anytime soon, so we would prefer to attempt to solve this issue as using VMWare distributed tools is the best case scenario for us.
Thanks for any help. Let me know if I can provide more information!
Last edited by spiffywiffy on Mon Feb 06, 2012 2:25 pm; edited 1 time in total |
|
Back to top |
|
|
Kaso_da_Zmok Apprentice
Joined: 19 Aug 2004 Posts: 189 Location: Zurich, Switzerland
|
Posted: Mon Jan 30, 2012 11:54 am Post subject: i can confirm the same behavior on ESXi 5.0.0 515841 |
|
|
i can confirm the same behavior on ESXi 5.0.0 515841
just finished funtoo with gentoo-sources 3.2.1-gentoo-r2
VMwareTools-8.6.0-515842.tar.gz
I have to issue the restart or shutdown guest twice from vSphere client
And also i cannot start the vmware-tools service with start command
Code: | /etc/init.d/vmware-tools start |
but have to use
Code: | /etc/init.d/vmware-tools restart |
to start the vmware tools.
so adding it to default runlevel does not start the vmware tools and i have to start it per hand with the restart of the service.
maybe related to my kernel config. |
|
Back to top |
|
|
ToeiRei Veteran
Joined: 03 Jan 2005 Posts: 1191 Location: Austria
|
Posted: Mon Jan 30, 2012 12:17 pm Post subject: |
|
|
Switched to open-vm-tools for that reason, keeping bumped ebuilds in my overlay to work wirh recent kernels. _________________ Please stand by - The mailer daemon is busy burning your messages in hell... |
|
Back to top |
|
|
Kaso_da_Zmok Apprentice
Joined: 19 Aug 2004 Posts: 189 Location: Zurich, Switzerland
|
Posted: Mon Jan 30, 2012 1:33 pm Post subject: looking at the esx logs |
|
|
here is the first attempt to restart guest from esx5 side
Code: | tail -f /var/log/hostd.log
2012-01-30T13:24:13.169Z [32EC2B90 info 'TaskManager' opID=34BF6CCF-00000620-56] Task Created : haTask-60-vim.VirtualMachine.rebootGuest-98092487
2012-01-30T13:24:13.170Z [32EC2B90 verbose 'vm:/vmfs/volumes/4e5d5ebc-9870d0cb-eadc-d8d3855ff8b4/rz-funtoo/rz-funtoo.vmx' opID=34BF6CCF-00000620-56] Rebooting guest
2012-01-30T13:24:13.170Z [32EC2B90 info 'vm:/vmfs/volumes/4e5d5ebc-9870d0cb-eadc-d8d3855ff8b4/rz-funtoo/rz-funtoo.vmx' opID=34BF6CCF-00000620-56] State Transition (VM_STATE_ON -> VM_STATE_REBOOTING)
2012-01-30T13:24:13.263Z [3319CB90 verbose 'vm:/vmfs/volumes/4e5d5ebc-9870d0cb-eadc-d8d3855ff8b4/rz-funtoo/rz-funtoo.vmx'] Handling message _vmxa: The request to Reset this virtual machine failed because the corresponding VMware Tools script did not run successfully. If you have configured a custom reset script in this virtual machine, make sure that it contains no errors. Attempting the operation again will ignore the script failure. You can also submit a support request to report this issue.
--> |
and vmware.log
Code: | 2012-01-30T13:24:13.175Z| vmx| TOOLS received reboot state change request; adding delayed poweron request
2012-01-30T13:24:13.175Z| vmx| TOOLS sending 'OS_Reboot' (state=2 callback=2B2E3A20 clientData=2C06D700) state change request
2012-01-30T13:24:13.175Z| vmx| Vix: [1888031 vmxCommands.c:677]: VMAutomation_ResetImpl: SoftReboot succeeded.
2012-01-30T13:24:13.261Z| vcpu-1| TOOLS state change 2 returned status 0
2012-01-30T13:24:13.261Z| vcpu-1| Msg_Post: Error
2012-01-30T13:24:13.261Z| vcpu-1| [msg.tools.rebootFailed] The request to Reset this virtual machine failed because the corresponding VMware Tools script did not run successfully. If you have configured a custom reset script in this virtual machine, make sure that it contains no errors. Attempting the operation again will ignore the script failure. You can also submit a support request to report this issue.
2012-01-30T13:24:13.261Z| vcpu-1|
2012-01-30T13:24:13.261Z| vcpu-1| ---------------------------------------- |
So that explains why the second attempt works |
|
Back to top |
|
|
Kaso_da_Zmok Apprentice
Joined: 19 Aug 2004 Posts: 189 Location: Zurich, Switzerland
|
Posted: Mon Jan 30, 2012 2:24 pm Post subject: sorted here |
|
|
Ok i have got it fixed here.
It fails on the script
Code: | /etc/vmware-tools/poweroff-vm-default |
which fails because it calls this
Code: | /etc/vmware-tools/scripts/vmware/network |
and this fails because it looks for init script called network or networking and also expects command ifup which is not in search path.
so the workaround/fix ( i created link from dhcpcd but you do link to whatever you use to start network)
Code: | rz-funtoo ~ # ln -s /etc/init.d/dhcpcd /etc/init.d/network |
And create /sbin/ifup ifdown
Code: | rz-funtoo ~ # nano -w /sbin/ifup
/sbin/ifconfig $1 up
rz-funtoo ~ # chmod 744 /sbin/ifup |
Code: | rz-funtoo ~ # nano -w /sbin/ifdown
/sbin/ifconfig $1 down
rz-funtoo ~ # chmod 744 /sbin/ifdown |
this makes the script happy and VM listens to the first restart/shutdown request
Last edited by Kaso_da_Zmok on Mon Feb 06, 2012 2:52 pm; edited 1 time in total |
|
Back to top |
|
|
spiffywiffy n00b
Joined: 08 May 2007 Posts: 21
|
Posted: Mon Jan 30, 2012 2:47 pm Post subject: |
|
|
Wow... well done! We haven't tested this yet, but it looks good. Just wanted to jump in and say thanks for the help - will confirm today hopefully and change to solved. |
|
Back to top |
|
|
Kaso_da_Zmok Apprentice
Joined: 19 Aug 2004 Posts: 189 Location: Zurich, Switzerland
|
Posted: Sat Feb 04, 2012 2:13 pm Post subject: did it work? |
|
|
did it work for you? |
|
Back to top |
|
|
spiffywiffy n00b
Joined: 08 May 2007 Posts: 21
|
Posted: Mon Feb 06, 2012 2:25 pm Post subject: |
|
|
Yep! Worked like a charm! Thanks.
Think there is anything we can do to make these changes part of Gentoo or at least the vmware-tools ebuild that is in portage? I doubt VMWare will do anything based on my conversations with support. |
|
Back to top |
|
|
libertytrek Apprentice
Joined: 18 Jul 2007 Posts: 258
|
Posted: Fri Apr 13, 2012 1:29 pm Post subject: |
|
|
Did you ever post a bug/enhancement request for this? We are getting ready to convert our servers to ESXi, and I'll be running my one gentoo mail/groupware server on it, and would love to not have to kludge anything on a long term ongoing basis... |
|
Back to top |
|
|
spiffywiffy n00b
Joined: 08 May 2007 Posts: 21
|
Posted: Fri Apr 13, 2012 1:47 pm Post subject: |
|
|
No, I didn't actually, but I shall do so right now.
Btw.. since you are going ESXi, we currently can't get any official version of VMWare tools to run on a v3 hardened kernel. Not sure if vanilla kernel is affected too. We have to run open-vm-tools on those, but this is not ideal at all. Just so you know going in. |
|
Back to top |
|
|
libertytrek Apprentice
Joined: 18 Jul 2007 Posts: 258
|
Posted: Fri Apr 13, 2012 2:00 pm Post subject: |
|
|
Ok, thanks for the heads up... are the drawbacks documented anywhere? I looked around on their website but didn't see anything comparing the two...
The main things I'm worried about are time syncing (looks like the open tools do that), snapshots, and automated safe shutdown/startup of the VMs via the hypervisor (ie, if the UPS issues a shutdown command)... |
|
Back to top |
|
|
spiffywiffy n00b
Joined: 08 May 2007 Posts: 21
|
Posted: Fri Apr 13, 2012 3:03 pm Post subject: |
|
|
Here's some detail:
http://open-vm-tools.sourceforge.net/faq.php
Basically, all of the Open Source components in the commercial VMware Tools are included in open-vm-tools. vmxnet3 and vmmemctl (balloon driver) are now included upstream, so those are less of an issue for folks. In reality, there aren't many drawbacks.
In our experience, the drawbacks for us are:
- Inability to snapshot virtual machine using VMware Data Recovery. Haven't investigated much yet - its a bit of a mystery considering that vmsync module is part of open-vm-tools and snapshot with quiesce enabled from vSphere client is successful.
- Inability to manage tools with VMware Update Manager. Any VM running open-vm-tools shows tools status as '3rd-party/Independant' in vSphere client and cannot be managed for updates to tools.
You should have no issues with graceful shutdown, restart, or time sync, etc. |
|
Back to top |
|
|
libertytrek Apprentice
Joined: 18 Jul 2007 Posts: 258
|
Posted: Fri Apr 13, 2012 3:53 pm Post subject: |
|
|
spiffywiffy wrote: | You should have no issues with graceful shutdown, restart, or time sync, etc. |
Ok thanks for confirming... but as to snapshots (I'm new to ESXi, and virtualization in general, so pardon my ignorance)...
With the open-vm-tools, will I be able to *automate* snapshots (for backup purposes)? |
|
Back to top |
|
|
spiffywiffy n00b
Joined: 08 May 2007 Posts: 21
|
Posted: Fri Apr 13, 2012 4:04 pm Post subject: |
|
|
Technically, yes. But, snapshot success is dependent on other factors. Sometimes snapshots fail if the filesystem cannot be quiesced, which is not necessarily the fault of vSphere, tools, etc. Frankly, we mostly only see automated snapshot problems with Windows systems. |
|
Back to top |
|
|
libertytrek Apprentice
Joined: 18 Jul 2007 Posts: 258
|
Posted: Fri Apr 13, 2012 5:59 pm Post subject: |
|
|
Ok, many thanks for the tips...
Oh - one more thing - when automating snapshots, do these tools allow me to define an email address (and/or SMS number) so I can get email and/or text notifications of success (but more importantly, failure)? If so, that would be sufficient... if not, I guess I'll have to be sure to perform manual snapshots regularly (ugh)...
Thanks again... |
|
Back to top |
|
|
spiffywiffy n00b
Joined: 08 May 2007 Posts: 21
|
Posted: Mon Apr 16, 2012 1:58 pm Post subject: |
|
|
The scheduled tasks feature in vSphere client has notification options.
I'd recommend using VMware Data Recovery - its free, has dedupe, and will offer you more overall flexibility. Plus, you won't suffer any of the possible performance issues having multiple snapshots on the VM object itself. |
|
Back to top |
|
|
honp Guru
Joined: 25 Sep 2006 Posts: 355 Location: Good old Prague, Czech rep.
|
Posted: Fri Oct 19, 2012 12:41 pm Post subject: |
|
|
It is old post but this didn`t solve my problem, but use of /et/local.d did. I added there script with /etc/init.d/vmware-tools restart... I don`t know if it is right, but it works..
H. |
|
Back to top |
|
|
honp Guru
Joined: 25 Sep 2006 Posts: 355 Location: Good old Prague, Czech rep.
|
Posted: Fri Oct 19, 2012 12:42 pm Post subject: |
|
|
This is old post but this didn`t solve my problem, but use of /et/local.d did. I added there script with /etc/init.d/vmware-tools restart... I don`t know if it is right, but it works..
H. |
|
Back to top |
|
|
|