Getting Gentoo to work in VMware (walkthrough)

[mastabog]

Here is everything I experienced these past few days trying to get VMware tools in Gentoo and Xorg (kde 3.3) working correctly in vmware. Note that I am writing this for all of you that hassled with getting Gentoo running properly in vmware with all bells and whistles but also for myself (as a future reference).

My sets:
- VMware Workstation v4.5.2.8848 running on an XP SP2 host
- Gentoo v2004.2 built from Stage 1; kernel gentoo-sources-2.4.26-r9; running as vmware guest
- X system in gentoo: Xorg and KDE 3.3 (I excluded all gnome/gtk support by the use of USE flags)

First thing's first. If you don't need the shared folders and the time sync feature from vmware then you can do without the vmware tools. You can get your KDE 3.3 running in any resolution and all other features (sound, USB hotplugging, etc) running beautifully without vmware tools.

Here is my /etc/X11/xorg.conf that is working with or without any vmware tools installed:

Code: Select all

Section "Module"
    Load        "dbe"   # Double buffer extension
    SubSection  "extmod"
      Option    "omit xfree86-dga"   # don't initialise the DGA extension
    EndSubSection
    Load        "type1"
    Load        "freetype"
EndSection
Section "Files"
    RgbPath     "/usr/X11R6/lib/X11/rgb"
    FontPath   "/usr/share/fonts/misc/"
    FontPath   "/usr/share/fonts/TTF/"
    FontPath   "/usr/share/fonts/Type1/"
    FontPath   "/usr/share/fonts/75dpi/"
    FontPath   "/usr/share/fonts/100dpi/"
    FontPath   "/usr/share/fonts/local/"
EndSection
Section "ServerFlags"
EndSection
Section "InputDevice"
    Identifier  "Keyboard1"
    Driver      "kbd"
    Option "AutoRepeat" "500 30"
    Option "XkbRules"   "xorg"
    Option "XkbModel"   "pc101"
    Option "XkbLayout"  "us"
EndSection
Section "InputDevice"
    Identifier  "Mouse1"
    Driver      "mouse"
    Option "Protocol"    "auto"
    Option "Device"      "/dev/mouse"
    Option "Emulate3Buttons"    "true"
    Option "Emulate3Timeout"    "50"
    Option "ZAxisMapping"       "4 5"
EndSection
Section "Monitor"
    Identifier  "vmMonitor"
    HorizSync   31.5 - 64.3
    VertRefresh 50-120
EndSection
Section "Device"
    Identifier  "Standard VGA"
    VendorName  "Unknown"
    BoardName   "Unknown"
    Driver     "vga"
EndSection
Section "Device"
    Identifier  "vmVideoCard"
    Driver      "vmware"
EndSection
Section "Screen"
    Identifier  "Screen 1"
    Device      "vmVideoCard"
    Monitor     "vmMonitor"
    DefaultDepth 24
    Subsection "Display"
        Depth       8
        Modes       "800x600" "1024x768" "640x480"
        ViewPort    0 0
    EndSubsection
    Subsection "Display"
        Depth       16
        Modes       "800x600" "1024x768" "640x480"
        ViewPort    0 0
    EndSubsection
    Subsection "Display"
        Depth       24
        Modes       "800x600" "1024x768" "640x480"
        ViewPort    0 0
    EndSubsection
EndSection
Section "ServerLayout"
    Identifier  "Simple Layout"
    Screen "Screen 1"
    InputDevice "Mouse1" "CorePointer"
    InputDevice "Keyboard1" "CoreKeyboard"
EndSection

You can also download it (with full comments) from here: http://bog.webhop.net/vmware/xorg.conf.txt

In case you don't have the same gentoo configuration that matches my xorg.conf above, here's how I obtained it:
Start the xorg configuration by typing "xorgconfig".
- mouse: auto, "yes" to emulate3buttons, /dev/mouse (or default) for the device
- keyb: generic 101-key, US, (default), "no" to additional XKB options
- monitor: horiz 31-65 KHz (or 1280x1024@60Hz), vert 40-150 Hz .. choose these values wisely, though i didn't have any problems coz i never went over the resolution i had in windows
- vga card: "yes" to look at the card database and choose the vmware driver (29 was the option number for me), 32Mb ram, then edit your modes and bit depth properly from the next screens
- "Yes" to write to /etc/X11/xorg.conf

Note: In order to enable the mouse wheel inside KDE, just add the following line to xorg.conf inside the mouse section somewhere:

Code: Select all

    Option "ZAxisMapping"       "4 5"

To properly start KDE when you type "startx" you need to:

Code: Select all

$ echo "exec startkde" > ~/.xinitrc

Or better yet, edit /etc/rc.conf and properly set the XSESSION and DISPLAYMANAGER variables. Mine are:

Code: Select all

DISPLAYMANAGER="kdm"
XSESSION="kde-3.3.0"

If you want a graphical login (runlevel 5), then:

Code: Select all

# rc-update add xdm default

Otherwise just start your kde with "startx" from the console.

Important: If you have framebuffer support enabled for your kernel and a vga=<some graphic mode here> option in your kernel line in lilo.conf or grub.conf file then you will experince a bug that is present in vmware (happens on all linux distros) when you exit from Xorg into the console (like ctrl-alt-backspace or nice logout): there would be 4 little screens with messed up colors. A quick way to fix this is to change the vga=<graphic mode> option from lilo.conf/grub.conf into:

Code: Select all

vga=normal

Which will use the normal text based console screen ... the downside is that you will get the standard 80x25 console and any splashscreen that you might have had enabled in your kernel will be gone ... (if anyone knows another fix for this then please reply)

Note: Even if you choose to install vmware tools (see below), all that was said above still stands.


Vmware tools installation:

If you wish to install the vmware-tools in gentoo, then do so from the vmware host OS menus, mount the cdrom in Gentoo, take the tarball and unpack it and optionally take the patch from http://forums.gentoo.org/viewtopic.php? ... 46#1620146 that supplanter posted (thanks ) or see the link in the p.s. of this post, edit it and remove the /dev/input/mice line (gentoo doesn't have such a dev) and apply the patch or modify the lines by hand (I personally did the second thing). Note that applying the patch is optional ... you don't need the xorg awareness of the vmware config script anyway (read on).

Create the fake rcN.d directories:

Code: Select all

# mkdir /etc/rc0.d
# mkdir /etc/rc1.d
# mkdir /etc/rc2.d
# mkdir /etc/rc3.d
# mkdir /etc/rc4.d
# mkdir /etc/rc5.d
# mkdir /etc/rc6.d

Then start the (patched) vmware-tools installation script and answer with the default answers to all questions (including to the compilation of the vmhgfs module). After everything is done do not attempt to start X as it won't work (xorg.conf file is still messed up). Instead, you can either use the xorg.conf file I provided above (editing maybe just the resolutions) or launch "xorgconfig" and follow the steps I described above for it. You'll be much better off than trying to get the vmware tools do it for you ... or at least I was.

Now, if you want to add the vmware-tools at startup, do not directly use "rc-update add vmware-tools default". If run, rc-update won't give any errors and it will add it to startup, but the script won't run at all when the machine is booted because it is not a gentoo init.d script (runscript) (someone correct me if i'm wrong). To fix this and still have the vmware tools start automatically at boot time, I created another script called simply "vmware" in /etc/init.d:

Code: Select all

# nano /etc/init.d/vmware

With the following contents:

Code: Select all

#!/sbin/runscript

depend() {
        after logger
}

start() {
        ebegin "Starting VMware tools (hack)"
        /etc/init.d/vmware-tools start
        eend $?
}

stop() {
        ebegin "Stopping VMware tools (hack)"
        /etc/init.d/vmware-tools stop
        eend $?
}

Add executable permissions to the file:

Code: Select all

chmod +x /etc/init.d/vmware
chown root:root /etc/init.d/vmware

You don't need to chown if you were already root.

Then add it to rc-update and it will start when booting just fine:

Code: Select all

# rc-update add vmware default
# rc-update del vmware-tools

Note: In some cases, when vmware tools are started, it can say that is still not configured and you need to run the config script (even if you did configure it and compiled the vmhgfs module). Do so and reboot the machine, it shouldn't happen again. I only exoerienced this when i tried to load the vmware tools twice or some other bad thing happened

Well, hope this helps someone esle

Cheers

p.s. The Xorg patch for the vmware tools config script can be found at kerneltrap.org in a blog of keithmo (he has helped (and still does) a lot in getting fedora to work ok with vmware). It was written originally for fedora: http://kerneltrap.org/node/view/3408

[Suicidal]

w00t - bookmarked

btw how does it run under vmware and what is the hardware speed you installed it on .

[mastabog]

I have a P4 2.4GHz with 1GB Ram and VMware installed on a WinXP SP2. The Gentoo vm machine has 256MB Ram, 1 hdd for / (8Gb), 1 hdd for swap (512MB) and 1 hdd for /usr/portage/{portage,distfiles} and /var/tmp/portage (4Gb). It also has USB and Sound.

This is the make.conf I used when building it:

Code: Select all

# Please consult /etc/make.conf.example for a more detailed example
CFLAGS="-O2 -march=i686 -fstack-protector -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
USE="tetex -gtk -gtk2 -gnome arts alsa kde qt -bonobo cdr esd java ldap -gtkhtml mozilla -mysql perl -postgres ruby tcltk acl cups ldap ssl tcpd -guile -directfb curl -svga"

I did a stage 1 installation following closely the gentoo installation handbook and used the --buildpkg option for all emerges (created my own binary repository for those CFLAGS, CHOST and USE).
Kernel was compiled using genkernel (yes, I also used to configure and compile it by hand, but the difference in speed or is almost none and I have all possible modules compiled and ready).

Edit: The sound works beautifully. As soon as you emerge alsa and its drivers (during the kde emerging) you will get a warning that the es1371 module could not be loaded during pci hotplugging detection. That is normal because the alse drivers (snd-es1371) were loaded before. Also, USB hotplugging works like a charm. I succesfully used a 512MB USB stick and an external 2.5" hdd ... All you need to make sure is that the vmware gentoo machine has focus when you plug the usb device in (otherwise it woul dbe the host OS that grabs its usage). You don't need to load any aditional modules (like usb-storage, vfat, etc), hotplug does that for you.

Just make sure you add "doscsi" in the kernel line in your grub.conf kernel line so that you can use USB storage devices or SCSI hdds in vmware. Here is my grub.conf:

Code: Select all

default 0
timeout 30
splashimage=(hd0,0)/grub/splash.xpm.gz

title=Gentoo Linux 2.4.26-r9
root (hd0,0)
kernel /kernel-2.4.26-gentoo-r9 root=/dev/ram0 init=/linuxrc ramdisk=8192 real_root=/dev/hda3 doscsi vga=normal
initrd /initrd-2.4.26-gentoo-r9

I can say without a doubt that this Gentoo vm is running faster than any other linux I installed in vmware, and the fact that you have the latest and revised versions of all packages makes it even nicer ... I am truly happy, it flies.

Cheers

[jakamaka]

Don't you kinda think what you did is stupid, what use does your system have now ?
It now runs with a unstable windowsxp base .
If i were you i just should have made a small partition of 5gig or so and practised with that.

Back in the days ( ) , when i installed gentoo the first time, i copied the whole gentoo manual on a usb sticky and had it all printed out in the local printcenter.
i still have the manual, somewhere ....

Never the less, it's a funny experiment.

[nyda]

I think I already had more kernel freezes and X lockups than Windows XP crashes. Admittedly my xp HDD died a few months ago (hardware failure), but still, I don't remember it freezing more then once or twice.
WinXP might be bad and all, but I'd be careful when comparing XP vs Linux stability. You would have to test on hundreds of different machines and configurations over several months to get any meaningful results.

[mastabog]

Don't you kinda think what you did is stupid, what use does your system have now ?
It now runs with a unstable windowsxp base .
If i were you i just should have made a small partition of 5gig or so and practised with that.

I have Gentoo and Mandrake and 2 other Windows XP machines all running at the same time on a 2Gb Ram P4 phy machine plus the Windows XP host. They all have access to the same network and you have access to all of them ... apart from not being able to play 3D games on a virtual machine, you can pretty much do anything with very little speed decrease. I work in supercomputing and testing clusters is not only a breeze with virtual machines but also a life, money, effort and time saver ... big time.

From your comments, I think you never truly worked with virtual machines. I use Windows as host ... take it as a personal choice. It has now an uptime of over 2 months, keeps on going smooth and is intensively used with a lot of services running on it. I won't go into a "windows sucks - linux rules" debate with you. BTW, what makes you all windows-haters so rude? Could it be the young age or lack of education? ... makes one wonder.
And also, if you're not insterested in the vmware matter, why post comments like "what you did was stupid"? That always intrigued me.

Cheers

[TheHermit]

Great job writing this.

I just don't seem to have a driver called "vmware". It doesn't show up on the driver list when I do xorgconfig or if I edit the config file manually and start X, I get an error then then it locks up, forcing me to restart.

[clutch]

Thanks for the write-up. I have been out a while (well, a long while) and I am now working alternate OS integration with Active Directory at my current job. I was trying to use VPC (free with MSDN Universal subscription) to run Linux boxes, but found that anything run more than Fluxbox was hopeless (the director likes SuSE, and is familiar with KDE). Just got VMWare and it has come a long way since I used it last. For those that are wondering, most *nix installs work really well (almost realtime response) when using something like fluxbox. When using KDE, it is a little bit slower (unless using VPC, and then it isn't worth using).

As for why someone would want to run complicated configurations of Linux on a Windows platform, there are many reasons for it. I use Windows as my base because it is stable, and has complete support for all of my hardware without much fuss (I did go completely Gentoo for a while last year, but constantly digging around for fixes to easy things grew tiresome). In addition, I missed Visual Studio.NET and Exchange too much and wanted to go back.

[mastabog]

Great job writing this.

I just don't seem to have a driver called "vmware". It doesn't show up on the driver list when I do xorgconfig or if I edit the config file manually and start X, I get an error then then it locks up, forcing me to restart.

That prolly because of an older Xorg version. I used 6.8.0-r1 (ebuild is called xorg-x11)

[lefou]

Which network driver do you use? I can't get eth0 up.

lefou

[mastabog]

Well, if you compile the kernel using genkernel, that it automatically detects all hardware and compiles all modules ... then the pci hotplug service will autoload any needed module.

But in case you manually compiled the kernel, or the pci hotplug is not loading yours, you should load the pcnet32 module. Just do a "modprobe pcnet32" then try bringing up your net. If this works, then add this to the automodules list.

If you don't have it compiled, then reconfigure your kernel, and compile the "AMD PCnet32 PCI support" as a module. Then load it and it should work.

Cheers

p.s. You were talking about vmware v4.5.2 and gentoo as a guest os, right?

[lefou]

Thank you, it works. Don't know why, but I thought, I have to use the lance module and so only got some error messages. pcnet32 is the right thing

lefou

[todd_]

thanks, this was excellent and got me on my way.. just what i was looking for..

my only question now is.. is it safe to go full screen (ctrl alt enter) even if im not sure the chosen xorg refresh rates match my monitor..

[mastabog]

thanks, this was excellent and got me on my way.. just what i was looking for..

my only question now is.. is it safe to go full screen (ctrl alt enter) even if im not sure the chosen xorg refresh rates match my monitor..

If you chose the same resolution that you have in your host OS, then it's perfectly safe .. the switch will be instant and refresh rate will be the same as the host's (regardless of what you chose for Xorg).

If not, just make sure your monitor supports the refresh rates you chose ... then going full screen will work just fine. Even if you've mistaken and chose some higher refresh rates then your monitor supports, you can always press ctrl-alt and get back to the oo then connect with putty to the gentoo guest and reconfigure Xorg; or press ctr-alt-backspace and get back into console.

[jakeWSYSA]

Hi,
Do you remeber which drivers you used when compiling the kernel?

Specifically for the ethernet?

[mastabog]

Hi,
Do you remeber which drivers you used when compiling the kernel?

Specifically for the ethernet?

uhmmm ... have you read 4 posts above?

[frank28_nfls]

In fact, i was trying to figure out my problem of Xorg for FreeBSD5.3 and Vmware. When i taped "Vmware Xorg" in google, i was led here, and found quickly that it was just the right answer for me--of cause, not 100% , but after the simple configuration by xorgconfig, i am now working in Gnome. So, just want to say,"thanks for your great job "

[mr_cheez]

Hi, i have read most of the information above and i appreciate all the work you guys have done on getting Gentoo to work in vmware. vmware is very cool and provides lots of functionallity.

I am really looking forward to getting that far in my "Gentoo hosted on XP adventure". But my problem is more basic; i am having trouble booting my newly compiled Gentoo 2.4.26-r13 kernel.

anyway. i downloaded the demo version of vmware... "Version 4.5.2 build-8848".

The Gentoo distribution booted fine from the CD, i went through all the instructions, i've done this before and have gotten reasonably competent with it. i'm sre i'm overlooking something small at this point. after i got to the end of the installation process, i saved a vmware snapshot. after saving the snapshot, i tried to boot the newly configured kernel.


I get a bunch of (normal) boot-time trace, then after loading a bunch of modules i get trace like this:

>> Block device /dev/sda3 is not a valid root device..
>> The root block device is unspecified or undetected.
>> Please specify a device to boot, or "shell" for a shell...
boot() ::

however, with the fancy vmware snapshot feature, i just click over there, and i'm back to the save point. Pretty handy!

Here are the pertinent facts:

my partition table in vmware looks like this:

/dev/sda1 2048m swap fs type 0x82
/dev/sda2 2048m /boot reiserfs
/dev/sda3 8192m / reiserfs
/dev/sda4 4096m /home reiserfs

when i set up GRUB i used these commands:
grub> root (hd0,1)
grub> setup (hd0,1)
grub> quit


the contents of my /boot/grub/menu.lst
timeout 10
default 0
title Gentoo
root (hd0,1)
kernel /kernel-2.4.26-gentoo-r13 real_root=/dev/sda3 root=/dev/ram0 ramdisk_size=8192 init=linuxrc
initrd /initrd-2.4.26-gentoo-r13

reiserfs is part of the kernel (not a module), and that's the only thing i could think of.

i can post the .config, but it's pretty verbose, i'll cut-n-paste any parts you guys think can help.


-- thanks for your help.

bob

[geten]

It is probably because your scsi-controller isn't loaded before it tries to mount the root-partition on sda3. If you have scsi-support and the buslogic driver (which vmware uses) compiled as modules, make sure they are loaded before it tries to use sda3. One thing that certainly will work is to have scsi-support and the buslogic driver compiled into the kernel

[mastabog]

simply add "doscsi" to your "kernel" line in the grub config file (it is mentioned in the gentoo docs ); in your case seems to be /boot/grub/menu.lst

[mr_cheez]

Thanks, that "doscsi" thing was the problem. i added "doscsi" to the kernel line, and i watched it load the buslogic module and finish booting, first time.

I'll recompile with buslogic in the kernel. i'm really cutting the kernel down; all i really want to do in Linux is development with C, GTK+ and MySQL.

Thanks for the replies and all your help. I really appreciate it.

This community - the forums and the guys building the packages are why Gentoo rocks.

--bob

[mastabog]

This community - the forums and the guys building the packages are why Gentoo rocks.

I feel the same. Weird maybe for some, but I first chose Gentoo because it has an official forum .. not mailing lists, not news servers but a web forum. Nice surprises from Gentoo kept pouring to my pleasure afterwards .. you simply have to love it.

[Zerodark]

Nice work writing this. It really helped me. I only have 1 problem. I used that script you wrote, but when i use rc-update it tells me vmware isn't executable, and skips it.

[Bob P]

I think I already had more kernel freezes and X lockups than Windows XP crashes. Admittedly my xp HDD died a few months ago (hardware failure), but still, I don't remember it freezing more then once or twice.
WinXP might be bad and all, but I'd be careful when comparing XP vs Linux stability.

no doubt. when in installed XP on my current box, the dual head installation went without a hitch. over a 12 month period i have had not one single lockup. my Gentoo installation is two months old, and i'm still fighting with video drivers and Xorg, and i'm stuck in a singlehead envoronment. lockups have been a frequent occurrence in X. my experience would be the opposite of yung's suggestions, but of course, yung's statements were completely in character.

[mastabog]

Nice work writing this. It really helped me. I only have 1 problem. I used that script you wrote, but when i use rc-update it tells me vmware isn't executable, and skips it.

You might need to add executable permissions on the file and make root the owner:

Code: Select all

chmod +x /etc/init.d/vmware
chown root:root /etc/init.d/vmware

i'll update the first post also with this info, thanks