Fix: old, broken, restricted, limited VPS install [SOLVED]

[HeXiLeD]

Here is the story. About 1 year ago i bought a VPS to ovh.net (ovh.net, ovh.com , ovh.pt, ovh.pt, etc etc they are all the same).

I first installed debian and for 1 year the vps was ok. Limited but ok. Last year i renewed the service for 2 more years and recently i tried to update the system. BIG mistake. Very regrettable.

To make a long story very short OVH debian broke; Their iscsi system was malfunctioning so bad that even most recovery options were not even working. Outdated kernels; udev problems ... this went to the point if seriously incompetent support service, technical service and customer service. All they care is if the machine responds to pings. I had email replies that not even a 15 year old that started to learn how to use linux will use to justify something.

In time they recognized their problem. Removed debian from the available install options and selections. Said they would compensate me for my downtimes. I had given up on using the service which almost caused me 2 years of data loss. (I will never EVER use OVH in my life and will not risk my work there again) but the box is paid for 2 more years. Although it wont be hosting my work i thought about putting it at the service to help other projects in regards of bandwidth and storage and decided to install gentoo in there. (available version from 2007)

Gentoo also faces the same problems as debian. In fact OVH as admitted that my service is now obsolete and no longer supported. They refuse to reimburse me or provide working solutions (ovh.pt) at no economic cost.

I am now stuck with a service that uses their own kernel which to what i was told i am stuck with. Uses their own portage ebuild system and custom packages which besides outdated and broken i was able to change. Any attempt of system or world update with their portage results in endless amount of fail installs/compile logs that i never seen in 7 years of gentoo.
With their install i am unable to use updated or even install a basic software package either because of dependencies or lack of dependencies. this counts for manually compile too.

Gentlemen... this company even provides 2.4 kernels and prevents clients to use their own kernel. Their solution is to upgrade to a better service which means pay for a new server package.

I have been trying to fix gentoo and at every fail i have to reinstall it and start from zero. I have lost my patience with their lousy and terrible customer/tech service which is unacceptable at all levels.

This now takes me to the following state:

I have tried:
http://www.gentoo.org/proj/en/portage/doc/manually-fixing-portage.xml
http://blog.jolexa.net/2009/03/gentoo-tips-to-upgrade-your-really-old-installation

Maybe given to exhaustion and or overlooking something i have hit a barrier and even to i was able to update quite a lot of the system, i have reached now some crucial updates that became hard to fix.

Right now i am stuck with:

[dol-sen]

Portage requires a minimum of python-2.6. The only package manager you'll get to work on that is pkgcore which can still operate on python-2.5.

But there are many newer pkgs that rely on newer kernels, so for that reason you are likely to run into other upgrade troubles.
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...

[Hypnos]

You could try Paludis, which only requires C++ and some small utilities in its minimal configuration. Then run the portage2paludis.bash script to migrate your Portage setup.

Of course, you should backup your system first.
_________________
Personal overlay | Simple backup scheme

[HeXiLeD]

Right now i was able to install python 2.7:

HOWTO: Manually install python (broken python)
https://forums.gentoo.org/viewtopic-t-814533-start-0.html

I am not sure how to take it from here as python-updater still behaves like before but thought about updating portage the same way.

The kernel in use is:

[HeXiLeD]

Very slowly i have been able to install one package here one package there from source playing eith their versions and using older versions while slowly upgrading them untill they fail and until i ran out of options.
Then i moved to use binhost from http://tinderbox.dev.gentoo.org which allowed me to continue with the same results; one package here one package there doing the same has before....

Right now at least gcc is at 4.5* but

[HeXiLeD]

When i almost got it fixed it broke. I have reinstalled the system and retried many times.
The tinderbox solution fails if i replace all 3 packages right away.
The gentoo handbook portage upgrade seems the most functional method and from there trying to install and upgrade packages slowly by old versions; one by one until they get to current date or closer.
I also created my own binhost using a almost the same machine environment but without success.

Right now i have created bin packages for all packages installed in that machine as a backup method and am stuck with upgrading portage since it needs bash 4. Bash 4 fails to compile.

[HeXiLeD]

After a few more retries where trial and error was the only option, i was able to update the system.
This took full time dedication. Had to upgrade packages slowly by using older versions and upgrading as fit and possible.
Sometimes using my own binary packages, other times downgrading and upgrading to be able to avoid circular dependencies and in the middle of all some "IT voodoo"also happened.

In the end there was udev, the kernel and openrc/baselayout. Once i changed baselayout i lost connectivity and had to boot the machine froma live recovery system where i am currently stuck.
The udev update; kernel sources and openrc were selected to match the running kernel and not higher versions.

This now takes me to:

[Jeffrey0]

How does iSCSI come into the picture for your setup? Do you even need it to boot? Everything I see points to /dev/sda being a simple virtual disk device that you set up a bootloader on with lilo. If that's the case, I suggest removing their initrd; it seems to be set up for a setup where you have the root filesystem on iSCSI and it doesn't seem to be working right for that case (note how udhcpd doesn't only fail to get an IP, it fails at seeing the interface entirely). Just booting with a simple root=/dev/sda1 commandline and no initrd should at least get vKVM working and it's much easier to debug any remaining issues from there on out.

The error you get in the screenshot makes only a little sense to me. It could be something similar to last year's stage3 issues, but at the same time, it doesn't seem to be mounting anything and pivot_root should never fail with something like that. Having the initrd's init script would help a lot in debugging the initrd if you can't remove it.

[HeXiLeD]

[kimmie]

HeXILeD,

I admire your style, stuck in the black sea on a rotten, sinking VPS battling monster sharks, night bats and scabies. I can only hope you eventually get the girl.

[HeXiLeD]

@ Kimmie
At least now i can see land. The last question is if those natives are friendly or cannibals.

Another error that i was having which prevented the update od some packages and dependencies is now solved too.
pkg config The pkg-config script could not be found or is too old
https://forums.gentoo.org/viewtopic-t-896624-highlight-.html

I have the whole install upgraded after doing emerge -e world and -e system while i keep udev compatible with the kernel and openrc.
ovh.net supplies their own flavour of useless custom kernels which i have not been able to to make work so far no matter what i download to the box and add to the bootloader i never see that kernel active for boot options.
I do however see other kernels on those boot options that do not exist in the box anywhere. (how is the company staying in business is beyond human understanding)

Their vKVMV options none works. Some present me a kernel that is not anywhere. Others boot a 32 bit kernel for a 64 bit install; Another boots the kernel if the hard drive is hda1 and not sda1 but fails to work with it.. and so on. vKVM option seems useless.
Their also have a boot from HD option that does nothing when active but their documentation says it works. (it does not. at max it allows to ping the box - nothing else happens from that boot option - no sshd, no access, nothing).

This now takes me to the initrd situation and maybe i am being a bit radical about what i am going to say but in all these years i have custom built many gentoos i never had the need of using that miserable option; specially on a server.

http://wiki.openvz.org/Modifying_initrd_image

[Jeffrey0]

Ok, your device nodes and everything look correct, not that surprising but good to get it out of the way.

Try changing your lilo image section to something like this:

[HeXiLeD]

I tried changing those details before with debian and i had no luck.
I now ditched lilo and went for grub which gave me better news after a few fails but at least initrd misery is gone.

I can confirm over vKVM crap that grub gets loaded and boots the machine.
I was told by OVH tiny microsoft mind support that i could not use my kernel and that they supply the kernel but i did a couple tests.

First test downloaded vanilla sources and just made a kernel to see if it actually was used. I renamed it to look like the crap they provide and it turns out it was loaded but with lots of errors.

For the second try; i used one of their kernels and the error i got helps understanding lilo's error too.

[HeXiLeD]

So here it goes... kimmie...
I got to land; killed the cannibals ... and got girl.

But before that

I tried a fast compiled generic kernel compiled: failed
I tried one their 2.6 kernels and booted it with grub: failed
Tried to recompile that kernel without NFS support: failed
Fetched one of their 3.2 kernels, renamed to a 2.6 kernel that they allow me to boot from (select menu) the machine menu "name" options.

Then i basically fell asleep on the keyboard, woke up, fast forward a crappy movie went on irc and saw that an eggdrop that i had left on the box ready to connect to irc in case the box was up; WAS UP.

The install is up and running, i was able to connect by telnet (backup login) since sshd was down. Now i am going for the gold.

Persistency; consistency and in this case 2 middle fingers to ovh.net for giving their clients 2007 operating systems installations in 2012. (now that is service excellency!!)

Edit/update:1/05/2012
This box is now at community service as a binhost @ Portage Binhost, Rsync & distcc (Public for netbooks) (100mbit)

[SOLVED]
_________________
Do you hear the sound of inevitability?
With age, comes great grumpiness and that, was 20 years ago...

CertFP: becbbd161d5a5c31de3c45171b77bf710911db29 / d985d21f89fe2977b593c4d381a1a86802e62990d9328d893db76d59f9935244

[kimmie]

NOOO, don't kill the cannibals, they're SO CUTE... anyway, sounds like you didn't really kill them, you just updated their asses into 2012. Congrats! I never would have persisted with that update... once I got stuck in Xen hell with a VPS provider who couldn't understand why my domU kernel needed to be able to see domU kernel modules in the domU filesystem; I just gave up and found me a new girl.

[VinzC]

HeXiLeD, I admire your patience and determination, really. Your reward is probably to now be able to open and maintain yourself a global hosting service and yet provide a much better support than these ass-holes.

I have just noticed one question about the need for an initrd in all of these years of Gentoo and that you weren't feeling comfortable with it. In fact I'd say the only one case Gentoo needs an initrd (on a headless server, basically) is when the root file system doesn't reside on a partition on the local disk, e.g. on LVM, on NFS or iSCSI. I haven't experimented the latter two however, only LVM.

Reading your posts I also learnt how to boot from NFS. So you might find this explanation interesting. As far as I have understood, booting from NFS is not automatic unless there's a kernel option, nfsroot along with root=/dev/nfs. But maybe these command line options are built into the kernel for I know it's possible with recent kernels.

Well I guess it's a bit late but I just wanted to add my 0.5¢ in case you still want to know.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!

[HeXiLeD]

I would like to thank you guys for all the support (specially the guys at #debian when the box had the debian crash before the gentoo install) and as a response to this event i started what i call:

The netbook server project (wiki page) and the topic (gentoo forum).
_________________
Do you hear the sound of inevitability?
With age, comes great grumpiness and that, was 20 years ago...

CertFP: becbbd161d5a5c31de3c45171b77bf710911db29 / d985d21f89fe2977b593c4d381a1a86802e62990d9328d893db76d59f9935244

[HeXiLeD]

Just thought about updating this topic with the latest that OVH has done to me.
After everything that is described above and finally being able to make the server working; they closed my service 2 weeks ago.

But not only the service was closed.

- I cannot access my account as my user and password are now invalid.
- The service was paid until December 2013. (5 more months)
- They emailed me a receipt that is under another name/user and the account identification is not mine
- They have not reimbursed me for the remaining 5 months

All activity done by the server was legit and within the rules.

This is OVH Web hosting solutions.
_________________
Do you hear the sound of inevitability?
With age, comes great grumpiness and that, was 20 years ago...

CertFP: becbbd161d5a5c31de3c45171b77bf710911db29 / d985d21f89fe2977b593c4d381a1a86802e62990d9328d893db76d59f9935244