| View previous topic :: View next topic |
| Author |
Message |
jlpoole
Guru
Joined: 01 Nov 2005
Posts: 483
Location: Salem, OR
|
 Posted: Mon Apr 11, 2022 5:02 am Post subject: xen-tools works in Gentoo Dom 0 |
 |
|
First off, let's distinguish some names which can be confusing. Gentoo has a package app-emulation/xen-tools which is entirely different from the project Xen-Tools at http://xen-tools.org/. The code for Xen-Tools is located at https://github.com/xen-tools/xen-tools . Xen-Tools is referenced on the Xen wiki at: https://wiki.xenproject.org/wiki/Xen_tools which tells us:
| Quote: | | xen-tools—a straightforward VM provisioning/installation tool |
See further: https://wiki.xenproject.org/wiki/Xen_tools#What_makes_xen-tools_special.3F
What has disappointed me for almost a decade is that running Debian had a neat tool "xen tools" which allowed you to easily create guest domains. Gentoo's Xen did not have such a feature; moreover, the package app-emulation/xen-tools created confusion because its name matches the project Xen-Tools. So, hereinafter, "xen tools" means the Xen-Tools project and their code.
I have wanted to look into some Artificial Intelligence ("ai") programs and its seems that Ubuntu is the preferred choice of platform. So I wanted to install Ubuntu as a VM on my Gentoo hypervisor... but there is scant out there about how to install Ubuntu as you might a Gentoo VM -- it seems the xen tools makes that housekeeping chore a snap.
So I cloned the xen tools project and installed in on a Dom0 instance. It immediately was broken. As I started to look into it, I discovered much of the script was done in Perl, despite the scripts not have the ".pl" suffix --- and the documentation within the scripts was plentiful. So I began to analyze why the xen tools project install was not working and determined that the problem is that it does not integrate with Gentoo's Perl layout. Why is that?? I spent several hours studying the structure of the xen tools scripts and concluded that they had simply scripted all the things we do by hand in building a system, but from a chroot perspective. I won't go into the details, but here's what you need to do to get the xen tools script to work. Simply define the PERL5LIB environmental variable when you call their scripts:
| Code: | | PERL5LIB="/usr/share/perl5:/root/perl5/lib/perl5" xen-create-image ... |
The reason I specify these two paths are 1) the xen-tools installs its Perl package under /usr/share/perl5. 2) Gentoo's redirection of cpan package installation necessitates that the packages installed in the user, e.g. root's, be included. The xen tools project requires some Perl packages which are not portage, so I had to resort to cpan and then learned that running cpan did not install the packages into the standard Perl tree, but into a local user tree, e.g. /root/perl5/lib/perl5.
So once I determined these two issues, running xen tools was as simple as prefixing the command with a specification of the PERL5LIB variable. Hence, I successfully created a Ubuntu "focal" build as follows:
| Code: | | --hostname=ai3 --lvm=vg0 --dhcp --memory=512mb --pygrub --kernel /boot/vmlinuz-5.15.11-gentoo-x86_64 --initrd /boot/initramfs-5.15.11-gentoo-x86_64.img --dist=focal |
Now, despite my specifying the kernel and ramdisk, xen tools went the route of pygrub which was a no-starter for me. I ended up modifying the configuration file and specifying the kernel and ramdisk. Here's my current version of ai3.cfg containing modification where noted form the configuration file created by xen-tools. (The creation of ai3 guest VM took 6 minutes -- I think, it was really quick (on an AMD Bulldozer from 2013).)
| Code: | hermes /etc/xen # cat ai3.cfg
#
# Configuration file for the Xen instance ai3, created
# by xen-tools 4.9.1 on Sun Apr 10 20:54:26 2022.
#
#
# Kernel + memory size
#
###bootloader = 'pygrub'
#
# from
# https://www.virtuatopia.com/index.php?title=Building_a_Debian_or_Ubuntu_Xen_Guest_Root_Filesystem_using_debootstrap
#
kernel = "/boot/vmlinuz-5.15.11-gentoo-x86_64"
ramdisk = "/boot/initramfs-5.15.11-gentoo-x86_64.img"
root = "/dev/xvda1 ro lvm"
vcpus = '1'
memory = '512'
#
# Disk device(s).
#
root = '/dev/xvda2 ro'
disk = [
'phy:/dev/vg0/ai3-disk,xvda2,w',
'phy:/dev/vg0/ai3-swap,xvda1,w',
]
#
# Physical volumes
#
#
# Hostname
#
name = 'ai3'
#
# Networking
#
dhcp = 'dhcp'
vif = [ 'mac=00:16:3E:4A:29:26' ]
#
# Behaviour
#
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
hermes /etc/xen #
|
Lastly, I wrote a Perl script which check's the xen-tools scripts and identifies which packages are missing... unfortuantely, you have to keep running the script until it all comes clean as when a package is not found in a script, that script's processing halts so you don't learn of other missing packages until a later run attempt. Here's the script. It could be cleaner what whatever, but I want to get on with learning ai, so I'm not too proud to share this current version:
| Code: | #!/usr/bin/perl
#
# Script to check all the Xen-Tools scripts to see if they need
# any additional packages. Xen-Tools installs scripts in two locations:
# 1) /usr/bin
# 2) /usr/share/xen-tools
#
# using this command:
# PERL5LIB="/usr/share/perl5" perl verify_xen-tools_scripts.pl
# causes the needed path to be inherited.
#
#
use strict;
use warnings;
use File::Find;
#
# from the xen-tools/bin directory:
# ls -1 /usr/local/src/xen-tools/bin
#
my $bin_root = '/usr/bin';
my $tools_dir = '/usr/share/perl5';
my $STD_OUT = "/tmp/check_OUT.txt";
my $STD_ERR = "/tmp/check_ERR.txt";
#
# The location where Xen-Tools installs the other helper scripts
#
my $scripts_dir = "/usr/share/xen-tools";
&test_main_scripts;
find(\&process,$scripts_dir);
sub process {
my $file = $File::Find::name; # full path
return if -d $file;
my $head = `head -n 1 $file`;
chomp $head;
if ($head =~ /perl/){
print "$file is a perl script\n";
my $contents = &fetch($file);
if ($contents =~ /Xen\/Tools\/Common\.pm/s){
print "$file needs Xen::Tools::Common\n";
}
} else {
print "$file\t$head\n";
}
}
#
# ----------------- main subs ------------------
#
sub test_main_scripts {
my @bin_scripts = qw{
xen-create-image
xen-create-nfs
xen-delete-image
xen-list-images
xen-resize-guest
xen-update-image
xt-create-xen-config
xt-customize-image
xt-guess-suite-and-mirror
xt-install-image
};
foreach my $file (sort @bin_scripts){
my $script = $bin_root."/$file";
if (-e $script){
#
# all output goes to STDERR
#
system("perl -c $script 1>$STD_OUT 2>$STD_ERR");
#my $result = &fetch($STD_OUT);
my $errors = &fetch($STD_ERR);
#print "[DEBUG] errors = $errors\n";
#print "result = $result\nerror = $errors\n";
#next;
#
# result will be empty
#
if ($errors =~ /syntax OK/s){
print "PASS\t$script\n";
} else {
if ($errors =~ /Xen\/Tools\/Common\.pm/s){
#
# retry providing the path
#
system("perl -c -I$tools_dir $script 1>$STD_OUT 2>$STD_ERR");
#my $result = &fetch($STD_OUT);
my $errors = &fetch($STD_ERR);
if ($errors =~ /syntax OK/){
#
# No other errors were reported when we added the
# include $tools_dir
#
print "Only needs XenTools dir\t$script\n";
} else {
print "FAIL\t$script\n\t$errors\nMay need Xen::Tools::Common, too.\n";
}
} else {
print "FAIL\t$script\n\t$errors\n";
}
}
} else {
print "$script not found; skipping.\n";
}
print "---------------------------\n";
}
}
#
# --------------------- subs ----------------
#
sub fetch {
my ($file) = @_;
#
# slurp up all lines setin the file
#
my $content;
{
local $/;
open(IN,$file) or die "Could not open $file";
$content = <IN>;
close(IN);
}
return $content;
}
|
[edit 2/7/2023 removed defeating "." from end of URL to xen-tools project link]
Last edited by jlpoole on Wed Feb 08, 2023 12:46 am; edited 1 time in total |
|
| Back to top |
|
 |
jlpoole
Guru
Joined: 01 Nov 2005
Posts: 483
Location: Salem, OR
|
| Posted: Mon Apr 11, 2022 5:14 am Post subject: |
|
|
| And... I have forgotten, I had to add to my Dom0 instance a directory and some information files for the scripts. I'll update this topic in the next few days with those details. Basically it was adding an /etc/apt directory, I think, and then specifying some URLs in a mirrors.conf file. |
|
| Back to top |
|
|
jlpoole
Guru
Joined: 01 Nov 2005
Posts: 483
Location: Salem, OR
|
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20067
|
|
| Back to top |
|
|
jlpoole
Guru
Joined: 01 Nov 2005
Posts: 483
Location: Salem, OR
|
| Posted: Tue Apr 12, 2022 5:25 pm Post subject: |
|
|
This "open source" project looks like it may be a bust since it relies upon a service, Maximo Visual Inspection, provided by IBM. See the Free Trial & Pricing Issue #53 I logged on the project's page. At the time this project was created, it appears IBM was offering trial subscriptions. It appears now, that is no longer the case.
| Quote: | PowerAI Vision
This code pattern requires Maximo Visual Inspection (formerly known as PowerAI Vision). |
Source: https://github.com/IBM/powerai-counting-cars#user-content-powerai-vision
My enthusiasm to "I wanna build that" caused me to overlook the gotcha that you need to use, and most likely pay for, IBM's service. |
|
| Back to top |
|
|
jlpoole
Guru
Joined: 01 Nov 2005
Posts: 483
Location: Salem, OR
|
| Posted: Mon Apr 18, 2022 9:28 pm Post subject: |
|
|
Here's the part I neglected to include that might take some time to hunt down. I had to find the files used for apt and did at the URL I added at the bottom of the sources.list.
| Code: | hermes /etc/xen # ls /etc/apt
sources.list
hermes /etc/xen # cat /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu/ focal main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ focal main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ focal-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ focal-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ focal-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
deb http://archive.canonical.com/ubuntu focal partner
deb-src http://archive.canonical.com/ubuntu focal partner
#
#
# from https://gist.github.com/ishad0w/788555191c7037e249a439542c53e170
#
hermes /etc/xen #
|
4/18/2022 - I got a Juypter notebook working on the Ubuntu installation... had to install a lot of programs using apt, but got there. Of interest is the fact that what took 27 seconds, an 11 second video clip, in Google Collab where you have a Jupyter notebook dedicated to a GPU, too over 1-1/2 hours on my Ubuntu VM without a GPU. This makes getting a GPU almost mandatory. But... there's Google's Coral development system; I have the Dev Board and it does the detections in a very snappy realtime mode ... using their samples. The problem with the Coral project is that it relies on gstreamer and Coral teams implementation of gstreamer for a headless server is want for success when using anything but the sample MP4 they provide. gstreamer is extremely complicated and finicky and I'm going to have to dive into it more to determine what it can parse for the Coral module. This is my word of warning for those who want to try our Coral: be prepared to do a lot of digging into gstreamer. I ended up buying a small HDMI monitor that would work with the Coral board so I could use their "device" mode instead of a headless server and that afforded my a lot of opportunities of seeing something of mine (after import/exporting through OpenShot Video editor which does something to make the *.mp4 files my Reolink cameras generated digestable), but still when I switch to headless mode and the sample code invokes gstreamer's rtsp server, nothing but failure. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Unsupported Software
