Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Request For Comments - How I make my LiveCD
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
bob doe
Apprentice
Apprentice


Joined: 11 Jun 2007
Posts: 207

PostPosted: Mon Dec 28, 2009 4:58 am    Post subject: Request For Comments - How I make my LiveCD Reply with quote

Hello folks! For over a year now, I have been building my LiveCD. I know some stuff but Im by far not an expert so I'd like to present to you how I build my LiveCD for people who want to learn how, or people who want to compare my approach to their own. What I would greatly appreciate is feedback on my method.

For a long time now, all of the scripts, configs, and buildscripts for making FLY have been available on linuxfly.net under the section 'Knowledge' but I like to post this in order to make sense of just how everything works, especially the buildscripts.

Firstly, FLY can be created on any Linux environment provided a few criterias are met:
- The kernel has squashfs support built in
- Internet connectivity is available
- Prefered if Gentoo is the Linux system running but not required, this is only because if you are running Gentoo, odds are you'll already have a populated /usr/portage/distfiles directory so you wont have to download as much source. Else, be prepared to download close to a gig of it. As well, non-Gentoo Linux users will have to modify the scripts a little as /usr/portage is binded to the FLY chroot and that is a Gentoo-only thing.

Ok, lets disect the buildscripts!

Code:
BUILD=/home/FLY/build/source/BUILD.log
EMERGE=/home/FLY/build/source/EMERGE.log
VERSION="0.7.0"
AUTOBUILD="20091013"

This is the very first part of the first script (view the whole first script here: http://linuxfly.net/knowledge/make/MAKE_NEW.070 ) This sets variables for the duration of the build process. The first two are static and are merely for log files that way I can refer to them as $BUILD and $EMERGE in the script rather than the full pathname. The VERSION variable is used to create the source directory as well as properly name the ISO and MD5 files with the version tag. At each new release, I modify this variable. The AUTOBUILD variable is used to identify the datestamp of the Gentoo autobuild that Im using. Unlike other distros, Gentoo is rolling and versionless. Previously, Gentoo would release a new stage3 tarball about once a year and new users would install starting with this tarball. The problem is, if that tarball is more than a few weeks/months old, you'll have a lot of updating to do just to get it up to date and some of those updates may be complex, such as a move to a newer GCC, which can cause headaches even for seasoned Gentoo users. With Gentoo autobuilds, new stage3 tarballs are automatically created and made available regularly. They originally planned for a new one every week but it hasnt always been so. They are usually released on a Tuesday or Wednesday and are available on any Gentoo mirror in the releases/x86/autobuilds/ directory. So anytime I change to a new autobuild, I update this variable. I could have avoided this and made the scripts use stage3* instead but if I have two autobuilds in the directory, that can cause an issue, and likewise, with the variable I can easily keep track of when the autobuild was created.

Code:
cd /home/FLY/
mkdir finished/$VERSION
touch tmpBUILD.log tmpEMERGE.log
echo "====  START OF BUILD LOG  ===================================================" >> tmpBUILD.log
echo "====  START OF EMERGE LOG  ==================================================" >> tmpEMERGE.log
date >> tmpBUILD.log
date >> tmpEMERGE.log
mkdir -pv build/source/{newroot,usr/portage,etc/portage}
cd build/source
mv -v ../../tmpBUILD.log BUILD.log
mv -v ../../tmpEMERGE.log EMERGE.log

This is the very beginning of the script. The directory structure I chose to use is as follows:

/home/FLY/
/home/FLY/build/
/home/FLY/finished/

Using a regular user account with permissions to the /home/FLY directory. The finished version directory is created, and this is where everything will be put once the script finishes. This includes the various files used to create FLY, the source of FLY, the ISO, MD5, and logs. Next we create the two logfiles in temporary locations, start the logs with a little section header and a datestamp in each. The two logfiles are BUILD and EMERGE. BUILD focuses on logging the progress of the LiveCD construction as well as any errors, warnings, notes, and exit code checks. The EMERGE logfile simply keeps a log of which ebuilds (packages) were installed and what version these were. So that if later down the line I have a problem with something, I can come back and compare versions. Generally, BUILD is the most important logfile. Finally we create the source directory where we will start building FLY, and then move the temporary logfiles into that directory. The reason for this is that we need the logfiles accessible inside chroot, which happens later.

Code:
tar jxvpf ../FILES/stage3-i686-$AUTOBUILD.tar.bz2 2>> $BUILD

This unpacks the Gentoo autobuild tarball to the source directory for FLY. The autobuild contains a nearly complete Gentoo installation.

Code:
mount -v -t proc none proc/
mount -v --bind /dev dev/
mount -v --bind /sys sys/
mount -v --bind /usr/portage/ usr/portage/
cp -v /etc/resolv.conf etc/resolv.conf
cp -v ../MAKE_c_NEW .
cp -vr ../FILES/MAKE .
mv -v MAKE/etc/portage/* etc/portage/
mv -v MAKE/etc/{locale.gen,make.conf} etc/

Then we mount various "system" partitions to the FLY source directory. We do this with --bind which links the mounted /dev to the new dev/. These mount points will be needed inside the chroot and that is why they are mounted here. /usr/portage is mounted as well so that the FLY source will have a fully updated Portage tree and distfiles from my Gentoo system. This eliminates the need to download a Portage tree snapshot (which takes time and bandwidth) and then update it. Typical guides just suggest you mount the distfiles in /usr/portage/distfiles/ but then you end up with two Portage trees which is useless when they are both identical (or nearly so). We copy /etc/resolv.conf so we can have nameserver information in the chroot. Finally, the chroot build script MAKE_c_NEW is copied so it can be run and complete the make of FLY without user intervention (except at the passwd prompts, which is annoying..) All the required script,config,code,whatever files that FLY will used are copied from /home/FLY/build/FILES/MAKE to /home/FLY/build/source and from there, a few files which are required immediately are put in their proper locations.

Code:
date +%m%d%H%M%Y > date.stamp
chroot . /bin/bash --login ./MAKE_c_NEW

Next, a datestamp is created with the current date and saved to a file. This is so when we chroot, we can update the date with the same date as the system by using the file. The date command is run to produce the same format as what would be required to enter to set a new date. Nifty! Then we chroot to the new FLY environment in a bash shell, and run the MAKE_c_NEW script. This script will then run until it is finished, at which point the chroot will have been exited and this script (MAKE_NEW) will continue to execute until it is also finished, which results in a FLY ISO.

So, moving to MAKE_c_NEW...
Code:
BUILD=/BUILD.log
EMERGE=/EMERGE.log
VERSION="0.7.0"
AUTOBUILD="20091013"
ROOTPASSWD="toor"
FLYPASSWD="fluffy"
DATE=`cat /date.stamp`

Just like the first script (MAKE_NEW), the second script (MAKE_c_NEW) also has variables. The first four are identical and then three more are added. The ROOTPASSWD and FLYPASSWD should be obvious. These are the passwords for the root account and the fly account respectively. The only purpose these variables serve is in an echo statement at the password prompt section further down in the script to remind me which passwords to set. The DATE variable displays the content of the date.stamp file we just created. It is used to set the date below.

Code:
env-update && source /etc/profile
echo "====  SETTING THE DATE  =====================================================" >> $BUILD
cp -v /usr/share/zoneinfo/Canada/Eastern /etc/localtime
  if [ "$?" = "0" ]; then echo "Localtime OK!" >> $BUILD; else echo "Localtime FAIL!" >> $BUILD; fi
date $DATE >> $BUILD
  if [ "$?" = "0" ]; then echo "Set Date OK!" >> $BUILD; else echo "Set Date FAIL!" >> $BUILD; fi
date >> $BUILD

Because we changed environments with the chroot, we need to update the environment and resource the contents of /etc/profile. Then we work on date/time information. First, copying the Canada/Eastern zoneinfo file to /etc/localtime so that FLY can take up that as it's timezone. Its my timezone so I use it. Next an exit code check to make sure the cp command finished correctly. At the end of the LiveCD, a grep is made in the logfile searching for any FAIL! exit code checks to report whether or not the buildscripts finished successfully or not. The date is then set using the date command and the DATE variable, which contains the contents of date.stamp in the proper format. Again an exit code check is made to make sure things are good. The date is then added to the BUILD logfile.

Code:
echo "====  EMERGING REQUIRED EBUILDS  ============================================" >> $BUILD
echo "Updating autobuild $AUTOBUILD tarball" >> $BUILD
echo "Updating autobuild $AUTOBUILD tarball" >> $EMERGE
emerge -uDNvp world >> $EMERGE
emerge -uDNq world 2>> $BUILD
  if [ "$?" = "0" ]; then echo "Updating OK!" >> $BUILD; else echo "Updating FAIL!" >> $BUILD; fi
echo -5 | update-etc
echo "Required merges" >> $BUILD
echo "Required merges" >> $EMERGE
emerge -vp sysklogd memtest86+ gentoolkit gentoo-sources livecd-tools localepurge slocate >> $EMERGE
emerge -vq sysklogd memtest86+ gentoolkit gentoo-sources livecd-tools localepurge slocate 2>> $BUILD
  if [ "$?" = "0" ]; then echo "Required OK!" >> $BUILD; else echo "Required FAIL!" >> $BUILD; fi

The next step is to update the Gentoo autobuild. This used to be a very long and sometimes painful process when stage3 tarballs were only released once a year. With the new Gentoo autobuilds, this is usually a very quick process and only once caused headaches, with, I believe, a GCC update. The initial emerge -uDNvp command will run a pretend (-p) and verbose (-v) update of the system and log the ebuilds to update to the EMERGE logfile. It will then run the actual update with the quiet (-q) option. Then an exit code check. If any config files need to be updated, these are done via the etc-update (or the older update-etc) command and the -5 option says to update all configs. This is the option to chose as none of my custom configs for FLY are in their proper directories yet so I dont have to worry about overwriting them. Next packages deemed 'required' to the LiveCD are installed. Notably, a kernel (gentoo-sources), and LiveCD tools. Another exit code check is performed.

Code:
echo "====  KERNEL AND MODULES  ===================================================" >> $BUILD
mv -v /MAKE/usr/src/linux/config /usr/src/linux/.config
cd /usr/src/linux
make clean && make && make modules_install
cp -v arch/i386/boot/bzImage /boot/vmlinuz
du -sh /lib/modules/* >> $BUILD

Next the kernel is compiled. We already downloaded the kernel sources in the 'required' installs above and the sources downloaded there match sources I had already downloaded on my Gentoo system and used to update the kernel .config file. So we're all set. We move the config file to its rightful place at /usr/src/linux/.config (/usr/src/linux is a symlink to whatever kernel sources are active, such as /usr/src/linux-2.6.31-gentoo-r6) Then the kernel is compiled with the make command. In the newer buildscripts, an exit code check is made here. Once the kernel is compiled and the modules installed to /lib/modules, the kernel itself is copied to /boot/vmlinuz. Then, to confirm everything went smoothly, du -sh is run on the modules to check that they are all (more or less) there. Generally there is about 20mb of modules. This was more of a problem in the beginnings of FLY and less so now.

Code:
echo "====  EMERGING NEW EBUILDS  =================================================" >> $BUILD
echo "0.2.0 merges" >> $BUILD
echo "0.2.0 merges" >> $EMERGE
emerge -vp dhcpcd elinks irssi yafc >> $EMERGE
emerge -vq dhcpcd elinks irssi yafc 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.2.0 OK!" >> $BUILD; else echo "0.2.0 FAIL!" >> $BUILD; fi
echo "0.3.0 merges" >> $BUILD
echo "0.3.0 merges" >> $EMERGE
#removed mplayer after splashutils
emerge -vp ntfs3g alsa-utils splashutils xorg-x11 mount-cifs tcpdump kismet >> $EMERGE
emerge -vq ntfs3g alsa-utils splashutils xorg-x11 mount-cifs tcpdump kismet 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.3.0 OK!" >> $BUILD; else echo "0.3.0 FAIL!" >> $BUILD; fi
echo "0.3.1 merges" >> $BUILD
echo "0.3.1 merges" >> $EMERGE
emerge -vp usbutils >> $EMERGE
emerge -vq usbutils  2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.3.1 OK!" >> $BUILD; else echo "0.3.1 FAIL!" >> $BUILD; fi
echo "0.3.2 merges" >> $BUILD
echo "0.3.2 merges" >> $EMERGE
emerge -vp ipw2200-firmware ndiswrapper wpa_supplicant >> $EMERGE
emerge -vq ipw2200-firmware ndiswrapper wpa_supplicant 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.3.2 OK!" >> $BUILD; else echo "0.3.2 FAIL!" >> $BUILD; fi
echo "0.3.4 merges" >> $BUILD
echo "0.3.4 merges" >> $EMERGE
emerge -vp eix >> $EMERGE
emerge -vq eix 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.3.4 OK!" >> $BUILD; else echo "0.3.4 FAIL!" >> $BUILD; fi
echo "0.4.0 merges" >> $BUILD
echo "0.4.0 merges" >> $EMERGE
emerge -vp twm xclock xterm fluxbox conky iptraf putty xf86-input-vmmouse xf86-video-ati xf86-video-i740 xf86-video-nv xf86-video-sis xf86-video-tdfx xf86-video-vesa xf86-video-vmware xf86-video-voodoo >> $EMERGE
emerge -vq twm xclock xterm fluxbox conky iptraf putty xf86-input-vmmouse xf86-video-ati xf86-video-i740 xf86-video-nv xf86-video-sis xf86-video-tdfx xf86-video-vesa xf86-video-vmware xf86-video-voodoo 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.4.0 OK!" >> $BUILD; else echo "0.4.0 FAIL!" >> $BUILD; fi
echo "0.4.1 merges" >> $BUILD
echo "0.4.1 merges" >> $EMERGE
emerge -vp madwifi-ng nvidia-drivers >> $EMERGE
emerge -vq madwifi-ng nvidia-drivers 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.4.1 OK!" >> $BUILD; else echo "0.4.1 FAIL!" >> $BUILD; fi
echo "0.5.0 merges" >> $BUILD
echo "0.5.0 merges" >> $EMERGE
emerge -vp mozilla-firefox abiword xchat pidgin tightvnc nmap wireshark xfe aircrack-ng feh gparted >> $EMERGE
emerge -vq mozilla-firefox abiword xchat pidgin tightvnc nmap wireshark xfe aircrack-ng feh gparted 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.5.0 OK!" >> $BUILD; else echo "0.5.0 FAIL!" >> $BUILD; fi
echo "0.5.1 merges" >> $BUILD
echo "0.5.1 merges" >> $EMERGE
emerge -vp acx-firmware clamav >> $EMERGE
emerge -vq acx-firmware clamav 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.5.1 OK!" >> $BUILD; else echo "0.5.1 FAIL!" >> $BUILD; fi
echo "0.5.2 merges" >> $BUILD
echo "0.5.2 merges" >> $EMERGE
emerge -vp adobe-flash scrot gqview >> $EMERGE
emerge -vq adobe-flash scrot gqview 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.5.2 OK!" >> $BUILD; else echo "0.5.2 FAIL!" >> $BUILD; fi
echo "0.5.3 merges" >> $BUILD
echo "0.5.3 merges" >> $EMERGE
emerge -vp tor epdfview scite php curl audacious >> $EMERGE
emerge -vq tor epdfview scite php curl audacious 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.5.3 OK!" >> $BUILD; else echo "0.5.3 FAIL!" >> $BUILD; fi
echo "0.5.4 merges" >> $BUILD
echo "0.5.4 merges" >> $EMERGE
emerge -vp transmission privoxy gtkterm >> $EMERGE
emerge -vq transmission privoxy gtkterm 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.5.4 OK!" >> $BUILD; else echo "0.5.4 FAIL!" >> $BUILD; fi
echo "0.5.5 merges" >> $BUILD
echo "0.5.5 merges" >> $EMERGE
emerge -vp acpi bind-tools >> $EMERGE
emerge -vq acpi bind-tools 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.5.5 OK!" >> $BUILD; else echo "0.5.5 FAIL!" >> $BUILD; fi
echo "0.6.0 merges" >> $BUILD
echo "0.6.0 merges" >> $EMERGE
emerge -vp truecrypt xf86-video-s3 xf86-video-savage >> $EMERGE
emerge -vq truecrypt xf86-video-s3 xf86-video-savage 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.6.0 OK!" >> $BUILD; else echo "0.6.0 FAIL!" >> $BUILD; fi
#echo "0.6.2 merges" >> $BUILD
#echo "0.6.2 merges" >> $EMERGE
#emerge -vp sun-jdk >> $EMERGE
#emerge -vq sun-jdk 2>> $BUILD
#  if [ "$?" = "0" ]; then echo "0.6.2 OK!" >> $BUILD; else echo "0.6.2 FAIL!" >> $BUILD; fi
echo "0.6.3 merges" >> $BUILD
echo "0.6.3 merges" >> $EMERGE
emerge -vp ufed screen >> $EMERGE
emerge -vq ufed screen 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.6.3 OK!" >> $BUILD; else echo "0.6.3 FAIL!" >> $BUILD; fi
echo "0.6.4 merges" >> $BUILD
echo "0.6.4 merges" >> $EMERGE
emerge -vp ettercap slim >> $EMERGE
emerge -vq ettercap slim 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.6.4 OK!" >> $BUILD; else echo "0.6.4 FAIL!" >> $BUILD; fi
echo "0.6.5 merges" >> $BUILD
echo "0.6.5 merges" >> $EMERGE
emerge -vp xf86-video-intel >> $EMERGE
emerge -vq xf86-video-intel 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.6.5 OK!" >> $BUILD; else echo "0.6.5 FAIL!" >> $BUILD; fi
echo "0.6.6 merges" >> $BUILD
echo "0.6.6 merges" >> $EMERGE
emerge -vp wgetpaste >> $EMERGE
emerge -vq wgetpaste 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.6.6 OK!" >> $BUILD; else echo "0.6.6 FAIL!" >> $BUILD; fi
echo "0.7.0 merges" >> $BUILD
echo "0.7.0 merges" >> $EMERGE
emerge -vp sudo vlc wine >> $EMERGE
emerge -vq sudo vlc wine 2>> $BUILD
  if [ "$?" = "0" ]; then echo "0.7.0 OK!" >> $BUILD; else echo "0.7.0 FAIL!" >> $BUILD; fi
echo "Rebuild" >> $BUILD
echo "Rebuild" >> $EMERGE
revdep-rebuild -- -vp >> $EMERGE
revdep-rebuild -- -q 2>> $BUILD
  if [ "$?" = "0" ]; then echo "Rebuild OK!" >> $BUILD; else echo "Rebuild FAIL!" >> $BUILD; fi

This next section is simply where everything on FLY is installed, broken down into when they are installed by version. Its just the same three commands that are repeated over and over: the pretend install and log to EMERGE, the install and log to BUILD, and the exit code check. At the end, revdep-rebuild is run to ensure that no package, dependancies, or libs are broken. They're not but this makes me feel better.

Code:
echo "==== FINISHING AND CLEANING UP  =============================================" >> $BUILD
echo -5 | update-etc
echo "Adding startups" >> $BUILD
rc-update add alsasound boot >> $BUILD
rc-update add fbcondecor boot >> $BUILD
rc-update add autoconfig default >> $BUILD
rc-update add xdm default >> $BUILD

Again, updates to config files are made, and then daemons we want starting at boottime are added. These are alsasound for audio, fbcondecor for the framebuffer, autoconfig for LiveCD configurations including hardware detection, and xdm for a graphical login manager, in FLY's case, SLiM is used.

Code:
echo "Running eix" >> $EMERGE
update-eix
eix -Ic >> $EMERGE
eix-test-obsolete brief >> $BUILD
emerge --depclean -p >> $EMERGE
emerge --depclean 2>> $BUILD
  if [ "$?" = "0" ]; then echo "Depclean OK!" >> $BUILD; else echo "Depclean FAIL!" >> $BUILD; fi

Eix is a utility to search the Portage tree. It's database is updated and then it will log to EMERGE all installed packages. This includes packages already installed in the stage3 tarball we extracted at the beginning as well as all packages we installed in the previous section, and their dependancies. eix-test-obsolete checks the Portage option files in /etc/portage/package.* to make sure no useless/outdated/broken entries exist. Just a cosmetic issue. Depclean is then run to un-install any packages which are needed, usually broken dependancies that no longer have their associated package.

Code:
echo "====  USER ADMINISTRATION  ==================================================" >> $BUILD
date >> $BUILD
echo "Password is $ROOTPASSWD"
passwd
date >> $BUILD
mv -v /MAKE/root/FLYinstaller /root/.FLYinstaller
mkdir /home
useradd -m -G wheel,users,wireshark,audio fly
  if [ "$?" = "0" ]; then echo "Useradd OK!" >> $BUILD; else echo "Useradd FAIL!" >> $BUILD; fi
echo "Password is $FLYPASSWD"
echo "Remember to change /usr/share/slim/themes/FLY/slim.theme"
passwd fly
mv -v /MAKE/home/fly/fluxbox /home/fly/.fluxbox
mv -v /MAKE/home/fly/irssi /home/fly/.irssi
mv -v /MAKE/home/fly/mozilla /home/fly/.mozilla
mv -v /MAKE/home/fly/xchat2 /home/fly/.xchat2
mv -v /MAKE/home/fly/xfe /home/fly/.xfe
mv -v /MAKE/home/fly/xinitrc /home/fly/.xinitrc
mv -v /MAKE/home/fly/vnc /home/fly/.vnc
mv -v /MAKE/home/fly/SciTEUser.properties /home/fly/.SciTEUser.properties
mv -v /MAKE/home/fly/gtktermrc /home/fly/.gtktermrc

This is where user interaction on my part is required (which is annoying). Because the buildscripts take about 10 hours to run, sometimes when it gets to this point, Im not there. Either Im away, or sleeping, and I dont want the built time to be screwed up, I use a date command at the start and the end so I know how much idle time there was. Then the root password is changed. Afterwards, the FLYinstaller files are copied to the root home directory. Then the fly regular account is created and assigned to groups. An exit code check again, and then setting the password for the account, as well as a reminder to edit slim.theme which contains an echo of the password during login. Then all the various files required in the fly user's home directory are moved.

Code:
echo "====  MOVING FILES AROUND  ==================================================" >> $BUILD
mv -v /MAKE/etc/X11/* /etc/X11/
mv -v /MAKE/etc/privoxy/* /etc/privoxy/
mv -v /MAKE/etc/conky/* /etc/conky/
mv -v /MAKE/etc/conf.d/* /etc/conf.d/
mv -v /MAKE/etc/init.d/* /etc/init.d/
cp -v /MAKE/etc/* /etc/
mv -v /MAKE/sbin/* /sbin/
mv -v /MAKE/usr/sbin/* /usr/sbin/
mv -v /MAKE/usr/share/slim/themes/* /usr/share/slim/themes/
rm -vr /usr/share/slim/themes/default
cp -v /etc/tor/torrc.sample /etc/tor/torrc
rm -rfv /mnt/cdrom
rm -rfv /etc/init.d/net.eth0
mkdir -v /mnt/linuxfly.net

Then all other files are moved into place, and a few are removed, such as net.eth0 which would otherwise attempt to run dhcpcd on eth0 at bootup and /mnt/cdrom as that is somewhat useless when on a LiveCD.

Code:
localepurge
makewhatis -u
find / -type f -name ".keep" -print -exec rm {} \;
cat /proc/mounts > /etc/mtab
updatedb
# As of this version, and Audacious version 1.5.1 which pulls in dbus,
# This command must be run to prevent Audacious from crashing
dbus-uuidgen --ensure

localepurge is then run to get rid of all locales except those defined in /etc/locale.gen This is to free up space, the only locale kept is English. Then a find command is run to locate all .keep files which are used in the stage3 tarball so that when Gentoo creates it, empty directories are not deleted. These files are no longer required for us and so we scrap them. Then we create /etc/mtab by populating it with /proc/mounts and then updating the slocate database.

Code:
# Making the tarball for the installer
cd /
tar cjfp installdata-$VERSION.tar.bz2 {var/{db,cache},usr/{lib/portage/,portage/profiles/,share/zoneinfo/},etc/portage/package.*}

To keep the LiveCD to a smaller size, various files are removed from it. Such as Portage, eix databases, etc etc. These are packaged into a tarball that is downloaded by FLYinstaller when you install FLY to your hard drive, giving you a complete distro.

Code:
echo "====  CHOWN AND CHMOD  ======================================================" >> $BUILD
chown fly:fly /home/fly/ -R
  if [ "$?" = "0" ]; then echo "1 of 11 OK!" >> $BUILD; else echo "1 of 11 FAIL!" >> $BUILD; fi
chmod 4500 /root/.FLYinstaller/ -R
  if [ "$?" = "0" ]; then echo "2 of 11 OK!" >> $BUILD; else echo "2 of 11 FAIL!" >> $BUILD; fi
chown root:fly /etc/conky/conkymaker.sh
  if [ "$?" = "0" ]; then echo "3 of 11 OK!" >> $BUILD; else echo "3 of 11 FAIL!" >> $BUILD; fi
chmod 4550 /etc/conky/conkymaker.sh
  if [ "$?" = "0" ]; then echo "4 of 11 OK!" >> $BUILD; else echo "4 of 11 FAIL!" >> $BUILD; fi
chown root:fly /home/fly/.fluxbox/dhcp/dhcp
  if [ "$?" = "0" ]; then echo "5 of 11 OK!" >> $BUILD; else echo "5 of 11 FAIL!" >> $BUILD; fi
chmod 4550 /home/fly/.fluxbox/dhcp/dhcp
  if [ "$?" = "0" ]; then echo "6 of 11 OK!" >> $BUILD; else echo "6 of 11 FAIL!" >> $BUILD; fi
chown root:fly /home/fly/.fluxbox/styleswitcher/{theme,blue,earth}
  if [ "$?" = "0" ]; then echo "7 of 11 OK!" >> $BUILD; else echo "7 of 11 FAIL!" >> $BUILD; fi
chmod 4550 /home/fly/.fluxbox/styleswitcher/{theme,blue,earth}
  if [ "$?" = "0" ]; then echo "8 of 11 OK!" >> $BUILD; else echo "8 of 11 FAIL!" >> $BUILD; fi
chmod 4111 /usr/bin/sudo
  if [ "$?" = "0" ]; then echo "9 of 11 OK!" >> $BUILD; else echo "9 of 11 FAIL!" >> $BUILD; fi
chmod 4711 /bin/su
  if [ "$?" = "0" ]; then echo "10 of 11 OK!" >> $BUILD; else echo "10 of 11 FAIL!" >> $BUILD; fi
chmod 777 /var/{lib,log}/clamav
  if [ "$?" = "0" ]; then echo "11 of 11 OK!" >> $BUILD; else echo "11 of 11 FAIL!" >> $BUILD; fi

Since a regular user account is used on FLY, various files need permission and ownership changes. This is it with exit code checks on each to make sure all is well.

Code:
echo "====  EMERGING GRUB  ========================================================" >> $BUILD
emerge -vp grub >> $EMERGE
emerge -vq grub 2>> $BUILD
  if [ "$?" = "0" ]; then echo "Grub OK!" >> $BUILD; else echo "Grub FAIL!" >> $BUILD; fi
rm -v /boot/grub/{menu.lst,splash.xpm.gz}
mv -v /MAKE/boot/grub/* /boot/grub/

The Grub bootloader is installed and config and graphics copied to the directory. Originals are removed first.

Code:
echo "====  INITRAMFS  ===========================================================" >> $BUILD
mkdir -vp /boot/initramfs/{proc,sbin,bin,etc,new,lib,dev,cdrom}
mv -v /MAKE/boot/init /boot/initramfs/init
mv -v /MAKE/boot/v86d /boot/initramfs/sbin/v86d
cp -v /usr/sbin/lspci /boot/initramfs/sbin/lspci
touch /boot/initramfs/etc/{mtab,fstab}
cd /bin
cp -v gzip sh mount chroot tar busybox /boot/initramfs/bin/
cp -v /usr/share/misc/pci.ids /boot/initramfs/bin/
cd /lib
#REQUIRED LIBS MAY CHANGE (use ldd <command>)
cp -v /usr/lib/libpci.so.3 libncurses.so.5 libdl.so.2 ld-linux.so.2 libc.so.6 libz.so.1 librt.so.1 libblkid.so.1 libpthread.so.0 libuuid.so.1 libresolv.so.2 /boot/initramfs/lib
cd /boot/initramfs/dev
mknod console c 5 1
mknod null c 1 3
mknod hda b 3 0
mknod hdb b 3 64
mknod hdc b 22 0
mknod hdd b 22 64
mknod sr0 b 11 0
mknod sda b 8 0
mknod sdb b 8 16
mknod sdc b 8 32
mknod sdd b 8 64
mknod tty c 4 0
mknod loop0 b 7 0
mknod cdrom c 11 0
mknod tty1 c 4 1
mknod mem c 1 1
mknod zero c 1 5
cd ..
find . | cpio -H newc -o > ../initramfs.cpio
cd ..
cat initramfs.cpio | gzip > initramfs.igz

The initramfs is created here. This is a rather complex and vitally important part of FLY and unfortunately I still dont fully understand it. At present Im having new problems with it and so Im tweaking it. Basically, before the actual FLY root can be mounted, a special small root has to be mounted first to load the CD and kernel. First, various files needed for the initramfs are copied into what looks like a mini distro (it sort of is). This has been tweaked to use busybox exclusively and so other binaries and their required libs are being removed. lspci and it's database file are added so that if initramfs fails to mount the CD drive, it can output an lspci list and the user can post that on bugzilla. Afterwards, various character and block devices are created. This is so that initramfs has somewhere to mount the CD drive to, and other things. Then the initramfs directory is packaged into a cpio archive and that is then compressed using gzip.

Code:
exit

The MAKE_c_NEW script is now finished and it exits chroot, giving back control to MAKE_NEW script and it continues where it left off:

Code:
env-update && source /etc/profile
rm -rfv MAKE
umount -fv sys dev usr/portage /home/FLY/build/source/proc
cd ..
mv source/installdata* .

Since we are now once again in a new environment, updating and sourcing are required. Then, since there is no longer a need for the MAKE directory which was used to copy files to FLY, it is removed. Mounted directories which were bound to directories in source for the chroot are unmounted and then the installdata tarball is taken out of the source and stored temporarily until it (along with everything else) will be moved to it's final resting place: /home/FLY/finished/$VERSION

Code:
rsync -avzdrhP --exclude={usr/{portage,src,local,i486-pc-linux-gnu,share/{doc,gtk-doc,man},lib/{portage,gcc/i486-pc-linux-gnu}},etc/portage,var/{tmp/*,run/*,lock/*,lib/clamav/*},tmp/*} source/ source.new/
cd source.new
rm -rfv boot/{initramfs,initramfs.cpio,boot} etc/mtab root/.bash_history MAKE_c_NEW var/db var/cache
touch etc/mtab

The entire source directory is copied using rsync to a new source.new directory with the exception of key directories and files, notably those placed in the installdata tarball earlier, as well as others such as logfiles, and manpages. Then in the source.new directory, a few more files are removed. Such as the useless initramfs directory and cpio archive (since we now have the gzip'ed file) as well as .bash_history, the MAKE_c_NEW script, and eix databases. I suppose if I wanted to I could simply add these to the rsync exclude line or into an exclude file. A new empty mtab file is created. This file is always automatically populated, but the file must first exist.

Code:
mkdir -pv ../target/files
cp -av boot ../target/
cd etc/
tar cvpzf ../../target/files/etc.tar.gz * .[[:alnum:]]* 2>> $BUILD
cd ../home/
tar cvpzf ../../target/files/home.tar.gz * .[[:alnum:]]* 2>> $BUILD
cd ../mnt/
tar cvpzf ../../target/files/mnt.tar.gz * .[[:alnum:]]* 2>> $BUILD
cd ../root/
tar cvpzf ../../target/files/root.tar.gz * .[[:alnum:]]* 2>> $BUILD
cd ../var/
tar cvpzf ../../target/files/var.tar.gz * .[[:alnum:]]* 2>> $BUILD
cd ../..
mv -v source.new/BUILD.log ../finished/$VERSION/BUILD.log
mv -v source.new/EMERGE.log ../finished/$VERSION/EMERGE.log

A final target/ directory is created. Within it, boot files, the kernel, and the initramfs are located in a target/boot directory, and then a target/files directory contains compressed tarballs of the /etc /home /mnt /root and /var directories are these ones all have files that are edited and need to be mounted re-write for FLY. The logfiles are also moved to their final resting places.

Code:
mksquashfs source.new/ target/files/source.img
mkisofs -R -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -boot-info-table -iso-level 4 -hide boot.catalog -o ../finished/$VERSION/FLY-$VERSION.iso target/ >> ../finished/$VERSION/BUILD.log

The SquashFS filesystem for the root of FLY is created and saved in target/files and then all of target/ is packed into a bootable ISO.

Code:
echo "====  GENERATING MD5SUM  ====================================================" >> ../finished/$VERSION/BUILD.log
cd ../finished/$VERSION/
md5sum FLY-$VERSION.iso >> BUILD.log
md5sum FLY-$VERSION.iso > FLY-$VERSION.iso.md5
md5sum -c FLY-$VERSION.iso.md5 >> BUILD.log
cp -rfp ../../build/{FILES/,MAKE*} . 2>> BUILD.log
mv ../../build/{source*,target,installdata*} . 2>> BUILD.log

The ISO is passed through MD5 and a MD5 check file is created, then compared and logged. The /home/FLY/build/FILES and /home/FLY/build/source directories are copied to /home/FLY/finished/$VERSION

Code:
echo "====  PASS OR FAIL  =========================================================" >> BUILD.log
failure=`grep "FAIL!" BUILD.log | wc -l`
if [ "$failure" = "0" ]; then
    echo "Autobuild OK!" >> BUILD.log
else
    echo "Autobuild FAIL! with $failure failure(s):" >> BUILD.log
    echo >> BUILD.log
    grep 'FAIL!' BUILD.log | grep -v Autobuild >> BUILD.log
fi
date >> BUILD.log
date >> EMERGE.log

A pass or fail mark is given to the build process and logged to BUILD. This greps all exit code failures and if there are none, a pass is given. If there are fails, then they are listed. Finally the finish datestamp is added.

The End,
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 31373
Location: 56N 3W

PostPosted: Fri Jan 01, 2010 9:53 pm    Post subject: Reply with quote

Moved from Gentoo Chat to Documentation, Tips & Tricks.

Will get more exposure here
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2726
Location: Bay Area, CA

PostPosted: Sat Jan 02, 2010 3:41 am    Post subject: Reply with quote

Use genkernel for initramfs creation. You can ask it to create just the initrd, with all the modules, as well as handling of volume manager, encryption, mdraid, dmraid, suspend etc. Its much easier that way.

Also, bind mount /tmp as well. If you need to open multiple terminal windows or run cfg-update in graphical mode from within the chroot.

And if you are using tmpfs on /var/tmp, bind mount that as well. You don't want to write disk over and over again with compile time files.

Do you repeat this every time you want a new livecd with newer programs and kernel?

How much are you able to compress it down to? I have about 516MB livecd with full KDE3.5 install and my favorite games and programs like firefox/thunderbird/virtualbox/vmplayer/vnc/nxclient/vpnc etc. in it. I have custom clean up scripts which remove a lot of stuff which is not needed for my setup.
Back to top
View user's profile Send private message
bob doe
Apprentice
Apprentice


Joined: 11 Jun 2007
Posts: 207

PostPosted: Sat Jan 16, 2010 3:31 pm    Post subject: Reply with quote

I was wondering where my thread went

devsk, to answer some of your questions:

I've never used genkernel before, and didnt want to start using it. It sure took me forever to figure out how to do this without genkernel, but that was fun (and at times frustrating)

Mounting /tmp to /home/FLY/build/source/tmp during the build? I never thought of that. I'll check that out.

I used to repeat this everytime I updated. I didnt want to take an existing build and update and remake it. Not sure why. Originally the compile times were around 4-5hours (Pentium-M 1.6GHz Dell Inspiron 6000 laptop), and eventually with Stage2 and emerge -ave it crept up to the 16-20 hour mark, but then Gentoo ditched Stage2 and went Stage3 so I didnt do the -ave anymore. Still, as I added more things to my LiveCD, the compile times went up. They're around 9-10 hours currently, and with the 0.7.1 version that Im working on, I was recompiling so often that I couldnt afford the wait (usually I'd run the scripts while Im sleeping, at work, at school, or away). I wrote an update script but havent used it much because of the complications Im having with 0.7.1 not booting. I havent found the cause yet.

This is the update script:

Code:
BUILD="/home/FLY/build/source/BUILD.log"
EMERGE="/home/FLY/build/source/EMERGE.log"
OLDVERSION="0.7.1.old"
NEWVERSION="0.7.1"
NEWROOTPASSWD="root"
NEWFLYPASSWD="kitty"

# Edit /usr/share/slim/themes/FLY/slim.theme line 29

cd /home/FLY/
mkdir finished/$NEWVERSION
touch tmpBUILD.log tmpEMERGE.log
echo "====  START OF BUILD LOG  ===================================================" >> tmpBUILD.log
echo "====  START OF EMERGE LOG  ==================================================" >> tmpEMERGE.log
date >> tmpBUILD.log
date >> tmpEMERGE.log
mkdir build/source
mv -v tmpBUILD.log build/source/BUILD.log
mv -v tmpEMERGE.log build/source/EMERGE.log
rsync -aq --progress --exclude={BUILD.log,EMERGE.log,MAKE_c_NEW,date.stamp} finished/$OLDVERSION/source/ build/source/
cd build/source
mount -v -t proc none proc/
mount -v --bind /dev dev/
mount -v --bind /sys sys/
mount -v --bind /usr/portage usr/portage/
echo "====  SYNCING NEW FILES  ====================================================" >> $BUILD
rsync -a --exclude=usr/src/ /home/FLY/build/FILES/MAKE/* .
cp -f ../FILES/MAKE/usr/src/linux/config usr/src/.config
echo "====  UPGRADE INSTRUCTIONS  =================================================" >> $BUILD
UPGRADE='#!/bin/bash\n'
UPGRADE+='\n'
UPGRADE+='BUILD="/BUILD.log"\n'
UPGRADE+='EMERGE="/EMERGE.log"\n'
UPGRADE+='VERSION="'$NEWVERSION'"\n'
UPGRADE+='ROOTPASSWD="'$NEWROOTPASSWD'"\n'
UPGRADE+='FLYPASSWD="'$NEWFLYPASSWD'"\n'
UPGRADE+='\n'
UPGRADE+='env-update && source /etc/profile\n'
UPGRADE+='echo "====  UPDATING WORLD  =======================================================" >> $BUILD\n'
#UPGRADE+='eselect profile set 1\n'
#UPGRADE+='USE="symlink" emerge -vp =sys-kernel/gentoo-sources-2.6.30-r8 >> $EMERGE\n'
#UPGRADE+='USE="symlink" emerge -q =sys-kernel/gentoo-sources-2.6.30-r8 2>> $BUILD\n'
#UPGRADE+='emerge -Cp =sys-kernel/gentoo-sources-2.6.31-r6 >> $EMERGE\n'
#UPGRADE+='emerge -C =sys-kernel/gentoo-sources-2.6.31-r6 2>> $BUILD\n'
UPGRADE+='emerge -uDNvp world >> $EMERGE\n'
UPGRADE+='emerge -uDNq world 2>> $BUILD\n'
UPGRADE+='  if [ "$?" = "0" ]; then echo "Updating OK!" >> $BUILD; else echo "Updating FAIL!" >> $BUILD; fi\n'
UPGRADE+='echo "====  KERNEL AND MODULES  ===================================================" >> $BUILD\n'
UPGRADE+='cd /usr/src/linux\n'
UPGRADE+='mv ../.config .\n'
UPGRADE+='make clean && make && make modules_install\n'
UPGRADE+='  if [ "$?" = "0" ]; then echo "Kernel make OK!" >> $BUILD; else echo "Kernel make FAIL!" >> $BUILD; fi\n'
UPGRADE+='cp -v arch/i386/boot/bzImage /boot/vmlinuz\n'
UPGRADE+='du -sh /lib/modules/* >> $BUILD\n'
UPGRADE+='echo "====  EMERGING NEW EBUILDS  =================================================" >> $BUILD\n'
UPGRADE+='#emerge -vp moo >> $EMERGE\n'
UPGRADE+='#emerge -q moo 2>> $BUILD\n'
UPGRADE+='#  if [ "$?" = "0" ]; then echo "'$NEWVERSION' OK!" >> $BUILD; else echo "'$NEWVERSION' FAIL!" >> $BUILD; fi\n'
UPGRADE+='echo "====  FINISHING AND CLEANING UP  ============================================" >> $BUILD\n'
UPGRADE+='echo -5 | etc-update\n'
UPGRADE+='echo "Running eix" >> $EMERGE\n'
UPGRADE+='eix -Ic >> $EMERGE\n'
UPGRADE+='eix-test-obsolete brief >> $BUILD\n'
UPGRADE+='emerge --depclean -p >> $EMERGE\n'
UPGRADE+='emerge --depclean 2>> $BUILD\n'
UPGRADE+='  if [ "$?" = "0" ]; then echo "Depclean OK!" >> $BUILD; else echo "Depclean FAIL!" >> $BUILD; fi\n'
UPGRADE+='echo "====  USER ADMINISTRATION  ==================================================" >> $BUILD\n'
#UPGRADE+='date >> $BUILD\n'
#UPGRADE+='echo "Password is $ROOTPASSWD"\n'
#UPGRADE+='passwd\n'
#UPGRADE+='echo "Password is $FLYPASSWD"\n'
#UPGRADE+='echo "Remember to change /usr/share/slim/themes/FLY/slim.theme"\n'
#UPGRADE+='passwd fly\n'
#UPGRADE+='date >> $BUILD\n'
UPGRADE+='cd /\n'
UPGRADE+='tar cjfp installdata-'$NEWVERSION'.tar.bz2 {var/{db,cache},usr/{lib/portage/,portage/profiles/,share/zoneinfo/},etc/portage/package.*}\n'
UPGRADE+='echo "====  INITRAMFS  ===========================================================" >> $BUILD\n'
UPGRADE+='cd /boot/initramfs/\n'
UPGRADE+='rm ../initramfs.*\n'
UPGRADE+='find . | cpio -H newc -o > ../initramfs.cpio\n'
UPGRADE+='cd ..\n'
UPGRADE+='cat initramfs.cpio | gzip > initramfs.igz\n'
UPGRADE+='exit\n'
echo -e $UPGRADE > UPGRADE
chmod 755 UPGRADE
chroot . /bin/bash --login ./UPGRADE
##############################################################################
env-update && source /etc/profile
umount -fv sys dev usr/portage /home/FLY/build/source/proc
cd ..
mv source/installdata* .
rsync -aq --exclude={usr/{portage,src,local,i486-pc-linux-gnu,share/{doc,gtk-doc,man},lib/{portage,gcc/i486-pc-linux-gnu}},etc/portage,var/{tmp/*,run/*,lock/*,lib/clamav/*},tmp/*} source/ source.new/
cd source.new
rm -rfv boot/{initramfs,initramfs.cpio,boot} etc/mtab root/.bash_history MAKE_c_NEW var/db var/cache UPGRADE
touch etc/mtab
mkdir -pv ../target/files
cp -av boot ../target/
cd etc/
tar cvpzf ../../target/files/etc.tar.gz * .[[:alnum:]]* 2>> $BUILD
cd ../home/
tar cvpzf ../../target/files/home.tar.gz * .[[:alnum:]]* 2>> $BUILD
cd ../mnt/
tar cvpzf ../../target/files/mnt.tar.gz * .[[:alnum:]]* 2>> $BUILD
cd ../root/
tar cvpzf ../../target/files/root.tar.gz * .[[:alnum:]]* 2>> $BUILD
cd ../var/
tar cvpzf ../../target/files/var.tar.gz * .[[:alnum:]]* 2>> $BUILD
cd ../..
mv -v source.new/BUILD.log ../finished/$NEWVERSION/BUILD.log
mv -v source.new/EMERGE.log ../finished/$NEWVERSION/EMERGE.log
mksquashfs source.new/ target/files/source.img
mkisofs -R -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -boot-info-table -iso-level 4 -hide boot.catalog -o ../finished/$NEWVERSION/FLY-$NEWVERSION.iso target/ >> ../finished/$NEWVERSION/BUILD.log
echo "====  GENERATING MD5SUM  ====================================================" >> ../finished/$NEWVERSION/BUILD.log
cd ../finished/$NEWVERSION/
md5sum FLY-$NEWVERSION.iso >> BUILD.log
md5sum FLY-$NEWVERSION.iso > FLY-$NEWVERSION.iso.md5
md5sum -c FLY-$NEWVERSION.iso.md5 >> BUILD.log
cp -rfp ../../build/{FILES/,MAKE*} . 2>> BUILD.log
mv ../../build/{source*,target,installdata*} . 2>> BUILD.log
echo "====  PASS OR FAIL  =========================================================" >> BUILD.log
failure=`grep "FAIL!" BUILD.log | wc -l`
if [ "$failure" = "0" ]; then
    echo "Autobuild OK!" >> BUILD.log
else
    echo "Autobuild FAIL! with $failure failure(s):" >> BUILD.log
    echo >> BUILD.log
    grep 'FAIL!' BUILD.log | grep -v Autobuild >> BUILD.log
fi
date >> BUILD.log
date >> EMERGE.log


At present, with the list of applications you can see in my initial post, with Fluxbox, and removing emerge from the liveCD (but you get it back when you do a HDD install) the LiveCD size is around 360mb for 0.7.0 and 373mb for 0.7.1 (unfinished) adding IPtables mostly. I have both the GTK+ and QT libs which add to the size though. I'd be curious about your clean up scripts.
Back to top
View user's profile Send private message
Bircoph
Apprentice
Apprentice


Joined: 27 Jun 2008
Posts: 258
Location: Moscow

PostPosted: Sun Jan 17, 2010 7:05 am    Post subject: Reply with quote

IMO the fastest way to create custom livecd will be to take sysresccd (www.sysresccd.org) as a basis and continue with your own customization. There is no need to create livecd from scratch, unless you are interested into research of all internals of the livecd creation process yourself.

The advantage of this approach is that you have not spend your time on all this *scripts burden. It already has development tools available and fairly good setup, so you may proceed almost like with normal gentoo system. This way I build several disks for different purposes.
_________________
Per aspera ad astra!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum