[TIP] Firefox and tmpfs: a surprising improvement

[truc]

copy the script somewhere, then modify the PROFILE variable, so that it's equal to "hzotbwlp.default"

close firefox, run the script, look at the output of the 'mount' command, then read this thread again, till you get something

[shgadwa]

localhost ~ # #!/bin/bash
localhost ~ #
localhost ~ # # Change this to match your correct profile
localhost ~ # PROFILE="profiles.ini"
localhost ~ #
localhost ~ # cd "${HOME}/.mozilla/firefox"
localhost firefox #
localhost firefox # if test -z "$(mount | grep -F "${HOME}/.mozilla/firefox/${PROFILE}" )"
> then
> mount "${HOME}/.mozilla/firefox/${PROFILE}"
> fi
[mntent]: line 20 in /etc/fstab is bad
mount: mount point /root/.mozilla/firefox/profiles.ini is not a directory
localhost firefox #
localhost firefox # if test -f "${PROFILE}/.unpacked"
> then
> tar --exclude '.unpacked' -cpf packed.tmp.tar "$PROFILE"
> mv packed.tar packed.tar.old
> mv packed.tmp.tar packed.tar
> else
> tar xpf packed.tar &&\
> touch "${PROFILE}/.unpacked"
> fi
bash: : command not found
touch: cannot touch `profiles.ini/.unpacked': Not a directory

[shgadwa]

And I do not even have a profile directory... I do have a file called profiles.ini in the /firefox folder.

EDIT: Sorry about the double post... I will have to read it a few times then when I get time. Later.

Thanks.

~Shawn

[truc]

No need to double-post.
here is a hint:

then read this thread again, till you get something

[shgadwa]

OK............... I read this thread a few times. I think it works, but then again, I think it does not work. That is to say, I do not get any errors but it never creates or mounts a directory anywhere that I can see and my .mozilla/firefox/ profile directory just keeps filling up with more cache and stuff. I take its not working. To be honest I'm tired of reading the same thing over and over again. Oh well, thats alright... its not like its super slow anyway.

[Enlight]

localhost ~ # #!/bin/bash
localhost ~ # # Change this to match your correct profile
localhost ~ # PROFILE="profiles.ini"

This is where you're wrong! profile is [something].default in ~/.mozilla/firefox/

[Sadako]

localhost ~ # #!/bin/bash
localhost ~ # # Change this to match your correct profile
localhost ~ # PROFILE="profiles.ini"

This is where you're wrong! profile is [something].default in ~/.mozilla/firefox/

Howsabouts this?

Code: Select all

PROFDIR="${HOME}/.mozilla/firefox/"
PROFILE="${PROFDIR}/$(grep Path= "${PROFDIR}/profiles.ini" | cut -d = -f 2)"

Assuming you only have one profile defined within profiles.ini...

[shgadwa]

Yes, I figured that out.... after reading the thread a few times.

I have used the right file (blahblah.default)... but it still does not work.

[Sadako]

localhost ~ # #!/bin/bash
localhost ~ # # Change this to match your correct profile
localhost ~ # PROFILE="profiles.ini"

This is where you're wrong! profile is [something].default in ~/.mozilla/firefox/

Howsabouts this?

Code: Select all

PROFDIR="${HOME}/.mozilla/firefox/"
PROFILE="${PROFDIR}/$(grep Path= "${PROFDIR}/profiles.ini" | cut -d = -f 2)"

Assuming you only have one profile defined within profiles.ini...

Actually, the above is completely unnecessary, and rather silly really.

All you really need is this;

Code: Select all

PROFILE="${HOME}/.mozilla/firefox/*.default/"

[Gentooser]

Cannot edit crontab

Code: Select all

/bin/sh: /usr/bin/vi: No such file or directory
crontab: "/usr/bin/vi" exited with status 127

[Dont Panic]

It looks like something has gone amiss with your default editor setting.

Normally, in Gentoo, this is set for /usr/bin/nano.

One possibility is that your default editor was changed to vim, but the package app-editors/vim was either not installed or un-installed.

You also may want to edit your /etc/rc.conf file, and change your default editor to one that is installed (/usr/bin/nano is usually the default).

[loudmax]

I rewrote stevenrobertson's script to include compression and some error checking. Compression isn't strictly necessary, but it's desirable for machines with SSD media such as netbooks.

Code: Select all

#!/bin/bash
################################################################################################
## Firefox tmpfs utility
#  see http://forums.gentoo.org/viewtopic-t-717117.html
#
# 1. check environment:
#   a. entry in /etc/fstab
#   b. profile dir is empty if not mounted tmpfs
#   if either condition is false, print setup info
# 2. check pack direction:
#   tmpfs is mounted
#       pack it to archive
#   tmpfs not mounted
#       mount and unpack
#
################################################################################################


## Change this to match your correct profile 
#  If blank, this script will grab whatever profile is most recent
#
PROFILE=""
FFOX_HOME="${HOME}/.mozilla/firefox"

## Preferred compression utility
#  We're going for speed, the options are: none, gzip or lzop
#  If blank, the script will select lzop if available or default to gzip
#
COMPRESSOR=""
export GZIP='--fast'    # option for gzip

## get more recent profile in home dir if not defined
#
if [[ ${PROFILE} == "" ]]; then
    PROFILE=$( basename $( ls -rdt ${HOME}/.mozilla/firefox/*.default | tail -n 1 ) )
fi

## Check that environment is set up right
#  a. We found the ffox profile
#  b. There should be an entry in /etc/fstab
#  c. The profile should be an empty directory if the tmps is unmounted
#
FAILED_CHECKS=0
if [[ ${PROFILE} == "" ]]; then
    echo "Cannot find firefox profile directory"
    exit 3
fi
if [[ $( grep "${FFOX_HOME}/${PROFILE}" /etc/fstab ) == "" ]]; then
    FAILED_CHECKS=1
fi
if test -z "$(mount | grep -F "${FFOX_HOME}/${PROFILE}" )"; then
    if [[ -d "${FFOX_HOME}/${PROFILE}" ]]; then
        FAILED_CHECKS=$( ls ${FFOX_HOME}/${PROFILE}/* 2> /dev/null| wc -l )
    fi
fi

if [[ ${FAILED_CHECKS} != 0 ]]; then
    echo "See the Gentoo forums for an explanation of how this works: http://forums.gentoo.org/viewtopic-t-717117.html"
    echo ""
    echo "Make sure a line like this one is in /etc/fstab:"
    GID=$( awk -v FS=: '/'"${USER}"'/{print $4 ; quit}' /etc/passwd )
    echo "firefox ${FFOX_HOME}/${PROFILE} tmpfs noauto,user,exec,uid=${UID},gid=${GID} 0 0"
    echo ""
    echo "Exit firefox and tar a copy of the profile:"
    echo "  cd ${FFOX_HOME}"
    echo "  tar -czpf ${PROFILE}.tar.gz ${PROFILE}   #(this is for gzip compression)"
    echo "      OR"
    echo "  tar -cp ${PROFILE} | lzop -c > ${PROFILE}.tar.lzo   #(this is for lzop compression)"
    echo "      OR"
    echo "  tar -cpf ${PROFILE}.tar ${PROFILE}   #(this is for no compression)"
    echo "Back up your profile and create an empty directory:"
    echo "  mv -i ${PROFILE} ${PROFILE}.bak && mkdir ${PROFILE}"
    exit 2
fi

## define how to pack archive
#
PACKLZO="tar --exclude '.unpackt' -C ${FFOX_HOME} -cp \"${PROFILE}\" | lzop -c > ${FFOX_HOME}/${PROFILE}.tar.lzo"
UNPACKLZO="lzop -cd \"${FFOX_HOME}/${PROFILE}.tar.lzo\" | tar -C ${FFOX_HOME} -xp"
PACKGZP="tar --exclude '.unpackt' -C ${FFOX_HOME} -czpf ${FFOX_HOME}/${PROFILE}.tar.gz \"${PROFILE}\""
UNPACKGZP="tar -C ${FFOX_HOME} -xzpf ${FFOX_HOME}/${PROFILE}.tar.gz"
PACKTAR="tar --exclude '.unpackt' -C ${FFOX_HOME} -cpf ${FFOX_HOME}/${PROFILE}.tar \"${PROFILE}\""
UNPACKTAR="tar -C ${FFOX_HOME} -xpf ${FFOX_HOME}/${PROFILE}.tar"

## set pack/unpack command based on available utilities
#
PACKCMD=""
UNPACKCMD=""
if [[ ${COMPRESSOR} == "" ]]; then
    if [[ $( which lzop ) != "" ]] && [[ -f ${FFOX_HOME}/${PROFILE}.tar.lzo ]] ; then
        PACKCMD=${PACKLZO} 
        UNPACKCMD=${UNPACKLZO}
    else
        PACKCMD=${PACKGZP}
        UNPACKCMD=${UNPACKGZP}
    fi
elif [[ ${COMPRESSOR} == "lzop" ]]; then
    if test -z "which lzop"; then
        echo "Could not find lzop"
        exit 4
    fi
elif [[ ${COMPRESSOR} == "gzip" ]]; then
    PACKCMD=${PACKGZP}
    UNPACKCMD=${UNPACKGZP}
elif [[ ${COMPRESSOR} == "none" ]]; then
    PACKCMD=${PACKTAR}
    UNPACKCMD=${UNPACKTAR}
fi

## Check mount point
#
if test -z "$(mount | grep -F "${FFOX_HOME}/${PROFILE}" )"; then

    mount "${FFOX_HOME}/${PROFILE}"

    if [[ $? != 0 ]]; then
        echo "Failed to mount tmpfs"
        exit 1
    fi
fi

## Things look good, so pack if running profile found, or unpack if profile empty
#
if [[ -f "${FFOX_HOME}/${PROFILE}/.unpackt" ]]; then 
    echo -n "tarring firefox... "
    START_TIME=$( date +%s )
    bash -c "${PACKCMD}"
    if [[ $? != 0 ]]; then
        echo "Failed to pack the profile"
        exit 4
    fi
    END_TIME=$( date +%s )
    PACK_TIME=$(( ${END_TIME} - ${START_TIME} ))
    echo "done in ${PACK_TIME} seconds!"
else
    bash -c "${UNPACKCMD}"
    echo "unpacking!"
    if [[ $? != 0 ]]; then
        echo "Failed to unpack the profile"
        exit 5
    fi
    touch "${FFOX_HOME}/${PROFILE}/.unpackt"
fi

[truc]

I haven't read the whole script, but, I don't see why you launch a new bash process to execute eg "${UNPACKCMD}"

Code: Select all

bash -c "${UNPACKCMD}"

Also, (as a side note ):
You can replace this:

Code: Select all

grep -m 1 ${USER} /etc/passwd | awk 'BEGIN{FS=":"}{print $4}'

with:

Code: Select all

awk -v FS=: '/'"${USER}"'/{print $4 ; quit}' /etc/passwd

EDIT: now that I think about what you really want, I think you can even replace the whole command with

Code: Select all

id -g

[loudmax]

I haven't read the whole script, but, I don't see why you launch a new bash process to execute eg "${UNPACKCMD}"

To tell the truth, I'm not sure why either. If I try to execute it directly without wrapping bash around it, I get this error:

Code: Select all

$ packfox.sh 
lzop: cannot use both `-c' and `-p'
Usage: lzop [-dxlthIVL19] [-qvcfFnNPkUp] [-o file] [-S suffix] [file..]
unpacking!

or this error:

Code: Select all

$ packfox.sh 
packfox.sh: line 137: lzop -cd "/home/user/.mozilla/firefox/abcd1234.default.tar.lzo" | tar -C /home/user/.mozilla/firefox -xp: No such file or directory

I get errors regardless of whether I'm using lzop, gzip, or no compression. If I wrap the command inside a spawned bash process, it comes out fine.

You can replace this:

Code: Select all

grep -m 1 ${USER} /etc/passwd | awk 'BEGIN{FS=":"}{print $4}'

with:

Code: Select all

awk -v FS=: '/'"${USER}"'/{print $4 ; quit}' /etc/passwd

Oh yes, that is nice. I'll update the script.

[boerKrelis]

Here's a rewrite which uses inotify, so it only tars whenever anything changes.
Read the comments in the file to find out how to handle this.

Code: Select all

#!/bin/bash

# Save this script as 'packfox' and hard/symlink it as 'packfox-daemon'
# (or the other way around) in the same dir.
# 'packfox' will invoke the packing.
# 'packfox-daemon' will run a loop, watching for changes, calling 'packfox' if any.
# both incantations will do checks, mounting, and unpacking.

# Change this to match your correct profile
PROFILE="bling"
PFDIR="${HOME}/.mozilla/firefox"
# Tar every .. seconds (regardless of changes)
TMOUT="900"
# But not more often than every .. seconds (regardless of changes)
TMMIN="60"
# Regex for which files not to act on when they're changed.
# Use inotifywait -m -e modify -e move -e create -e delete --exclude '(/Cache/)' -r your_profile_dir
# and do some browsing to determine which regex will be right for YOU.
IEXCL="(.sqlite-journal$)|(\-log.txt$)|(cookies.sqlite$)|(sessionstore\-[0-9].js$)|(/weave/)|(/Cache/)"
# Have you read everything and have you made the necessary adjustments? Then remove the line below ;-)
echo "I should read the comments and adjust the variables before running this script" && exit 2


# No user servicable parts below this line.
TGT="${PFDIR}/${PROFILE}"

function seppuku(){
  echo "${1}"
  exit 2
}

test -d "${PFDIR}" || seppuku "Profile dir doesn't exist"

if [ -z "$(mount -t tmpfs | grep -F "${TGT}" )" ]
then
    mount "${PFDIR}/${PROFILE}" || seppuku "Mounting of profile's tmpfs failed; check fstab"
fi

test -f "${TGT}/.unpacked" || tar -xpf "${PFDIR}/packed.tar" -C "${PFDIR}" && touch "${TGT}/.unpacked"

if [ "$(basename ${0})" == "packfox-daemon" ]
then
  which inotifywait >/dev/null 2>&1 || seppuku "Install sys-fs/inotify-tools first".
  while true
    do inotifywait -q -q -t ${TMOUT} -e modify -e move -e create -e delete --exclude "${IEXCL}" \
    -r "${PFDIR}/${PROFILE}"; "$(dirname "${0}")/packfox"; sleep ${TMMIN}
    done
  exit
fi

cd "${PFDIR}"
tar --exclude '.unpacked' -cpf "${PFDIR}/packed.tmp.tar" "${PROFILE}"
mv "${PFDIR}/packed.tar" "${PFDIR}/packed.tar.old"
mv "${PFDIR}/packed.tmp.tar" "${PFDIR}/packed.tar"

[iamboredr]

it would affect the browsers speed?

[gigio]

Hi,
I'm very interested in this,
Could anyone please post the shell commands to run the script in Mac OS X 10.4 ?
I tried to follow all of your instructions, but i get this error when mounting tmpfs: "unknown special file or file system."
It probably has to do with fstab,

Anyone can help?

Thank you very much in advance

[truc]

Hi,
I'm very interested in this,
Could anyone please post the shell commands to run the script in Mac OS X 10.4 ?
I tried to follow all of your instructions, but i get this error when mounting tmpfs: "unknown special file or file system."
It probably has to do with fstab,

Anyone can help?

Thank you very much in advance

googling for tmpfs osx ~brings this link: Blazing Fast Firefox using OSX RamDisk

[Ch00k]

Well, I wonder why would you need a cron entry for the script. Wouldn't it be enough to execute the script for just two times: on login (boot) and on logout (shutdown). What files do I have to add this script to? Are they .bash_profile and .bash_logout?

[Ch00k]

One thing to bear in mind is that variables in these files override what you may find in make.conf, rather than add to them, so if you have just 'CFLAGS="-O3"' in the above example ffmpeg will be compiled without any -march set.

I also have a question regarding this post. Does the above mean that if you do not set CHOST, MAKEOPTS, GENTOO_MIRRORS etc, then they will leave unset for this specific package? Well, without GENTOO_MIRRORS you will not be able to even fetch it

[boerKrelis]

Ch00k, you're right wrt needing to run the script >2 times. It depends on how much of your browsing history, open tabs, bookmarks, entered passwords, stored cookies you are willing to lose if the unforeseen happens (eg you crash your machine).
If you want to avoid doing unnecessary work (tarring even though nothing's changed) and want to control upper and lower bounds to how often the tarring should take place I'd recommend my inotify script posted in this thread.
Your ~/.bash_login only gets sourced when you execute bash as a login shell. From this it follows that that is not a good place to call the script from, unless you don't boot into X and bash is your login shell.
I run my 'packfox-daemon' from an XDG entry in ~/.config/autostart .

I also have a question regarding this post. Does the above mean that if you do not set CHOST, MAKEOPTS, GENTOO_MIRRORS etc, then they will leave unset for this specific package? Well, without GENTOO_MIRRORS you will not be able to even fetch it

If you don't override stuff on the commandline they will be taken from make.conf . If not found there, portage will leave them unset or, in some cases, make up something by itself.
This can be illustrated by doing something like

Code: Select all

CFLAGS="-foobar" emerge --info

and observing the difference between this and a regular

Code: Select all

emerge --info

You can override any variable. This is a generic mechanism. Observe the output from

Code: Select all

env

That's all the variables assigned in your current environment (such as your current bash shell).

You can figure out how it works by observing the difference in output between

Code: Select all

/usr/bin/which ls

and

Code: Select all

PATH="/tmp" /usr/bin/which ls

You can make variables persistent in the current environment by using export. For instance, try

Code: Select all

export PATH="/tmp"

and observe that it has now become quite difficult to do stuff in this particular shell ;-)

[Ch00k]

boerKrelis, many thanks for your reply. Though, I actually mean the following:

Sure, tbh I can't even remember how/where I learned about it, but I know full well about the lack of documentation for this feature.

Take (for example) media-video/ffmpeg, if you have CFLAGS="-O2 -march=k8" in make.conf, but wish to have ffmpeg compiled with -O3, you'd create the file (and parent directories) '/etc/portage/env/media-video/ffmpeg', and simply add 'CFLAGS="-O3 -march=k8"' to it, basically the same format as make.conf, which is really just for setting shell variables.

That's basically all there is to it, and it has lots of uses, you can override many variables which you might set in make.conf, CFLAGS just being the most obvious one, but another fantastic one is EXTRA_ECONF, to which you can add extra --enable-foo or --disable-foo arguments to ./configure (presuming the ebuild uses econf).

One thing to bear in mind is that variables in these files override what you may find in make.conf, rather than add to them, so if you have just 'CFLAGS="-O3"' in the above example ffmpeg will be compiled without any -march set.

Another thing is not everything seems to work, one example of something which doesn't is ALSA_PCM_PLUGINS for media-libs/alsa-lib, and probably the same holds true for VIDEO_CARDS, LINGUAS and the like (run `emerge -v --info | grep USE_EXPAND` for a list), but most likely the reason for this is they are in fact just USE flags, and their real format is alsa_pcm_plugins_FOO, so adding alsa_pcm_plugins_rate and similar to /etc/portage/package.use should do the trick there.

So... if you create this /etc/portage/env/media-video/ffmpeg file and post there, say, CFLAGS="-O2 -march=k8", then will all the other variables, like CHOST, MAKEOPTS, GENTOO_MIRRORS be taken from /etc/make.conf, or will they be left unset. From the "One thing to bear in mind is that variables in these files override what you may find in make.conf, rather than add to them, so if you have just 'CFLAGS="-O3"' in the above example ffmpeg will be compiled without any -march set." I concluded that they will be left unset. Or maybe the files in /etc/portage/env are intended to be used for posting into them the CFLAGS variable only, and all the rest is taken from /etc/make.conf?
Sorry to post this in an unrelated thread, but I'm really concerned in learning this feature

[loki99]

Thx for the tip! It reduced the initial startup time by 50% for me and the whole thing just feel a tad faster.

[Sadako]

boerKrelis, many thanks for your reply. Though, I actually mean the following:

Sure, tbh I can't even remember how/where I learned about it, but I know full well about the lack of documentation for this feature.

Take (for example) media-video/ffmpeg, if you have CFLAGS="-O2 -march=k8" in make.conf, but wish to have ffmpeg compiled with -O3, you'd create the file (and parent directories) '/etc/portage/env/media-video/ffmpeg', and simply add 'CFLAGS="-O3 -march=k8"' to it, basically the same format as make.conf, which is really just for setting shell variables.

That's basically all there is to it, and it has lots of uses, you can override many variables which you might set in make.conf, CFLAGS just being the most obvious one, but another fantastic one is EXTRA_ECONF, to which you can add extra --enable-foo or --disable-foo arguments to ./configure (presuming the ebuild uses econf).

One thing to bear in mind is that variables in these files override what you may find in make.conf, rather than add to them, so if you have just 'CFLAGS="-O3"' in the above example ffmpeg will be compiled without any -march set.

Another thing is not everything seems to work, one example of something which doesn't is ALSA_PCM_PLUGINS for media-libs/alsa-lib, and probably the same holds true for VIDEO_CARDS, LINGUAS and the like (run `emerge -v --info | grep USE_EXPAND` for a list), but most likely the reason for this is they are in fact just USE flags, and their real format is alsa_pcm_plugins_FOO, so adding alsa_pcm_plugins_rate and similar to /etc/portage/package.use should do the trick there.

So... if you create this /etc/portage/env/media-video/ffmpeg file and post there, say, CFLAGS="-O2 -march=k8", then will all the other variables, like CHOST, MAKEOPTS, GENTOO_MIRRORS be taken from /etc/make.conf, or will they be left unset. From the "One thing to bear in mind is that variables in these files override what you may find in make.conf, rather than add to them, so if you have just 'CFLAGS="-O3"' in the above example ffmpeg will be compiled without any -march set." I concluded that they will be left unset. Or maybe the files in /etc/portage/env are intended to be used for posting into them the CFLAGS variable only, and all the rest is taken from /etc/make.conf?
Sorry to post this in an unrelated thread, but I'm really concerned in learning this feature

Sorry I didn't reply sooner, updates to this thread weren't bumping it in my "egosearch".

Anyways, variables like CHOST & co will only be overridden if you set them in the /etc/portage/env/cat/package file, ie if you don't set them they'll use what's set in make.conf.

The point I was trying to make was that you can't just add new flags to a variable like CFLAGS, ie if you have CFLAGS="-march=i686 -O2" in make.conf and you want to add -fomit-frame-pointer for one particular package via /etc/portage/env, you need to add CFLAGS="-march=i686 -O2 -fomit-frame-pointer" rather than just CFLAGS="-fomit-frame-pointer", as with the latter there'll be no -march or optimization switch passed to gcc during compile.

However, only the flags you explicitly set in the env file will override those in make.conf, meaning if you don't set CHOST or whatever in that file it'll still use the CHOST setting in make.conf, regardless of whatever else you have set in the env file.

A hope this clears it up a little, sorry it was kinda vague before.

[El_Goretto]

Hi everyone,

I wanna use this script, but I dont understand why using so dirty "hacks" (local.stop or crontab... argh) to ensure that the script is run after ff is closed...
Why not simply use a simple "wrapper", such as this totally complex:

Code: Select all

#!/bin/bash

/usr/local/bin/pack_ffox.sh
/usr/bin/firefox $*
/usr/local/bin/pack_ffox.sh

With a generic pack_ffox.sh using Hopeless modification:

Code: Select all

#!/bin/bash

PROFDIR="${HOME}/.mozilla/firefox"
PROFILE="$(grep Path= "${PROFDIR}/profiles.ini" | cut -d = -f 2)"

cd "${HOME}/.mozilla/firefox"

if test -z "$(mount | grep -F "${HOME}/.mozilla/firefox/${PROFILE}" )"
then
    mount "${HOME}/.mozilla/firefox/${PROFILE}"
fi

if test -f "${PROFILE}/.unpacked"
then
    tar --exclude '.unpacked' -cpf packed.tmp.tar "$PROFILE"
    mv packed.tar packed.tar.old
    mv packed.tmp.tar packed.tar
else
    tar xpf packed.tar &&\
    touch "${PROFILE}/.unpacked"
fi 

Or did I miss something?