Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Emerge (Verbose) Output Parser - A Script
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
ahsan.ae
n00b
n00b


Joined: 02 Apr 2007
Posts: 2

PostPosted: Mon Apr 02, 2007 11:54 am    Post subject: Emerge (Verbose) Output Parser - A Script Reply with quote

Hi all,

I've written a command-line utility to parse the output of the 'emerge -pv world|system' command. You just need ruby installed on your system.

The output of ./evoparser.rb -h gives a good idea of what it does.

Code:
Usage: evoparser.rb [options] [FILENAME]
       (Invoke without the filename for standard input)

Specific options:
    -c, --command                    Print emerge command
    -r, --report                     Print a tabular report (default)
    -e, --existing                   Show only downloaded pkgs
    -d, --pending-dl                 Show pkgs pending download
    -t, --status [STAT]              Show packages with the specified status
    -s, --size                       Sort by package size (descending)
    -S, --size-asc                   Sort by package size (ascending)
    -A, --alphasort                  Sort alphabetically
    -p, --precise                    Print emerge command with package versions
    -x, --execute-command            Execute 'emerge -pv' command on resulting pkgs (if -c)
    -g, --debug                      Debug - print the lines skipped by the parser. Ignore all other options.


It generates a tabular report as follows:

Code:
SIZE    STATUS  NAME
(kb)

0       N       sys-libs/zlib-1.2.3-r1
0       N       sys-libs/gpm-1.20.1-r5
0       N       sys-libs/ncurses-5.5-r3
0       N       app-shells/bash-3.1_p17
0       N       sys-libs/readline-5.1_p4
0       N       virtual/libiconv-0
0       N       dev-libs/expat-1.95.8
0       N       sys-devel/gettext-0.16.1
0       N       sys-apps/diffutils-2.8.7-r1
0       N       sys-devel/gnuconfig-20060702
0       N       sys-libs/db-4.2.52_p4-r2
0       N       sys-libs/gdbm-1.8.3-r3
0       N       sys-devel/libperl-5.8.8-r1
0       N       dev-lang/perl-5.8.8-r2
0       N       dev-libs/openssl-0.9.8d
69      N       perl-core/Test-Harness-2.64
0       N       app-admin/perl-cleaner-1.04.3
95      N       perl-core/PodParser-1.35


Alternatively, if I want to just emerge the packages that are already on disk, I can do:

Code:
emerge -pv world --deep | ./evoparser.rb -ec


which will output the following:

Code:
emerge -pv sys-libs/zlib sys-libs/gpm sys-libs/ncurses app-shells/bash sys-libs/readline virtual/libiconv dev-libs/expat sys-devel/gettext sys-apps/diffutils sys-devel/gnuconfig sys-libs/db sys-libs/gdbm sys-devel/libperl dev-lang/perl dev-libs/openssl perl-core/Test-Harness app-admin/perl-cleaner perl-core/PodParser


I can also tell evoparser to execute the command with the -x flag. Another useful flag is -p which tells evoparser to output the command with the precise package versions as follows:

Code:
emerge -pv =sys-libs/zlib-1.2.3-r1 =sys-libs/gpm-1.20.1-r5 =sys-libs/ncurses-5.5-r3 =app-shells/bash-3.1_p17 =sys-libs/readline-5.1_p4 =virtual/libiconv-0 =dev-libs/expat-1.95.8 =sys-devel/gettext-0.16.1 =sys-apps/diffutils-2.8.7-r1 =sys-devel/gnuconfig-20060702 =sys-libs/db-4.2.52_p4-r2 =sys-libs/gdbm-1.8.3-r3 =sys-devel/libperl-5.8.8-r1 =dev-lang/perl-5.8.8-r2 =dev-libs/openssl-0.9.8d =perl-core/Test-Harness-2.64=app-admin/perl-cleaner-1.04.3 =perl-core/PodParser-1.35


Since it is time-consuming to pipe the output of emerge to evoparser every time, it is also possible to do 'emerge -pv world > tmp.txt' and then: './evoparser.rb tmp.txt'.

Here's the code (please copy and paste into an editor and save it as evoparser.rb).

Code:
#!/usr/bin/ruby

# evoparser-0.1 (Emerge (Verbose) Output Parser) by Ahsan Ali (ahsanali (dot) ae (at) gmail (dot) com)

#  evoparser parses the output of
#  the 'emerge -pv world|system --deep' command to report on packages that are downloaded
#  or pending download, their portage status (N|UD|F|f|B|R|S), and their sizes.
#  The report can be output in tabular format, or as a emerge command ready to be executed

require 'optparse'

class GentooPackage
  attr_reader :name, :size, :useflags, :installed_version, :status, :category
  attr_writer :name, :size, :useflags, :installed_version, :status, :category
 
  def initialize(name)
    @name = name
    @size = 0
    @useflags = []
    @installed_version = nil
    @status = ''
    @category = @name.split('/').first
  end

  def <=>(other)
    @size <=> other.size
  end

  def to_s
    self.name
  end

  def name
    packageName = %r{([\_\w\-0-9\.\/\+\-]+)}
    pkg_regexp = %r{([\w\_0-9\-\/\+\-]+)-([rc\-\.0-9btestalpha]+)} # limited, but works for most packages
   
    mdata = pkg_regexp.match(@name)
    mdata[1] if mdata.class == MatchData
  end

  def name_with_version
    @name
  end

  def get_report_row
    "#@size\t#@status\t#@name"
  end
end

class EVOParser # Emerge Output Parser
  def initialize(options, files=[])
    @options = options
    @files = files
    @packages = []
  end

  def parse(fn, stdin=false)
    # [ebuild     U ] gnome-base/gnome-vfs-2.16.3 [2.14.2-r1] USE="hal ssl -avahi -debug -doc -gnutls -ipv6 -samba" 0 kB [1]
    patstr = '\[ebuild\s+([R UNFfBSUD]+)\s*\]\s+([\_\w\-0-9\.\/\+\-]+)\s*(\[[a-z0-9\.\_\-\w]+\])?\s*((USE|LINGUAS|VIDEO_CARDS|INPUT_DEVICES)=\"(.+)\")*\s+([0-9\,]+)\s+kB.*'

    pat = Regexp.new(patstr)

    if stdin then
      input_file = $stdin
    else
      input_file = File.open(fn)
    end

    input_file.each_line do |line|
      m = pat.match(line)
      if m.class == MatchData then
    gp = GentooPackage.new(m[2])
    gp.status = m[1]
    gp.installed_version = m[3] unless m[4].nil?
    gp.useflags = m[6].split(' ') unless m[6].nil?
    gp.size = m[7].delete(',').to_i
    @packages << gp
      else
   puts line if @options[:debug]
      end
    end

    input_file.close unless stdin

  end

  def get_emerge_command(list)
      return "" if list.empty?
      "emerge -pv #{list.join(' ')}" # otherwise ;)
  end

  def start
    parse(@files.first, @files.empty?) # parse just the first file
    puts "Warning: Ignoring #{@files.slice(1,files.length).join(' ')}" if @files.length > 1
     
    # IGNORE all actions if in debug mode, or if there are no packages to process
    unless @options[:debug] or @packages.empty?
   
      # the interdependencies of the options are taken care of in the earlier code
      @packages.reject! {|p| p.size > 0} if @options[:existing]
      @packages.reject! {|p| p.size == 0 } if @options[:todownload]
      @packages.reject! {|p| p.status != @options[:status]} unless @options[:status].nil?
      @packages.sort! if @options[:size]
      @packages.sort! {|one,two| two <=> one} if @options[:sizeasc]
      @packages.sort! {|one,two| one.name <=> two.name} if @options[:alphasort]

      # must be last command before printing
      # as it replaces GentooPackage instances with Strings
      @packages.collect! {|p| "=#{p.name_with_version}"} if @options[:precise]

      # OUTPUT the report/command
      if @options[:command] then
   cmd = get_emerge_command(@packages)
   puts cmd unless @options[:execute]
      elsif @options[:report] then
   puts "SIZE \tSTATUS\tNAME"
   puts "(kb)"
   puts
   @packages.each {|p| puts p.get_report_row}
      end

      system(cmd) if @options[:execute]

    end
  end
end

def init

  options = {}
  op = OptionParser.new do |opts|
    opts.banner = "Usage: evoparser.rb [options] [FILENAME]"
    opts.separator  "       (Invoke without the filename for standard input)"
    opts.separator  ""
    opts.separator  "Specific options:"

    opts.on('-c', '--command', 'Print emerge command ') do |e|
      raise 'Specify -c OR -r' if options[:report]
      options[:command] = e
    end

    opts.on('-r', '--report', 'Print a tabular report (default)') do |e|
      raise 'Specify -c OR -r' if options[:command]
      options[:report] = e
    end

    opts.on('-e', '--existing', 'Show only downloaded pkgs') do |e|
     
      raise "Specify -e  OR -d" if options[:todownload]
      options[:existing] = e
    end

    opts.on('-d', '--pending-dl', 'Show pkgs pending download') do |e|
      raise "Specify -e OR -d" if options[:existing]
      options[:todownload] = e
    end

    opts.on('-t', '--status [STAT]', String ,'Show packages with the specified status') do |e|
      options[:status] = e # true by default
    end

    opts.on('-s', '--size', 'Sort by package size (descending)') do |s|
      raise 'Specify either -s OR -S or -A' if options[:sizeasc] or options[:alphasort]
      options[:size] = s
    end

    opts.on('-S', '--size-asc', 'Sort by package size (ascending)') do |s|
      raise 'Specify either -s OR -S or -A' if options[:size] or options[:alphasort]
      options[:sizeasc] = s
    end

    opts.on('-A', '--alphasort', 'Sort alphabetically') do |s|
      raise 'Specify either -s OR -S or -A' if options[:size] or options[:sizeasc]
      options[:alphasort] = s
    end

    opts.on('-p', '--precise', 'Print emerge command with package versions ') do |e|
      if options[:command] then
   options[:precise] = e
      else
   puts "Warning: ignoring -p (needs -c)"
      end
    end
   
    opts.on('-x', '--execute-command', "Execute 'emerge -pv' command on resulting pkgs (if -c)") do |e|
      if options[:command] then
   options[:execute] = e
      else
   puts "Warning: ignoring -x (needs -c)"
      end
    end

    opts.on('-g', '--debug', 'Debug - print the lines skipped by the parser. Ignore all other options.') do |s|
      options[:debug] = s
      puts 'Warning: ignoring all actions due to debug mode'
    end
  end

  files = op.parse!

  # if neither -c or -r have been specified, assume -r
  options[:report] = true unless options[:report] or options[:command]

  parser = EVOParser.new(options, files)
  parser.start
end

init


Please comment !


Last edited by ahsan.ae on Tue Apr 03, 2007 6:28 am; edited 1 time in total
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 2160
Location: The Peanut Gallery

PostPosted: Mon Apr 02, 2007 2:27 pm    Post subject: Re: Emerge (Verbose) Output Parser - A Script Reply with quote

Quote:
Code:
SIZE    STATUS  NAME
(kb)

0       N       sys-libs/zlib-1.2.3-r1
0       N       sys-libs/gpm-1.20.1-r5
0       N       sys-libs/ncurses-5.5-r3
0       N       app-shells/bash-3.1_p17
0       N       sys-libs/readline-5.1_p4
0       N       virtual/libiconv-0
0       N       dev-libs/expat-1.95.8
0       N       sys-devel/gettext-0.16.1
0       N       sys-apps/diffutils-2.8.7-r1
0       N       sys-devel/gnuconfig-20060702
0       N       sys-libs/db-4.2.52_p4-r2
0       N       sys-libs/gdbm-1.8.3-r3
0       N       sys-devel/libperl-5.8.8-r1
0       N       dev-lang/perl-5.8.8-r2
0       N       dev-libs/openssl-0.9.8d
69      N       perl-core/Test-Harness-2.64
0       N       app-admin/perl-cleaner-1.04.3
95      N       perl-core/PodParser-1.35

...
Please comment !

Please don't take this offensively, but I don't see what that output has over emerge. TBH bash is much better at this sort of thing, but well done for learning Ruby and getting it working.
Back to top
View user's profile Send private message
ahsan.ae
n00b
n00b


Joined: 02 Apr 2007
Posts: 2

PostPosted: Tue Apr 03, 2007 3:58 am    Post subject: Why I wrote it Reply with quote

Thanks for the reply, Steve.

Without a net connection on my gentoo machine I update my portage with the nightly portage snapshots, but when it comes to updating a package, I use this script to filter out the packages that need to be downloaded (I do that separately using distgrabber).

Usually, there are a lot of packages that need to be updated/installed, and emerge itself does not provide a facility to JUST emerge the packages whose source files are already on my system, or be able to run a command like 'emerge -pF' on JUST the packages that are NOT already downloaded. [see the -e and -d options]

At other times I decide to install the smallest packages first, and the big ones (qt, eclipse) later, and I need the output Sorted to decide that ;) [see the -S and -s options]

There are also instances when certain packages are shown to need updating but when you do 'emerge packagename' minus the version it picks up some other slot. That is also taken care of by the script [see the -p option]

That was why I wrote the script. I hope someone else also finds it helpful. If you can think of any other functionality, please drop a line.

And YES ! I'm beginning to LOVE ruby :)

Cheers,

AA
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum