Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
how to transform a column list into usable command line?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
_______0
Guru
Guru


Joined: 15 Oct 2012
Posts: 521

PostPosted: Sun Jan 06, 2013 8:59 pm    Post subject: how to transform a column list into usable command line? Reply with quote

hi,

I want to generate a list of of packages (or any files for that matter) with eix. Currently is possible but if I want to use that list in command line is not possible because it's a column and command line can't process columns.

What I am trying to do is with eix to list packages from a certain category that are NOT installed, but shockingly has options for everything except for 'NOT installed' :/

This list is needed because some packages I want to discard but most not, so eix alone can't do it. I was passing eix to emerge like this emerge `eix fumanchu`.

Anyone know how to transform a list like this:

Code:
a
b
c
d
e
f
.
.
.


to this??

Code:
a b c d e ...


My current crudish solution is to do eix -I -C >foo then eix -C > bar and diff foo bar > final, and at last manually tweak final and pass it to emerge with `cat final`. I know, is not the smartest way to achieve this.

thnks
Back to top
View user's profile Send private message
kimmie
Guru
Guru


Joined: 08 Sep 2004
Posts: 531
Location: Australia

PostPosted: Sun Jan 06, 2013 11:01 pm    Post subject: Reply with quote

When you use bash command substitution $(..) (or even the old shell backticks `..`), all whitespace is coalesced to single spaces. Newlines are treated as whitespace, so they disappear.

So:
Code:
$ equery -q l -f x11-drivers
x11-drivers/nvidia-drivers-310.19
x11-drivers/xf86-input-evdev-2.7.3
x11-drivers/xf86-input-synaptics-1.6.2-r1
$ echo $(equery -q l -f x11-drivers)
x11-drivers/nvidia-drivers-310.19 x11-drivers/xf86-input-evdev-2.7.3 x11-drivers/xf86-input-synaptics-1.6.2-r1


If your output contains non-newline whitespace, you'll have to be careful about quoting.
Back to top
View user's profile Send private message
_______0
Guru
Guru


Joined: 15 Oct 2012
Posts: 521

PostPosted: Mon Jan 07, 2013 8:39 am    Post subject: Reply with quote

amazing it works!!

thanks a lot.

just one more thing. Why can't command line process columns? Is more natural than serialized items. For example I do ls on a directory with ogg files, then I highlight the list and it becomes a column:

Code:

1.mp3
2.mp3
3.mp3
...


there's no way to paste columns to a command on the terminal. If not current shell design is fundamentally flawed. Serialized items, visually, are a pain to parse (humanly), and wastes a lot of time.
Back to top
View user's profile Send private message
kimmie
Guru
Guru


Joined: 08 Sep 2004
Posts: 531
Location: Australia

PostPosted: Mon Jan 07, 2013 11:23 am    Post subject: Reply with quote

I see the argument pasting issue as the terminal, the shell and the user all having different views of a stream of text, more than as a flaw in the shell. The terminal is 2D, the shell is 1D, and only you can say whether the end of your selection should terminate the command when pasted....

I think some xterm variants may translate newlines into spaces (I'm not sure which ones, and it probably depends on a using ctrl or alt-paste). Or, app-misc/screen will virtualise your terminal session, and has a vi-like copy mode where you can scroll and search terminal output and also copy and paste. It will do that sort of translation. screen is really handy if you do lots of terminal work, especially if your session is on a remote system. Bit of a learning curve though.

From the shell end, I find myself using cat or xargs fairly often. xargs takes arguments from stdin, and builds a command, so:
Code:
$ mkdir test
$ cd test
$ touch a b abc "def ghi"
$ ls -1
a
abc
b
def ghi
$ cat | xargs -d '\n' du -sh
a    <--- cut and paste above output
abc
b
def ghi
^D    <--- hit ^D to terminate input to cat
0       a
0       abc
0       b
0       def ghi

The -d option tells xargs that each line is an argument, so that will handle the file "def ghi" correctly.

You can use a shell alias to save typing:
Code:
$ alias pastie="cat | xargs -d '\n'"
$ pastie ls -lh
a  <--- copy and paste
abc
b
def ghi
^D   <--- hit ^D
-rw-r--r-- 1 gid gid 0 Jan  7 21:47 a
-rw-r--r-- 1 gid gid 0 Jan  7 21:47 abc
-rw-r--r-- 1 gid gid 0 Jan  7 21:47 b
-rw-r--r-- 1 gid gid 0 Jan  7 21:47 def ghi

Sometimes, just using $(cat) is handier:
Code:
$  for x in $(cat) ; do echo "Processing $x ..." ; done
a   <--- paste
abc
b
def ghi
^D   <-- hit ^D
Processing a ...
Processing abc ...
Processing b ...
Processing def ...
Processing ghi ...

Since that won't handle "def ghi" correctly, sometimes you need:
Code:
$ while read x ; do echo "Processing $x ..." ; done
a
abc
b
def ghi
^D
Processing a ...
Processing abc ...
Processing b ...
Processing def ghi ...

:)
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4316

PostPosted: Mon Jan 07, 2013 2:01 pm    Post subject: Reply with quote

You should really get used to read manuals. Of course, eix is able to print uninstalled packages (e.g. ... --and --not --installed), and it can produce it in almost any format you want; it could even print you uninstalled slots if you dive into details of the documentation, although this would be a bit more tricky and require to build your custom FORMAT string.
Back to top
View user's profile Send private message
_______0
Guru
Guru


Joined: 15 Oct 2012
Posts: 521

PostPosted: Mon Jan 07, 2013 4:22 pm    Post subject: Reply with quote

it was tricky though, looks like the option order matters here, by crazy recombination of option I discovered something that appears to yield the proper results:

Code:
eix --only-names -C fonts --not -I


by the way, I dunno wtf this means:

Code:
       EXPRESSION ::= [ --not | -! ] BRACE_OR_TEST |
                      EXPRESSION [ --and| -a ] EXPRESSION |
                      EXPRESSION [ --or | -o ] EXPRESSION

       BRACE_OR_TEST ::= --open|-( EXPRESSION --close|-) |
                      TEST_WITH_OPTIONS

       TEST_WITH_OPTIONS ::= [TEST_OPTIONS] [PATTERN]


why the hell are brackets, parethesis and pipes in there?? are they usable or what? dis crazy!!
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4316

PostPosted: Mon Jan 07, 2013 4:46 pm    Post subject: Reply with quote

_______0 wrote:
it was tricky though, looks like the option order matters here

Of course. It is a list of expressions. In a shell the list of expressions
Code:
a && ! b
also differs from
Code:
! b && a

Quote:
By the way, I dunno wtf this means:

It is a grammar written in (a somewhat sloppy variant of) Backus-Naur Form (BNR) which is a wide-spread form (and essentiall the only simple form) to describe the syntax of complex expressions, see e.g. the Wiki entry on BNR. You will meet this in many manpages, e.g. in
Code:
man sudoers
Back to top
View user's profile Send private message
py-ro
Veteran
Veteran


Joined: 24 Sep 2002
Posts: 1511
Location: St. Wendel

PostPosted: Mon Jan 07, 2013 8:49 pm    Post subject: Reply with quote

If you use bash, try CTRL-X CTRL-E, your favorite editor will open and you can paste your arguments easily.

[EDIT]Swapped keys[/EDIT]
Back to top
View user's profile Send private message
_______0
Guru
Guru


Joined: 15 Oct 2012
Posts: 521

PostPosted: Tue Jan 08, 2013 12:42 pm    Post subject: Reply with quote

py-ro wrote:
If you use bash, try CTRL-X CTRL-E, your favorite editor will open and you can paste your arguments easily.

[EDIT]Swapped keys[/EDIT]


uh???


k, found out a simpler way to transform columnated list to serial:

Code:
echo $(echo '<paste huge column paste>a
cmdsubst>b
cmdsubst>c
cmdsubst>d
cmdsubst>e
cmdsubst>f
cmdsubst>g
cmdsubst>h')


Compared with cat | xargs ... is simpler since you can save two extra commands, | and xargs with complicated escaped option. But nice to know different ways of achieving this.

kimmie wrote:
The terminal is 2D, the shell is 1D


^^^ that made me lol!! poor shell


What about the reverse? how to transform a serial list into a column?? IS IT TOO MUCH ASKING THIS NOW???
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 7888
Location: Somewhere over Atlanta, Georgia

PostPosted: Tue Jan 08, 2013 3:07 pm    Post subject: Reply with quote

TIMTOWTDI. Here're just a few:

From the command line:
Code:
~ $ awk 'BEGIN { for (i=1; i <= ARGC; i++) print ARGV[i] }' The quick brown fox jumped over the lazy dog.
The
quick
brown
fox
jumped
over
the
lazy
dog.
Or, from a pipe:
Code:
~ $ echo "The quick brown fox jumped over the lazy dog." | awk '{ for (i=1; i <= NF; i++) print $i }'
The
quick
brown
fox
jumped
over
the
lazy
dog.
Or, more simply (but more abstrusely), with sed:
Code:
~ $ echo "The quick brown fox jumped over the lazy dog." | sed 's/\s/\n/g'
The
quick
brown
fox
jumped
over
the
lazy
dog.
There are more but I'm going to quit now. :wink:

- John
_________________
I can confirm that I have received between 0 and 999 National Security Letters.
Back to top
View user's profile Send private message
papu
Guru
Guru


Joined: 25 Jan 2008
Posts: 393
Location: Sota algun pi de l'empordà

PostPosted: Mon Jan 14, 2013 1:29 pm    Post subject: Reply with quote

hi, i like pass to file a list of filter atoms bases/atoms versions , with all dependencies of a previous just installed package.

something like this:
Code:
enric@Ordinador ~ $ sudo emerge -pv kscd

These are the packages that would be merged, in order:

Calculating dependencies                                   ... done!
[ebuild  N     ] media-libs/libdiscid-0.3.0  USE="-static-libs" 0 kB
[ebuild  N     ] media-libs/musicbrainz-3.0.3:3  USE="{-test}" 0 kB
[ebuild  N     ] kde-base/libkcompactdisc-4.9.5:4  USE="alsa (-aqua) -debug" 0 kB
[ebuild  N     ] kde-base/kscd-4.9.5:4  USE="(-aqua) -debug" 0 kB

Total: 4 packages (4 new), Size of downloads: 0 kB

Code:

enric@Ordinador ~ $ sudo emerge -pv kscd|cut -d']' -f2 |cut -d' ' -f2 > uninstall_kscd
enric@Ordinador ~ $ cat uninstall_kscd

are

dependencies
media-libs/libdiscid-0.3.0
media-libs/musicbrainz-3.0.3:3
kde-base/libkcompactdisc-4.9.5:4
kde-base/kscd-4.9.5:4

4



but in an optimized and clean manner, i don't know how to use text filters like sed o awk to do this.
and then uninstall this packages with this:

Code:
enric@Ordinador ~ $ sudo emerge -Cp `< uninstall_kscd`

 This action can remove important packages! In order to be safer, use
 * `emerge -pv --depclean <atom>` to check for reverse dependencies before
 * removing packages.

>>> These are the packages that would be unmerged:

 media-libs/libdiscid
    selected: 0.3.0
   protected: none
     omitted: none

 media-libs/musicbrainz
    selected: 3.0.3
   protected: none
     omitted: none

 kde-base/libkcompactdisc
    selected: 4.9.5
   protected: none
     omitted: none

 kde-base/kscd
    selected: 4.9.5
   protected: none
     omitted: none

All selected packages: kde-base/kscd-4.9.5 kde-base/libkcompactdisc-4.9.5 media-libs/libdiscid-0.3.0 media-libs/musicbrainz-3.0.3

>>> 'Selected' packages are slated for removal.
>>> 'Protected' and 'omitted' packages will not be removed.


thanks...a lot! ad1
_________________
--Intel i5 3570k --Asrock z77 ext. 4 --Skill 16GB dd3 --Radeon r9 270 --S.O Gentoo & Win 8.1 64bits
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4316

PostPosted: Mon Jan 14, 2013 2:07 pm    Post subject: Reply with quote

The eix --pipe parsing algorithm is quite smart, so you can simply use eix:
Code:
sudo emerge -pv kscd | eix '-|*' --format '<markedversions:EQNAMEVERSION>'
Back to top
View user's profile Send private message
papu
Guru
Guru


Joined: 25 Jan 2008
Posts: 393
Location: Sota algun pi de l'empordà

PostPosted: Mon Jan 14, 2013 2:48 pm    Post subject: Reply with quote

mv wrote:
The eix --pipe parsing algorithm is quite smart, so you can simply use eix:
Code:
sudo emerge -pv kscd | eix '-|*' --format '<markedversions:EQNAMEVERSION>'


thanks, but this only read first package of the list:

Code:

enric@Ordinador ~ $ sudo emerge -pv kscd | eix '-|*' --format '<markedversions:EQNAMEVERSION>'
=media-libs/libdiscid-0.3.0


Code:
enric@Ordinador ~ $ sudo emerge -pv kscd

These are the packages that would be merged, in order:

Calculating dependencies                                   ... done!
[ebuild  N     ] media-libs/libdiscid-0.3.0  USE="-static-libs" 0 kB
[ebuild  N     ] media-libs/musicbrainz-3.0.3:3  USE="{-test}" 0 kB
[ebuild  N     ] kde-base/libkcompactdisc-4.9.5:4  USE="alsa (-aqua) -debug" 0 kB
[ebuild  N     ] kde-base/kscd-4.9.5:4  USE="(-aqua) -debug" 0 kB

Total: 4 packages (4 new), Size of downloads: 0 kB


:roll:
_________________
--Intel i5 3570k --Asrock z77 ext. 4 --Skill 16GB dd3 --Radeon r9 270 --S.O Gentoo & Win 8.1 64bits
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4316

PostPosted: Mon Jan 14, 2013 3:54 pm    Post subject: Reply with quote

papu wrote:
thanks, but this only read first package of the list:

You are right, this is due to changed output of portage: The appended slot in the versionnumber is confusing eix. I guess eix needs to be fixed.
Back to top
View user's profile Send private message
papu
Guru
Guru


Joined: 25 Jan 2008
Posts: 393
Location: Sota algun pi de l'empordà

PostPosted: Mon Jan 14, 2013 5:36 pm    Post subject: Reply with quote

mv wrote:
papu wrote:
thanks, but this only read first package of the list:

You are right, this is due to changed output of portage: The appended slot in the versionnumber is confusing eix. I guess eix needs to be fixed.


:lol: ,

i ask you, ¿what do you recomended to me to start learning filter text, sed or awk or... ? i don't need advanced think, only to take some control of this stuff on command line :)


thanks, a lot...
_________________
--Intel i5 3570k --Asrock z77 ext. 4 --Skill 16GB dd3 --Radeon r9 270 --S.O Gentoo & Win 8.1 64bits
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4316

PostPosted: Mon Jan 14, 2013 6:18 pm    Post subject: Reply with quote

In general, you can use any of sed, awk, perl (or even python or ruby) for simple modifications of such a type. However, dedicated tools like eix are preferrable if they are suited to your task, because you can expect that they will be adopted to changes of portage. For instance, although it is unfortunate, that just now it turned out that eix has problems with the new portage output, this will be fixed in the next eix release (>=0.28.1 which will be soon in the tree - the fix is already in git): For your own scripts you cannot expect that they will fix themselves automatically when portage changes its output format in a new version...

Edit: Until the new version is available to you, you can use something like
Code:
sudo emerge -pv kscd | sed -n -e 's!^.* \([^ ]*/[^ ]*\).*$!=\1!p'
Back to top
View user's profile Send private message
papu
Guru
Guru


Joined: 25 Jan 2008
Posts: 393
Location: Sota algun pi de l'empordà

PostPosted: Tue Jan 15, 2013 12:17 am    Post subject: Reply with quote

mv wrote:
In general, you can use any of sed, awk, perl (or even python or ruby) for simple modifications of such a type. However, dedicated tools like eix are preferrable if they are suited to your task, because you can expect that they will be adopted to changes of portage. For instance, although it is unfortunate, that just now it turned out that eix has problems with the new portage output, this will be fixed in the next eix release (>=0.28.1 which will be soon in the tree - the fix is already in git): For your own scripts you cannot expect that they will fix themselves automatically when portage changes its output format in a new version...

Edit: Until the new version is available to you, you can use something like
Code:
sudo emerge -pv kscd | sed -n -e 's!^.* \([^ ]*/[^ ]*\).*$!=\1!p'


ok, thank you very much my friend,it is i was looking for :), i am going to learn about sed.
with the today update, the eix way works properly too.

:D
_________________
--Intel i5 3570k --Asrock z77 ext. 4 --Skill 16GB dd3 --Radeon r9 270 --S.O Gentoo & Win 8.1 64bits
Back to top
View user's profile Send private message
paulj
Guru
Guru


Joined: 30 Sep 2004
Posts: 342
Location: Wales, UK

PostPosted: Fri Jan 18, 2013 6:16 am    Post subject: Reply with quote

_______0 wrote:

by the way, I dunno wtf this means:

Code:
       EXPRESSION ::= [ --not | -! ] BRACE_OR_TEST |
                      EXPRESSION [ --and| -a ] EXPRESSION |
                      EXPRESSION [ --or | -o ] EXPRESSION

       BRACE_OR_TEST ::= --open|-( EXPRESSION --close|-) |
                      TEST_WITH_OPTIONS

       TEST_WITH_OPTIONS ::= [TEST_OPTIONS] [PATTERN]


why the hell are brackets, parethesis and pipes in there?? are they usable or what? dis crazy!!


It's a grammar definition. The brackets, parenthesis and pipes are read as follows: the brackets include an either/or part, so
Code:
 [ --not |  -!]
means you can use --not or -!. The pipe symbol is the OR symbol in this case. I would read the EXPRESSION part as follows: Where EXPRESSION can be used, I can also use either --not or -! followed by a BRACE_OR_TEST OR EXPRESSION followed by --and or -a followed by an EXPRESSION, OR an EXPRESSION followed by --or or -o followed by EXPRESSION. It is a of course recursive in nature, so you can nest the terms. For example:
Code:
eix gnome
produces a long list of 143 matches. I want to find packages containing gnome and extras:
Code:
 eix gnome --and extras
This lists three packages. If I didn't want packages containing python:
Code:
eix gnome --and extras --and --not python
As expected based on the previous result, this lists two packages.

I can recommend the Dragon Compiler Book if you want to learn more about grammar and syntax definition!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum