Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Let’s fund pkgcore work
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
ArneBab
Guru
Guru


Joined: 24 Jan 2006
Posts: 424
Location: Graben-Neudorf, Germany

PostPosted: Sun May 07, 2017 4:37 pm    Post subject: Let’s fund pkgcore work Reply with quote

pkgcore is the fastest tool to install a program from the portage tree. On my box
Code:
time pmerge -p pkgcore
finishes within 1.3s, compared to one full minute with emerge.

It’s search is so fast, that
Code:
pquery --raw -nv --attr=keywords
can replace eix without needing a separate database update step.
And it provides a drop-in replacement for emerge, including commandline options. Simply replace emerge with pmerge to get an instant speed-bump.
pkgcore achieves this by doing just the work that’s actually needed for the task. Taken together, that’s conceptual and technical excellence, deeply optimized to the requirements, and I pull my hat to its developerss.

However the PMS addition of subslots moved the target for that optimization. The dependency-space exploded, which pushes pkgcore into a costly path of its optimization when doing a world update. To fix this, it needs a restructuring of its dependency resolution, but that’s a large task which is hard to do in pure hobby time.

Therefore, after a year of prodding from me, radhermit set up a patreon account where we can provide funding. The initial goal is just 25$ per month: To show that there is actual interest in this work. I pledged $3. If just 8 of you join me, we’ll already have reached that goal.

If you want to help fund pkgcore, please join up on https://www.patreon.com/pkgcore
_________________
Being unpolitical means being political without realizing it. - Arne Babenhauserheide ( http://draketo.de )

pkgcore: So fast that it feels unreal - by doing only what is needed.
Back to top
View user's profile Send private message
geki
Advocate
Advocate


Joined: 13 May 2004
Posts: 2319
Location: Germania

PostPosted: Sun May 07, 2017 7:58 pm    Post subject: Reply with quote

Hmm, looks like a worthwhile project to me, since I am too short on time, unlike before. :o

Though, somehow I cannot calculate anymore or I miss something.
The project amount before and after my pledge did not add up right,
although the fees were added on top of my pledge on the bill. :roll:
_________________
boost|select libs to build|slotable|python-buildid

hear hear
Back to top
View user's profile Send private message
ArneBab
Guru
Guru


Joined: 24 Jan 2006
Posts: 424
Location: Graben-Neudorf, Germany

PostPosted: Sun May 07, 2017 9:16 pm    Post subject: Reply with quote

geki wrote:
Hmm, looks like a worthwhile project to me, since I am too short on time, unlike before. :o

Though, somehow I cannot calculate anymore or I miss something.
The project amount before and after my pledge did not add up right,
although the fees were added on top of my pledge on the bill. :roll:

Could that be because I increased my pledge from $3 to $4?
_________________
Being unpolitical means being political without realizing it. - Arne Babenhauserheide ( http://draketo.de )

pkgcore: So fast that it feels unreal - by doing only what is needed.
Back to top
View user's profile Send private message
geki
Advocate
Advocate


Joined: 13 May 2004
Posts: 2319
Location: Germania

PostPosted: Mon May 08, 2017 11:04 am    Post subject: Reply with quote

Well, I do not know. It was at 2 patreons and $6, I pledge $5, now it is at 3 patreons and $10.
_________________
boost|select libs to build|slotable|python-buildid

hear hear
Back to top
View user's profile Send private message
ArneBab
Guru
Guru


Joined: 24 Jan 2006
Posts: 424
Location: Graben-Neudorf, Germany

PostPosted: Mon May 08, 2017 8:01 pm    Post subject: Reply with quote

geki wrote:
Well, I do not know. It was at 2 patreons and $6, I pledge $5, now it is at 3 patreons and $10.

It is now at 4 patrons and $11.

I’m paying $4, you pay $5, so we’re together at $9. The other two might be pledging 1$ each.

@other patrons: If you are here, could you give your pledge amount?
_________________
Being unpolitical means being political without realizing it. - Arne Babenhauserheide ( http://draketo.de )

pkgcore: So fast that it feels unreal - by doing only what is needed.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10156
Location: Somewhere over Atlanta, Georgia

PostPosted: Mon May 08, 2017 8:10 pm    Post subject: Reply with quote

Not saying how I know, but someone is mistaken. ;)

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
geki
Advocate
Advocate


Joined: 13 May 2004
Posts: 2319
Location: Germania

PostPosted: Mon May 08, 2017 9:04 pm    Post subject: Reply with quote

Well, as long as it is just alright as is, that's fine. I just wonder. :o
_________________
boost|select libs to build|slotable|python-buildid

hear hear
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6875

PostPosted: Mon May 08, 2017 9:16 pm    Post subject: Reply with quote

going faster, but doing less, you don't need any good algorithm to achieve this :)
are you sure adding subslots and missing features would still make the dependencies resolution faster? Could be why author drops dev of pkgcore
Back to top
View user's profile Send private message
ArneBab
Guru
Guru


Joined: 24 Jan 2006
Posts: 424
Location: Graben-Neudorf, Germany

PostPosted: Fri May 12, 2017 5:57 pm    Post subject: Reply with quote

krinn wrote:
going faster, but doing less, you don't need any good algorithm to achieve this :)
are you sure adding subslots and missing features would still make the dependencies resolution faster? Could be why author drops dev of pkgcore

Yes, I am sure: subslots are already supported.

pkgcore was already much faster than portage when there were no subslots. It is still much faster, and it supports most of the latest features (i.e. EAPI 6: development is still ongoing, but in small steps due to limited free time of the devs), but a world update (pmerge -uDN world) requires too much memory.

To fix that high memory consumption on a world update, the core datastructures need a redesign. And that’s the task for which the core dev needs a larger slab of time.
_________________
Being unpolitical means being political without realizing it. - Arne Babenhauserheide ( http://draketo.de )

pkgcore: So fast that it feels unreal - by doing only what is needed.
Back to top
View user's profile Send private message
NerdIII
n00b
n00b


Joined: 10 Oct 2010
Posts: 10

PostPosted: Sun May 14, 2017 11:57 am    Post subject: Reply with quote

time emerge -p pkgcore: 5.0s
time pmerge -p pkgcore: 0.5s

time emerge -puDN @world: 111s
time pmerge -puDN @world: 6.4s

Yes, that's awesome. It just does not find any changed USE flags or downgraded packages at this time. :P
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6875

PostPosted: Sun May 14, 2017 2:59 pm    Post subject: Reply with quote

Quiet impressive, this speak better NerdIII, i might gives it a try.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5327

PostPosted: Sun May 14, 2017 5:13 pm    Post subject: Reply with quote

Maybe I'll finally give this a try. I could tolerate emerge's slowness but only because I'd been conditioned to accepting it was fast after using paludis for far too long (not gonna miss those half-hour dependency resolving times...)
Back to top
View user's profile Send private message
NerdIII
n00b
n00b


Joined: 10 Oct 2010
Posts: 10

PostPosted: Mon May 15, 2017 9:51 am    Post subject: Reply with quote

I said it wasn't coming up with the correct package list to merge. I was using the live ebuild and the merge list was just empty, missing some packages with changed use flags and some downgrades. So all I know is that it is fast at doing nothing at the moment. :lol:
It could be my config files. They have grown messy over the years with lots of entries in package.accept_keywords, package.use and package.env.

EDIT:

I tried the live ebuild again today and it found a package to upgrade with
Code:
pmerge -uDNav @world
:

sys-boot/refind-0.10.7-r1::gentoo [0.10.7::gentoo]

This time I had no changed USE flags, but the output is missing downgrading packages again (feature or bug?). The times for my actual system upgrade emerge command look less fishy now:

emerge -uDNav @world: 143s
pmerge -uDNav @world: 40s
Back to top
View user's profile Send private message
tranquilcool
Veteran
Veteran


Joined: 25 Mar 2005
Posts: 1149

PostPosted: Mon May 15, 2017 4:31 pm    Post subject: Reply with quote

it never worked for me.

pmerge pkgcore

ERROR: sys-apps/pkgcore-0.9.4 failed (unpack phase):
* S was null- , path=/usr/lib/pkgcore/ebd/helpers/5:/usr/lib/pkgcore/ebd/helpers/4:/usr/lib/pkgcore/ebd/helpers/common:/sbin:/bin:/usr/sbin:/usr/bin
_________________
this is a strange strange world.
Back to top
View user's profile Send private message
ArneBab
Guru
Guru


Joined: 24 Jan 2006
Posts: 424
Location: Graben-Neudorf, Germany

PostPosted: Mon May 15, 2017 9:12 pm    Post subject: Reply with quote

If you want to help debug that, please join in IRC¹ or report the bug on github: https://github.com/pkgcore/pkgcore/issues

¹: https://webchat.freenode.net/?randomnick=1&channels=pkgcore
_________________
Being unpolitical means being political without realizing it. - Arne Babenhauserheide ( http://draketo.de )

pkgcore: So fast that it feels unreal - by doing only what is needed.
Back to top
View user's profile Send private message
ArneBab
Guru
Guru


Joined: 24 Jan 2006
Posts: 424
Location: Graben-Neudorf, Germany

PostPosted: Sun May 21, 2017 1:29 pm    Post subject: Reply with quote

tranquilcool wrote:
it never worked for me.

pmerge pkgcore

ERROR: sys-apps/pkgcore-0.9.4 failed (unpack phase):
* S was null- , path=/usr/lib/pkgcore/ebd/helpers/5:/usr/lib/pkgcore/ebd/helpers/4:/usr/lib/pkgcore/ebd/helpers/common:/sbin:/bin:/usr/sbin:/usr/bin

You’ll want to use the
Code:
=sys-apps/pkgcore-9999
at the moment.
_________________
Being unpolitical means being political without realizing it. - Arne Babenhauserheide ( http://draketo.de )

pkgcore: So fast that it feels unreal - by doing only what is needed.
Back to top
View user's profile Send private message
tranquilcool
Veteran
Veteran


Joined: 25 Mar 2005
Posts: 1149

PostPosted: Mon May 22, 2017 4:13 am    Post subject: Reply with quote

ArneBab wrote:
tranquilcool wrote:
it never worked for me.

pmerge pkgcore

ERROR: sys-apps/pkgcore-0.9.4 failed (unpack phase):
* S was null- , path=/usr/lib/pkgcore/ebd/helpers/5:/usr/lib/pkgcore/ebd/helpers/4:/usr/lib/pkgcore/ebd/helpers/common:/sbin:/bin:/usr/sbin:/usr/bin

You’ll want to use the
Code:
=sys-apps/pkgcore-9999
at the moment.


yes, did use pkgcore-9999 and it worked. thanks.
_________________
this is a strange strange world.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10156
Location: Somewhere over Atlanta, Georgia

PostPosted: Wed Nov 08, 2017 1:15 pm    Post subject: Reply with quote

Been funding through Patreon now for several months. As of yet, though, the required funding goal for the developer to continue work hasn't been reached.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Fri Dec 08, 2017 11:45 am    Post subject: Reply with quote

pgkcore is user-friendly and impressive. It is great: we can avoid resorting to the plethora of "complementary" tools related at Portage!
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5327

PostPosted: Fri Dec 08, 2017 6:01 pm    Post subject: Reply with quote

If that's all you care about, you can use paludis instead.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10156
Location: Somewhere over Atlanta, Georgia

PostPosted: Fri Dec 08, 2017 7:26 pm    Post subject: Reply with quote

Also, isn't pkgcore one of the, "... plethora of "complementary" tools related at Portage?"

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Sun Dec 10, 2017 8:15 am    Post subject: Reply with quote

Should we modify the system to only integrate pkgcore? I do not know. pkgcore is qualified as a framework manager, it has its own configuration but uses the ebuilds repository.

I want to be able to study the package manager to answer some questions and solve problems. The design of Portage is undocumented (at least for the user), the technical
documentation is partial (scattered, not up to date) and the code may be of poor quality (obscure parts, lines of code not readable, complex algorithms...).

At the point where I am, I do not exclude to try several solutions.
Back to top
View user's profile Send private message
The_Document
Apprentice
Apprentice


Joined: 03 Feb 2018
Posts: 244

PostPosted: Sun Feb 25, 2018 6:42 am    Post subject: Reply with quote

helecho wrote:
Should we modify the system to only integrate pkgcore? I do not know. pkgcore is qualified as a framework manager, it has its own configuration but uses the ebuilds repository.

I want to be able to study the package manager to answer some questions and solve problems. The design of Portage is undocumented (at least for the user), the technical
documentation is partial (scattered, not up to date) and the code may be of poor quality (obscure parts, lines of code not readable, complex algorithms...).

At the point where I am, I do not exclude to try several solutions.


This means portage is not well coded, to say the least it had plenty of time to increase its performance and it hardly did. I tried pkgcore and its incredibly fast compared to portage however it doesn't honor useflag changes when doing updates, I would totally use if I found out how to do a portage equivalent of --jobs which I am sure pkgcore doesn't support.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum