PFL looking for new owner!

[tuxmainy]

Hi,
as the subject says: I, the founder and current owner of PFL (https://portagefilelist.de/), am looking for a new owner. You may have noticed the recent lack of maintenance and support of PFL. The reasons are mainly private and shouldn't be discussed here.

So here I am after ~15 years asking for anyone who want's to take over PFL. This includes the domain as well. The software can be found in github (https://github.com/portagefilelist). The website code is missing because it contains code which is not licensed by me. So I guess the web site has to be rewritten. And to be honest the website really needs a refresh

So from my point of view your job would be:
1. install all the server side stuff on your / a server
2. create a new website for doing queries (remember the API used by e-file)
3. take over the domain
4. have fun with maintaining the data and PFL client side stuff

Anyone interested? Otherwise I have to shutdown PFL. Sorry

regards
Daniel

PS: May an admin pin this post to the top? I think this is an important request.

[pjp]

I'm sorry to hear you have to move on from the project (it sounds like maybe you would have preferred to not).

I've made the post an Announcement, so it will remain at the top until there is a hand off or you've had to shut it down.


Could you be more specific about the work involved? How is the list of files generated? What would that look like for a person hosting this, or paying for compute time somewhere?

You mention a new front end would be needed. Is the old front end not transferable to the new owner until they are able to create a new one?

Not knowing what is involved, I'd personally like to see something like that provided by Gentoo.

[tuxmainy]

I've made the post an Announcement, so it will remain at the top until there is a hand off or you've had to shut it down.

Thanks

Could you be more specific about the work involved?

I try to. Please everyone don't hesitate to ask further questions. As I have come up with most of the stuff I don't understand which parts need more explanations.

How is the list of files generated?

The files are collected by all the gentoo users who have PFL emerged. The package adds a python script which is periodically called by cron. This script collects all new packages since last run, packs them into XML and sends them to portagefilelist.de. On the server runs a second script periodically (every hour), unpacks the XML and imports it into a postgres database. Using another database should be fairly easy but please keep in mind that we are talking about lot of data. I have switched between multiple databases (also tried ldap at some point). But anyway, the website simply requests the database.

What would that look like for a person hosting this, or paying for compute time somewhere?

You need to host it by your own. I have rented a dedicated root server from a common provider. So yes, payment for resources would be involved. If someone is more familiar with cloud computing this might be an option. But as I don't have enough knowledge about cloud computing I cannot decide on this. Of course I will provide the current database as a SQL dump.

You mention a new front end would be needed. Is the old front end not transferable to the new owner until they are able to create a new one?

The code of the old frontend is not transferable at all because of license reasons. If you mean "switching over the domain" by "transferable" you have to keep in mind that the domain serves two purposes:
1. collecting data from gentoo users
2. making the data "queryable"
So once you get the domain you should be ready to get the data from point 1. Not providing point 2 will result in some unhappy users but might be ok for one week or so.

Hope this helps.

regards
Daniel

[pjp]

Thanks for clarifying. Hopefully that will be useful information for someone. It's a bit beyond my comfort level and budget (0), otherwise I would consider it.

The code of the old frontend is not transferable at all because of license reasons.

That is what I was wondering as it seems you were specifically indicating it was not available.

[flexibeast]

@tuxmainy,

i just want to say thanks for maintaining PFL, and for so long - i regularly make use of it. Hope someone is able to take it over soon!

[eccerr0r]

Curious of the bandwidth required to maintain the service?

Though I love running services on my computer, I have neither the network bandwidth or the funds to maintain the DNS name after it expires, sorry :(

[spica]

What are the monthly costs for servers?
How much disk storage needed for database?
If someone can start replicating web ui now, how much time he has?

[NeddySeagoon]

spica,

Server Auction is one place to look for a whole server. I have one of those.
It hosts my Raspberry Pi arm64 binhost among other things.

Maybe you don't need a whole server and a VM or web hosting would work?

[spica]

NeddySeagoon,
No, no. I want to know the current spendings of the project, how much it costs now. If I know current numbers then I can compare with my infrastructure and then I can understand can I give a shelter or not.
It's a question about requirements for the service
I guess a $5 t3.micro should be enough

[ARomaSH]

Anyone interested? Otherwise I have to shutdown PFL. Sorry

PM-ed

[tuxmainy]

Generally on the hardware: I experienced most trouble with CPU and RAM while inserting new records because of index regeneration and users doing wildcard queries which couldn't be handled be the index (e.g. *foo). The database should have a fair amount of RAM available to handle the indexes. Disk I/O might be a problem but using standard SSDs should be fine.

Using another DBMS which is build for the amount of data might resolve some performance problems (like using NO-SQL?). But my database knowledge is limited to SQL. So...

What are the monthly costs for servers?

Hard to say. It's a single dedicated root server which runs multiple services (not just PFL). So I never cared about the PFL specific costs. But to give you an idea:

• Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
• 64GB RAM (~16GB is used by the database)
• 2 4TB SSDs in RAID 1

Server is hosted by a common german hoster (hetzner) and currently costs ~50€/month. But as said there are running multiple services so the resources are not exclusive for running PFL! The domain is a .de domain (no, I also don't understand why I registered a .de domain ). So you should check your local domain dealer for prices.

How much disk storage needed for database?

Current size of PFL database on disk is 235GB.

Curious of the bandwidth required to maintain the service?

~3GB of uploads per month. I don't have numbers for the website itself but I guess that's not that much. Said hoster grants unlimited traffic so I never cared about this.

If someone can start replicating web ui now, how much time he has?

I'd like to hand over until 1st of July. If there is a plan which takes more time it depends on the plan if I am fine with

[szatox]

Current size of PFL database on disk is 235GB.

Wow, that's a lot more than I thought it would be.
Is it the actual volume of data, or perhaps there were a lot of updates/deletions over the years and it just requires vacuum full? AFAIR postgres does not delete dead records unless explicitly ordered to.

[Banana]

Well I'm from germany, I do have a host and I do know PHP, but the DB size would be a pain.

Are there are historical data in it which could be "archived"?

[tuxmainy]

Well I'm from germany, I do have a host and I do know PHP, but the DB size would be a pain.

Are there are historical data in it which could be "archived"?

Yes, I think so. As said, it's not well maintained currently, archiving included. But I have no clue how much data could be archived. Sorry

[pjp]

It may help to post an announcement on the PFL website itself.

Maybe even an announcement / update to the cli tool? I'm not sure how feasible that is.

[Banana]

I will have a look. Currently I see the website which is able to do a search and display the results, python on the client side which creates a xml and uploads, python on the client which can query, php on the server side to accept the uploaded xml and another php file to import those files.

My idea would be to create a fork and some new server stuff and go from there. I will respond here if my research/experiment is working or not.

[geki]

..., I'd personally like to see something like that provided by Gentoo.

Using another DBMS which is build for the amount of data might resolve some performance problems (like using NO-SQL?).

I had a look at how I would implement a portage file list. I had a look at using SQL and found a Closure-Table[0] to manage hierarchy. Seems to work fairly good for a simple approach. But the more I looked around, the more I got to the point that querying file/path metadata is no good fit for a SQL DB. I may be wrong, though. Do you know better?

Instead I looked at what is there already. There is the Portage Database already, full of metadata for each installed package on the user system located at /var/db/pkg. Either parse the data by cli tools and send a subset of its metadata as compressed tarball of updated packages or use python's portage API as e-file is doing it already. On the server manage a modified, stripped Portage Database similar to /var/db/pkg within a tmpfs. Then, I remembered about locate and updatedb. It is been doing file search since decades and came to this thread of stack exchange[1] with a very interesting comment of just using grep with a big text file consisting of all files, which shall be way faster. Once I get my gentoo box running again, I may play around with this approach and see, if I get anything scalable resource- and WebUI performance-wise. I have no idea, but fun to see, where I may end up with.

[0] https://dirtsimple.org/2010/11/simplest ... eries.html
[1] https://unix.stackexchange.com/question ... 254_379729

[pjp]

...

I hadn't considered an implementation, only that with devs building things or through CI, it shouldn't be difficult to add something that tracked which files were installed by a given program. No need for a client tool to submit user data.

[Hu]

I believe one of the motivations for user submissions was the prevalence of packages where USE=foo controls whether /usr/bin/foo is built at all. For such packages, any one build may not be representative of all the files the package can generate, so the database needs a sampling of different build configurations. Some packages have many USE flags, so building all possible combinations is infeasible (and not all combinations will actually vary the file list). Arguably, an extension on top of the USE system where the ebuild tracks which flags enable/disable building particular files could help there, but presently Gentoo maintainers are not asked to compile that list in any sort of machine-readable form.

[geki]

It gets all the more fun when we collect files not only from portage tree packages, but also from user overlay packages. Anyway, it's just something I spend a little of my spare time doing for fun. See, if my variant of NoSQL approach gets somewhere. But I think we better wait for what the other(s) come up with.

[pjp]

I believe one of the motivations for user submissions was the prevalence of packages where USE=foo controls whether /usr/bin/foo is built at all. For such packages, any one build may not be representative of all the files the package can generate, so the database needs a sampling of different build configurations. Some packages have many USE flags, so building all possible combinations is infeasible (and not all combinations will actually vary the file list). Arguably, an extension on top of the USE system where the ebuild tracks which flags enable/disable building particular files could help there, but presently Gentoo maintainers are not asked to compile that list in any sort of machine-readable form.

I presumed automation, not manual enumeration. I suppose lacking an upstream manifest, whatever upstream build options could be the expected "creates these files." But I'm not a dev, so it would seem none have thought it worth their time..

[tuxmainy]

Hi,
the current dump of the SQL database can be found at https://portagefilelist.de/pfl-files.sql.gz

PFL is used for 3 different questions:

• Which file is own by which package? Sometimes it's not easy to see even if you don't have to enable a special useflag
• Which file is build by which useflag and architecture? Use case obvoius, I hope
• How did the files changed over time? Sometimes you lose a file after an update and it would be nice to see where it is gone.

Point 2 is exactly where you need a huge user base for collecting because covering all useflags and architectrues won't be possible for a singe person, eh?

The current pfl collect script ignores user repositories and I really recommend to not collect user repositories data. This data is not helpfully as it doesn't help someone to know that the file they are looking for is in a package which they cannot emerge. Also this doesn't seem to be a good idea from a data privacy point of view. Imagine you have written a tool called "mirror my favorite p*rnh*b videos" which then gets collected (not the tool itself but the name) and published on a website. Or even more sensitive data like usernames or hashs which are coded into a filenames.
And to be honest: It happened in an early version of PFL (not intended, just never thought about). So I have seen data from user repositoies. I promise to you, this data is mostly garbage at all

Have a look into the current collect script. I think it's easy to understand (at least I hope so )


regards
Daniel

[Banana]

I've currently "rewrote" some of the stuff to get the idea how each part works. I'll have a working example in the next couple of days.

[geki]

Hi,
the current dump of the SQL database can be found at https://portagefilelist.de/pfl-files.sql.gz

Hi tuxmainy, would you mind to share a tarball with only the db and db tables schemas dump to get a quick look on layout and such. I would not need the content for my tinkering. Then, I would not need to download that much data, a~hem. ... Just to make sure I did not miss something...

[Banana]

I've got a working example (first draft) over here: https://delta-labs.org/portagefilelist/

What it does: you can search for a filename based on my local data collection. Here are some names you can search for: hugo, apache, kernel, mupdf, vim

What it does NOT (yet): show packages, versions. Data endpoint is disabled right now.
I've just implementend the import and frontend. Clientside will be untouched for now, since I do not see any problems with it so far.

@tuxmainy
How do you plan any further steps? Do you want a complete example before switching over?

@geki
If you have also ideas or examples, or whatever, we can join forces.