Portage website - gentoo-portage.com

Retriever II · Posted: Thu Oct 09, 2003 7:05 pm Post subject:

It's quicker than emerge -s package

thrasher6670 · Posted: Thu Oct 09, 2003 7:15 pm Post subject:

Retriever II · Posted: Thu Oct 09, 2003 7:53 pm Post subject:

I think I grew a beard by the time I finished running emerge with -S :lol:

Haro · n00b Joined: 17 May 2003 Posts: 71 Location: West Bend, WI

thrasher6670, still need web design work?

I'll be in windows the next couple of days unfortunately so let me know and i'll whip some pages up.

Sorry if someone else is already doing this, I don't have time to sift through 5 pages while at work

Haro

carambola5 · Apprentice Joined: 10 Jul 2002 Posts: 214

Wrote up a huge enhancements list on my laptop. Next time I boot that machine I'll post it. Which should be very very soon.

thrasher6670 · Posted: Fri Oct 10, 2003 5:04 am Post subject:

thrasher6670 · Posted: Fri Oct 10, 2003 5:10 am Post subject:

also, i beefed up the search a little, it will now display exact matchs first. figured out how to do that

. also there is an option on the search page to search for an exact package, this will bring you to the package directly. if you know that package name that is.

enjoy the search upgrade. more to come later.
_________________
Gentoo-Portage.com - Where the portage meets the web
Gentoo Wiki

carambola5 · Apprentice Joined: 10 Jul 2002 Posts: 214

Suggestions for the future of the Portage Website

Preface
About this post: I was riding the bus on the way home one day when I started inadvertantly brainstorming on the Portage Website. So, I sat there for a few minutes working things out in my head. Eventually, I got to the point where I realized that if I didn't write this stuff down soon, it would be gone. So, I cracked out my laptop and started writing. So, without further ado...

Disclaimer: These are just ideas for enhancements. If implemented, I seriously think they will make the Portage Website such an invaluable tool that it could beat out the "Big Guys." I refrain from defining "Big Guys" because that will be different for everyone. Just think of it as usurping whoever is your biggest pain in terms of OSs. And no, I am not trying to implicate one company... there are plenty of companies out there in the OS game. The ease of administration resulting from the inclusion of these ideas should trump anything that's out there. Also, I am not necessarily volunteering myself to implement these changes. God knows I have plenty of other stuff to work on (ie: Senior year in Engineering and CompSci).

Table of Contents:

Security
Execution of emerge statements on the server
Queueing/Scheduling of statements
Predictive state
Remote administration basics
Remote Gentoo installation

Security:

Let's face it, we live in a world of hackers. Well, that's what the media wants us to believe. And I despise that use of the word "hackers." *shudder* Anyways... If any of this higher-level stuff is even considered, security is a must. This means SSL, SSH, and liberal applications of addslashes. Obviously SSL and SSH are not ours to implement, but they, or equivalent technologies, must be used at all times. I don't care if you only administer your box from inside a NAT'ed, firewalled LAN, if any access point in your network is compromised and exploited, internal LAN traffic is no longer safe. And I wouldn't want my root password being sent in cleartext over a network being watched by a hostile user. If you don't think I'm being reasonable, think of a company LAN that is securely firewalled with no incoming ports open. Secure right? Only until the first employee brings in a wireless access point so that they can work on their laptop sans wires. What am I getting at? Cryptography should be used... nay, required for the Portage Website (henceforth, as "PW"). Each time a user with a cookie indicating "I'm an admin" wants a page, the code will check for https. If not present, dump a blank page.

In the upcoming sections, I will consider remote command execution. In other words, there are potentially three computers involved: the website user ("webclient"), the server ("server"), and a client computer on which the command will be executed ("remote client"). I think it goes without saying that SSH (or equivalent) should be used between the server and the remote client.

With respect to sending shell commands, I believe that the entire command should be built with predefined strings. Nothing that the user inputs into a form should be used directly to aid in the construction of a command. No matter how safe you think you can make user input, something will always go wrong. Murphy's Law.

Execution of emerge statements on the server:

Simply put, the PW, should be able to send an internally generated command to the shell, parse the response, and output to the next page. Seeing as most of the searching and retrieving has already been implemented in the DB backend to the site, I don't really see the need to implement 'emerge -s', 'emerge -S', 'emerge -p', or any qpkg command. That essentially leaves '[optional variables] emerge <package-name>' and 'emerge sync'. In accordance with the last point made in the Security section, these functions are (more or less) easily implemented.

When dealing with Gentoo, as opposed to systems such as Debian, the duration of a command is usually quite lengthy. Not only will the HTTP Request likely get timed out, but having the user wait with a blank screen is something that should be avoided. Therefore, I propose that 'screen' be a PW dependency. I am not entirely familiar with all of the nuances of screen, but I have used it. And I really think this is the way to go. And since no two emerge statements should ever be executing simultaneously, only the basic functions of screen are needed. And for those of us who like to sit in front of scrolling compile messages, I think that could be implemented with a refreshing webpage.

Queueing/Scheduling of statements

One of my favorite tips that I've learned is the 'Ctrl-Z; fg && emerge' trick. It's queueing in it's most basic sense. Obviously, this can be expanded upon in a GUI setting, such as a website. My ideal method of implementing this would be to use XML as the job description language and have page requests in conjunction with a cron job to execute the queue. If that sounds complicated, let me explain:

First let's look at why I choose XML over, say, a DB entry: Both methods have mature handling routines built into PHP. But XML's huge benefit comes when I discuss administration of non-server computers. When a job needs to be run on a remote machine, the entire job description is contained within an XML element which is encapsulated within a file and sent over the network. If the job was stored in the server's DB, it would need to be serialized, sent, deserialized, and saved. And what if the client computer doesn't have a database engine? I sure wouldn't want my firewall running MySQL. If the client computer needs the job description in non-DB format, you might as well start with a non-DB format. An added benefit of using XML is its plaintext nature. No messing around with binary files.

OK, so we have the job defined in an XML file. How do we execute it? My idea is that an external program (assume it is /usr/sbin/raboof, for example) would be called every time an "immediate job" is submitted (ie: you want to emerge foo now, rather than at 1am when no one is around). The job would be saved to an "immediate job" file, and raboof would be called with a switch telling it to scan the "immediate job" file. Otherwise, if a scheduled job is submitted, that job is saved to a file, and a cron job is added that simply calls the raboof at the desired time. raboof would simply scan the "scheduled jobs" file for jobs which need to be run.

Viewing the pending jobs on PW would be as easy as writing an XSL file.

Predictive State

This means that before actually emerging any group of packages, the --pretend output would be parsed to check for blocking errors. IIRC, xft at some point blocked xfree (right around 4.3.0 came out). If the user submitted a batch job containing both xft and xfree, the website should inform the user of the conflict. Not much more to say here.

Remote Administration Basics

This is it. The coup de grace. This will set Gentoo coupled with PW over any competing administration software/OS combo. The idea is simple. Execute statements over SSH inside a screen virtual terminal (on the server). Notice how I didn't say 'emerge statments.' We're talking any statement over SSH. Which, of course, paves the way to the last section, Remote Gentoo Installation, which covers full Gentoo installs.

First of all, when you have multiple computers, you might as well use distcc. Though distcc is completely transparent to PW, it should be noted on the site that distcc will greatly improve the user experience. In addition, buildpkg should be intelligently and transparently used. This should significantly reduce compile times for groups of similar remote clients.

Organization of remote clients: Client computers should be organized in a recursively linked structure. Essentially, all computers should co-exist in a flat, sorted structure, then linked to an appropriate position in the relationship tree. For the following example, all filenames, folders, virtual folders, etc should be customizable and are defined in the example only for clarification.

Example: You have 5 remote clients A, B, C, D, and E. In addition, you have a server S. Finally, for completion's sake, the webclient is W, which can be defined as either A, B, C, D, E, or another computer connected via SSL. It can also be S run on a non-SSL connection to localhost. All 6 affected computers (ie: not W) should exist in a flat structure called 'computers.' Outside the 'computers' structure, will exist a separate virtual folder 'Gentoo Herd.' Inside 'Gentoo Herd' can exist any number of subfolders. In this example, we will have 'servers' and 'clients.' Inside 'servers' can exist any folders or links. In this example, there should be a single link to S. For computational robustness, the link should be doubly linked. Next, inside 'clients' can, once again, exist any number of folders or links. For this example, we will have two folders: 'i686' and 'i486.' This example of sorting by architecture is purely a decision by the user. It could just as well be 'east wing' and 'west wing.' Let's say that A and B are linked in 'i686' and C is linked in 'i486.' Also existing in 'i486' will be a virtual folder, 'Bob's LAN.' Within 'Bob's LAN,' are links to D and E (note, D and E are in /Gentoo Herd/i486/Bob's Lan). While my example is probably not the best way to do things, the ability to link computers to multiple locations can make for interesting possibilities:

carambola5 · Apprentice Joined: 10 Jul 2002 Posts: 214

p\/\/n3d!

thrasher6670 · Posted: Fri Oct 10, 2003 3:00 pm Post subject:

god damn, um yeah i'll read that more fully after work.... damn.

p.s. registered gentoo-portage.com should have it up in 48 hours or something like that
_________________
Gentoo-Portage.com - Where the portage meets the web
Gentoo Wiki

thrasher6670 · Posted: Sat Oct 11, 2003 3:28 pm Post subject:

gentoo-portage.com is now working Yay!

i have added the ability to use [indent] and [space] to your comments
and the news is now dinamic.

carambola5 --- finally sat down and read your essay, alot of it sounds good, and it is mostly doable, eventually. i copied that document and i'll go over a few more times when i reach a point when i can start implementing some of that.
_________________
Gentoo-Portage.com - Where the portage meets the web
Gentoo Wiki

Retriever II · Posted: Sat Oct 11, 2003 4:29 pm Post subject:

bug with [indent] - it shows an extra incomplete

thrasher6670 · Posted: Sat Oct 11, 2003 10:58 pm Post subject:

ok im looking for a little help here...

this is the test version of gentoo-portage http://gentoo-portage.com/projects/gentoo-portage-test/index.php

i am currently working on migrating over the design. im having one small problem thou, the bars on the left and right can spill over if they are too long. does anyone know how i can fix this?

***** EDIT ******
fixed my lil' problem
_________________
Gentoo-Portage.com - Where the portage meets the web
Gentoo Wiki

thrasher6670 · Posted: Sun Oct 12, 2003 5:28 am Post subject:

NEW DESIGN

Just finished the new website look, i would like to thank Chris Norris of ToadWeb for the design. he will be doing the design for the site from here on in.

I have also implemented the ability to track packages.

If you notice any problems with the new design, could you please email me with the page and browser. thank you. enjoy the new design.

www.gentoo-portage.com
_________________
Gentoo-Portage.com - Where the portage meets the web
Gentoo Wiki

Vishruth · Posted: Sun Oct 12, 2003 7:27 am Post subject:

The new design on gentoo-portage rocks! Way to go!

Though, you could improve your grammar on the main page.

_________________
http://freeshell.in/~vish/

thrasher6670 · Posted: Sun Oct 12, 2003 11:47 am Post subject:

OdinsDream · Veteran Joined: 01 Jun 2002 Posts: 1057

I'd like to suggest something pretty simple, probably just nit-picking...

When providing any links that go outside gentoo-portage.com, it looks like you're targetting a new window. I'm sure there's a reason behind this, but, from my perspective, it's just tiresome, as I regularly middle-click to open windows in a separate tab, if I so desire.

My philosophy behind this being, if the user wanted a new window, they would have handled the link themselves to create the window, as they saw fit. Of course, there are plenty of reasons to target new windows... things like micro-sites as part of your site, or advertisements...

Just something I noticed.

The site is looking amazing. I'm fully behind your design, it really is very well done. I especially like that it's image-less for the most part (totally? not sure...) It loads quick, gets to the point, and it's developing at an incredible rate. Superb job.
_________________
s/(?<!gnu\/)linux(?! kernel)/GNU\/Linux/gi

Don't blame me. I didn't vote for him.

http://john.simplykiwi.com

thrasher6670 · Posted: Sun Oct 12, 2003 3:06 pm Post subject:

NoUseForAName · Posted: Sun Oct 12, 2003 7:20 pm Post subject:

An rss feed maybe?

thrasher6670 · Posted: Tue Oct 14, 2003 1:05 am Post subject:

Today's updates:

RSS feed

Thanks to:

http://www.oopsilon.com

http://www.sysrage.net

If you have any questions or comments regarding the changes, please post something about it, i can always use more comments and suggestions
_________________
Gentoo-Portage.com - Where the portage meets the web
Gentoo Wiki

carambola5 · Apprentice Joined: 10 Jul 2002 Posts: 214

implement my stuff yet? :twisted:

Xiderpunk · Posted: Thu Oct 16, 2003 11:30 pm Post subject:

Thanks for the design comments, the idea was just to keep things simple and fast by using css layout and creative use of colour instead of images, so hopefully we have pulled it off.

All credit though for this project goes to Thrasher, his ideas. :-)

Some people have reported some display bugs with the layout, I will aim to address these over the coming weekend. One issue present is minimum width requirement on the "browse portage" page, to allow the data table to display. This unfortunately can't be fixed too easily, without reducing the number of columns.

Alternative stylesheets are also forthcoming this weekend (hopefully!) for those that prefer a different look/style for the site.
_________________
Gentoo-Portage - the only way to view/search/explore portage

Kernel: 2.6.7--rc1-mm1
gcc: 3.3.2
desktop: kde-3.2.2 / XFCE4

thrasher6670 · Posted: Fri Oct 17, 2003 12:36 am Post subject:

Today's a day of landmarks in the world of gentoo-portage. One hundred people are currently registered for gentoo-portage.com! Congratulations to magxe for being the hundredth user. We now have a completed FAQ thanks to James Gray and over fifty comments posted.

also:
i am thinking about starting gentoo-update in the next few weeks, but i am going to need some help.... if anyone is interested, contact me. i'll definitly need someone who can code a browser plugin.

gentoo-update will be something similar to windows update, but obviosly better.
_________________
Gentoo-Portage.com - Where the portage meets the web
Gentoo Wiki

thrasher6670 · Posted: Sat Oct 18, 2003 4:26 am Post subject:

ok ive started on a dependancy tree tonight, but i could use some help in parsing the dependancies.

thrasher6670 · Posted: Sat Oct 18, 2003 10:29 pm Post subject:

Dependency lists are now available for all ebuilds. If now dependencies show for a build, there are none. The current dependency lists are preliminary only, I will have something better looking soon.

The source will be available either later tonight or tomorrow night. for those that have been waiting.
_________________
Gentoo-Portage.com - Where the portage meets the web
Gentoo Wiki