Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Portage emerge binaries and source
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Duplicate Threads
View previous topic :: View next topic  
Author Message
dsims
n00b
n00b


Joined: 08 Aug 2008
Posts: 6

PostPosted: Tue Aug 12, 2008 6:07 pm    Post subject: Portage emerge binaries and source Reply with quote

Was thinking as I always seem to do. IF you have a user base that compiles source would it be an idea to scan the source packages being emerged in a users computer and scan it for warnings, errors and which use flags were enabled, ask the user if he would like to contribute their binary form ( if no errors or warnings were reported ) of the program to a binary package tree separated by different architectures and certain use flags. Of course virus checked and root kit checked. The person on the computer looking for a new binary would then have the computer architecture and drivers of their device to make there use flags show which binary package would be good for them. Of course this tree would be huge with the same programs compiled for different uses and architectures, but the fear of binary failure would be diminished depending on the idea base of the user group. Just a random thought , something I will probably have to do for myself.
Back to top
View user's profile Send private message
mikegpitt
Advocate
Advocate


Joined: 22 May 2004
Posts: 3224

PostPosted: Tue Aug 12, 2008 8:04 pm    Post subject: Reply with quote

It's been proposed many times and also exists. Look into portage BINHOSTS. The only issue with the method you are proposing, is that there are many variations of USE flags, which would mean that there would need to exist many binaries of the same packages... and it would all need to be maintained.

Personally I use a custom script with emerge -K functionality to update a bunch of machines I need to manage that are all in the same reasonable configuration.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9507
Location: beyond the rim

PostPosted: Tue Aug 12, 2008 8:59 pm    Post subject: Re: Portage emerge binaries and source Reply with quote

dsims wrote:
Of course virus checked and root kit checked.

You're too much thinking in Windows terms. The problem with user-contributed binaries is that anybody could inject a few lines of code into the source of any package to create a backdoor, that won't be detected at all by virus- or rootkit-scanners (those deal with separate binaries and/or binary modifications, not source-modifications, and only detect known malware). There is unfortunately no (automated) way to check for such things. Something that's a backdoor in one program can be completely legit in another, assuming you can even detect such things, which itself is next to impossible.
If you even remotely care about security you only install binaries provided by people you trust. And unfortunately you cannot trust random people on the internet ...
Back to top
View user's profile Send private message
mikegpitt
Advocate
Advocate


Joined: 22 May 2004
Posts: 3224

PostPosted: Tue Aug 12, 2008 9:19 pm    Post subject: Reply with quote

One way to establish trust is to compare the binaries to others in a community... if they all have the same checksum, then they are probably legit. Of course this could also be circumvented by creating a number of false users who are all serving the backdoored binaries.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9507
Location: beyond the rim

PostPosted: Tue Aug 12, 2008 9:56 pm    Post subject: Reply with quote

mikegpitt wrote:
One way to establish trust is to compare the binaries to others in a community... if they all have the same checksum, then they are probably legit. Of course this could also be circumvented by creating a number of false users who are all serving the backdoored binaries.

And doesn't work all that well as binaries built on two different systems will rarely have the same checksum. Some binaries even include the system name of the build host, or timestamps of the build (like the Linux kernel, though that's a bad example in this context). Also note that binaries are not only affected by configure options, CFLAGS or gcc versions, there is a huge number of variables affecting the final result, many of them not affecting functionality and not tracked by the package manager.
Back to top
View user's profile Send private message
dsims
n00b
n00b


Joined: 08 Aug 2008
Posts: 6

PostPosted: Wed Aug 13, 2008 12:36 pm    Post subject: Reply with quote

so you mean to tell me that there is no way to check for back-doors or have some program that would test the binary in a virtual box to see if it calls out to some malicious code.
Quote:
there is a huge number of variables affecting the final result, many of them not affecting functionality and not tracked by the package manager.
You are correct. That's why this would be a challenging project. we have already established that their would be many high and low binaries. portage takes the source and compiles it, right then and their, of course the portage sources could be md5sumed. And given that a user is compiling the source on their computer through portage, we would already get the different configurations for the result binary. All in one fell swoop the only way to submit a binary is right after compilation, if you hit the space bar or anything other than y or n , bang you missed your chance to send the binary of course you could also add a 5 second time after completion of compilation where the user could submit the binary. These can also be id'd by the compiling user. Of course their is no 100% failsafe but their are measures that could be taken to catch certain malicious binaries. Well to make a long story short Are you saying this could not be done?
Back to top
View user's profile Send private message
AllenJB
Veteran
Veteran


Joined: 02 Sep 2005
Posts: 1285

PostPosted: Wed Aug 13, 2008 2:03 pm    Post subject: Reply with quote

It's not just USE flags that can vary - there's also CFLAGS, CXXFLAGS, CBUILD, CHOST and LDFLAGS (off the top of my head - I bet there's more). Things like GCC version and glibc version could also affect things. Kernel version affects modules. Not to mention the versions of any other dependencies.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9507
Location: beyond the rim

PostPosted: Wed Aug 13, 2008 3:49 pm    Post subject: Reply with quote

dsims wrote:
so you mean to tell me that there is no way to check for back-doors or have some program that would test the binary in a virtual box to see if it calls out to some malicious code.

Not in an automated way. And doing it manually isn't an option.
Of course you can try to install something in a sandbox and monitor what it's doing, but that requires additional resources, can only detect known patterns, still requires someone to decide if something (like opening a port) is legit or not, and doesn't help if the attacker knows how the system works (he can just delay the activation of the backdoor for a few hours/days).

Quote:
portage takes the source and compiles it, right then and their, of course the portage sources could be md5sumed. And given that a user is compiling the source on their computer through portage, we would already get the different configurations for the result binary. All in one fell swoop the only way to submit a binary is right after compilation, if you hit the space bar or anything other than y or n , bang you missed your chance to send the binary of course you could also add a 5 second time after completion of compilation where the user could submit the binary. These can also be id'd by the compiling user. Of course their is no 100% failsafe but their are measures that could be taken to catch certain malicious binaries. Well to make a long story short Are you saying this could not be done?

That's what I'm saying. You assume that uploads to the system can only come from portage, but there is no way to enforce such a restriction (not that such a restriction would be useful anyway). Also checksums of the source code are useless as there are many other ways to inject code during the build process that are independent of the package itself (using a modified compiler for example), changes to the source code of a package are just the most obvious way.

I realize that this probably sounds quite paranoid to you, but for this kind of service you can't be careful enough. If the system accepts just one compromised package, any client using the system to install that package could be hijacked by the attacker. It basically becomes a malware distribution system. It would be equivalent to a malicious party uploading a trojaned patch to "Windows Update".
Back to top
View user's profile Send private message
dsims
n00b
n00b


Joined: 08 Aug 2008
Posts: 6

PostPosted: Thu Aug 14, 2008 3:04 pm    Post subject: Reply with quote

well if you use portage to get your source then yes only source code in portage can be compiled and returned as binary. I'm not saying let someone else's code be compiled as such. I'm also not saying after someone compiles it auto magically be released to the wild, that would be nice and I'm sure there is a way to automate that effect. If the source code is in the main branches, I'm pretty sure that it is not full of back doors and such. I'm thinking of an application that would lock down the instance of compiling and sending the resulting binary. I don't assume anything. Like every project you have constants and variables. Obviously the main constant is security, you are dealing with anyone out their. Like in windows when you open up an e-mail from someone you do not know. I kind of thought we had the software or knowledge to automate tasks like this. Actually the only thing I assume is that my title of noob in this forum would warrant someone else to keep using window like terms so I can understand it better, which is the wrong assumption on that other parties part.
Back to top
View user's profile Send private message
AllenJB
Veteran
Veteran


Joined: 02 Sep 2005
Posts: 1285

PostPosted: Thu Aug 14, 2008 3:49 pm    Post subject: Reply with quote

dsims wrote:
well if you use portage to get your source then yes only source code in portage can be compiled and returned as binary. I'm not saying let someone else's code be compiled as such. I'm also not saying after someone compiles it auto magically be released to the wild, that would be nice and I'm sure there is a way to automate that effect. If the source code is in the main branches, I'm pretty sure that it is not full of back doors and such. I'm thinking of an application that would lock down the instance of compiling and sending the resulting binary. I don't assume anything. Like every project you have constants and variables. Obviously the main constant is security, you are dealing with anyone out their. Like in windows when you open up an e-mail from someone you do not know. I kind of thought we had the software or knowledge to automate tasks like this. Actually the only thing I assume is that my title of noob in this forum would warrant someone else to keep using window like terms so I can understand it better, which is the wrong assumption on that other parties part.


The source code is taken from /usr/portage/distfiles, matched against the manifest in the main tree (which can be edited by users, or bypassed replacing the source code in the compile location and telling portage not to wipe the working directory) to get the package file to attempt to compile any source they wish). There's currently no way to verify that these files haven't been altered which can't be tampered with (ie. you request the Manifest file to verify what package I compiled against - fine, I just create some code that sends you the manifest data which you accept - now you've got my malicious package).

Even if you could somehow verify the manifests of all the source files used in the compilation, a malicious user could modify the build tools to compile manipulate the code during compilation itself, an attack against which to my knowledge there is no possible defense.

Even if it were possible to protect against these attacks, it would require a huge amount of time to implement the infrastructure to do it - time which, in my opinion, would be better spent on resolving bugs and implementing new features that a larger part of the community would be interested in using.
Back to top
View user's profile Send private message
node_one
Apprentice
Apprentice


Joined: 07 Apr 2008
Posts: 165

PostPosted: Thu Aug 14, 2008 5:22 pm    Post subject: Reply with quote

I think doing this on a small(er) scale is definitely worth it. In situations where you have less __FLAGS, USE, etc. variety and trust between the participants. Large scale, I do not see it either.
Back to top
View user's profile Send private message
dsims
n00b
n00b


Joined: 08 Aug 2008
Posts: 6

PostPosted: Thu Aug 14, 2008 8:54 pm    Post subject: Reply with quote

Quote:
I think doing this on a small(er) scale is definitely worth it. In situations where you have less __FLAGS, USE, etc. variety and trust between the participants. Large scale, I do not see it either
This is exactly what I mean you can always start out small. Let's say you have a website that disperses the source code files and the software on the computer to lock the program with out any user interference except a Gui, so you could choose which flags or whatever you wanted to be compiled with. only switch scenarios to keep any form of text away from the user. all the source would be on the website that disperses it. On sending the source we would grab a unique id and ip address of course you could grab more information if they filled out a form or snag it. if you cannot grab these off the users computer then they cannot participate. On program start up the libraries needed to compile are md5summed against the previous versions and the new versions. If all is well the program starts the compilation process if errors pop up the binary is not a candidate and is not sent back if their are no errors the binary is sent to a read only file or mysql database. The binary file will have the program name the date/time and the users computer identification as an added text file. as these files pile in then developers can test these on only the bare minimalist of computers. you can always rearrange the date time of any system if you do not get the date from the server. Then the developer could test these out and scan for malicious activity even maybe work with the F.B.I to catch persons who steal ids and accounts. if after the period of 5 days no malicious activity is detected or change the server time to speed up the process. But if you could get the program to lock all user interference and check the md5 sum of the libraries on the users side used to compile it with. I am sure their is a better way though. In the word of time, mine will be used.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9507
Location: beyond the rim

PostPosted: Thu Aug 14, 2008 9:53 pm    Post subject: Reply with quote

dsims wrote:
Quote:
I think doing this on a small(er) scale is definitely worth it. In situations where you have less __FLAGS, USE, etc. variety and trust between the participants. Large scale, I do not see it either
This is exactly what I mean you can always start out small.

You missed the key part in that quote: "trust between the participants"
As long as you trust all contributors in such a system the big security problem doesn't exist. But that means that the system becomes less useful, as the number of contributors becomes (very) limited. I's not about "starting" small, it's about keeping it small (by not opening it to the public).
There simply is no technical solution to replace trust in this context.

In other words: The system needs some authentication that an uploaded package is "good" before accepting an upload.
The "simple" solution is to use ordinary user authentication, but only grant access to trusted individuals.
Benefit: relatively easy to implement and audit, acceptable protection from malware (obviously there's always the chance that trust is abused)
Drawback: less contributors -> less packages uploaded -> less useful

What you want is to base authentication on the binary package itself, which I claim is impossible, no matter how complex you make it.
Benefit: many contributors -> many packages uploaded -> more useful
Drawback: complex to implement and audit (I doubt it's possible at all), questionable protection from malware

Anyway, unless someone actually is going to implement something related to this and cares about my opinion I'm done with this discussion.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Fri Aug 15, 2008 12:13 pm    Post subject: Reply with quote

Genone wrote:
You missed the key part in that quote: "trust between the participants"
As long as you trust all contributors in such a system the big security problem doesn't exist. But that means that the system becomes less useful, as the number of contributors becomes (very) limited. I's not about "starting" small, it's about keeping it small (by not opening it to the public).
There simply is no technical solution to replace trust in this context.

In other words: The system needs some authentication that an uploaded package is "good" before accepting an upload.

Well gpg signing of packages would give that, and enable the same trust model that's used for kernel sources. It's then up to the individual user to decide whom to trust (for a distro I'd enable the releng teams' keys as a default.) So long as certficates can be revoked (and such would be downloaded on sync, same as news items) there's no real issue imo.

Quote:
Anyway, unless someone actually is going to implement something related to this and cares about my opinion I'm done with this discussion.

Yeah that's the big one ;)

AllenJB wrote:
Even if you could somehow verify the manifests of all the source files used in the compilation, a malicious user could modify the build tools to compile manipulate the code during compilation itself, an attack against which to my knowledge there is no possible defense.

Well that's already there via signed portage snapshots, so we're down from a man in the middle to a local user, which is far better as a class of attack (cf the GLSA severity model.)
Quote:
Even if it were possible to protect against these attacks, it would require a huge amount of time to implement the infrastructure to do it - time which, in my opinion, would be better spent on resolving bugs and implementing new features that a larger part of the community would be interested in using.

Well I think a fairly large part of the community would be interested in using a proper binary Gentoo, straight from Gentoo. It would also make it much easier to attract new users and developers.
Back to top
View user's profile Send private message
dsims
n00b
n00b


Joined: 08 Aug 2008
Posts: 6

PostPosted: Fri Aug 15, 2008 2:04 pm    Post subject: Reply with quote

Yes I understand trustworthiness, That's why you start out small with a trusted group of people were you can pass data and in a way play a technical version of tag to see who can screw whose computer up the best. No just kidding we could relay data across the group and see if their like minds can point out flaws in the system. I've written a couple of programs that the first thing is security you want absolutely no user interference besides choosing in switch form as in a select box or radio button and that is it. seeing as this sort of program will be a binary download and not source. Of course their could always be reverse engineering, but that will have to be taken into the whole equation. If I could make a base program to this effect. As it is not in my best interest to jeopardize my future in learning. It's never about money it's about making the naysayers see something in a different light.
Back to top
View user's profile Send private message
timeBandit
Bodhisattva
Bodhisattva


Joined: 31 Dec 2004
Posts: 2719
Location: here, there or in transit

PostPosted: Fri Aug 15, 2008 3:14 pm    Post subject: Reply with quote

dsims wrote:
Yes I understand trustworthiness, That's why you start out small .... It's about making the naysayers see something in a different light.
There's a difference between nay-saying and recognizing the infeasible. This idea has been discussed many times, for years, in various forms, since almost the dawn of Gentoo. To see it in a different light we'd have to look beyond the visible spectrum.

It is like a debate over perpetual motion machines with "entropy" replaced by "trust" as the insurmountable obstacle. You cannot design away trust any more than entropy.

Moved from Gentoo Chat to Duplicate Threads.
_________________
Plants are pithy, brooks tend to babble--I'm content to lie between them.
Super-short f.g.o checklist: Search first, strip comments, mark solved, help others.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Duplicate Threads All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum