Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Gentoo Package Repository now using Git
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
ppurka
Advocate
Advocate


Joined: 26 Dec 2004
Posts: 3256

PostPosted: Sat Sep 05, 2015 2:38 pm    Post subject: Reply with quote

Update Got my answer. Renamed the gentoo.conf.rsync to .gentoo.conf.rsync and it worked. It seems the rsync configuration was overriding the git configuration. Strange, because I thought it only reads files ending in .conf



I can not seem to sync using git. Can someone explain the reason?

Code:
~» mount | grep  '/usr/portage'   # portage is on a ext2 formatted file
/mnt/usb/portage/portageFile on /usr/portage type ext2 (rw,noatime)


~» ls -a /usr/portage  # the entire directory is empty
.  ..  lost+found



~» l /etc/portage/repos.conf   # the repository information is on gentoo.conf
total 8.0K
4.0K gentoo.conf  4.0K gentoo.conf.rsync



~» cat /etc/portage/repos.conf/gentoo.conf   # the contents
[DEFAULT]
main-repo = gentoo

[gentoo]
location    = /usr/portage
sync-type   = git
sync-uri    = git://anongit.gentoo.org/repo/gentoo.git
auto-sync   = yes



~» grep -i repo /etc/portage/make.conf   # nothing else in make.conf
     -fam -ruby -harfbuzz -crashreporter -cvs avx -mercurial gtk2 \



~» emaint sync -a
>>> Syncing repository 'gentoo' into '/usr/portage'...
!!! getaddrinfo failed for 'rsync.gentoo.org': [Errno -2] Name or service not known
>>> Starting rsync with rsync://rsync.gentoo.org/gentoo-portage...
Why does emaint still try to use rsync?
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/
Back to top
View user's profile Send private message
hasufell
Retired Dev
Retired Dev


Joined: 29 Oct 2011
Posts: 429

PostPosted: Sat Sep 05, 2015 5:14 pm    Post subject: Reply with quote

I don't know if it can properly initialize the repo if the "/usr/portage" directory already exists. In addition, if it's gone, the "profiles/" subdirectory of the portage tree is missing which means you don't even have a working profile. That can break/confuse some tools.

I'd do the initial clone manually.

edit: tested in a docker environment, it should work if you remove /usr/portage first
Back to top
View user's profile Send private message
ppurka
Advocate
Advocate


Joined: 26 Dec 2004
Posts: 3256

PostPosted: Sun Sep 06, 2015 1:35 am    Post subject: Reply with quote

hasufell wrote:
I don't know if it can properly initialize the repo if the "/usr/portage" directory already exists. In addition, if it's gone, the "profiles/" subdirectory of the portage tree is missing which means you don't even have a working profile. That can break/confuse some tools.

I'd do the initial clone manually.

edit: tested in a docker environment, it should work if you remove /usr/portage first
Yes. It works with a completely empty /usr/portage. In my case, it wasn't even using the git protocol. It was because I had two files
Code:
~» l /etc/portage/repos.conf   # the repository information is on gentoo.conf
total 8.0K
4.0K gentoo.conf  4.0K gentoo.conf.rsync
The second one was a backup of the rsync config. Unfortunately, emaint reads all files in /etc/portage/repos.conf - not just those files which are ending in .conf.

Once I removed the gentoo.conf.rsync (actually renamed it to .gentoo.conf.rsync), it started using git. But then it again failed because I had newly formatted /usr/portage and the formatting created a lost+found folder. I had to delete that folder, and then git sync worked. And git sync was incredibly fast - I eventually used the github repo.
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/
Back to top
View user's profile Send private message
WWWW
Tux's lil' helper
Tux's lil' helper


Joined: 30 Nov 2014
Posts: 143

PostPosted: Fri Sep 11, 2015 10:49 am    Post subject: Reply with quote

I was reading about wtf was $harepoint, and apparently can be used for code versioning.
Back to top
View user's profile Send private message
Maitreya
Guru
Guru


Joined: 11 Jan 2006
Posts: 441

PostPosted: Fri Sep 11, 2015 11:22 am    Post subject: Reply with quote

WWWW wrote:
I was reading about wtf was $harepoint, and apparently can be used for code versioning.


How is this related? Are you trying to fake you post count for some reason?
Back to top
View user's profile Send private message
Yamakuzure
Advocate
Advocate


Joined: 21 Jun 2006
Posts: 2282
Location: Adendorf, Germany

PostPosted: Wed Sep 16, 2015 8:30 am    Post subject: Reply with quote

Just a question:

Is it normal that when ever I sync I get full profile updates?
Code:
Performing Global Updates
(Could take a couple of minutes if you have a lot of binary packages.)
  .='update pass'  *='binary update'  #='/var/db update'  @='/var/db move'
  s='/var/db SLOT move'  %='binary move'  S='binary SLOT move'
  p='update /etc/portage/package.*'
/usr/portage/profiles/updates/1Q-2010..........................................
................................
/usr/portage/profiles/updates/2Q-2010.......................................
/usr/portage/profiles/updates/3Q-2010............
/usr/portage/profiles/updates/4Q-2010............................
/usr/portage/profiles/updates/1Q-2011..........................................
.................................
/usr/portage/profiles/updates/2Q-2011..........................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
......................................................
/usr/portage/profiles/updates/3Q-2011..........................
/usr/portage/profiles/updates/4Q-2011..........................................
...................................................................
/usr/portage/profiles/updates/1Q-2012.......................
/usr/portage/profiles/updates/2Q-2012........................
/usr/portage/profiles/updates/3Q-2012..........................................
...........
/usr/portage/profiles/updates/4Q-2012..........................................
...................
/usr/portage/profiles/updates/1Q-2013..........................................
........................................................
/usr/portage/profiles/updates/2Q-2013...............
/usr/portage/profiles/updates/3Q-2013............
/usr/portage/profiles/updates/4Q-2013......
/usr/portage/profiles/updates/1Q-2014.......
/usr/portage/profiles/updates/2Q-2014..
/usr/portage/profiles/updates/3Q-2014................
/usr/portage/profiles/updates/4Q-2014.............
/usr/portage/profiles/updates/1Q-2015..........................................
.........................
/usr/portage/profiles/updates/2Q-2015..........................................
...............................................................................
...............................................................................
.........................................
/usr/portage/profiles/updates/3Q-2015......................
#
It finishes in about a second, but how is this triggered and how can I tell the system that those updates are already performed?

Oh, and btw: where are all ChangeLog files gone?

Edit: Got my answer:
hasufell wrote:
miket wrote:
(and lots of luck sorting it out if the committer commited several things at once)

There is no luck involved. Git can show you _any_ commit message that has touched a given file (or directory), no matter if it has been part of a bigger commit. It is far superior to any ChangeLogs we had.
Yes, that is of course right. The ChangeLog files are just a bit cleaner and easier to get with a structure nicer to read.
But as that is a pure personal preference and has nothing to do with the information level, which, of course, isn't any higher in ChangeLog files than the git log, I am completely content with the ChangeLog files removal.
_________________
Important German:
  1. "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
  2. "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
Back to top
View user's profile Send private message
dweezil-n0xad
Apprentice
Apprentice


Joined: 30 Oct 2006
Posts: 156
Location: Ostend, Belgium

PostPosted: Thu Sep 17, 2015 7:02 am    Post subject: Reply with quote

now also with dtd, glsa and news:
https://github.com/gentoo-mirror/gentoo/tree/master/metadata
_________________
i7-4790K | 16GB DDR3 | GTX 970 | 500GB SSD
ASUS N56VV | i7-3630QM | 12GB DDR3 | GT 750M | 256GB SSD
Back to top
View user's profile Send private message
charles17
Advocate
Advocate


Joined: 02 Mar 2008
Posts: 3664

PostPosted: Thu Sep 17, 2015 11:58 am    Post subject: Reply with quote

dweezil-n0xad wrote:
now also with dtd, glsa and news:
https://github.com/gentoo-mirror/gentoo/tree/master/metadata

So we could finally use this without any postsync stuff.
Code:
/etc/portage/repos.conf/gentoo.conf

[DEFAULT]
main-repo = gentoo
 
[gentoo]
location = /usr/portage
sync-type = git
sync-uri = https://github.com/gentoo-mirror/gentoo.git
auto-sync = yes

Is this sync-uri unique or will there be uris to select as are for sync selected by mirrorselect -i -r -o >> /tmp/foobar?
Back to top
View user's profile Send private message
hasufell
Retired Dev
Retired Dev


Joined: 29 Oct 2011
Posts: 429

PostPosted: Thu Sep 17, 2015 12:07 pm    Post subject: Reply with quote

charles17 wrote:
dweezil-n0xad wrote:
now also with dtd, glsa and news:
https://github.com/gentoo-mirror/gentoo/tree/master/metadata

So we could finally use this without any postsync stuff.
If you don't care about git history and gpg signatures...
Back to top
View user's profile Send private message
Yamakuzure
Advocate
Advocate


Joined: 21 Jun 2006
Posts: 2282
Location: Adendorf, Germany

PostPosted: Thu Sep 17, 2015 12:09 pm    Post subject: Reply with quote

Oh! I am still using hasufell's postsync stuff, so that might have caused the constant repeating of the update checks, right?
_________________
Important German:
  1. "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
  2. "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
Back to top
View user's profile Send private message
hasufell
Retired Dev
Retired Dev


Joined: 29 Oct 2011
Posts: 429

PostPosted: Thu Sep 17, 2015 12:15 pm    Post subject: Reply with quote

Yamakuzure wrote:
Oh! I am still using hasufell's postsync stuff, so that might have caused the constant repeating of the update checks, right?

I doubt it, haven't experienced that here.

The updates are part of the main repository, no matter what repo source you use.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sat Sep 19, 2015 7:10 am    Post subject: Reply with quote

hasufell wrote:
If you don't care about git history and gpg signatures...

Funny, just on the previous page, you were saying that the spiffing new git repo does not, and will not, have history:
hasufell wrote:
The old history will not be re-added to the current development repository

Now here you are claiming how wonderful it is, "because history."
Back to top
View user's profile Send private message
hasufell
Retired Dev
Retired Dev


Joined: 29 Oct 2011
Posts: 429

PostPosted: Sat Sep 19, 2015 9:37 am    Post subject: Reply with quote

steveL wrote:
hasufell wrote:
If you don't care about git history and gpg signatures...

Funny, just on the previous page, you were saying that the spiffing new git repo does not, and will not, have history:

That is wrong, the repo does have history, it was just reset. Please refrain from spreading your negativity, hate and drama for no reason. You are neither productive, nor helpful. You should seriously reconsider your behavior.

PS: I'm going to unsubscribe from this topic... if anyone needs further help wrt the scripts, create an issue on github
Back to top
View user's profile Send private message
ct85711
Veteran
Veteran


Joined: 27 Sep 2005
Posts: 1791

PostPosted: Sat Sep 19, 2015 3:43 pm    Post subject: Reply with quote

dweezil-n0xad wrote:
now also with dtd, glsa and news:
https://github.com/gentoo-mirror/gentoo/tree/master/metadata


I noticed, even though it's in the tree on github, it's still not being pulled.
According to the .gitignore file in metadata
Code:

# These are added during tree generation only.
/cache
/dtd
/glsa
/md5-cache
/news
/herds.xml
/timestamp
/timestamp.chk
/timestamp.x
/pkg_desc_index

They are still on the ignore list, seems that .gitignore needs to be updated so that those directories will start being pulled once again.
Back to top
View user's profile Send private message
miket
Guru
Guru


Joined: 28 Apr 2007
Posts: 488
Location: Gainesville, FL, USA

PostPosted: Sat Sep 19, 2015 3:51 pm    Post subject: Reply with quote

I wish I had had enough time to review the patching-in technique you described for the older history. Here is the big point, though, one I wish you would have kept in mind: older history matters.

It is not acceptable to have the older history available. Unless that git-replace technique you described can do a seamless merge of the old history, you're not done.

hasufell wrote:
That is wrong, the repo does have history, it was just reset. Please refrain from spreading your negativity, hate and drama for no reason. You are neither productive, nor helpful. You should seriously reconsider your behavior.

PS: I'm going to unsubscribe from this topic... if anyone needs further help wrt the scripts, create an issue on github

Great way to avoid the issue. Recognizing that we need the old history is not being negative or counterproductive.
Back to top
View user's profile Send private message
ct85711
Veteran
Veteran


Joined: 27 Sep 2005
Posts: 1791

PostPosted: Sat Sep 19, 2015 4:38 pm    Post subject: Reply with quote

miket, think it more of falls down to time vs value. Right now, how I am seeing it, doing all that work doesn't payoff the time spent. Now if this was a development tree, I can see where it would pay off for the extra amount of time however for this tree it doesn't.

Then it also runs into, how many people really need that long of history for the portage tree? Making the assumption that a change now makes something wrong does not also make the past wrong too (especially at the scale you are wanting).
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sat Sep 19, 2015 5:17 pm    Post subject: Reply with quote

hasufell wrote:
Please refrain from spreading your negativity, hate and drama for no reason.

WTF are you talking about? It was a simple statement, alluding to the content miket posted above in response to your negativity.

Projecting your issues on me, then asserting that I behave in manner X, does not make it true.

Though you're not alone in trying it.

Funny how often it happens with formerly-lucid developers who become paludroids; normally after about 12-18 months indoctrination in the myth that the cult-members are a "persecuted minority".
Typically they then start kicking up a fuss about how no-one wants paludis as a default (boo-hoo.)

There were several years of very good reasons why McCreesh was forcibly ejected from Gentoo, and why his behaviour thereafter formed the background to the Developer CoC, including its effect on developers after they leave Gentoo.

You really should reconsider both your approach, your manners, and the people you associate with.

I'll return to the actual topic next week; I had no intention of even posting to this thread until then, then I found out you'd done your usual "ban him, because I don't like him, let me count the made-up reasons" nonsense.

Consider whether you are not just a troll, however wonderful you might think you are (before you actually have to run your ideas by others, let alone real-life.)
Back to top
View user's profile Send private message
miket
Guru
Guru


Joined: 28 Apr 2007
Posts: 488
Location: Gainesville, FL, USA

PostPosted: Sat Sep 19, 2015 5:29 pm    Post subject: Reply with quote

I'll concede that not often will people need to look back 10 years, but it does occur quite often that people need that history from just a few months ago. I see that kind of thing a lot in the forums when someone is trying to go back to some version no longer in the tree so that they can unbreak their machines. Or, as in my case, try to figure out where something went wrong so that I can file a meaningful bug report.

Didn't, as I understand, someone already go through the big effort to make that happen? What happened to that?
Back to top
View user's profile Send private message
ct85711
Veteran
Veteran


Joined: 27 Sep 2005
Posts: 1791

PostPosted: Sat Sep 19, 2015 6:21 pm    Post subject: Reply with quote

I do know they did try to go through the effort to port the history over, when this was originally thought up. I couldn't really say on what happen on that part, the most I can do is guess on the reasons. I suspect they more of had in issue where they can't get the commits from CVS to line up to git (most likely the case) or more of space usage.
Back to top
View user's profile Send private message
miket
Guru
Guru


Joined: 28 Apr 2007
Posts: 488
Location: Gainesville, FL, USA

PostPosted: Sat Sep 19, 2015 9:38 pm    Post subject: Reply with quote

ct85711 wrote:
I do know they did try to go through the effort to port the history over, when this was originally thought up. I couldn't really say on what happen on that part, the most I can do is guess on the reasons. I suspect they more of had in issue where they can't get the commits from CVS to line up to git (most likely the case) or more of space usage.

A few years ago at work I wrote and ran a script to grab our most ancient CVS history and bring it into a Mercurial repository. One factor that made it more difficult is one to which you allude: lining up CVS commits in order to make Mercurial commits (Mercurial has been our main VCS since 2007). The problem, of course, is that CVS commits are simply per-file; there is no good way to find everything that happened for the commit of a large changeset. My solution was to look for all the CVS commits from a given author within a short time interval and stage them into a single Mercurial commit.

In general, this is a bit less of an issue with Portage than it was in our system: while we needed the changes to many files to appear coherently in changsets, in Portage the commits are often just for single ebuilds.

The process is certainly doable. With git it would be faster than with Mercurial and would take up less space. One thing that made the conversion easer for me was that I had direct access to the RCS files themselves.

I did this because our main Mercurial repository encompassed only those changes since we switched from CVS to Subversion in 2005. That switch involved a packaging change from a set of a couple of dozen separate repositories into a single repository. My work also coalesced those separate CVS repositories into a directory structure like the one we adopted at the time of the switch to Subversion. (Funny, we were on Subversion only 2 years before switching to Mercurial.)

There are many times I've found my reconstructed repository to be useful--even if it is for commits more than 10 years old.

The whole Portage CVS repository manages to sit on a Gentoo server. It certainly is of finite size. As to how large and how long it would take to transmit this if converted to git, I have no good guess.

Our main system at work has maybe 10,000 files in version control; the average size is around 10Kb. It does not take unmanagably long to get a checkout with that 10 years of history behind it (my reconstruction of the CVS history is not in that main repository). Portage has about 15 times that number of files, but the average size is more like 5Kb. (No wonder syncing takes so long!)

The whole of Portage history might be more than most users might want to slurp up, but at least it ought to be available. Would not have it been good to obtain the whole converted repository but be able to slice off part of it for distribution? At reasonable-enough intervals (say once a year), grab a patch of the last n years or months from the end of the big repository. For normal syncing, send that.

--Or-- (I think this would work but I haven't used git enough to be sure), make users' the initial clones be shallow clones at whatever convenient depth so that subsequent pull requests would pick up only newer revisions. That would involve no periodic splitting from the big repository. The pull requests at every normal sync would pick up only the newer changes.

There would still be a good basis for people who wanted to pull the whole authoritative history and use it (perhaps for routine uses like displaying good reconstructed-from-commit-history changelogs on sites like znurt.org).

After all, even those developers complaining it would take too long to transmit and store lots of really old Portage history and therefore took the expedient of making this unhappy August 2015 chop-point are going to be up against a new reality three or four years hence: the Portage repo will have accumulated lots of old history and those initial clones are going to take longer than maybe they want it to take.

I think that either of my proposals for using a grand authoritative repository is much more sustainable in the long term. It would be more honest, at least.
Back to top
View user's profile Send private message
desultory
Bodhisattva
Bodhisattva


Joined: 04 Nov 2005
Posts: 9410

PostPosted: Mon Sep 21, 2015 4:03 am    Post subject: Reply with quote

hasufell wrote:
steveL wrote:
hasufell wrote:
If you don't care about git history and gpg signatures...

Funny, just on the previous page, you were saying that the spiffing new git repo does not, and will not, have history:

That is wrong, the repo does have history, it was just reset. Please refrain from spreading your negativity, hate and drama for no reason. You are neither productive, nor helpful. You should seriously reconsider your behavior.

PS: I'm going to unsubscribe from this topic... if anyone needs further help wrt the scripts, create an issue on github
It hardly seems to be "spreading your negativity, hate and drama for no reason" to cite what you yourself wrote . Granted, there have been some comments which reflect being somewhat less than fully informed on the git migration, but that also hardly seems to qualify, and certainly does not justify your claims made against steveL. If, by some chance, you do eventually lodge a valid complaint against steveL, it would be rather less likely to be dismissed out of hand if you would stop spuriously reporting him for doing what you are actually doing.
Back to top
View user's profile Send private message
alinefr
Tux's lil' helper
Tux's lil' helper


Joined: 05 Jul 2009
Posts: 113
Location: São Paulo, Brasil

PostPosted: Thu Feb 18, 2016 1:29 am    Post subject: Reply with quote

Someone in this thread suggested to set:
CACHE_METHOD="/usr/portage/ parse|ebuild*"

... as eix-update was not working by default when 'sync-type = git'. But it was taking ages to build the database. So I did try different values for CACHE_METHOD, and in the end 'assign' is working as it should. A lot more faster than parse|ebuild:

So my CACHE_METHOD is set like this now:
CACHE_METHOD="/usr/portage/ assign"
Back to top
View user's profile Send private message
ExecutorElassus
Veteran
Veteran


Joined: 11 Mar 2004
Posts: 1435
Location: Berlin, Germany

PostPosted: Tue Oct 11, 2016 9:52 pm    Post subject: Reply with quote

I have /usr/portage/distfiles as a separate partition. When I try to sync, I get an error:

Code:
# emerge --sync
>>> Syncing repository 'gentoo' into '/usr/portage'...
/usr/bin/git clone --depth 1 https://github.com/gentoo-mirror/gentoo .
fatal: destination path '.' already exists and is not an empty directory.
!!! git clone error in /usr/portage
q: Updating ebuild cache in /usr/portage ...
q: Finished 0 entries in 0.000037 seconds


Is there a way to use git without having to re-jigger all my partitions?

Cheers,

EE
Back to top
View user's profile Send private message
grumblebear
Apprentice
Apprentice


Joined: 26 Feb 2008
Posts: 202

PostPosted: Wed Oct 12, 2016 5:24 pm    Post subject: Reply with quote

Your problem is the initial cloning? Just do:

umount /usr/portage/distfiles
emerge --sync
mount /usr/portage/distfiles
Back to top
View user's profile Send private message
ExecutorElassus
Veteran
Veteran


Joined: 11 Mar 2004
Posts: 1435
Location: Berlin, Germany

PostPosted: Wed Oct 12, 2016 5:30 pm    Post subject: Reply with quote

It was just the initial cloning, but then a dev told me to move what used to be /usr/portage to /var/portage/repos and distfiles to /var/portage/distfiles, so that it's no longer within the tree that git syncs to. I had to move mountpoints around and edit my fstab, but now it works.

But thanks for the tip. It's good to know I can also just unmount the partition.

Cheers,

EE
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page Previous  1, 2, 3, 4, 5  Next
Page 4 of 5

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum