Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
rsync exclude/include lists confusion
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
NickDaFish
Tux's lil' helper
Tux's lil' helper


Joined: 12 Sep 2002
Posts: 112
Location: Boston, USA

PostPosted: Tue May 13, 2003 8:54 pm    Post subject: rsync exclude/include lists confusion Reply with quote

Please excuse me if this is a dupe. I have tryed searching the forums and google but I can't find the info I need.
I have a few minimal gentoo systems that don't need to rsync half the things that they do. I only need about 11 of the 69 portage branches. I wanted to put together a exclude list (refed from make.conf) to lighten up the load on the rsync servers they hit and our bandwidth. What I realised is that I need more of an include list than an exclude list, so rather than specifying the 58 un-used branches I could just specify the 11 used ones. However I can't find any good doc on this. After reading the docs I *think* I can do it like so...

Code:

+ app-admin/
+ app-admin/*
+ app-arch/
+ app-arch/*
...
- /*


However I can't *quite* get my head around the doc and I'm not 100% sure that this will work. Also it seems odd that I have to specify each dir and it's contents, also I'm not sure if I'm somehow excluding subdirs. I've tryed working this out but I know little about rsync and I'm just confusing myself at this point.

So.. calling all rsync gurus.....
Is this the best way to do this?
Will this work at all?
Can this be simplyfied?
Are there any pitfalls I'm not seeing?

Thanks all, when (if) I get it all figured out I'll write a post in the Tips forum.
Back to top
View user's profile Send private message
NickDaFish
Tux's lil' helper
Tux's lil' helper


Joined: 12 Sep 2002
Posts: 112
Location: Boston, USA

PostPosted: Wed May 14, 2003 7:53 pm    Post subject: Reply with quote

Answering my own question... I should do this:
Code:

+ app-admin/
+ app-admin/**
+ app-arch/
+ app-arch/**
...
- *


Basicly it's the recursive way that the excludes are applyed that give rise to the confusion. If you have a path /foo/bar and you want to sync the bar dir only you still have to explicitly allow access to the foo directory (+ foo/) other wise when the foo dir is checked the only match it will make is with the final '- *' and /foo/bar will never even be examined. This is kinda clear in the docs but a little hard to get the head around untill it clicks.

There is annother logic that can be used outlined in this post http://www.mail-archive.com/rsync@lists.samba.org/msg06864.html.

For shts n' giggles I wrote a small Python script using qpkg (from the gentoo tool kit) that makes a very locked down exclude list....
Code:

# prlock.py
# Portage Rsync Lockdown tool by Nick Fisher 5/03
import os, re

# Run qpkg and gather a list
pkgl = os.popen('/usr/bin/qpkg -nc -I','r').readlines()

# Format qpkg for rysync (Var for dirs only, Extras)
fmt = re.compile(r'[\w|-]*/')

# Add the branches of portage that we use
list=[]
for line in pkgl:
    m = fmt.search(line)
    # Add the branch with leading '+ ' if it's not there
    if not('+ '+m.group()+'\n' in list): list.append('+ '+m.group()+'\n')

# Format and duplicate the package dirs
for x in range(len(pkgl)):
    pkgl[x] = '+ '+pkgl[x].strip()+'/\n'
    pkgl.append(pkgl[x].strip()+'**\n')

# Add the pakages to the branches
list.extend(pkgl)

# Sort the list for easy reading and best logic
list.sort()

# Out put....
print ''.join(list), '+ /*\n- *'


Make sure that the /etc/portage dir exsists then run the following command...

Code:
/path/to/prlock.py > /etc/portage/rsync_excludes


Oh and make sure that the rsync_excludes line is uncommented in make.conf.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum