Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How do you manage /etc/portage for multiple targets?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18125

PostPosted: Thu Apr 25, 2019 3:05 am    Post subject: How do you manage /etc/portage for multiple targets? Reply with quote

Elsewhere, I decided Chef, Puppet, Salt and I think something else weren't great solutions for Reasons. Ansible seemed to address some of those, but is still a bigger footprint than I'd prefer to use.

For now I'm starting with directories common and <client>: "{common,<client>}/etc/portage/...". That almost necessitates a need for other directories such as X and VM. At that point or soon after, it is probably at its limits of "reasonable" to manage. Initial distribution will likely be clients pulling with scp or similar.

So I'm curious what others are doing and hoping to learn of a method better than the directories, but with less of a footprint than the listed tools.
_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
berferd
Tux's lil' helper
Tux's lil' helper


Joined: 13 May 2004
Posts: 117

PostPosted: Fri Apr 26, 2019 1:11 am    Post subject: Reply with quote

I don't have an answer for you, but I'm hoping you'll entertain a question. What's so big about Ansible's footprint? All it requires on the configured machine is an ssh daemon, and even most docker containers will have one of those.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18125

PostPosted: Fri Apr 26, 2019 3:20 am    Post subject: Reply with quote

Distribution is really a secondary issue. When compared with simpler solutions for distribution, it seems like overkill. I don't recall if I was surprised by the number of dependencies it had, or just that it had more than I expected at the time (I installed it on a server but haven't used it there yet).

When I last used ansible, I thought getting a functional environment seemed a bit like laws and sausages (better to not see how it is done). YAML is pretty ugly and seemed fragile. Maintaining that environment is quite a bit of work. Also, in my experience, python was a requirement, and I thought there was some other ansible module that was preferred (bootstrap or something like that?).

With ansible, I still have to solve my original problem / question. Unless I'm forgetting something, I'd still have the same files in a similar sprawl of directories. And for right now, it doesn't seem like enough of a benefit compared to my current "first, make it work" solution.

The only other options I've thought of were using prefixed numbered files for sequence (00-foo, 10-bar), or a database. Neither of those seem like an improvement.
_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
berferd
Tux's lil' helper
Tux's lil' helper


Joined: 13 May 2004
Posts: 117

PostPosted: Sat Apr 27, 2019 12:12 am    Post subject: Reply with quote

Well with Ansible I'd abstract the common pieces of the /etc/portage denizens into a set of Jinja2 templates, and use the Ansible template module to generate the correct file for each client.

Jinja2 is meh at best, but it's no worse than any other text templating engine I've ever had to use.

Totally agree with you on YAML. The nicest thing I can say about it is that I hate it less than XML. It does seem that non-programmers pick it up easier than Python, though.

I'm working at a small place right now, and the plain-text host inventory has worked just fine for us so far. I wouldn't try it on setups with more than 40 machines or so. We just check the inventory file into git along with all the playbooks.

We all use Mac workstations. I just did a "brew install ansible". I don't remember a long list of dependencies, but it was some time ago.

I'm puzzled about your comment about a server. Ansible is serverless unlike Puppet and Chef (and maybe Salt). You can even configure Windows machines with it if you're so inclined. The Ansible tools themselves will not run on Windows, though.

I'm not trying to annoy you here, so just say the word if you want me to shut up about Ansible already.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18125

PostPosted: Sat Apr 27, 2019 5:12 am    Post subject: Reply with quote

Are Jinja2 templates different from Ansible templates? I've used the latter, and i don't think it seems appropriate. How I thought Ansible templates would be used (and how I started using them) was to take a "well defined" config file and provide variables. Such as /etc/ssh/sshd_config or even /etc/hosts. Depending on Defined Factors, each of those files would turn an option on or off or make sure a line existed in the file.

That would work for make.conf and probably the repo related files, but I don't see how it would work for most anything else in /etc/portage: env, package.{accept_keywords,env,license,mask,provided,unmask,use} all seem "unpredictable" as far as a template is concerned. My use of them was limited, so maybe I'm just not aware of how that would look.


Almost anything is better than XML :). And yes, the "non-programmer" factor was about the only reason Ansible was viable. I also didn't care for the "not really open" model the others use (although I read one of them changed from that).


For just the hosts file, I was working with only a handful of hosts and thought that was almost too much for plain text. I hadn't gotten to breaking that one down. The inheritance model seemed like a nasty mess and that it made it easy to make poor decisions early that could be difficult to undo.


In the version of Ansible I used (not long after RH bought it) the files were stored on a server and pushed with Ansible software to clients. On that server, the Ansible software had dependencies of more than ssh. Some simple tasks could be performed on clients with only ssh, but for other capabilities, a certain version of python was required. I just don't remember the specifics and would have to dig up my notes. And I also recall having an issue for which I remember something to do with a "bootstrap" process, but I don't recall what. A few searches aren't turning up anything obviously related. I want to say it had to do with getting the preferred minimum version of python installed on the client, but that wasn't a priority issue at the time.


You're not being annoying at all. As I mentioned, I have ansible installed, I just didn't think it was well suited for this specific task. Although I may eventually settle on using it.

Also in that same time period, Mitogen was released. I've thought about trying to do something with it, but that would be a lot more involved (and probably too big of a leap for me to undertake).

Thanks for the comments. Making me think about it is only helpful.
_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
Dr.Willy
Guru
Guru


Joined: 15 Jul 2007
Posts: 500
Location: NRW, Germany

PostPosted: Sat Apr 27, 2019 9:22 am    Post subject: Reply with quote

When I last looked into this I was consedering cdist, but ultimately I ended up with a prefix-solution.
Settings common across all hosts start with an underscore, settings specific to a machine with an upper-case letter. So yeah, basically it's all manual.
Back to top
View user's profile Send private message
berferd
Tux's lil' helper
Tux's lil' helper


Joined: 13 May 2004
Posts: 117

PostPosted: Sat Apr 27, 2019 2:26 pm    Post subject: Reply with quote

Yep, Ansible uses Jinja2 templates. You can do a lot more than set variables in a file with Jinja2. Here is how I generate a Nagios host config file, for example:

Code:
{% for host in groups['ds-servers'] %}

define host {
        use                     linux-server
        host_name               {{ host }}
        hostgroups              linux-remote-hosts
        address                 {{ host }}
}

{% endfor %}

{% for host in groups['tableau-linux-servers'] %}

define host {
        use                     linux-server
        host_name               {{ host }}
        hostgroups              linux-remote-hosts
        address                 {{ host }}
        contact_groups          tab-admins
}

{% endfor %}

{% for host in groups['tableau-windows-servers'] %}

define host {
        use                     linux-server
        host_name               {{ host }}
        hostgroups              all-hosts
        address                 {{ host }}
        contact_groups          tab-admins
        max_check_attempts      2
}

{% endfor %}


I configure Windows machines to get ping monitoring only using an Ansible inventory group called "tableau-windows-servers". This is what that looks like in the inventory file:
Code:
[tableau-windows-servers]
tableau-prod
tableau-dev


Yes, the ini-style format is brittle, but again, I'm trying to make this accessible to non-programmers.

pjp wrote:

That would work for make.conf and probably the repo related files, but I don't see how it would work for most anything else in /etc/portage: env, package.{accept_keywords,env,license,mask,provided,unmask,use} all seem "unpredictable" as far as a template is concerned. My use of them was limited, so maybe I'm just not aware of how that would look.


Well if you have files that are bespoke for some hosts there's no alternative to keeping several different files on a file system somewhere. The Ansible copy module at least gives you flexibility in how to name them. You can keep them in directories like you suggest, you could use file prefixes or suffixes, etc.

pjp wrote:
I also didn't care for the "not really open" model the others use (although I read one of them changed from that).

Not sure what you mean here. I'm allergic to proprietary software. We're not using Tower, and I fight new vendors tooth and nail. Sometimes they're unavoidable, unfortunately.

pjp wrote:
In the version of Ansible I used (not long after RH bought it) the files were stored on a server and pushed with Ansible software to clients.

That sounds like Ansible Tower. Yep, won't touch it. We keep the inventory files in a directory called "inventory" and we have one file per environment (prod, test, dev). Our workflow is git clone or pull -> ansible-playbook -i inventory/test some-playbook.yml.

pjp wrote:
On that server, the Ansible software had dependencies of more than ssh. Some simple tasks could be performed on clients with only ssh, but for other capabilities, a certain version of python was required. I just don't remember the specifics and would have to dig up my notes. And I also recall having an issue for which I remember something to do with a "bootstrap" process, but I don't recall what. A few searches aren't turning up anything obviously related. I want to say it had to do with getting the preferred minimum version of python installed on the client, but that wasn't a priority issue at the time.

Yep, it does seem picky about its Python, but that requirement seems reasonable to me. I expect all machines to have at least Python 2.7 installed. Here's what I found about Ansible bootstrapping:
https://docs.ansible.com/ansible/latest/user_guide/intro_bsd.html#bootstrapping-bsd

pjp wrote:
Also in that same time period, Mitogen was released. I've thought about trying to do something with it, but that would be a lot more involved (and probably too big of a leap for me to undertake).

Mitogen looks gee-whiz cool. I know what I'm doing for part of today.

pjp wrote:
Thanks for the comments. Making me think about it is only helpful.

My pleasure. Walking through this makes me think and re-think my decisions. Always a good exercise.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18125

PostPosted: Sat Apr 27, 2019 10:58 pm    Post subject: Reply with quote

Dr.Willy wrote:
When I last looked into this I was consedering cdist, but ultimately I ended up with a prefix-solution.
Settings common across all hosts start with an underscore, settings specific to a machine with an upper-case letter. So yeah, basically it's all manual.
It's certainly looking like manual. I'm sure it is a difficult to solve problem with a simple solution. cdist looks interesting, but seems more complicated than I'd prefer, given other more commonly used solutions.

I'll most likely have to live with the manual solution to see if I get any bright ideas on something "better." Thanks for the comments.
_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18125

PostPosted: Sat Apr 27, 2019 11:17 pm    Post subject: Reply with quote

berferd wrote:
The Ansible copy module at least gives you flexibility in how to name them. You can keep them in directories like you suggest, you could use file prefixes or suffixes, etc.
Yeah, that concept is what I'll have to kick around.

berferd wrote:
Not sure what you mean here. I'm allergic to proprietary software. We're not using Tower, and I fight new vendors tooth and nail. Sometimes they're unavoidable, unfortunately.
As I recall, all of the primary alternatives to Ansible had a portion of the tool that was not $free or completely open source, or required licensing over a certain number of hosts. I don't consider those solutions truly open and I avoid that model if at all possible. And for personal use, it is always possible. But I might also be mixing configuration management tools with others, as the primary reason for not using Ansible alternatives was the necessity of programming. (At least at the time of my review.)

berferd wrote:
That sounds like Ansible Tower. Yep, won't touch it. We keep the inventory files in a directory called "inventory" and we have one file per environment (prod, test, dev). Our workflow is git clone or pull -> ansible-playbook -i inventory/test some-playbook.yml.
No, not Tower. I've never used it. Take a server, install ansible. That is a server even if it pushes stuff to clients.

berferd wrote:
Yep, it does seem picky about its Python, but that requirement seems reasonable to me. I expect all machines to have at least Python 2.7 installed.
Eventually I'd prefer to live without it. I'm not fond of chasing the Dynamic Runtime Version Dragon.

berferd wrote:
Here's what I found about Ansible bootstrapping:
https://docs.ansible.com/ansible/latest/user_guide/intro_bsd.html#bootstrapping-bsd
Yeah, that's probably it. I think I was looking to automate installing or updating python.

berferd wrote:
Mitogen looks gee-whiz cool. I know what I'm doing for part of today.
I thought so too. I was mainly thinking of a simplified solution rather than the complex ecosystems.
_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18125

PostPosted: Sun Apr 28, 2019 8:36 pm    Post subject: Reply with quote

I think Gentoo inherently supports a solution: profiles. I just have to finally untangle how they work.

https://wiki.gentoo.org/wiki/Profile_(Portage)#custom

Since profiles support inheritance, the common -> vm? -> GUI layering ought to work. Then a client can have its files in /etc/portage.

That should also work well with a custom repo I'd been thinking of using.
_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
berferd
Tux's lil' helper
Tux's lil' helper


Joined: 13 May 2004
Posts: 117

PostPosted: Sat May 04, 2019 4:50 am    Post subject: Reply with quote

You made me curious so I decided to run emerge -pv ansible on my last Gentoo box. Whoa that's a long list of stuff. Weird stuff too like dev-python/jupyter_client and net-libs/zeromq. I was puzzled so I ran a equery depgraph ansible. Nothing untoward there. Then I looked at requirements.txt in the Ansible source. Nope, just three lines. Maybe those are minimal runtime dependencies and the build dependencies are more involved, so I looked at the ebuild for it. I'm still not seeing what's dragging in the kitchen sink.

I might try to pull the ebuild into my overlay, remove all the deps and add them back in one by one to see what's causing the exotica to be required. Sounds like a whole lot of work right now, though.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18125

PostPosted: Sat May 04, 2019 6:28 am    Post subject: Reply with quote

I was surprised as well. IIRC, there is a GUI component of some kind (I thought I saw something gtk). Although I only have it installed on a GUI-less system.
_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 14297

PostPosted: Sat May 04, 2019 4:51 pm    Post subject: Reply with quote

If you have not yet, try adding --tree to your emerge call. The resulting indentation gives some insight into package dependency relations, and will be less trouble than modifying the ebuild.
Back to top
View user's profile Send private message
berferd
Tux's lil' helper
Tux's lil' helper


Joined: 13 May 2004
Posts: 117

PostPosted: Sat May 04, 2019 6:43 pm    Post subject: Reply with quote

Thanks for the tip Hu! Turns out the culprit was the "cli" use flag to netaddr. Adding
Code:
dev-python/netaddr -cli
To my package.use made the dependencies sane:
Code:
[ebuild  N     ] dev-python/netaddr-0.7.19  USE="-cli -test" PYTHON_TARGETS="python2_7 python3_6 -python3_5 (-python3_7)"
[ebuild  N     ] net-misc/sshpass-1.06
[ebuild  N     ] dev-python/jinja-2.10  USE="-doc -examples -test" PYTHON_TARGETS="python2_7 python3_6 (-pypy) (-pypy3) -python3_5 (-python3_7)"
[ebuild  N     ] dev-python/pyyaml-3.13  USE="-examples -libyaml" PYTHON_TARGETS="python2_7 python3_6 (-pypy) (-pypy3) -python3_5 (-python3_7)"
[ebuild  N     ] dev-python/httplib2-0.10.3-r1  PYTHON_TARGETS="python2_7 python3_6 (-pypy) (-pypy3) -python3_5"
[ebuild  N     ] dev-python/pyparsing-2.2.0  USE="-doc -examples" PYTHON_TARGETS="python2_7 python3_6 (-pypy) (-pypy3) -python3_5 (-python3_7)"
[ebuild  N     ] dev-python/bcrypt-3.1.3  USE="-test" PYTHON_TARGETS="python2_7 python3_6 (-pypy) -python3_5"
[ebuild  N     ] dev-libs/libsodium-1.0.16-r2  USE="asm urandom -minimal -static-libs" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="-aes -sse4_1"
[ebuild  N     ] dev-python/packaging-16.8  USE="-test" PYTHON_TARGETS="python2_7 python3_6 (-pypy) (-pypy3) -python3_5"
[ebuild  N     ] dev-python/pynacl-1.2.1  USE="-test" PYTHON_TARGETS="python2_7 python3_6 -python3_5 (-python3_7)"
[ebuild  N     ] dev-python/paramiko-2.4.2  USE="-doc -examples (-server) -test" PYTHON_TARGETS="python2_7 python3_6 -python3_5 (-python3_7)"
[ebuild  N     ] app-admin/ansible-2.7.10  USE="-doc -test" PYTHON_TARGETS="python2_7 python3_6 -python3_5 (-python3_7)"
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum