| View previous topic :: View next topic |
| Author |
Message |
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
 Posted: Sun Jun 08, 2014 8:49 am Post subject: Free job scheduler [solved] |
 |
|
Perhaps the subject just uses the wrong terminology, but I do not know an appropriate term:
I am looking for a program which can "schedule tasks", but not in the sense of cron ("at a given time") but in the sense of sequences/conditionals/...
For instance, I want to start an emerge process. When this has finished succesfully, I want to start a kernel compilation in my 32bit chroot and emerge nvidia-drivers in the chroot, and simultaenously do the same on my main system.
When this is running, I find that when all has finished a "revdep-rebuild" on both systems should be scheduled.
And perhaps after a while, I change my mind and want to remove one of the two revdep-rebuilds from the schdule.
Yes, I know for some of these tasks, I can write shell scripts (although this is already not so easy with a 32bit chroots) or sequences in one shell command line. However, once such a sequence started running, I cannot modify it easily. In particular, it is not easy to "add" something to such a sequence at a certain place.
Yes, there are many ad-hoc workarounds: one could create/remove files as flags, use pgrep or simliar things, already "for later adding" execute a not-yet-written script within a shell script etc. I am not looking for such tips which all are somewhat hackish: I am looking for a "general" such scheduler which can do all this "cleanly" without requiring the user to remember the "schedule tree" and adapting the methods on how this tree loks like.
I am also aware that there is a "starter" script in the mv overlay which is able to schedule tasks sequentially, adding a new task "at the end".
However, this is not powerful enough for a "full" scheduler (with && || conditionals, adding tasks "in the middle", simultaneous execution of some tasks, removal of tasks, etc).
I suppose, using some "scheduler" daemon (e.g. basing on dbus) might be the right tool, 'though there are of course many problems with permissions/policies, and also the 32bit-chroot might be an issue with such a solution.
Anyway, perhaps such an animal (which at least can some of these things) already does exist?
Is it perhaps even built in into some existing DE and I just did not understand that this is the case?
Last edited by mv on Sun Jul 13, 2014 2:21 pm; edited 3 times in total |
|
| Back to top |
|
 |
Maitreya
Guru
Joined: 11 Jan 2006
Posts: 441
|
| Posted: Sun Jun 08, 2014 9:44 am Post subject: |
|
|
Something like "at"?
man at |
|
| Back to top |
|
|
Maitreya
Guru
Joined: 11 Jan 2006
Posts: 441
|
| Posted: Sun Jun 08, 2014 9:45 am Post subject: |
|
|
And on the other end of the extreme you got buildbots/CI systems.
But I guess you want something in between? |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Jun 08, 2014 1:08 pm Post subject: |
|
|
AFAIK, "at" triggers events only based on time. I want some "sequential" scheduling.
To make perhaps clearer what I mean, here is an example: Let us say, I have 10 commands which I want to execute in a certain order (some parallel, some conditional etc.) then I could write a shell script like e.g.
| Code: | cmd1; cmd2 && {
cmd3 &
c3=$!
if cmd4
then cmd5
else cmd6 ; cmd7
fi &
cmd8 && wait $c3
} && cmd9 || wait; cmd10 |
(the point is not, how exactly the shell-script looks like - it is a "random" example).
The above shell-script would already be "almost" the scheduler I like to have. However, it has the problem, that one the script is started, I cannot modify it anymore. For instance, if cmd2 is an long-running command (say, "emerge libreoffice"), and I decide for some reason in that moment that I want to replace the last line by
| Code: | | } && cmd9 || wait; cmd10a && cmd10; cmd11 |
I cannot do this without stopping and then restarting the whole script (and thus "emerge libreoffice"), modifying it correspondingly so that cmd1 is not executed again. This is not the "natural" thing to do when all I want is to modify the "scheduling" of some command in some distant "future".
| Maitreya wrote: | | buildbots/CI systems. |
I do not mean building some particular project: I really mean "random" tasks which should not all be started in parallel (partially due to processor/memory restrictions, partially due to the fact that some tasks can only be started after others have succesfully completed). A typical example for a gentoo user is kernel compilation followed by emerge @module-rebuild followed by initiating a backup followed by rebooting. But suddenly it turns out that before making the backup or before the rebooting something other important has to be done...) |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sun Jun 08, 2014 1:11 pm Post subject: Re: Looking for a "task scheduler" |
|
|
| mv wrote: | | Perhaps the subject just uses the wrong terminology, but I do not know an appropriate term: I am looking for a program which can "schedule tasks", but not in the sense of cron ("at a given time") but in the sense of sequences/conditionals/... |
mv ... I think the term you are looking for is "program", but we can see the obvious catch with such a term as this "program program" exists already in a variety of forms (every "programming" language has some form of "sequences/conditionals", etc). Yes, I know this is not what you have in mind, but I think the reason we don't have something (even to the extent of a descriptive term) of this nature is that this is what programming logic provides, so such a "program program" is in the realms of meta-programming.
best ... khay |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Jun 08, 2014 2:34 pm Post subject: Re: Looking for a "task scheduler" |
|
|
| khayyam wrote: | | ]mv ... I think the term you are looking for is "program" |
The difficulty is that this "programm" would have the need to be modified dynamically, and this is where all standard languages fail.
In fact, the "language" for this "program" can be very poor (the example I posted already contains all directives which it would have to know), but the point is the possibility to modify it dynamically.
Thus, here is the way how I thought about implementing it: There are two executables "schedule" and "start".
Then you would call e.g. | Code: | schedule 1 emerge ...
schedule 2 'sh -c "cd /usr/src/linux && make && make install"'
schedule 3 ... |
this would not start the corresponding tasks but just cause some daemon to record these commands somewhere. Once is executed, the recorded task "2" would run. Then you could just run the shell-script in the example above, just with "cmd1" ... "cmd10" replaced by "start 1" ... "start 10".
The crucial difference to running the commands directly is that, since the programs are run by some daemon, you can just stop the scheduling script, modify it, and restart it again, without stopping the actuall tasks.
Of course, there are a lot of difficulties in this approach: The tasks might not be appropriate to be run separate from a terminal, there are issues if it should be possible to run "schedule" by different users or even on different chroots etc.
I can imagine many approaches how to actually implement "schedule" and "start":
1. "schedule" could be somethnig like offereing a service over dbus and start is just sending a message (this approach has the problem that either you can use only a session dbus with limited permissions or, if you want to use the system dbus, you must modify policy files on the run).
2. Instead of dbus, one could also use "magic files" in some directory to communicate (this is actually what the "starter" implementation does).
3. "schedule" and "start" might be implemented using (or reimplementing parts of) screen, tmux, nohup.
4. One could also make the starting prorgam "smarter" by catching signals which stop the starting program but not the running child processes (not sure how to implement this, but certainly this is possible).
Anyway, all is a lot of work. Maybe somebody has already implemented this, or there exists other solutions which follow a completely different approach for a "scheduler". |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Jun 08, 2014 3:20 pm Post subject: |
|
|
Meanwhile I have found that what I was looking for is called Enterprise job scheduling,
It seems that there exist some professional (and rather expensive) solutions but no free solution yet. |
|
| Back to top |
|
|
John R. Graham
Administrator
Joined: 08 Mar 2005
Posts: 10587
Location: Somewhere over Atlanta, Georgia
|
| Posted: Sun Jun 08, 2014 4:08 pm Post subject: |
|
|
I have briefly looked at OpenJCS in the past. Does something like this fit your requirements?
- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Jun 08, 2014 6:42 pm Post subject: |
|
|
| Thanks. OpenJCS looks like overkill to me (apache, php, java, zope), but the result also seems to be a huge superset of what I had requested... |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Tue Jun 10, 2014 9:34 pm Post subject: |
|
|
| This all sounds like something that a makefile and a few short scripts could do... the restarting/editing in-flight part could be done with inotifywait. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Wed Jun 11, 2014 9:50 am Post subject: |
|
|
| Ant P. wrote: | | This all sounds like something that a makefile and a few short scripts could do... the restarting/editing in-flight part could be done with inotifywait. |
Yes, I was never claiming that it is hard to do. In fact, with the "starter" script I already have a partial solution (Starter uses one file to "communicate" which is checked on a timer basis instead of inotifywait). However, "starter" is limited to sequential execution, and later reordering is not possible with this concept. It would require a few days of work to write a "generic" solution. This is not extraordinarily much, and so I thought maybe somebody else might already have done it.
The "easy" implementation I was actually thinking of is the following: You run in different shells (possibly with different users/chroots) the commands "schedule [command you want to schedule]". The commands are not immediately executed but wait first for a signal of some kind; when they are finished, they signal back there exit status.
The "scheduler" then just sends the corresponding signals and collects the results. It can be controlled by a simple shell script which, if required, is just stopped rewritten and restartet...
For the signals there are several possibilities: They can be "unix" (kill) signals, "magic" files, sockets, dbus, ... each choice has some disadvantages: kill signals might not work within chroots (at least for hardened kernels) and there are permission issues, common files or sockets also have problems with chroots (and possibly raise permission issues if world-writable directories are created for them), dbus also has permission problems (and additionally needs sockets to communicate).
Certainly, everything can be solved, but implementing a "safe" solution probably requires a few days of work...
The "starter" script does a lot of these things already quite safe (with file-locking etc.), but its concept is too simple to be extended for a "full" editing of a dependence tree. After all, the "starter" script was written only in half an afternoon... |
|
| Back to top |
|
|
desultory
Bodhisattva
Joined: 04 Nov 2005
Posts: 9410
|
| Posted: Thu Jun 12, 2014 3:17 am Post subject: |
|
|
| mv wrote: | | AFAIK, "at" triggers events only based on time. I want some "sequential" scheduling. |
Might I suggest a little light reading? http://linux.die.net/man/1/batch
Additionally, incrond (sys-process/incron) might be of interest to you, depending upon quite how you want to work and how you want your solution to work |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Mon Jul 07, 2014 2:54 pm Post subject: |
|
|
Sorry, somehow I missed that the previous reply...
No, this is not what I am looking for: It is missing the "dependencies" of the tasks.
| Quote: | | Additionally, incrond (sys-process/incron) might be of interest to you |
This seems closer, but could only be used as a back-end for the front-end I am looking for: One would need a bunch of wrapper scripts which generate and control "magic" files. This is essentially what the "starter" script does (not using inotify but mucher simpler by checking in regular intevals). |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Jul 13, 2014 2:20 pm Post subject: |
|
|
After one evening and one morning of work, the presumably first free job scheduler sees the light of the world:
Here it is.
As usual, gentoo users can install it by using the mv overlay (available over layman).
It is now written in perl and uses TCP sockets for IPC. It would probably make sense to rewrite at least the server part in C or C++, but it is sufficient for my current needs.
Probably you can still find quite a few bugs, since testing was only very rudimentarily.
Edit: Fix typo
Last edited by mv on Sun Jul 13, 2014 4:28 pm; edited 1 time in total |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sun Jul 13, 2014 2:27 pm Post subject: |
|
|
| mv wrote: | | It is now written in perl and uses TCP sockets for IRC. |
mv ... hehe, more grist for the mills of nitpickery :) Oh, and good work as always, just taking a look-see now.
best ... khay |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3131
|
| Posted: Sun Jul 13, 2014 5:23 pm Post subject: |
|
|
Well, I've just stumbled over this thread, and....
Wouldn't 'at' be enough?
| Quote: |
eix -I at$
sys-process/at
Available versions: 3.1.13-r2 ~3.1.14 {pam selinux}
Installed versions: 3.1.13-r2(14:27:16 05/09/14)(pam -selinux)
Homepage: http://packages.qa.debian.org/a/at.html
Description: Queues jobs for later execution |
Honestly never needed it enough to bother with learning it's command line, but you might want to have a look |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sun Jul 13, 2014 5:38 pm Post subject: |
|
|
| szatox wrote: | | Wouldn't 'at' be enough? |
"at" cannot start a job "triggered" by the finishing or exitstatus of another, not to speak about wating on the finishing of several jobs. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Wed Jul 23, 2014 9:13 pm Post subject: |
|
|
A small update: Meanwhile, the most obvious bugs of "schedule" should be elminiated so that a beta-test phase can be announced.
It is now also possible (and reasonably configurable) to use file sockets instead of TCP sockets for IPC.
The advantages/disadvantages of this choice concerning securtiry/convenience are sketched in the README. Please, read it! |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Thu Jul 24, 2014 5:24 pm Post subject: |
|
|
| mv wrote: | Meanwhile I have found that what I was looking for is called Enterprise job scheduling,
It seems that there exist some professional (and rather expensive) solutions but no free solution yet. |
My argument has always been that so much of this is handled by a shell; ie batching and sequential/conditional execution, with dependencies handled via makefiles. Yes this is very granular, but that's the beauty of it. Each user only needs to know sh, and make is simply a direct (ie transparent) layer on top of that. In the enterprise arena, there's not much point in shutting down, and the interesting part is job-scheduling across a grid, though my feeling is that that's overcomplexified.
However as I've said, I've done something similar with workflows, and I do like the idea of batching jobs with dep chains, though as I said that's what make does too. Still you clearly have a use-case or you wouldn't have bothered to implement anything, and in that regard I like the idea of waiting for all the jobs the machine has to do and then shutting down.
With regard to the specifics, I think you should separate out the shutdown, so it can't ever happen when there are jobs still underway, by default. And certainly a normal user should not be able to move it around as mentioned in the readme.
I'd also switch the -l option to mean 'local' (ie insecure) and disable it by default. The UI can be made easier perhaps with a session daemon similar to ssh-agent, so that the user doesn't need to specify what type of backend they're working with when scheduling jobs, since that's really an admin task. It's early days, so it's not like there's any problem changing cli-switches. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri Jul 25, 2014 8:03 am Post subject: |
|
|
Thanks for your comments.
| steveL wrote: | | In the enterprise arena, there's not much point in shutting down, and the interesting part is job-scheduling across a grid, though my feeling is that that's overcomplexified. |
My aim was not to compete with enterprise servers but to solve my problems ;)
Anyway, if you make tcp not listen to a local host, you can even distribute jobs over the network, though you have to use shell or make or whatever if you want to resolve complex dependencies automatically: The idea of "schedule" is just to provide the necessary command line tool for IPC (which is hopefully flexible enough for whatever one needs). More or less, it is only a "backend", but plain shell is a good enough "frontend" for the purposes I have in mind.
| Quote: | | With regard to the specifics, I think you should separate out the shutdown, so it can't ever happen when there are jobs still underway, by default. |
This is just a question of writing the example: Of course, you could write in some shell | Quote: | | schedule wait && shutdown -h now |
instead. Maybe I should mention this in the README. | Quote: | | And certainly a normal user should not be able to move it around as mentioned in the readme. |
Even if he cannot move it around in the modified example, he could cancel the other jobs, setting their exit status to 0. There is not really an easy possibility to do a reliable authorization over sockets, and although I was thinking about using a server password, the implementation would be ugly, since one would have to establish a https-type connection if the password should not be transferred in clear text. Somehow it appeared to be overkill for me, since if security is an issue for a certain applicatoin case, one should use a local socket instead and do the authorization by its permissions.
| Quote: | | I'd also switch the -l option to mean 'local' (ie insecure) and disable it by default. |
I think it is rather intuitive for "local" to mean "local connection/mode" (hence "secure").
Concerning the default, I was thinking for quite a while, but since the main idea of "schedule" is to allow "shared" execution of tasks of different users/chroots, it would be counterintuitive to just disable this functionality by default. It should be very clear to everybody by its very nature what this implies concerning security.
| Quote: | | The UI can be made easier perhaps with a session daemon similar to ssh-agent, so that the user doesn't need to specify what type of backend they're working with when scheduling jobs, since that's really an admin task. |
No, it's a local task: You (as the user) must know whether you want to access the system-wide server (if there is any) or whether you want to run your own, independent of other users. So a shell variable containing the corresponding options appears to be the right "ssh-agent" information to me: The defaults of this variable can be set in a system-wide profile, according to the server running (if any). |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Fri Jul 25, 2014 7:03 pm Post subject: |
|
|
The new release schedule-0.10 extends the README correspondingly.
It also contains (still) surprisingly many essential bugfixes. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Sat Aug 09, 2014 12:26 pm Post subject: |
|
|
schedule-1.0 is released.
Finally, the security issues are solved satisfactory by using encryption.
Please see the current README for security-related recommendations.
Although tne version is 1.0, don't expect it to be completely bug-free, yet: This is the first (still experimental) release which I expect to be usable for production. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6747
|
| Posted: Mon Sep 15, 2014 3:19 pm Post subject: |
|
|
Meanwhile schedule-6.0.1 is reasonably matured.
I am surprised that I use it much more often than I had expected, originally: It really changed my workflow.
Prepending "s s" (s is an alias for "schedule") to any emerge command, any start of a video encryption, practically almost everything non-interactively, turned out to be rather convenient, even if no real scheduling is needed:
will give you an overview of all commands and in which directory/changroot/user you started them, the windows title is appropriate, you can see the exit status etc.
And once you really want to say: "and after this do ..." you have to no problems to do so.
I can really just recommend anybody to try whether this workflow is something for you too... whenever you are running non-interactive tasks, schedule seems to be the "right" thing to use ...  |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3339
Location: Rasi, Finland
|
| Posted: Thu Jul 29, 2021 9:51 pm Post subject: |
|
|
Well. Now I know this exists. Thanks mv!
Bumping old topic, but I think this needs more attention. Awesome little piece of software!
I didn't knew I needed this. I will definitely start using this!
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
Fitzcarraldo
Advocate
Joined: 30 Aug 2008
Posts: 2034
Location: United Kingdom
|
| Posted: Fri Jul 30, 2021 12:35 pm Post subject: |
|
|
I didn't know about Schedule. I had been looking for something like this in the past. Will have to try it out.
_________________
Clevo W230SS: amd64, VIDEO_CARDS="intel modesetting nvidia".
Compal NBLB2: ~amd64, xf86-video-ati. Dual boot Win 7 Pro 64-bit.
OpenRC udev elogind & KDE on both.
Fitzcarraldo's blog |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
