Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Free job scheduler [solved]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Sun Jun 08, 2014 8:49 am    Post subject: Free job scheduler [solved] Reply with quote

Perhaps the subject just uses the wrong terminology, but I do not know an appropriate term:

I am looking for a program which can "schedule tasks", but not in the sense of cron ("at a given time") but in the sense of sequences/conditionals/...

For instance, I want to start an emerge process. When this has finished succesfully, I want to start a kernel compilation in my 32bit chroot and emerge nvidia-drivers in the chroot, and simultaenously do the same on my main system.
When this is running, I find that when all has finished a "revdep-rebuild" on both systems should be scheduled.
And perhaps after a while, I change my mind and want to remove one of the two revdep-rebuilds from the schdule.

Yes, I know for some of these tasks, I can write shell scripts (although this is already not so easy with a 32bit chroots) or sequences in one shell command line. However, once such a sequence started running, I cannot modify it easily. In particular, it is not easy to "add" something to such a sequence at a certain place.

Yes, there are many ad-hoc workarounds: one could create/remove files as flags, use pgrep or simliar things, already "for later adding" execute a not-yet-written script within a shell script etc. I am not looking for such tips which all are somewhat hackish: I am looking for a "general" such scheduler which can do all this "cleanly" without requiring the user to remember the "schedule tree" and adapting the methods on how this tree loks like.

I am also aware that there is a "starter" script in the mv overlay which is able to schedule tasks sequentially, adding a new task "at the end".
However, this is not powerful enough for a "full" scheduler (with && || conditionals, adding tasks "in the middle", simultaneous execution of some tasks, removal of tasks, etc).

I suppose, using some "scheduler" daemon (e.g. basing on dbus) might be the right tool, 'though there are of course many problems with permissions/policies, and also the 32bit-chroot might be an issue with such a solution.

Anyway, perhaps such an animal (which at least can some of these things) already does exist?
Is it perhaps even built in into some existing DE and I just did not understand that this is the case?


Last edited by mv on Sun Jul 13, 2014 2:21 pm; edited 3 times in total
Back to top
View user's profile Send private message
Maitreya
Apprentice
Apprentice


Joined: 11 Jan 2006
Posts: 222

PostPosted: Sun Jun 08, 2014 9:44 am    Post subject: Reply with quote

Something like "at"?

man at
Back to top
View user's profile Send private message
Maitreya
Apprentice
Apprentice


Joined: 11 Jan 2006
Posts: 222

PostPosted: Sun Jun 08, 2014 9:45 am    Post subject: Reply with quote

And on the other end of the extreme you got buildbots/CI systems.
But I guess you want something in between?
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Sun Jun 08, 2014 1:08 pm    Post subject: Reply with quote

AFAIK, "at" triggers events only based on time. I want some "sequential" scheduling.

To make perhaps clearer what I mean, here is an example: Let us say, I have 10 commands which I want to execute in a certain order (some parallel, some conditional etc.) then I could write a shell script like e.g.
Code:
cmd1; cmd2 && {
  cmd3 &
  c3=$!
  if cmd4
  then cmd5
  else cmd6 ; cmd7
  fi &
  cmd8 && wait $c3
} && cmd9 || wait; cmd10

(the point is not, how exactly the shell-script looks like - it is a "random" example).

The above shell-script would already be "almost" the scheduler I like to have. However, it has the problem, that one the script is started, I cannot modify it anymore. For instance, if cmd2 is an long-running command (say, "emerge libreoffice"), and I decide for some reason in that moment that I want to replace the last line by
Code:
} && cmd9 || wait; cmd10a && cmd10; cmd11

I cannot do this without stopping and then restarting the whole script (and thus "emerge libreoffice"), modifying it correspondingly so that cmd1 is not executed again. This is not the "natural" thing to do when all I want is to modify the "scheduling" of some command in some distant "future".

Maitreya wrote:
buildbots/CI systems.

I do not mean building some particular project: I really mean "random" tasks which should not all be started in parallel (partially due to processor/memory restrictions, partially due to the fact that some tasks can only be started after others have succesfully completed). A typical example for a gentoo user is kernel compilation followed by emerge @module-rebuild followed by initiating a backup followed by rebooting. But suddenly it turns out that before making the backup or before the rebooting something other important has to be done...)
Back to top
View user's profile Send private message
khayyam
Advocate
Advocate


Joined: 07 Jun 2012
Posts: 2950

PostPosted: Sun Jun 08, 2014 1:11 pm    Post subject: Re: Looking for a "task scheduler" Reply with quote

mv wrote:
Perhaps the subject just uses the wrong terminology, but I do not know an appropriate term: I am looking for a program which can "schedule tasks", but not in the sense of cron ("at a given time") but in the sense of sequences/conditionals/...

mv ... I think the term you are looking for is "program", but we can see the obvious catch with such a term as this "program program" exists already in a variety of forms (every "programming" language has some form of "sequences/conditionals", etc). Yes, I know this is not what you have in mind, but I think the reason we don't have something (even to the extent of a descriptive term) of this nature is that this is what programming logic provides, so such a "program program" is in the realms of meta-programming.

best ... khay
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Sun Jun 08, 2014 2:34 pm    Post subject: Re: Looking for a "task scheduler" Reply with quote

khayyam wrote:
]mv ... I think the term you are looking for is "program"

The difficulty is that this "programm" would have the need to be modified dynamically, and this is where all standard languages fail.
In fact, the "language" for this "program" can be very poor (the example I posted already contains all directives which it would have to know), but the point is the possibility to modify it dynamically.

Thus, here is the way how I thought about implementing it: There are two executables "schedule" and "start".
Then you would call e.g.
Code:
schedule 1 emerge ...
schedule 2  'sh -c "cd /usr/src/linux && make && make install"'
schedule 3 ...

this would not start the corresponding tasks but just cause some daemon to record these commands somewhere. Once
Code:
start 2
is executed, the recorded task "2" would run. Then you could just run the shell-script in the example above, just with "cmd1" ... "cmd10" replaced by "start 1" ... "start 10".

The crucial difference to running the commands directly is that, since the programs are run by some daemon, you can just stop the scheduling script, modify it, and restart it again, without stopping the actuall tasks.

Of course, there are a lot of difficulties in this approach: The tasks might not be appropriate to be run separate from a terminal, there are issues if it should be possible to run "schedule" by different users or even on different chroots etc.

I can imagine many approaches how to actually implement "schedule" and "start":
1. "schedule" could be somethnig like offereing a service over dbus and start is just sending a message (this approach has the problem that either you can use only a session dbus with limited permissions or, if you want to use the system dbus, you must modify policy files on the run).
2. Instead of dbus, one could also use "magic files" in some directory to communicate (this is actually what the "starter" implementation does).
3. "schedule" and "start" might be implemented using (or reimplementing parts of) screen, tmux, nohup.
4. One could also make the starting prorgam "smarter" by catching signals which stop the starting program but not the running child processes (not sure how to implement this, but certainly this is possible).

Anyway, all is a lot of work. Maybe somebody has already implemented this, or there exists other solutions which follow a completely different approach for a "scheduler".
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Sun Jun 08, 2014 3:20 pm    Post subject: Reply with quote

Meanwhile I have found that what I was looking for is called Enterprise job scheduling,
It seems that there exist some professional (and rather expensive) solutions but no free solution yet.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 8417
Location: Somewhere over Atlanta, Georgia

PostPosted: Sun Jun 08, 2014 4:08 pm    Post subject: Reply with quote

I have briefly looked at OpenJCS in the past. Does something like this fit your requirements?

- John
_________________
I can confirm that I have received between 0 and 999 National Security Letters.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Sun Jun 08, 2014 6:42 pm    Post subject: Reply with quote

Thanks. OpenJCS looks like overkill to me (apache, php, java, zope), but the result also seems to be a huge superset of what I had requested...
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 2952
Location: UK

PostPosted: Tue Jun 10, 2014 9:34 pm    Post subject: Reply with quote

This all sounds like something that a makefile and a few short scripts could do... the restarting/editing in-flight part could be done with inotifywait.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Wed Jun 11, 2014 9:50 am    Post subject: Reply with quote

Ant P. wrote:
This all sounds like something that a makefile and a few short scripts could do... the restarting/editing in-flight part could be done with inotifywait.

Yes, I was never claiming that it is hard to do. In fact, with the "starter" script I already have a partial solution (Starter uses one file to "communicate" which is checked on a timer basis instead of inotifywait). However, "starter" is limited to sequential execution, and later reordering is not possible with this concept. It would require a few days of work to write a "generic" solution. This is not extraordinarily much, and so I thought maybe somebody else might already have done it.

The "easy" implementation I was actually thinking of is the following: You run in different shells (possibly with different users/chroots) the commands "schedule [command you want to schedule]". The commands are not immediately executed but wait first for a signal of some kind; when they are finished, they signal back there exit status.
The "scheduler" then just sends the corresponding signals and collects the results. It can be controlled by a simple shell script which, if required, is just stopped rewritten and restartet...
For the signals there are several possibilities: They can be "unix" (kill) signals, "magic" files, sockets, dbus, ... each choice has some disadvantages: kill signals might not work within chroots (at least for hardened kernels) and there are permission issues, common files or sockets also have problems with chroots (and possibly raise permission issues if world-writable directories are created for them), dbus also has permission problems (and additionally needs sockets to communicate).
Certainly, everything can be solved, but implementing a "safe" solution probably requires a few days of work...
The "starter" script does a lot of these things already quite safe (with file-locking etc.), but its concept is too simple to be extended for a "full" editing of a dependence tree. After all, the "starter" script was written only in half an afternoon...
Back to top
View user's profile Send private message
desultory
Administrator
Administrator


Joined: 04 Nov 2005
Posts: 8229

PostPosted: Thu Jun 12, 2014 3:17 am    Post subject: Reply with quote

mv wrote:
AFAIK, "at" triggers events only based on time. I want some "sequential" scheduling.
Might I suggest a little light reading? http://linux.die.net/man/1/batch

Additionally, incrond (sys-process/incron) might be of interest to you, depending upon quite how you want to work and how you want your solution to work
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Mon Jul 07, 2014 2:54 pm    Post subject: Reply with quote

Sorry, somehow I missed that the previous reply...
desultory wrote:
Might I suggest a little light reading? http://linux.die.net/man/1/batch

No, this is not what I am looking for: It is missing the "dependencies" of the tasks.
Quote:
Additionally, incrond (sys-process/incron) might be of interest to you

This seems closer, but could only be used as a back-end for the front-end I am looking for: One would need a bunch of wrapper scripts which generate and control "magic" files. This is essentially what the "starter" script does (not using inotify but mucher simpler by checking in regular intevals).
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Sun Jul 13, 2014 2:20 pm    Post subject: Reply with quote

After one evening and one morning of work, the presumably first free job scheduler sees the light of the world:

Here it is.

As usual, gentoo users can install it by using the mv overlay (available over layman).
It is now written in perl and uses TCP sockets for IPC. It would probably make sense to rewrite at least the server part in C or C++, but it is sufficient for my current needs.

Probably you can still find quite a few bugs, since testing was only very rudimentarily.

Edit: Fix typo


Last edited by mv on Sun Jul 13, 2014 4:28 pm; edited 1 time in total
Back to top
View user's profile Send private message
khayyam
Advocate
Advocate


Joined: 07 Jun 2012
Posts: 2950

PostPosted: Sun Jul 13, 2014 2:27 pm    Post subject: Reply with quote

mv wrote:
It is now written in perl and uses TCP sockets for IRC.

mv ... hehe, more grist for the mills of nitpickery :) Oh, and good work as always, just taking a look-see now.

best ... khay
Back to top
View user's profile Send private message
szatox
l33t
l33t


Joined: 27 Aug 2013
Posts: 673

PostPosted: Sun Jul 13, 2014 5:23 pm    Post subject: Reply with quote

Well, I've just stumbled over this thread, and....
Wouldn't 'at' be enough?
Quote:

eix -I at$
sys-process/at
Available versions: 3.1.13-r2 ~3.1.14 {pam selinux}
Installed versions: 3.1.13-r2(14:27:16 05/09/14)(pam -selinux)
Homepage: http://packages.qa.debian.org/a/at.html
Description: Queues jobs for later execution

Honestly never needed it enough to bother with learning it's command line, but you might want to have a look
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Sun Jul 13, 2014 5:38 pm    Post subject: Reply with quote

szatox wrote:
Wouldn't 'at' be enough?

"at" cannot start a job "triggered" by the finishing or exitstatus of another, not to speak about wating on the finishing of several jobs.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Wed Jul 23, 2014 9:13 pm    Post subject: Reply with quote

A small update: Meanwhile, the most obvious bugs of "schedule" should be elminiated so that a beta-test phase can be announced.

It is now also possible (and reasonably configurable) to use file sockets instead of TCP sockets for IPC.
The advantages/disadvantages of this choice concerning securtiry/convenience are sketched in the README. Please, read it!
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 3648
Location: The Peanut Gallery

PostPosted: Thu Jul 24, 2014 5:24 pm    Post subject: Reply with quote

mv wrote:
Meanwhile I have found that what I was looking for is called Enterprise job scheduling,
It seems that there exist some professional (and rather expensive) solutions but no free solution yet.

My argument has always been that so much of this is handled by a shell; ie batching and sequential/conditional execution, with dependencies handled via makefiles. Yes this is very granular, but that's the beauty of it. Each user only needs to know sh, and make is simply a direct (ie transparent) layer on top of that. In the enterprise arena, there's not much point in shutting down, and the interesting part is job-scheduling across a grid, though my feeling is that that's overcomplexified.

However as I've said, I've done something similar with workflows, and I do like the idea of batching jobs with dep chains, though as I said that's what make does too. Still you clearly have a use-case or you wouldn't have bothered to implement anything, and in that regard I like the idea of waiting for all the jobs the machine has to do and then shutting down.

With regard to the specifics, I think you should separate out the shutdown, so it can't ever happen when there are jobs still underway, by default. And certainly a normal user should not be able to move it around as mentioned in the readme.

I'd also switch the -l option to mean 'local' (ie insecure) and disable it by default. The UI can be made easier perhaps with a session daemon similar to ssh-agent, so that the user doesn't need to specify what type of backend they're working with when scheduling jobs, since that's really an admin task. It's early days, so it's not like there's any problem changing cli-switches.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Fri Jul 25, 2014 8:03 am    Post subject: Reply with quote

Thanks for your comments.
steveL wrote:
In the enterprise arena, there's not much point in shutting down, and the interesting part is job-scheduling across a grid, though my feeling is that that's overcomplexified.

My aim was not to compete with enterprise servers but to solve my problems ;)
Anyway, if you make tcp not listen to a local host, you can even distribute jobs over the network, though you have to use shell or make or whatever if you want to resolve complex dependencies automatically: The idea of "schedule" is just to provide the necessary command line tool for IPC (which is hopefully flexible enough for whatever one needs). More or less, it is only a "backend", but plain shell is a good enough "frontend" for the purposes I have in mind.
Quote:
With regard to the specifics, I think you should separate out the shutdown, so it can't ever happen when there are jobs still underway, by default.

This is just a question of writing the example: Of course, you could write in some shell
Quote:
schedule wait && shutdown -h now
instead. Maybe I should mention this in the README.
Quote:
And certainly a normal user should not be able to move it around as mentioned in the readme.
Even if he cannot move it around in the modified example, he could cancel the other jobs, setting their exit status to 0. There is not really an easy possibility to do a reliable authorization over sockets, and although I was thinking about using a server password, the implementation would be ugly, since one would have to establish a https-type connection if the password should not be transferred in clear text. Somehow it appeared to be overkill for me, since if security is an issue for a certain applicatoin case, one should use a local socket instead and do the authorization by its permissions.
Quote:
I'd also switch the -l option to mean 'local' (ie insecure) and disable it by default.

I think it is rather intuitive for "local" to mean "local connection/mode" (hence "secure").
Concerning the default, I was thinking for quite a while, but since the main idea of "schedule" is to allow "shared" execution of tasks of different users/chroots, it would be counterintuitive to just disable this functionality by default. It should be very clear to everybody by its very nature what this implies concerning security.
Quote:
The UI can be made easier perhaps with a session daemon similar to ssh-agent, so that the user doesn't need to specify what type of backend they're working with when scheduling jobs, since that's really an admin task.

No, it's a local task: You (as the user) must know whether you want to access the system-wide server (if there is any) or whether you want to run your own, independent of other users. So a shell variable containing the corresponding options appears to be the right "ssh-agent" information to me: The defaults of this variable can be set in a system-wide profile, according to the server running (if any).
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Fri Jul 25, 2014 7:03 pm    Post subject: Reply with quote

The new release schedule-0.10 extends the README correspondingly.
It also contains (still) surprisingly many essential bugfixes.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Sat Aug 09, 2014 12:26 pm    Post subject: Reply with quote

schedule-1.0 is released.

Finally, the security issues are solved satisfactory by using encryption.
Please see the current README for security-related recommendations.

Although tne version is 1.0, don't expect it to be completely bug-free, yet: This is the first (still experimental) release which I expect to be usable for production.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4710

PostPosted: Mon Sep 15, 2014 3:19 pm    Post subject: Reply with quote

Meanwhile schedule-6.0.1 is reasonably matured.

I am surprised that I use it much more often than I had expected, originally: It really changed my workflow.

Prepending "s s" (s is an alias for "schedule") to any emerge command, any start of a video encryption, practically almost everything non-interactively, turned out to be rather convenient, even if no real scheduling is needed:
Code:
s l

will give you an overview of all commands and in which directory/changroot/user you started them, the windows title is appropriate, you can see the exit status etc.
And once you really want to say: "and after this do ..." you have to no problems to do so.

I can really just recommend anybody to try whether this workflow is something for you too... whenever you are running non-interactive tasks, schedule seems to be the "right" thing to use ... :wink:
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum