Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
kdbus in the kernel
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 7, 8, 9 ... 25, 26, 27  Next  
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
depontius
Advocate
Advocate


Joined: 05 May 2004
Posts: 3509

PostPosted: Mon Jan 26, 2015 8:15 pm    Post subject: Reply with quote

khayyam wrote:
I could have opted not to enter the kingdom of gnucash and instead entered the forest of sqlite ...


But sqlite is a lightweight database, not a double-entry ledge accounting system. There's just a bit of difference.

OTOH, "grisb" is in portage, and it will read gnucash files - but it's not a double-entry ledger. This pits financial goodness against computing goodness. However grisb can export QIF and CSV, so it might end up being an export tool if I really need to get out of gnucash.

In the meantime, as mentioned before, I've determined that in my situation the gnome-keyring can't work, because it's directory was a dangling symlink, and I can certainly make it so, again. Better yet, I can make it dangle into a place where as an ordinary user, I can't write, so it can't be automagically fixed by any sort of smart scripting. In this way, it's just dead, useless code. I have to think through if there is any possible exposure, here.

Come to think of it, given that I've been running for years with an impossible-to-work gnome-keyring, maybe the best shim of all is a dead shim. Maybe that's what I want to add - a way to close the hole caused by an unused feature.
_________________
.sigs waste space and bandwidth
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Mon Jan 26, 2015 8:25 pm    Post subject: Reply with quote

depontius wrote:
khayyam wrote:
I could have opted not to enter the kingdom of gnucash and instead entered the forest of sqlite ...

But sqlite is a lightweight database, not a double-entry ledge accounting system. There's just a bit of difference.

depontius ... that's not the point though, the point is "design", and where that design will lead (dead ends, or what-have-you). Now, unless you are saying that its impossible to make a "double-entry ledge accounting system" without inevitably relying on dbus, then that point is valid.

best ... khay
Back to top
View user's profile Send private message
depontius
Advocate
Advocate


Joined: 05 May 2004
Posts: 3509

PostPosted: Mon Jan 26, 2015 8:43 pm    Post subject: Reply with quote

khayyam wrote:
depontius wrote:
khayyam wrote:
I could have opted not to enter the kingdom of gnucash and instead entered the forest of sqlite ...

But sqlite is a lightweight database, not a double-entry ledge accounting system. There's just a bit of difference.

depontius ... that's not the point though, the point is "design", and where that design will lead (dead ends, or what-have-you). Now, unless you are saying that its impossible to make a "double-entry ledge accounting system" without inevitably relying on dbus, then that point is valid.


In fact, from the ebuild it appears that gnucash is equipped to use sqlite, postgres, or mysql. Funny thing is, I don't see something in the ebuild requiring at least one of those to be set, so I'm not sure if it's native store is really one of those, or if they're really there for import.

The dbus is used by gnome-keyring, which is used by libsecret, which is used to store passwords for accessing home banking web sites, which I'm not doing. So that stuff is dead weight to me.

Over ten years ago, when I picked up gnucash, I simply wanted a double-entry ledge bookkeeping program for my personal finances. I wasn't as worried about the software design side of things. I've been cognizant of security issues, but at the time thought of them as being more implementation issues. It's not that many years that events have forced me to think of problems (other than Windows, of course) being architectural and even worse, platform-architectural issues.

I am where I am, and for the moment I'm going to make sure gnome-keyring can't work. In the longer term I may look at some "dead shims" to make sure that not only can they not work, they can't do anything at all.
_________________
.sigs waste space and bandwidth
Back to top
View user's profile Send private message
juggling_ben
n00b
n00b


Joined: 25 Nov 2007
Posts: 15

PostPosted: Tue Jan 27, 2015 2:59 am    Post subject: Reply with quote

depontius wrote:

The dbus is used by gnome-keyring, which is used by libsecret, which is used to store passwords for accessing home banking web sites, which I'm not doing. So that stuff is dead weight to me.


Hello fellow gnucash lover

Probably getting offtopic, but just playing around, gnucash doesn't really depend on libsecret or gnome-keyring. It will build fine without it (ie outside of portage). There is no way to disable it at configure time, however, and will just automatically detect it if it exists.

I've created an enhancement: bug 537930
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Tue Jan 27, 2015 3:02 am    Post subject: Reply with quote

steveL wrote:
Mind, I'm all for an easy-ipc lib; I just think it's only interesting if the admin can configure it, similarly to /etc/services, on a domain-specific basis.

tclover wrote:
This shouldn't be difficult to implement with such an easy API... I remove the "difficult API" said previously!

Yay :-)
Quote:
I second a lib for that kind of things... to give at least a "named service" (string) translated to a (TIPC) "named port" {TYPE,INSTANCE} (two 32-bits integer.) It will certainly help...

Indeed it'd be simple; though I think it should be more extensive so we could have say (for the two common uses):
Code:
# notify on file if required
email: file "/var/somepath" with data: count
mixer: [
  ctrl: mqueue "alsa/mixer"
  data: tipc 1000:0:1000
]/mixer
audio: [
  ctrl: fifo "/var/run/alsa/volume"
  data: tipc 2000:0:1000
]/audio

The point being that we should be able to tell a generic API that we want to use one of the N variants of IPC available, including fifos, mqueues and AF_UNIX sockets, depending on the service and the semantics (so a service might not support being used with "file" for example, which would result in a hard error at startup, whatever else happened.) Really this starts to overlap with a stay-around pid2 as well, since this implies dependency and configuration checks.

Though I think that's also necessary, if we consider containers and cgroups, quite apart from monitoring which is the use-case I'd like satisfied more neatly.

In any event, the app shouldn't care since it's down to the lib to establish the channel, so long as the semantic is fulfilled. The admin or distro can then tweak in the light of concrete experience, and system constraints.

RT signals are also an important part of the mix, though that tends to be within a specific domain, usually within one application amongst threads, or related processes. Really it's up to the app to wire that, but we need to consider it wrt mqueues, which can be configured to trigger RT signals, as well as a thread. So the app may need to know what mechanism is in-use, in specialist situations (which we'd likely just write a simple shim to the existing lib for.)
Quote:
What about a "bearer" (link) implementation?

I don't think we need to worry about that, since our primary focus is local-machine. TIPC gives us the ability to extend that across the LAN (at least for SOHO users) very easily, should we need to extend in the future; atm however the focus is on efficient intra-node comms, certainly as far as the proposed kdbus changes go, which are at best very speculative in ABI terms. (Not something we want to base our computing platforms on, at least round here.)
Quote:
I'm yet to finish reading the specs, but, so far so good. The API looks almost more easy to grasp... for even the "simpler" (than D-Bus) kDbus.. the ZONE/CLUSTER/NODE address space & one-to-many(multicast+broadcast)/many-to-one/one-to-one(unicast (low latency)) make kDbus' domain/bus/connection counterpart... almost a toy.

Yeah, I need to get my head round, and it would be good to document, the whole addressing ranges idea, and how that works in practice with simple pub/sub testcases.
Quote:
(Some facts: kernel(good config)+JACK gives a ~20ms latency; or ~<10ms for RT kernel.)

Direct peer-to-peer (or one-to-one) communication (for low latency) on ~weak~ hardware achieve a ~100us for Round Trip Time (send+receive message) is not that bad... (The 4us of kDbus is not trustworthy considering the politics of systemD cabals.)

These figures are from testing using ticputils? Cool :)

The other aspect of it is not to send blobs of data down what are really meant to be control channels, either at lib or app level (metadata iow.) For instance it would be foolish to start using a different IPC, be that kdbus or anything else, for X direct-rendering data. Similarly it's foolish to start using anything other than jack to actually handle the audio data, though as above we might want a way to tap into it (we'd be using "jack/foo" not "alsa/foo" for anything pro-audio, though.)

As Ant said, and the kernel folks continually pointed out in the previously-linked dialogue, if you rethink the uses of the metadata bus, with knowledge of what else is already provided, you come up with a much cleaner, and much more efficient design, since you're not shoving everything down it. Instead you let the kernel multiplex for you, since that is its main task in life.
Quote:
I added net-misc/tipcutils-2.0.3 on my overlay (it has a cmdline utils for AF_TIPC ("ports") sockets creation compared to version 2.0.0 available in the official tree); and dev-perl/IO-Socket-TIPC in order to test a few things for those interested...

Nice work :-) Thanks muchly.

Would be good to see what mv thinks of the perl API.
Back to top
View user's profile Send private message
tclover
Guru
Guru


Joined: 10 Apr 2011
Posts: 516

PostPosted: Tue Jan 27, 2015 11:28 am    Post subject: Reply with quote

steveL wrote:
Indeed it'd be simple; though I think it should be more extensive so we could have say (for the two common uses):
Code:
# notify on file if required
email: file "/var/somepath" with data: count
mixer: [
  ctrl: mqueue "alsa/mixer"
  data: tipc 1000:0:1000
]/mixer
audio: [
  ctrl: fifo "/var/run/alsa/volume"
  data: tipc 2000:0:1000
]/audio

The point being that we should be able to tell a generic API that we want to use one of the N variants of IPC available, including fifos, mqueues and AF_UNIX sockets, depending on the service and the semantics (so a service might not support being used with "file" for example, which would result in a hard error at startup, whatever else happened.) Really this starts to overlap with a stay-around pid2 as well, since this implies dependency and configuration checks.


That's fine but... it's a little hard to read. So, what about--hoping to keep `/etc/services' simplicity--
Code:
"NAME"    "FACILITY"    "[OPTIONS]" # Just like the extra fields in `/etc/services'
"NAME"    "ADRESS:PORT/IPC"    "[OPTIONS]" # To put it in another manner
ladi    128.1.1000/tipc # To lmake an example (colon `:' or dot `.'?)
ladi    128.1/tipc  [RDM|DGRAM|STREAM|SEQPACKET] # To illustrate a possible option (socket type)
ladi    128.1:STREAM/tipc  [OPTIONS] # Or <ADDRESS:PORT|SOCKET(-TYPE)/IPC> makes more sense?
# WARN: 0 in <ZONE.CLUSTER.NODE> is special and are not permited in a "node" address or "named port" or "name service"

EDIT(PREVIOUS WITH):TIPC Network Address: This address is a 32-bit integer, structured into three fields, zone (8 MSB), cluster, (12 bits), and node (12 LSB).

This syntax is more readable and has the optional flexibility to add extra fields for particular IPC.

steveL wrote:
Though I think that's also necessary, if we consider containers and cgroups, quite apart from monitoring which is the use-case I'd like satisfied more neatly.


Yes, this is what I have in mind as a primary concern monitoring in order to supervise and control the availability of services/functionalities. And this what most users want when using D-Bus. However...

steveL wrote:
The other aspect of it is not to send blobs of data down what are really meant to be control channels, either at lib or app level (metadata iow.) For instance it would be foolish to start using a different IPC, be that kdbus or anything else, for X direct-rendering data. Similarly it's foolish to start using anything other than jack to actually handle the audio data, though as above we might want a way to tap into it (we'd be using "jack/foo" not "alsa/foo" for anything pro-audio, though.)


Indeed, it would be foolish to jump blindly in a new toy when existant protocols satisfy particular needs e.g X or JACK. Although with a low latency & data capable communication IPC (TIPC set max packet size to 66000 bytes),--this is where some are waiting to threw away licence concerns (who said RH already?... Oracle?... the starting block are full of chalengers),--data transactions become an interesting perspective. I provided the above figures to put some perspective on the latency concerns. (I did not, yet, have the time to test TIPC--TIPC latency figure is on the spec papaer; JACK latency data are commonly found on the intertubes.) Of course, some politics become almost transparent with this... as if audio transport data is the _real_ motive to write a new in-kernel IPC when there are at least:
* a low latency & network friendly audio transport (where mixer concerns vanish);
* an already good low latency in-kernel IPC;
good implementation available.

But this just provide (other) justifications of their... pitiful failure with PulseAudio... when it was _first_ sold as a mixer capable solution implying that... as if there weren't any available.
Stopping here. There are way too many deceiving schemes behind this, and GKH finally said it all in a short sentence--"trust us, we know what we're doing!"
_________________
home/:mkinitramfs-ll/:supervision/:e-gtk-theme/:overlay/


Last edited by tclover on Tue Jan 27, 2015 1:59 pm; edited 4 times in total
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6095
Location: Dallas area

PostPosted: Tue Jan 27, 2015 11:52 am    Post subject: Reply with quote

Quote:
On Tue, 20 Jan 2015 09:13:59 +0800
> That's because people have not done anything really needing performance
> on the desktop over D-Bus in the past due to how slow the current
> implementation is.

The desktop is a performance critical environment, even though certain
desktop developers think 2GB is a bit small to run an application and
5000 dbus transactions to start something is "ok"

Alan


LoL
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6095
Location: Dallas area

PostPosted: Tue Jan 27, 2015 11:59 am    Post subject: Reply with quote

tclover wrote:
There are way too many deceiving schemes behind this,


Indeed

Quote:
and GKH finally said it all in a short sentence--"trust us, we know what we're doing!"


We know what you're trying to do, too...subvert the kernel and put it under RH's control.

Trust you...don't make me laugh.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
WWWW
Tux's lil' helper
Tux's lil' helper


Joined: 30 Nov 2014
Posts: 143

PostPosted: Tue Jan 27, 2015 2:15 pm    Post subject: Reply with quote

To solve this problem the IPC one, engineers should implement IPC in hardware.

CPU designers should come up with a cpu layer that has hardwired IPC paths, within cpu, or even extending to cpu-mem.

This way the complexity of this issue would be resolved. IPC appears to be a stand alone paradigm unrelated to software, arch, languge etc.

Imagine the posibilities!! Real time ipc, zero latency, etc...
Back to top
View user's profile Send private message
ct85711
Veteran
Veteran


Joined: 27 Sep 2005
Posts: 1791

PostPosted: Tue Jan 27, 2015 9:15 pm    Post subject: Reply with quote

As much it would be nice to see a hardware based IPC, I don't think it's feesible. My understanding of IPC is that if you do hardware based, it's going to end up as static routes. This in turn effectively boils down to Apple, Google, M$ paying for static routes for their software only, and turn into hardware lock-in. So I'd have to say, No to hardware-based IPC.

Now my understanding of IPC can also be wrong, which can be the case.
Back to top
View user's profile Send private message
tclover
Guru
Guru


Joined: 10 Apr 2011
Posts: 516

PostPosted: Wed Jan 28, 2015 11:14 am    Post subject: Reply with quote

tclover wrote:
steveL wrote:
Indeed it'd be simple; though I think it should be more extensive so we could have say (for the two common uses):
Code:
# notify on file if required
email: file "/var/somepath" with data: count
mixer: [
  ctrl: mqueue "alsa/mixer"
  data: tipc 1000:0:1000
]/mixer
audio: [
  ctrl: fifo "/var/run/alsa/volume"
  data: tipc 2000:0:1000
]/audio

The point being that we should be able to tell a generic API that we want to use one of the N variants of IPC available, including fifos, mqueues and AF_UNIX sockets, depending on the service and the semantics (so a service might not support being used with "file" for example, which would result in a hard error at startup, whatever else happened.) Really this starts to overlap with a stay-around pid2 as well, since this implies dependency and configuration checks.


That's fine but... it's a little hard to read. So, what about--hoping to keep `/etc/services' simplicity--
Code:
"NAME"    "FACILITY"    "[OPTIONS]" # Just like the extra fields in `/etc/services'
"NAME"    "ADRESS:PORT/IPC"    "[OPTIONS]" # To put it in another manner
ladi    128.1.1000/tipc # To lmake an example (colon `:' or dot `.'?)
ladi    128.1/tipc  [RDM|DGRAM|STREAM|SEQPACKET] # To illustrate a possible option (socket type)
ladi    128.1:STREAM/tipc  [OPTIONS] # Or <ADDRESS:PORT|SOCKET(-TYPE)/IPC> makes more sense?
# WARN: 0 in <ZONE.CLUSTER.NODE> is special and are not permited in a "node" address or "named port" or "name service"

EDIT(PREVIOUS WITH):TIPC Network Address: This address is a 32-bit integer, structured into three fields, zone (8 MSB), cluster, (12 bits), and node (12 LSB).

This syntax is more readable and has the optional flexibility to add extra fields for particular IPC.

steveL wrote:
Though I think that's also necessary, if we consider containers and cgroups, quite apart from monitoring which is the use-case I'd like satisfied more neatly.


Yes, this is what I have in mind as a primary concern monitoring in order to supervise and control the availability of services/functionalities. And this what most users want when using D-Bus. However...

steveL wrote:
The other aspect of it is not to send blobs of data down what are really meant to be control channels, either at lib or app level (metadata iow.) For instance it would be foolish to start using a different IPC, be that kdbus or anything else, for X direct-rendering data. Similarly it's foolish to start using anything other than jack to actually handle the audio data, though as above we might want a way to tap into it (we'd be using "jack/foo" not "alsa/foo" for anything pro-audio, though.)


Indeed, it would be foolish to jump blindly in a new toy when existant protocols satisfy particular needs e.g X or JACK. Although with a low latency & data capable communication IPC (TIPC set max packet size to 66000 bytes),--this is where some are waiting to threw away licence concerns (who said RH already?... Oracle?... the starting block are full of chalengers),--data transactions become an interesting perspective. I provided the above figures to put some perspective on the latency concerns. (I did not, yet, have the time to test TIPC--TIPC latency figure is on the spec papaer; JACK latency data are commonly found on the intertubes.) Of course, some politics become almost transparent with this... as if audio transport data is the _real_ motive to write a new in-kernel IPC when there are at least:
* a low latency & network friendly audio transport (where mixer concerns vanish);
* an already good low latency in-kernel IPC;
good implementation available.

But this just provide (other) justifications of their... pitiful failure with PulseAudio... when it was _first_ sold as a mixer capable solution implying that... as if there weren't any available.
Stopping here. There are way too many deceiving schemes behind this, and GKH finally said it all in a short sentence--"trust us, we know what we're doing!"


EDIT2:
SteveL wrote:
Would be good to see what mv thinks of the perl API.

Just take a look to the module doc (`perldoc $PATH_TO_MODULE') to see how easy it is to actually create connection[-less] sockets and then send "messages" easily with a couple of lines. Couldn't be easier... I don't know if you ever tried to write something for D-Bus, be it directly or indirectly with language binding,--I did in python,-- to see what I mean. It's not exactly difficult,--well, this depends on what the "client" would be doing,--but it's far from being strait forward and simple. And there are test programs (in C for server, client et al along with doc for this end) in the source files of tipcutils package. Should be easy enough to write anything on it...
_________________
home/:mkinitramfs-ll/:supervision/:e-gtk-theme/:overlay/
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6095
Location: Dallas area

PostPosted: Fri Feb 06, 2015 6:04 pm    Post subject: Reply with quote

I've been keeping up with the mailing list and it's getting funny.

People are questioning the speed of kdbus and now GKH is saying that they don't want kdbus for the speed it's for the other things

I do remember when they were pushing for it simply for the speed aspect, and that includes comments from that lying POS GKH.
Talk about moving the goal posts.

The good thing, is if it's so transparent that I can see it, I'm pretty sure others do to.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
depontius
Advocate
Advocate


Joined: 05 May 2004
Posts: 3509

PostPosted: Fri Feb 06, 2015 6:29 pm    Post subject: Reply with quote

Anon-E-moose wrote:
I've been keeping up with the mailing list and it's getting funny.

People are questioning the speed of kdbus and now GKH is saying that they don't want kdbus for the speed it's for the other things

I do remember when they were pushing for it simply for the speed aspect, and that includes comments from that lying POS GKH.
Talk about moving the goal posts.

The good thing, is if it's so transparent that I can see it, I'm pretty sure others do to.


I follow it occasionally, but not terribly faithfully. The stuff I've been noticing more about is handling of metadata, whether kdbus connections are connectionless or not, and whether the in-between spot they seem to be aiming at constitutes a giant security problem waiting to happen.

As for the talking-out-both-sides nature of the debate, I've had that happen to me. In one case, probably 6-9 months ago I was arguing Unix Philosophy vs systemd. One responder argued the Unix Way was outmoded, obsolete, and holding Linux back. Another responder on the same subthread argued that SysV-Init was flawed from a Unix Philosophy point of view, and systemd was actually more adherent to the Unix Way, once you realized how modular it really is.

Yes, they argue both sides of every issue, but never against each other.
_________________
.sigs waste space and bandwidth
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6095
Location: Dallas area

PostPosted: Tue Feb 10, 2015 11:06 am    Post subject: Reply with quote

I'm waiting for the reality of what it means to be included in the kernel to hit the kdbus devs upside the head.

They keep on saying "we'll be able to change things down the road" not little things but major interface changes
such as whether to include metadata or not depending on how fast the kdbus code is.
And the kernel devs have been trying to tell them that things won't change if it ever gets to be included.
So far those statements seem to have gone right over the kdbus devs heads.

They seem to think that because they "created" the code they'll be able to change things as if it were user-space code.
We know different, we've heard "we don't break user-space" for a long time.
Hell, it's the main reason that Linus banned Kay from being able to submit any patches to the kernel, he got tired of it not sinking in to Kay.

Right now, I'm still seeing the need for kdbus re:speed being questioned.

Oh well, it'll be an interesting train wreck if it ever gets that far.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
depontius
Advocate
Advocate


Joined: 05 May 2004
Posts: 3509

PostPosted: Tue Feb 10, 2015 5:59 pm    Post subject: Reply with quote

The 3.19 kernel was released last night. I built and booted it this morning, so I could build AMDKFA. Even if I don't have any HSA userspace yet, I just wanted to get this far. (I'm having AMDIOMMU_V2 problems that are probably BIOS-related, but that's another story.)

In the release article on Phoronix they were talking about how once Linus said that 3.20 probably wouldn't happen, and it would be 4.0 instead. It appears that hasn't happend, whereupon a call rang out for 4.0, so they could break that nasty backwards compatibility. No mention of systemd, but I suspect it was behind the covers. Anyone for replacing syscalls with a kdbus message?
_________________
.sigs waste space and bandwidth
Back to top
View user's profile Send private message
229566
Tux's lil' helper
Tux's lil' helper


Joined: 16 Aug 2010
Posts: 127

PostPosted: Tue Feb 10, 2015 6:38 pm    Post subject: Reply with quote

depontius wrote:
Anyone for replacing syscalls with a kdbus message?


Ugh.

But in another article from Phoronix, it would appear that live kernel patching lands in 3.20. That's something I'm looking forward to.

Personally I believe it would make little sense to have a major version bump now. What kind of major new development will the kernel see? kdbus? That hasn't happened yet. Frankly I don't even remember if there was any when 2.6.x switched to 3.x, I think it was just for some numbering convenience as 2.6 part was supposed to stay for a while.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54096
Location: 56N 3W

PostPosted: Tue Feb 10, 2015 7:04 pm    Post subject: Reply with quote

GrueXYZ,

Some have it that 3.0 was to celebrate the dawn of the third decade of the kernel.
If thats true, its a while before we get to 4.0
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Shamus397
Apprentice
Apprentice


Joined: 03 Apr 2005
Posts: 218
Location: Ur-th

PostPosted: Wed Feb 18, 2015 2:54 pm    Post subject: Reply with quote

This seems to be a recurring theme:

In response to GKH, Johannes Stezenbach wrote:
Well, IMHO you got it backwards. Before adding a complex new IPC API to the kernel you should do the homework and gather some evidence that there will be enough users to justify the addition.

But maybe I misunderstood the purpose of this thread and you're just advertising it to find possible users instead of already suggesting to merge it? If someone has some convincing story to share why kdbus would solve their IPC needs, I'm all ears.

(I'm sorry this implies your responses so far were not convincing: not verifiable facts, no numbers, no testimonials etc.)

FWIW, my gut feeling was that the earlier attempts to add a new IPC primitve like multicast UNIX domain sockets http://thread.gmane.org/gmane.linux.kernel/1255575/focus=1257999 were a much saner approach. But now I think the comments from this old thread have not been addressed, instead the new approach just made the thing more complex and put in ipc/ instead of net/ to bypass the guards.

Found here. That whole thread is quite interesting, and it gives me a least a little hope that the kernel devs aren't going to be buffaloed by GKH and friends. :)

EDIT: Interestingly enough, seems the mailing list has fallen silent on the topic for close to a month now.
Back to top
View user's profile Send private message
ct85711
Veteran
Veteran


Joined: 27 Sep 2005
Posts: 1791

PostPosted: Wed Feb 18, 2015 4:52 pm    Post subject: Reply with quote

I found this rather funny, to see the claims that kdbus is going to be faster, and yet it's slower than regular kernel methods doing.
http://lkml.iu.edu/hypermail/linux/kernel/1502.0/02862.html
Quote:

From: Andy Lutomirski
Date: Wed Feb 04 2015 - 18:03:37 EST

I see "latencies" of around 20 microseconds with lockdep and context
tracking off. For example:

stats (UNIX): 226730 packets processed, latency (nsecs) min/max/avg
3845 // 34828 // 4069
stats (KDBUS): 37103 packets processed, latency (nsecs) min/max/avg
19123 // 99660 // 20696

This is IMO not very good. With memfds off:

stats (UNIX): 226061 packets processed, latency (nsecs) min/max/avg
3885 // 32019 // 4079
stats (KDBUS): 83284 packets processed, latency (nsecs) min/max/avg
10525 // 42578 // 10932

With memfds off and the payload set to 8 bytes:

stats (KDBUS): 77669 packets processed, latency (nsecs) min/max/avg
9963 // 64325 // 11645
stats (UNIX): 253695 packets processed, latency (nsecs) min/max/avg
2986 // 56094 // 3565

Am I missing something here? This is slow enough that a lightweight
userspace dbus daemon should be able to outperform kdbus, or at least
come very close.


Then a comment Andy made later is a nice catcher: http://lkml.iu.edu/hypermail/linux/kernel/1502.1/00114.html
Quote:

Date: Sun Feb 08 2015 - 11:55:13 EST

I tried to disable metadata. I may have failed.

Regardless, if metadata is very slow, then that's more reason not to
use it on send. And if you shouldn't use it, then maybe the kernel
shouldn't provide it.

I assumed there was a context switch in there. I can try to test
differently. If UDS is twice as fast *with* a contest switch, then a
userspace solution should be faster.

Also, UDS can use memfds, too.

>
> A few notes on that:
>
> * kdbus is a bus layer. We don't intend to replace UDS, but improve
> dbus. Comparing roundtrip times with UDS is tempting, but in no way
> fair. To the very least, a bus layer has to perform peer-lookup, which
> UDS does not have to do. Imo, 2.5us vs. 1.5us is already pretty nice.
> Compare this to ~77us for dbus1 without marshaling.

This makes me wonder what dbus1 is doing wrong.


Shamus397: The mailing list hasn't gone quiet, it's just broken up into sections (based on the date). If you click on Other mail archives (on top of the page, in thread view) -> Kernel Mailing List -> select which date
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6095
Location: Dallas area

PostPosted: Wed Feb 18, 2015 6:21 pm    Post subject: Reply with quote

Quote:
This makes me wonder what dbus1 is doing wrong.


Most of us are wondering this. :lol:

As far as the posts, there was only one last week and that was from Andy with no replies, and none this week (as of this morning when I looked).

also from that post (the one that I quoted at the top of this post)
Quote:
>
> * We have not optimized kdbus code-paths for speed, yet. Our main
> concerns are algorithmic challenges, and we believe they've been
> improved considerably with kdbus. I have constantly measured kdbus
> performance with 'perf' and flame-graphs, and there're a lot of
> possible optimizations (especially on locking). However, I think this
> can be done afterwards just fine. Neither API nor ioctl overhead has
> shown up in my measurements. If anyone has counter evidence, please
> let us know. But I'm a bit reluctant to change our API solely based on
> performance guesses.

But removal of send-time metadata can't be done after the fact.

--Andy


Optimizations are one thing, removing or adding fields ie. breaking user space code is quite another.
Andy and other kernel devs keep trying to get this point across and it still seems like the kdbus devs don't understand
or perhaps they're so arrogant that they think they'll be able to do whatever they want if it gets included in the kernel.

Re: the performance guesses, performance is the original reason that they claimed they needed kdbus for. :roll:
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
ct85711
Veteran
Veteran


Joined: 27 Sep 2005
Posts: 1791

PostPosted: Wed Feb 18, 2015 8:13 pm    Post subject: Reply with quote

I find it funny, every time someone goes through tests and proves their original claim for the use of kdbus wrong, they go start claiming that's completely wrong. That they are NOT going for that, but for something else. It will be funny when someone and tests out how secure kdbus really is; then they'll have lost all their original claims for kdbus.
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6095
Location: Dallas area

PostPosted: Wed Feb 18, 2015 9:12 pm    Post subject: Reply with quote

ct85711, it's the same thing that was done with justifying sysd.

First it was marvelously fast boot up times, then when it only proved marginally faster in many cases,
that quit being touted and other improvements were pushed in it's place.
Notice that fast boot times is hardly ever used as justification for sysd nowadays.

In the world of having arguments it's called "moving the goal posts" so as to avoid "losing the argument".

I'm more curious as to why all the parties that were trying to justify kdbus have quit posting.
Maybe LP is getting ready to write another of his whining screeds about being attacked because they won't let him in the kernel. :roll:

Edit to add: I went googling to see what was going on with kdbus outside the mailing list and
ran across a statement claiming that kdbus was needed for properly sandboxed applications in X (and wayland).
Of course the one making such a statement was a gnome dev, "surprise, surprise, surprise"
The only other thing I've seen mentioned is how it would make "ta da" pulseaudio work better.
I suppose thats the reason that we've seem multi-meg data transfers mentioned re: the need for kdbus

So basically it sounds like gnome is the main driver for kdbus.
Makes me wonder what they promised GKH to get it into the kernel. A job with RH/gnome perhaps. :roll:
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Thu Feb 19, 2015 6:39 am    Post subject: Reply with quote

Anon-E-moose wrote:
ct85711, it's the same thing that was done with justifying sysd.

First it was marvelously fast boot up times, then when it only proved marginally faster in many cases,
that quit being touted and other improvements were pushed in it's place.
Notice that fast boot times is hardly ever used as justification for sysd nowadays.

In the world of having arguments it's called "moving the goal posts" so as to avoid "losing the argument".

Indeed; we've been here before with paludis, which has a similar history of ever-changing selling points, "advantages" that turned out not to be.
Quote:
Makes me wonder what they promised GKH to get it into the kernel. A job with RH/gnome perhaps.

I imagine he already has stock options, as do quite a few kernel devs from what I recall; something about RH giving them all a load of stock in the early days/first floatation. Mind, I've no idea how accurate that is.
Back to top
View user's profile Send private message
tld
Veteran
Veteran


Joined: 09 Dec 2003
Posts: 1812

PostPosted: Fri Feb 20, 2015 12:05 am    Post subject: Reply with quote

Anon-E-moose wrote:
ct85711, it's the same thing that was done with justifying sysd.

First it was marvelously fast boot up times, then when it only proved marginally faster in many cases,
that quit being touted and other improvements were pushed in it's place.
Notice that fast boot times is hardly ever used as justification for sysd nowadays.

In the world of having arguments it's called "moving the goal posts" so as to avoid "losing the argument".

Bingo...everything these folks dish out ends up being a cure in search of a disease...and not a nice cure either...more the chemotherapy variety...

Tom
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sun Feb 22, 2015 11:18 am    Post subject: Reply with quote

steveL wrote:
Indeed it'd be simple; though I think it should be more extensive so we could have say (for the two common uses):
Code:
# notify on file if required
email: file "/var/somepath" with data: count
mixer: [
  ctrl: mqueue "alsa/mixer"
  data: tipc 1000:0:1000
]
audio: [
  ctrl: fifo "/var/run/alsa/volume"
  data: tipc 2000:0:1000
]
edit: removed the optional trailers as crufty here.
The point being that we should be able to tell a generic API that we want to use one of the N variants of IPC available, including fifos, mqueues and AF_UNIX sockets, depending on the service and the semantics (so a service might not support being used with "file" for example, which would result in a hard error at startup, whatever else happened.) Really this starts to overlap with a stay-around pid2 as well, since this implies dependency and configuration checks.

tclover wrote:
That's fine but... it's a little hard to read. So, what about--hoping to keep `/etc/services' simplicity--
Code:
"NAME"    "FACILITY"    "[OPTIONS]" # Just like the extra fields in `/etc/services'
"NAME"    "ADRESS:PORT/IPC"    "[OPTIONS]" # To put it in another manner
ladi    128.1.1000/tipc # To lmake an example (colon `:' or dot `.'?)
ladi    128.1/tipc  [RDM|DGRAM|STREAM|SEQPACKET] # To illustrate a possible option (socket type)
ladi    128.1:STREAM/tipc  [OPTIONS] # Or <ADDRESS:PORT|SOCKET(-TYPE)/IPC> makes more sense?
# WARN: 0 in <ZONE.CLUSTER.NODE> is special and are not permited in a "node" address or "named port" or "name service"

EDIT(PREVIOUS WITH):TIPC Network Address: This address is a 32-bit integer, structured into three fields, zone (8 MSB), cluster, (12 bits), and node (12 LSB).

This syntax is more readable and has the optional flexibility to add extra fields for particular IPC.

Hmm the issue is when we want to group settings to apply them to one service. I'm not especially fussed about the exact format, so long as it does enough: the simple cases should stay simple one-liners, but if we need more, then we have to be able to accommodate it. For some things that are in the IPC domain, we need to cope with several stanzas per-logical item (or the conf is going to get messy, and people will ask "why didn't you use a standard init parser?", or even worse "XML".)

miroR linked us to the postfix docs via a thread on grsec. I find the setup of master.cnf to be exactly along the lines we've been discussing.

So you have a Service type = inet | unix | fifo | pass (= single conn AF_UNIX; pass accepted fd), where the "service name syntax depends on the service type" eg specifying the smtp port (-D means run under debugger):
Code:
smtp      inet  n       -       n       -       -       smtpd -D

postfix master is a daemon manager, xinetd on steroids (not so surprising when we consider the author is Wietse Venema.) I really like the approach, because it constrains the complexity to the domain.

As well as for an easy-ipc lib, whereby the admin configures the ports, protocols and any pub/sub ranges for TIPC, I'd also like us to look at using similar configs within openrc, for the xinetd functionality in combination with monitoring; then we can easily inspect the postfix config, or any other like it.

That ties into the cgroup handling we added over a year ago; we may need to look at the Gentoo kernel options for openrc, in case we need to stop systemd idiocy from preventing us using cgroups sanely.

Note that the master.cf is more like the runscript-specification, and there is another main.cf where the admin configures variables used by it, along the lines of /etc/rc.conf or /etc/conf.d/svcfoo.conf. spawn handles the "one-shot" daemons of xinetd.
Quote:
Yes, this is what I have in mind as a primary concern monitoring in order to supervise and control the availability of services/functionalities. And this what most users want when using D-Bus. However...

Hmm I think we need to look at that use-case with scepticism; most apps (and thus most users) should not care about services going up or down. The things that do, tend to be things like systray apps, which we should treat as one class of thing, as it's quite simple.

Thereafter, each use-case should be explained and justified, before we end up with yaf spaghetti-junction of things that need to be recompiled every time a so-called convenience layer changes.
We must use a stable API like LADSPA for any such layer, so that the shim layers have a stable ABI to support, and not some junk design bolted onto, every time some nub has a brainwave. I don't much care what's in it, so long as it's well-thought through, and we separate by domain, so that we're not shoving everything down the one, inappropriate, IPC mechanism.

Service dependencies that are affected are of course the domain of the init-manager, pid2 (makes a nice working name, anyhow;)
steveL wrote:
The other aspect of it is not to send blobs of data down what are really meant to be control channels, either at lib or app level (metadata iow.) For instance it would be foolish to start using a different IPC, be that kdbus or anything else, for X direct-rendering data. Similarly it's foolish to start using anything other than jack to actually handle the audio data, though as above we might want a way to tap into it (we'd be using "jack/foo" not "alsa/foo" for anything pro-audio, though.)

Quote:
Indeed, it would be foolish to jump blindly in a new toy when existant protocols satisfy particular needs e.g X or JACK.

Although with a low latency & data capable communication IPC (TIPC set max packet size to 66000 bytes),--this is where some are waiting to threw away licence concerns (who said RH already?... Oracle?... the starting block are full of challengers),--data transactions become an interesting perspective. I provided the above figures to put some perspective on the latency concerns. (I did not, yet, have the time to test TIPC--TIPC latency figure is on the spec papaer; JACK latency data are commonly found on the intertubes.)
Of course, some politics become almost transparent with this... as if audio transport data is the _real_ motive to write a new in-kernel IPC when there are at least:
* a low latency & network friendly audio transport (where mixer concerns vanish);
* an already good low latency in-kernel IPC;
good implementation available.

But this just provide (other) justifications of their... pitiful failure with PulseAudio... when it was _first_ sold as a mixer capable solution implying that... as if there weren't any available.
Stopping here. There are way too many deceiving schemes behind this, and GKH finally said it all in a short sentence--"trust us, we know what we're doing!"

Heh. You're right though, the pieces are already in place, and we have a lovely in-kernel IPC mechanism, that's been through many iterations and serious real-world testing by Ericcson for 20 years.
There's zero upside in allowing kdbus to infect our machines, imo.

memsealing we can use to implement mmapped message-queues, where the receiver gets a struct iovec (or equivalent with const void *base: see man uio.h); we'd need to support usage with aio_suspend/lio_listio for POSIX (see man aio.h) which can be done in userland, since glibc does them in userland.

wrt io_submit for linux, that would have to be supported from the kernel side, but would be a much nicer basis for a userland wrapper, since there's no extra threading.
Though there are caveats; see also Arvid Norberg's excellent answers on stackoverflow.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Goto page Previous  1, 2, 3 ... 7, 8, 9 ... 25, 26, 27  Next
Page 8 of 27

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum