Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[Solved] Not so randomly missing warnings, due to ratelimit
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
Tout
n00b
n00b


Joined: 18 Jul 2020
Posts: 11

PostPosted: Fri Jul 24, 2020 3:39 pm    Post subject: [Solved] Not so randomly missing warnings, due to ratelimit Reply with quote

Since the start of this month, I have been migrating my dear desktop to Gentoo. I must say, I really like Gentoo so far! It is clean and I feel I am in control mostly :) So that is nice, since I like being in control in this regard.

So in my quest for perfection, Gentoo is pushing me to new levels. And that made me aware of a message in the log, I have not noticed before:

Code:
[    4.114611] random: 7 urandom warning(s) missed due to ratelimiting


This is a message telling me (and please tell me if I misunderstand the situation), that there were 7 warnings suppressed due to ratelimiting.

Ratelimiting makes sure the system does not register too many warnings in a certain period. (?)

I would really like to know what 7 warnings I missed. So I searched around places a bit and tried some actions to find out. So far I have:

Emerged haveged, and put the daemon in the default runlevel, and later in the sysinit runlevel to get it up earlier
From what I understand, this helps expanding the entropy pool.

Emerged rng-tools and put the daemon in the sysinit level. This should change the way random numbers are put into the pool.

Boot the kernel with the printk.devkmsg=on

So you might already have guessed, these attempts did not bring me closer to solving this. The messages related to random, I find in dmesg, are:

Code:
[    0.000000] random: get_random_bytes called from start_kernel+0x30e/0x4d9 with crng_init=0
[    0.809766] random: fast init done
[    3.934726] random: udevd: uninitialized urandom read (16 bytes read)
[    3.934835] random: udevd: uninitialized urandom read (16 bytes read)
[    3.934856] random: udevd: uninitialized urandom read (16 bytes read)
[    4.114610] random: crng init done
[    4.114611] random: 7 urandom warning(s) missed due to ratelimiting


These messages I think mean the following:

Start kernel calls it
Initial fast init had been done
Udev asks for random number, but urandom was not initialized to be read (3x)
crng init was finished <-- I guess this is where random is ready
And then after it's ready it notifies me that there were 7 warnings missed

Could you help me understanding this?


Last edited by Tout on Tue Jul 28, 2020 7:52 pm; edited 1 time in total
Back to top
View user's profile Send private message
Ionen
Veteran
Veteran


Joined: 06 Dec 2018
Posts: 1155

PostPosted: Fri Jul 24, 2020 3:42 pm    Post subject: Reply with quote

As far as I'm aware skipped messages are typically only for repeated messages, so you've missed 7 lines of the same message you already seen 3 times.

Also, the messages probably happened before the init was finished, as this is a late summary made from buffered messages.
Back to top
View user's profile Send private message
Tout
n00b
n00b


Joined: 18 Jul 2020
Posts: 11

PostPosted: Fri Jul 24, 2020 3:46 pm    Post subject: Reply with quote

Aha, that sounds logical. So you think the message:

random: udevd: uninitialized urandom read (16 bytes read)

Has been thrown 7 more times?

Yes, I am also quite sure it is a summary after it happened.

edit: I tried to find examples of this behaviour of suppressing repeating warnings, and could not find any. So not proven yet. If I look at the code, which is actually quite fun to do, I notice the following:

Some snippets from the description of random.c:
https://elixir.bootlin.com/linux/v5.4.48/source/drivers/char/random.c

* Computers are very predictable devices. Hence it is extremely hard
* to produce truly random numbers on a computer

So instead, we must try to
* gather "environmental noise" from the computer's environment, which
* must be hard for outside attackers to observe, and use that to
* generate random numbers.

Randomness from these sources are
* added to an "entropy pool"

* When random bytes are desired, they are obtained by taking the SHA
* hash of the contents of the "entropy pool"

* The userspace interfaces are two character devices /dev/random and
* /dev/urandom.

/dev/random is suitable for use when very high
* quality randomness is desired ..., as it will only return a maximum of the number of
* bits of randomness

* The /dev/urandom device does not have this limit, and will return
* as many bytes as are requested.

Then in the code I see the following happening:

First it creates a ratelimit state, called urandom_warning:

ratelimit_state urandom_warning =
Code:
RATELIMIT_STATE_INIT("warn_urandom_randomness", HZ, 3);



Then after some time is checks if there are missed warnings, and throws the message:

Code:
      if (urandom_warning.missed) {
         pr_notice("random: %d urandom warning(s) missed "
              "due to ratelimiting\n",
              urandom_warning.missed);
         urandom_warning.missed = 0;
      }


While looking how a warning is thrown I noticed this code:

Code:
   static int maxwarn = 10;
   int ret;

   if (!crng_ready() && maxwarn > 0) {
      maxwarn--;
      if (__ratelimit(&urandom_warning))
         printk(KERN_NOTICE "random: %s: uninitialized "
                "urandom read (%zd bytes read)\n",
                current->comm, nbytes);


So I guess this is where at __ratelimit(&urandom_warning), the magic is happening :)
https://elixir.bootlin.com/linux/v5.4.48/source/lib/ratelimit.c#L27

That function, I think (?), is called from ratelimit.c. That is stating the following:

Code:
   /*
    * If we contend on this state's lock then almost
    * by definition we are too busy to print a message,
    * in addition to the one that will be printed by
    * the entity that is holding the lock already:
    */


And setting missed:
Code:
rs->missed++;


Question that remains: how can I see the missed warnings? Could you, the one reading this :), help me answer this?


Copyright notice on random.c in kernel 5.4.48: (I think I have to include this here :) )


* Copyright (C) 2017 Jason A. Donenfeld <Jason@zx2c4.com>. All
* Rights Reserved.
*
* Copyright Matt Mackall <mpm@selenic.com>, 2003, 2004, 2005
*
* Copyright Theodore Ts'o, 1994, 1995, 1996, 1997, 1998, 1999. All
* rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, and the entire permission notice in its entirety,
* including the disclaimer of warranties.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. The name of the author may not be used to endorse or promote
* products derived from this software without specific prior
* written permission.
*
* ALTERNATIVELY, this product may be distributed under the terms of
* the GNU General Public License, in which case the provisions of the GPL are
* required INSTEAD OF the above restrictions. (This clause is
* necessary due to a potential bad interaction between the GPL and
* the restrictions contained in a BSD-style copyright.)
*
* THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
* WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
* WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
* OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
* BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
* USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
* DAMAGE.
Back to top
View user's profile Send private message
Tout
n00b
n00b


Joined: 18 Jul 2020
Posts: 11

PostPosted: Tue Jul 28, 2020 6:58 pm    Post subject: Reply with quote

Because I think it was save to do, after understanding the RATELIMIT_STATE_INIT macro, I decided to alter the random.c code, replacing:
Code:
RATELIMIT_STATE_INIT("warn_urandom_randomness", HZ, 3);


with:
Code:
RATELIMIT_STATE_INIT("warn_urandom_randomness", HZ, 10);


Compiled the kernel, booted, and indeed the missed warnings were gone :) And 10 instead of 3 warnings from udevd

Code:
[    4.018346] random: udevd: uninitialized urandom read (16 bytes read)
[    4.018393] random: udevd: uninitialized urandom read (16 bytes read)
[    4.018401] random: udevd: uninitialized urandom read (16 bytes read)
[    4.018409] random: udevd: uninitialized urandom read (16 bytes read)
[    4.018422] random: udevd: uninitialized urandom read (16 bytes read)
[    4.018445] random: udevd: uninitialized urandom read (16 bytes read)
[    4.045218] random: udevd: uninitialized urandom read (16 bytes read)
[    4.045237] random: udevd: uninitialized urandom read (16 bytes read)
[    4.045262] random: udevd: uninitialized urandom read (16 bytes read)
[    4.045303] random: udevd: uninitialized urandom read (16 bytes read)


I'm still not sure if it skips unitialized reads from one process, or if it would skip them from all processes after reaching the rate limit. Ideas are welcome.

Next I will try to understand why udevd is having this uninitialized reads. edit: it is because it starts before crng is ready.
Back to top
View user's profile Send private message
halcon
Apprentice
Apprentice


Joined: 15 Dec 2019
Posts: 249

PostPosted: Tue Jul 28, 2020 11:41 pm    Post subject: Reply with quote

Tout wrote:
Next I will try to understand why udevd is having this uninitialized reads. edit: it is because it starts before crng is ready.

I digged into this a little a while ago, and I can say it's not that simple to fix it. The most helpful resources that I found about the matter, are:
That comment in Bugzilla
That topic in forum
Slackware initrd patches

If in short,
    if the boot process continues, it does not harm much, but, ideally, of course, it should be fixed anyway;
    OpenRC has no tools to wait until crng is initialized by kernel;
    userspace daemons (like haveged), which helped me with other services (like syslog-ng), didn't help me with udevd (and lvm either) because it is launching very early and I couldn't make udevd start after haveged with standard OpenRC's "after";
    CONFIG_RANDOM_TRUST_CPU is dangerous after all the CPU exploits from the last few years;
    CONFIG_CRYPTO_JITTERENTROPY=y, as is, did not help either, may be because I had to install the corresponding userspace program along with it but I haven't yet tried it;
    other options that I had not yet time to try are CONFIG_GCC_PLUGIN_LATENT_ENTROPY (may be it will help) and, as a last resort, Slackware initrd patches :? .
Back to top
View user's profile Send private message
Tout
n00b
n00b


Joined: 18 Jul 2020
Posts: 11

PostPosted: Wed Jul 29, 2020 5:57 am    Post subject: Reply with quote

Wow, your search skills are good :) Thanks for posting these links. Did not encounter them myself :) I read through them, and will dive into it more when I have the time to do so. Will update here. From what I've understood now, I might try to do the patches, if I have and find the courage to do so ;)

For LVM, that was the start of this quest. I use btrfs and and not LVM. Noticed the LVM warnings at boot and via a question I posted on LQ, found the answer in LVM being pulled in as a dependency. Not a very logical one as I remember. Was not able to loose it as a dependency, so ended up removing the initrd scripts regarding LVM :)

Post at LQ, which I enjoyed a lot :) :

https://www.linuxquestions.org/questions/linux-kernel-70/lvm-referenced-during-boot-not-my-choice-4175679023/

edit: as for the LVM2 dependencies:

sys-fs/lvm2-2.02.187-r2 pulled in by:
sys-fs/cryptsetup-2.3.2 requires sys-fs/lvm2

sys-fs/cryptsetup-2.3.2 pulled in by:
sys-libs/libblockdev-2.24

Libblock dev has the LVM use flag, but that (-lvm) does not remove the dependency of cryptsetup... Maybe after this, I will look into that one :)
Back to top
View user's profile Send private message
halcon
Apprentice
Apprentice


Joined: 15 Dec 2019
Posts: 249

PostPosted: Wed Jul 29, 2020 11:58 am    Post subject: Reply with quote

Tout wrote:
I might try to do the patches, if I have and find the courage to do so ;)

If/when you do, please write back about the result, I'm interested.
Back to top
View user's profile Send private message
Tout
n00b
n00b


Joined: 18 Jul 2020
Posts: 11

PostPosted: Wed Jul 29, 2020 1:34 pm    Post subject: Reply with quote

Setting CONFIG_GCC_PLUGIN_LATENT_ENTROPY to */Y in the kernel didn't work for me, so now we know that :)
Back to top
View user's profile Send private message
halcon
Apprentice
Apprentice


Joined: 15 Dec 2019
Posts: 249

PostPosted: Wed Jul 29, 2020 1:49 pm    Post subject: Reply with quote

Tout wrote:
Setting CONFIG_GCC_PLUGIN_LATENT_ENTROPY to */Y in the kernel didn't work for me, so now we know that :)

Well, there is only one way left :)
Back to top
View user's profile Send private message
Tout
n00b
n00b


Joined: 18 Jul 2020
Posts: 11

PostPosted: Sat Aug 01, 2020 8:36 am    Post subject: Reply with quote

The Slackware patch route is a bit too much for me. Diving into it, there are two things that are stopping me: the need to compile a non Gentoo package (twuewand). And also in a way the Slackware patch route persons found necessary to adjust for the patch route. Second block was that the patches are for older versions of mkinit and haveged. I guess it will take too much effort, which I better put into something else. So for now I will leave this path unfortunately... Maybe later ;)
Back to top
View user's profile Send private message
halcon
Apprentice
Apprentice


Joined: 15 Dec 2019
Posts: 249

PostPosted: Sat Aug 01, 2020 12:03 pm    Post subject: Reply with quote

Tout wrote:
I guess it will take too much effort, which I better put into something else. So for now I will leave this path unfortunately... Maybe later ;)

OK. There are things for each of us that would take too much effort, comparing the expected benefits and time spent...

As for me, I'm interested in the matter because, first, I have always built my initrd automatically with genkernel and never even looked into it, and, second, the mentioned idea of transferring Haveged's entropy to the kernel does not look difficult to me: as far as I understand it, all that rndaddentropy tool does, is taking entropy from stdin and seeding it to /dev/random (49, 55 and 58 lines):
Code:
if((randfd = open("/dev/random", O_WRONLY)) < 0) {
...
while((count = fread(entropy.buf, 1, sizeof(entropy.buf), stdin)) > 0) {
...
if(ioctl(randfd, RNDADDENTROPY, &entropy) < 0) {

Yes, twuewand is not neither in Gentoo tree nor in overlays, but there is no need in the whole twuewand package, rndaddentropy tool does not look complicated - less than 100 lines, I also found another similar tool with a similar code: rngseed. May be, there is still something in Gentoo tree that can do this work.

The difficulty is that I (almost) haven't learned C and low-level operations, like this, and don't know which pitfalls / security risks could be there...

At least, after reading your last message, I unpacked my genkernel's initrd with xz (had to rename <filename>.img to <filename>.xz before it), then cpio and looked into it :) Looked at init shell script where the Slackware's command
Code:
/sbin/haveged -F -n 512 -f - | /sbin/rndaddentropy
would have to be inserted...

If/when I am ready for the following steps, probably, makes sense to create a new topic for discussing the details? Not sure.

EDIT:
Looks like I found a tool that can do this work in Gentoo: rng-tools (it is mentioned in rngseed's README: "can do what this utility does and much more").
Back to top
View user's profile Send private message
Tout
n00b
n00b


Joined: 18 Jul 2020
Posts: 11

PostPosted: Sat Aug 01, 2020 6:06 pm    Post subject: Reply with quote

Hmmm, sounds like you have things a bit more clear indeed. A new topic might get more people into the specific topic of getting random numbers available early in the boot process. That also might attract some people with knowledge about C and security related matters. If you do decide to continue, would you please leave a link here, so people following this topic and people who reach this at a later point in time, can continue there?
Back to top
View user's profile Send private message
halcon
Apprentice
Apprentice


Joined: 15 Dec 2019
Posts: 249

PostPosted: Sat Aug 01, 2020 6:50 pm    Post subject: Reply with quote

Tout wrote:
If you do decide to continue, would you please leave a link here, so people following this topic and people who reach this at a later point in time, can continue there?

Sure that I will post the link if I decide to create a new topic! I think it will make sense only if nobody more experienced reply here... Still there are chances.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum