Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Slow I/O on OctoPrint on Gentoo ARM64 causing print pauses
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo on ARM
View previous topic :: View next topic  
Author Message
salfter
n00b
n00b


Joined: 02 Jan 2003
Posts: 61

PostPosted: Thu Mar 07, 2019 9:41 pm    Post subject: Slow I/O on OctoPrint on Gentoo ARM64 causing print pauses Reply with quote

I'd like to move my OctoPrint installations off of Raspbian and onto Gentoo. I started with https://github.com/sakaki-/gentoo-on-rpi3-64bit, stripped out all of the desktop stuff, brought in my overlay with an ebuild for OctoPrint, and currently have the following in my world set:

app-editors/joe
app-portage/layman
dev-embedded/avrdude
dev-embedded/rpi3-64bit-meta
media-video/mjpg-streamer
www-apps/octoprint
www-servers/nginx

This setup drives an Anet A8; for printer control electronics, I'm using an Arduino Mega 2560 and a RAMPS 1.6. The Arduino provides a USB serial port, configured to run at 250 kbps. Since the needed include files to get recent versions of mjpg-streamer (the ones with RPi camera support) to build aren't available in raspberrypi-userland, I switched from the RPi camera to a UVC webcam (a Microsoft LifeCam HD6000 I had sitting around) plugged into a USB port.

At the same print speeds I've always used, the printer is pausing occasionally for a few seconds at a time during even simple prints. I've rewritten /etc/init.d/octoprint to pass "--nicelevel -2 --ionice 1" to start-stop-daemon when it launches OctoPrint. I've cut resolution on the webcam back from 1280x720 to 640x480 to 320x240. I've even shut off mjpg-streamer entirely and unplugged the camera. I still get pauses during the print. When I had the same hardware running Raspbian, only the most complex print jobs (lots of curves, etc.) caused the printer to stutter briefly, and never anything like pausing for 3-5 seconds at a time.

Streaming 250 kbps across a USB serial connection shouldn't be that difficult. top isn't even showing that much of a load...maybe 5-10% for the OctoPrint process, and mjpg-streamer was about the same or lower. Is there something I'm missing?
Back to top
View user's profile Send private message
Sakaki
Apprentice
Apprentice


Joined: 21 May 2014
Posts: 278

PostPosted: Fri Mar 08, 2019 12:55 am    Post subject: Reply with quote

salfter,

sorry to hear you're having problems with this - not something I've had reports about before ><

Which kernel are you running? Incidentally, the latest (1.4.0) release of gentoo-on-rpi3-64bit has a 4.19 kernel, which supports accessing the Pi's camera module via ffmpeg (although the libraries you need may not be properly integrated yet).

One thing you might try though, would be to install your components on my raspbian-nspawn-64 image. Although the main point of that image is to host a 64-bit Debian guest OS within Raspbian-32 using systemd-nspawn (eek!), you also get an essentially vanilla 32-bit Raspbian userland available, just booted under a 64-bit kernel (the same kernel used by 1.3.x gentoo-on-rpi3-64bit).

Accordingly, if you try installing and running your octoprint system on this 32-bit Raspbian userland (i.e. not via a 64-bit "container shell", just via a regular Raspbian terminal window), do you experience the same issues? This test would help to narrow down whether the problem lies somewhere in the 64-bit userland on the (Gentoo) image, or the 64-bit kernel.

In other words:

32-bit kernel, 32-bit Raspbian userland (stock Raspbian): your system works OK
64-bit kernel, 32-bit Raspbian userland (raspbian-nspawn-64): ??
64-bit kernel, 64-bit Gentoo userland (stripped gentoo-on-rpi3-64bit): your system exhibits the problem
_________________
Regards,

sakaki
Back to top
View user's profile Send private message
salfter
n00b
n00b


Joined: 02 Jan 2003
Posts: 61

PostPosted: Fri Mar 08, 2019 7:46 am    Post subject: Reply with quote

Sakaki wrote:

Which kernel are you running?


/proc/version reports this:
Linux version 4.19.25-v8-78eb13b25d5e-bis+ (sakaki@kurosawa) (gcc version 7.3.0 (Gentoo 7.3.0-r3 p1.4)) #2 SMP PREEMPT Tue Feb 26 13:27:47 GMT 2019

Quote:

Incidentally, the latest (1.4.0) release of gentoo-on-rpi3-64bit has a 4.19 kernel, which supports accessing the Pi's camera module via ffmpeg (although the libraries you need may not be properly integrated yet).


I saw that earlier today. I've plugged the Raspberry Pi camera back in and have been figuring out how to stream from it. OctoPrint expects Motion JPEG, and while I can get ffmpeg to produce it, I'm not sure how I'd get it to stream, and transcoding H.264 to a bunch of JPEGs is CPU-intensive even at low framerates.

I've gotten the raw H.264 video to stream via RTMP and HTTP Live Streaming, both with the assistance of nginx as the streaming-server component. 720p at 30000/1001 fps is running about 2.5 Mbps, and CPU load is minimal. Of those, it's more likely I can persuade the OctoPrint maintainers to support HTTP Live Streaming, as it's easy to incorporate into a webpage.

(One neat trick with HTTP Live Streaming is to write the files to a tmpfs mounted within your web server's file structure. Only a few seconds' worth of storage is needed, so 16 MB should be sufficient. It keeps live video in RAM until it's discarded, instead of wearing out your SD card.)

Quote:

One thing you might try though, would be to install your components on my raspbian-nspawn-64 image. Although the main point of that image is to host a 64-bit Debian guest OS within Raspbian-32 using systemd-nspawn (eek!), you also get an essentially vanilla 32-bit Raspbian userland available, just booted under a 64-bit kernel (the same kernel used by 1.3.x gentoo-on-rpi3-64bit).


Might need to try that out. I think I have another MicroSD card kicking around to hold it. (Still have the original Raspbian system on one and Gentoo on another.)
Back to top
View user's profile Send private message
salfter
n00b
n00b


Joined: 02 Jan 2003
Posts: 61

PostPosted: Fri Mar 08, 2019 5:02 pm    Post subject: Reply with quote

One more thing I found with the just-added camera support: since the camera shows up as a V4L2 device, it works with the old (2012) version of mjpg_streamer that's in Portage. Just use input_uvc.so instead of input_raspicam.so (or whatever it is) as the input driver. CPU usage is still down in the weeds. I'm using this in /etc/conf.d/mjpg-streamer:

Code:

INPUT_PLUGIN="uvc"
INPUT_PLUGIN_OPTS="-d /dev/video0 -r 1280x720 -f 5 -n"
OUTPUT_PLUGIN="http"
OUTPUT_PLUGIN_OPTS="-p 31337"
MJPG_STREAMER_USER="nobody"
MJPG_STREAMER_GROUP="video"


This streams Motion JPEG on port 31337. It's not as bandwidth-efficient as H.264 delivered by RTMP or HLS, but it's lower latency. Perhaps if the H.264 encoding parameters for the camera were adjustable... (They probably are somehow; I just don't know what knobs to turn.)

Now to look into that 32-bit Raspbian userland on a 64-bit kernel thing...
Back to top
View user's profile Send private message
Sakaki
Apprentice
Apprentice


Joined: 21 May 2014
Posts: 278

PostPosted: Fri Mar 08, 2019 6:33 pm    Post subject: Reply with quote

There appears to be an issue with the 4.19 kernel and USB-based webcams on the RPi3 (isochronous mode). See e.g. this thread.
If that's what was causing the problem in your original tests, then you should find your external webcam works OK on the raspbian-nspawn-64 image I mentioned (since that has a 4.14 64-bit kernel).

If that proves so, then unfortunately, you may have to choose: roll back to 4.14, which removes the Pi camera module support (and aiui there are no plans to backport that), but gives you more robust USB, or use the internal camera module with 4.19 (and forgo any USB-connected webcam).
_________________
Regards,

sakaki
Back to top
View user's profile Send private message
salfter
n00b
n00b


Joined: 02 Jan 2003
Posts: 61

PostPosted: Mon Mar 11, 2019 12:30 am    Post subject: Reply with quote

Sakaki wrote:
There appears to be an issue with the 4.19 kernel and USB-based webcams on the RPi3 (isochronous mode). See e.g. this thread.
If that's what was causing the problem in your original tests, then you should find your external webcam works OK on the raspbian-nspawn-64 image I mentioned (since that has a 4.14 64-bit kernel).

If that proves so, then unfortunately, you may have to choose: roll back to 4.14, which removes the Pi camera module support (and aiui there are no plans to backport that), but gives you more robust USB, or use the internal camera module with 4.19 (and forgo any USB-connected webcam).


IIRC, the print stuttering happened whether the USB webcam was in use or not. It even happened when the webcam was unplugged.

I've gotten the RPi cam working...can stream H.264 over RTMP or HLS, or I can use the method OctoPrint expects to stream MJPEG over HTTP. I've put away the USB webcams.

That said, OctoPrint communicates with the Arduino in the printer over an emulated serial connection, currently configured to run at 250 kbps. An ATMEGA16U2 on the Arduino is configured as a USB serial interface. It uses the cdc_acm kernel module. Would the isochronous-mode issue mentioned above have any effect here?

I tested your 64-bit Raspbian config the other day with the same OctoPrint configuration (within the 32-bit Raspbian userland) as my usual 32-bit build; it worked as expected, with no hiccups during printing. I switched back to 64-bit Gentoo and printed the same gcode I'd printed with Raspbian on the 64-bit kernel; got the same stuttering as before.

A couple more things I'm considering checking are to diff the kernel configs between Gentoo and Raspbian and see if there's any difference, and a speed test: upload gcode to the printer's SD-card storage through OctoPrint on the two systems. I expect I'll find a discrepancy there.
Back to top
View user's profile Send private message
Sakaki
Apprentice
Apprentice


Joined: 21 May 2014
Posts: 278

PostPosted: Mon Mar 11, 2019 1:25 am    Post subject: Reply with quote

The kernels on the raspbian-nspawn-64 and current gentoo-on-rpi3-64bit systems are materially different: the first uses 4.14.93, the latter 4.19.25. So the USB-handling kernel change that happened between 4.14 and 4.19 may still be the culprit, based on your tests.

One thing you could try, to isolate this, would be to forcibly install a prebuilt 4.19 kernel tarball on your raspbian-nspawn-64 image, reboot, and see if the problem then emerges. To do this, boot the image, become root in a standard (not container) shell, and issue:
Code:
# cp /boot/kernel8.img{,.old}
# wget -c https://github.com/sakaki-/bcmrpi3-kernel-bis/releases/download/4.19.25.20190226/bcmrpi3-kernel-bis-4.19.25.20190226.tar.xz
# tar -xJf bcmrpi3-kernel-bis-4.19.25.20190226.tar.xz -C /
# sync && reboot

This will install the same kernel as used on the current gentoo-on-rpi3-64bit system, so if you get the stuttering after that, it's a kernel, not userland issue.
_________________
Regards,

sakaki
Back to top
View user's profile Send private message
salfter
n00b
n00b


Joined: 02 Jan 2003
Posts: 61

PostPosted: Mon Mar 11, 2019 5:29 am    Post subject: Reply with quote

It's beginning to look like a hardware issue, with either the USB cable or the Arduino to blame. Checking the OctoPrint logfile shows lots of communication timeouts. I've dropped the port speed down from 250 kbps to 115.2 kbps. I've created a gcode file that just moves the printhead in circles without actually printing anything. I'm testing it remotely, but even from here, I can see the printer sometimes pause not even a minute in. The logfile shows a communication timeout, then the head starts moving again. This keeps on repeating at random. It also showed up during a firmware upgrade; it took several tries for avrdude to successfully burn the image that slowed down the port.

I have another printer I'm building at home...not quite done yet, but the electronics suite is nearly identical: Arduino Mega, RAMPS 1.4 (vs. 1.6, but the differences in the two are trivial), and a Raspberry Pi 3B+ (vs. 3B). It's not glitching like the other printer.

Right now, I have just an Arduino Mega (technically, a clone with a CH340 USB-serial chip instead of an ATMEGA16U2) plugged into the Raspberry Pi, running the same firmware image that's on the printer at the office (the one that's been troublesome). It's a fourth of the way through my running-in-circles gcode without any glitches. If it finishes up with no glitches, I'm going to consider this resolved as a hardware issue, swap out Arduinos on the printer, and press on from there.
Back to top
View user's profile Send private message
salfter
n00b
n00b


Joined: 02 Jan 2003
Posts: 61

PostPosted: Mon Mar 11, 2019 8:09 pm    Post subject: Reply with quote

Tried replacing the Arduino and the USB cable...no dice. Tried replacing the power supply...went from the +5VSB line on an ATX power supply (which has been giving me undervoltage warnings) to a 5.25V wall wart. Still nothing. I've now swapped out the Raspberry Pi 3B for a 3B+ I had on hand for other purposes. Just had a glitch three-and-a-half minutes into the test gcode. Swapped in the new Arduino and USB cable, so this setup should be identical to what ran flawlessly at home. It's still glitching. The only thing left that I can think to try is to clone the SD card on the home printer; maybe there's some subtle difference in the images such that one works right, but the other doesn't.
Back to top
View user's profile Send private message
salfter
n00b
n00b


Joined: 02 Jan 2003
Posts: 61

PostPosted: Tue Mar 12, 2019 4:56 am    Post subject: Reply with quote

Imaged the working system and copied it to the one that was having problems...looks like that did the trick.

One thing I noticed, though: this is the kernel version string that's now on both systems:

Linux version 4.14.90-v8-6d68e517b3ec-bis+ (sakaki@kurosawa) (gcc version 7.3.0 (Gentoo 7.3.0-r3 p1.4)) #2 SMP PREEMPT Sat Dec 22 01:32:32 GMT 2018

I've not changed any hardware, other than the SD card with the different image.

Quote:

The kernels on the raspbian-nspawn-64 and current gentoo-on-rpi3-64bit systems are materially different: the first uses 4.14.93, the latter 4.19.25. So the USB-handling kernel change that happened between 4.14 and 4.19 may still be the culprit, based on your tests.


Having ruled everything else out (userland, hardware, etc.), it would seem that there is a regression of some sort in the 4.19 kernel USB code.
Back to top
View user's profile Send private message
erm67
Guru
Guru


Joined: 01 Nov 2005
Posts: 329
Location: EU

PostPosted: Fri Mar 15, 2019 6:55 am    Post subject: Reply with quote

What about the dtbs? Did you make sure you were using the correct dtb for the kernel? Did they change anything related to usb in the dtb between 4.14 and 4.19?
Sounds like a IRQ handling problem.
_________________
True ignorance is not the absence of knowledge, but the refusal to acquire it.
Ab esse ad posse valet, a posse ad esse non valet consequentia

My fediverse account: @erm67@erm67.dynu.net
Back to top
View user's profile Send private message
Sakaki
Apprentice
Apprentice


Joined: 21 May 2014
Posts: 278

PostPosted: Fri Mar 15, 2019 9:55 pm    Post subject: Reply with quote

erm67 wrote:
What about the dtbs? Did you make sure you were using the correct dtb for the kernel? Did they change anything related to usb in the dtb between 4.14 and 4.19?
Sounds like a IRQ handling problem.

The dtbs get built alongside the kernel and are included in the binary kernel release tarball. So they should be OK.
Interrupt handling is at the bottom of this though, I agree: see e.g. this post (ff) by an RPF engineer:
6by9 wrote:
...
One big but - Usb support on 64 bit kernels is iffy. Aiui the usb controller isn't the best and needs very fast interrupt handling. On 32bit this has been implemented using a fiq (arm's fast interrupt handler), but this has been deprecated in aarch64 linux.
I tried a uvc webcam yesterday. It works fine on 32 bit, but drops frames galore on 64, with anything above vga dropping >90% of frames.

_________________
Regards,

sakaki
Back to top
View user's profile Send private message
erm67
Guru
Guru


Joined: 01 Nov 2005
Posts: 329
Location: EU

PostPosted: Sat Mar 16, 2019 8:05 am    Post subject: Reply with quote

Sakaki wrote:
erm67 wrote:
What about the dtbs? Did you make sure you were using the correct dtb for the kernel? Did they change anything related to usb in the dtb between 4.14 and 4.19?
Sounds like a IRQ handling problem.

The dtbs get built alongside the kernel and are included in the binary kernel release tarball. So they should be OK.
Interrupt handling is at the bottom of this though, I agree: see e.g. this post (ff) by an RPF engineer:
6by9 wrote:
...
One big but - Usb support on 64 bit kernels is iffy. Aiui the usb controller isn't the best and needs very fast interrupt handling. On 32bit this has been implemented using a fiq (arm's fast interrupt handler), but this has been deprecated in aarch64 linux.
I tried a uvc webcam yesterday. It works fine on 32 bit, but drops frames galore on 64, with anything above vga dropping >90% of frames.


Also uboot uses the dtb to initialize some hardware, it is important to make sure the kernel and uboot uses the same dtb otherwise there could pe problems since not always HW can be initialized differently by the kernel once the peripheral has been initialized by the bootloader or uboot. He solvea his problem replacing the SD card which means he used a new uboot (I think also the raspi has a uboot in the first blocks of the SD card), do the cards use the same uboot and do they use the same dtb? Maybe it's a bug in uboot, it attempts to boot from usb so it has to initialize it before the kernel.


FIQ sucks apparently:
https://www.st.com/resource/en/application_note/cd00259791.pdf
_________________
True ignorance is not the absence of knowledge, but the refusal to acquire it.
Ab esse ad posse valet, a posse ad esse non valet consequentia

My fediverse account: @erm67@erm67.dynu.net
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42569
Location: 56N 3W

PostPosted: Sat Mar 16, 2019 9:32 am    Post subject: Reply with quote

erm67,

uboot is optional on the Pi. The GPU loads a binary blob called bootcode.bin that runs on the GPU to load the kernel, dtb and initrd if there is one.
The behaviour of bootcode.bin can be influenced by config.txt.
All the time bootcode.bin runs, the ARM CPU is held reset.

bootcode.bin can be told to load uboot, or anything, instead of a kernel but most users don't do that.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo on ARM All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum