View previous topic :: View next topic |
Author |
Message |
The_Great_Sephiroth Veteran
Joined: 03 Oct 2014 Posts: 1602 Location: Fayetteville, NC, USA
|
Posted: Mon Mar 11, 2019 3:54 pm Post subject: "tee" failing every time... |
|
|
I use "dd" over SSH with "tee" to clone systems for deployment. This worked flawlessly, until today. Now, when I issue the same command I have used for eons, I get "tee 'standard output' resource temporarily unavailable". I have never had this before and do not know what to do.
Code: |
dd if=/dev/sda bs=512 | tee >(ssh root@192.168.2.29 "dd of=/dev/sda bs=512") >(ssh root@192.168.2.30 "dd of=/dev/sda bs=512") | ssh root@192.168.2.31 "dd of=/dev/sda bs=512"
|
In the past we have cloned up to 30 systems this way. Now whether I do one system or ten, I get the error.
Also, the last system in the line, the one which is not tee'd but is on the end of the pipe, will stop cloning when tee errors out. Does not matter which system is in which order, once tee errors out one the ones inside the tee statement will continue cloning. How do I fix this? _________________ Ever picture systemd as what runs "The Borg"? |
|
Back to top |
|
|
The_Great_Sephiroth Veteran
Joined: 03 Oct 2014 Posts: 1602 Location: Fayetteville, NC, USA
|
Posted: Mon Mar 11, 2019 4:18 pm Post subject: |
|
|
OK, my desktop guy had burned the latest System Rescue CD. When I found out we went to the last version of the 5 series and now the command works like a charm, Is there a bug in the newer versions of tee? Odd that it fails in System Rescue CD 6 but not in System Rescue CD 5. Either way it is cloning so my big issue has been resolved, but this does prevent me from upgrading to newer live distros. System Rescue CD is Gentoo-based, if my memory serves me. _________________ Ever picture systemd as what runs "The Borg"? |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54266 Location: 56N 3W
|
Posted: Mon Mar 11, 2019 4:24 pm Post subject: |
|
|
The_Great_Sephiroth,
System Rescue CD is Arch based from version 6. Older ones are indeed Gentoo based. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
mike155 Advocate
Joined: 17 Sep 2010 Posts: 4438 Location: Frankfurt, Germany
|
|
Back to top |
|
|
toralf Developer
Joined: 01 Feb 2004 Posts: 3922 Location: Hamburg
|
Posted: Mon Mar 11, 2019 8:18 pm Post subject: |
|
|
Hhm, under Gentoo indeed just the last file is filled: Code: | $ ls | tee >a >b >c >d | wc
0 0 0
$ ls -l ?
-rw-r--r-- 1 tinderbox tinderbox 0 Mar 11 21:17 a
-rw-r--r-- 1 tinderbox tinderbox 0 Mar 11 21:17 b
-rw-r--r-- 1 tinderbox tinderbox 0 Mar 11 21:17 c
-rw-r--r-- 1 tinderbox tinderbox 34 Mar 11 21:17 d
|
|
|
Back to top |
|
|
mike155 Advocate
Joined: 17 Sep 2010 Posts: 4438 Location: Frankfurt, Germany
|
Posted: Mon Mar 11, 2019 8:31 pm Post subject: |
|
|
toralf: you're right that
Code: | $ ls | tee >a >b >c >d
|
won't work. But try
Code: | $ ls | tee a >(cat > b) >(cat > c) >(cat > d) >e
|
You will get the result below:
Code: | $ ls -l ?
-rw-r----- 1 tinderbox tinderbox 857 11. Mär 21:27 a
-rw-r----- 1 tinderbox tinderbox 857 11. Mär 21:27 b
-rw-r----- 1 tinderbox tinderbox 857 11. Mär 21:27 c
-rw-r----- 1 tinderbox tinderbox 857 11. Mär 21:27 d
-rw-r----- 1 tinderbox tinderbox 857 11. Mär 21:27 e
|
It's a combination of tee's ability to write to multiple files and the shell's feature 'process substitution'.
Last edited by mike155 on Mon Mar 11, 2019 8:51 pm; edited 1 time in total |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54266 Location: 56N 3W
|
Posted: Mon Mar 11, 2019 8:50 pm Post subject: |
|
|
The_Great_Sephiroth,
When you get that working, you will find that its a lot faster with a bigger block size. Try bs=1M
Most drives these days have a 4k block size too. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
toralf Developer
Joined: 01 Feb 2004 Posts: 3922 Location: Hamburg
|
Posted: Mon Mar 11, 2019 8:53 pm Post subject: |
|
|
mike155 wrote: | It's a combination of tee's ability to write to multiple files and the shell's feature 'process substitution'. | Thx! |
|
Back to top |
|
|
Naib Watchman
Joined: 21 May 2004 Posts: 6051 Location: Removed by Neddy
|
Posted: Mon Mar 11, 2019 9:15 pm Post subject: |
|
|
NeddySeagoon wrote: | The_Great_Sephiroth,
System Rescue CD is Arch based from version 6. Older ones are indeed Gentoo based. | but why would tee fail, thats part of coreutils. Surely Arch couldn't have screwed that up...
just means I need to pull my finger out _________________
Quote: | Removed by Chiitoo |
|
|
Back to top |
|
|
mike155 Advocate
Joined: 17 Sep 2010 Posts: 4438 Location: Frankfurt, Germany
|
Posted: Mon Mar 11, 2019 9:22 pm Post subject: |
|
|
Naib wrote: | but why would tee fail |
I guess it's not tee, but the shell, which fails. 'info tee' shows the details: process substitution (the ‘>(command)’ syntax), works with ‘zsh’, ‘bash’, and ‘ksh’, but not with ‘/bin/sh’. |
|
Back to top |
|
|
Naib Watchman
Joined: 21 May 2004 Posts: 6051 Location: Removed by Neddy
|
Posted: Mon Mar 11, 2019 9:43 pm Post subject: |
|
|
how did they screw that up? this is an embarrassment _________________
Quote: | Removed by Chiitoo |
|
|
Back to top |
|
|
mike155 Advocate
Joined: 17 Sep 2010 Posts: 4438 Location: Frankfurt, Germany
|
Posted: Mon Mar 11, 2019 10:19 pm Post subject: |
|
|
Hmm. I just started SystemRescueCD 6.0.2 in a virtual machine. The defaults shell is bash and the statement below works as expected:
Code: | ls | tee a >(cat > b) >(cat > c) >(cat > d) >e |
It must be something else... |
|
Back to top |
|
|
mike155 Advocate
Joined: 17 Sep 2010 Posts: 4438 Location: Frankfurt, Germany
|
Posted: Mon Mar 11, 2019 10:56 pm Post subject: |
|
|
I tried to reproduce that error. Since I don't have 30 remote servers, I used the statements below to copy a tar archive from a client machine to my server:
Code: | cd /
tar cf - /usr/share | tee >(ssh user@10.0.0.1 "dd of=/d/scratch/a") >(ssh user@10.0.0.1 "dd of=/d/scratch/b") | ssh user@10.0.0.1 "dd of=/d/scratch/c" |
If I run that statement on my Gentoo Linux client, I get the error message:
Code: | tee: 'standard output': Resource temporarily unavailable
|
A 'ls' on the destination directory shows:
Code: | $ ls -la a b c
-rw-r--r-- 1 user user 1327042560 11. Mär 23:41 a
-rw-r--r-- 1 user user 1327042560 11. Mär 23:41 b
-rw-r--r-- 1 user user 2107392 11. Mär 23:40 c
|
It's probably better to run:
Code: | tar cf - /usr/share | tee >(ssh user@10.0.0.1 "dd of=/d/scratch/a") >(ssh user@10.0.0.1 "dd of=/d/scratch/b") >(ssh user@10.0.0.1 "dd of=/d/scratch/c") >/dev/null
|
If I do it that way, I don't get any errors an a 'ls' on the destination directory shows:
Code: | $ ls -la a b c
-rw-r--r-- 1 user user 1327042560 11. Mär 23:49 a
-rw-r--r-- 1 user user 1327042560 11. Mär 23:49 b
-rw-r--r-- 1 user user 1327042560 11. Mär 23:49 c
|
Conclusion: the error is not related to SystemRescueCD. It occurs on Gentoo Linux machines, too.
Last edited by mike155 on Mon Mar 11, 2019 11:09 pm; edited 2 times in total |
|
Back to top |
|
|
The_Great_Sephiroth Veteran
Joined: 03 Oct 2014 Posts: 1602 Location: Fayetteville, NC, USA
|
Posted: Mon Mar 11, 2019 10:57 pm Post subject: |
|
|
Crap, does that move mean systemd? If so, I just stopped using them. Also, these were 500GB disks with 512b sectors, and I prefer to copy a sector at a time. I even saw a new 1TB disk with 512b sectors. Strange, I know. _________________ Ever picture systemd as what runs "The Borg"? |
|
Back to top |
|
|
mike155 Advocate
Joined: 17 Sep 2010 Posts: 4438 Location: Frankfurt, Germany
|
Posted: Mon Mar 11, 2019 11:31 pm Post subject: |
|
|
I tried to find out what happened and ran the statement below:
Code: | cd /
tar cf - /usr/share | strace -f /tmp/log tee >(ssh user@10.0.0.1 "dd of=/d/scratch/a") >(ssh user@10.0.0.1 "dd of=/d/scratch/b") | ssh user@10.0.0.1 "dd of=/d/scratch/c"
|
The strace log shows what happened:
Code: | read(0, "...", 8192) = 8192
write(1, "...", 8192) = 8192
write(3, "...", 8192) = 8192
write(4, "...", 8192) = 8192
read(0, "...", 8192) = 8192
write(1, "...", 8192) = 8192
write(3, "...", 8192) = 8192
write(4, "...", 8192) = 8192
read(0, "...", 8192) = 8192
write(1, "...", 8192) = -1 EAGAIN (Resource temporarily unavailable)
write(2, "tee: ", 5) = 5
write(2, "'standard output'", 17) = 17
openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, ": Resource temporarily unavailab"..., 34) = 34
write(2, "\n", 1) = 1
write(3, "...", 8192) = 8192
write(4, "...", 8192) = 8192
read(0, "...", 8192) = 8192
write(3, "...", 8192) = 8192
write(4, "...", 8192) = 8192
read(0, "...", 8192) = 8192
write(3, "...", 8192) = 8192
write(4, "...", 8192) = 8192
|
tee happily reads 8kB blocks from STDIN and writes them to STDOUT (which is a pipe to the last ssh command) and to its two output files (which are asynchronously running ssh processes).
After a while, the write to STDOUT fails with EAGAIN - which means: pipe is full. And that's strange. I wouldn't expect EAGAIN in a normal pipe. I looked at the source code of tee. There's no handler for EAGAIN.
The culprit seems to be SSH. According to this thread: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209509, SSH sets STDIN to non-blocking mode, which would explain why tee receives EAGAIN if the pipe is full.
Last edited by mike155 on Tue Mar 12, 2019 12:14 am; edited 1 time in total |
|
Back to top |
|
|
mike155 Advocate
Joined: 17 Sep 2010 Posts: 4438 Location: Frankfurt, Germany
|
Posted: Mon Mar 11, 2019 11:57 pm Post subject: |
|
|
@The_Great_Sephiroth: the problem is not related to SystemRescueCD. I can't tell you why the command you posted worked in the past - maybe it was sheer luck?
But I think you can safely use the statement below. Just send the output of the pipe to /dev/null:
Code: | dd if=/dev/sda bs=1M | tee >(ssh root@192.168.2.29 "dd of=/dev/sda bs=1M") >(ssh root@192.168.2.30 "dd of=/dev/sda bs=1M") >(ssh root@192.168.2.31 "dd of=/dev/sda bs=1M") >/dev/null
|
|
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Tue Mar 12, 2019 12:25 am Post subject: |
|
|
The_Great_Sephiroth wrote: | Crap, does that move mean systemd? If so, I just stopped using them. Also, these were 500GB disks with 512b sectors, and I prefer to copy a sector at a time. I even saw a new 1TB disk with 512b sectors. Strange, I know. |
Not that strange, there's no need for advanced format disks until you get to 2TB (2³¹ × 512). Even so, it's more efficient to use a larger bs= value like 4K (syscalls still add up, and you're multiplying that by the overhead of three ssh connections...) |
|
Back to top |
|
|
The_Great_Sephiroth Veteran
Joined: 03 Oct 2014 Posts: 1602 Location: Fayetteville, NC, USA
|
Posted: Thu Apr 11, 2019 6:48 pm Post subject: |
|
|
I meant to reply here and forgot to do so. Using any of the 4 and 5 series System Rescue discs my command works fine. Starting on 6, it fails. That is all I need to know for now. I did try piping to /dev/null, but that too failed on 6.
The reason I used 512b blocks is because the HDDs have 512b physical sectors. Older machines. I do one sector at a time so, should it fail, I know which sector failed and I can manually test it and even resum starting on that sector or a few before it, thus saving time. _________________ Ever picture systemd as what runs "The Borg"? |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3347 Location: Rasi, Finland
|
Posted: Thu Apr 11, 2019 9:18 pm Post subject: |
|
|
This is certainly an interesting problem.
I just posted this because I want to know how this ends up... _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
mike155 Advocate
Joined: 17 Sep 2010 Posts: 4438 Location: Frankfurt, Germany
|
Posted: Thu Apr 11, 2019 10:23 pm Post subject: |
|
|
The_Great_Sephiroth:
I tested the statement below under SystemRescueCD 6 and it works.
Code: | dd if=/dev/sda | tee >(ssh root@192.168.2.29 "dd of=/dev/sda") >(ssh root@192.168.2.30 "dd of=/dev/sda") >/dev/null |
I also explained in my last post why it works and why your original statement doesn't work: OpenSSH sets STDIN to non-blocking mode and therefore we cannot pipe the output of tee to ssh.
Do I understand correctly that this statement doesn't work for you under SystemRescueCD 6? |
|
Back to top |
|
|
Akkara Bodhisattva
Joined: 28 Mar 2006 Posts: 6702 Location: &akkara
|
Posted: Fri Apr 12, 2019 8:07 am Post subject: |
|
|
This thread reminds me of some tee troubles I had in the past.
I believe the problem comes from when the pipe(s) aren't being read in a more or less synchronous manner by all the processes. In my case I was trying to transcode audio into multiple formats in parallel with only a single disk-read/decode feeding the chain. And I ran into incomplete encoding and various errors.
Unfortunately this was a long time ago and I don't recall clearly what the solution was. But it involved reading up on the "-p" option to tee (use info tee which has better documentation), and, if I recall, creative use of dd, probably with the conv=fullblock option in there somewhere.
Sorry this is all rather vague. But I hope it might give some ideas on how to proceed. _________________ Many think that Dilbert is a comic. Unfortunately it is a documentary. |
|
Back to top |
|
|
The_Great_Sephiroth Veteran
Joined: 03 Oct 2014 Posts: 1602 Location: Fayetteville, NC, USA
|
Posted: Wed Apr 24, 2019 7:21 pm Post subject: |
|
|
Thank you for the info. I am on my Android tablet now, but will check it out later today.
Also, is that a gnome I see? You know, I'm Tauren... _________________ Ever picture systemd as what runs "The Borg"? |
|
Back to top |
|
|
|