Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
"tee" failing every time...
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
The_Great_Sephiroth
Veteran
Veteran


Joined: 03 Oct 2014
Posts: 1602
Location: Fayetteville, NC, USA

PostPosted: Mon Mar 11, 2019 3:54 pm    Post subject: "tee" failing every time... Reply with quote

I use "dd" over SSH with "tee" to clone systems for deployment. This worked flawlessly, until today. Now, when I issue the same command I have used for eons, I get "tee 'standard output' resource temporarily unavailable". I have never had this before and do not know what to do.
Code:

dd if=/dev/sda bs=512 | tee >(ssh root@192.168.2.29 "dd of=/dev/sda bs=512") >(ssh root@192.168.2.30 "dd of=/dev/sda bs=512") | ssh root@192.168.2.31 "dd of=/dev/sda bs=512"

In the past we have cloned up to 30 systems this way. Now whether I do one system or ten, I get the error.

Also, the last system in the line, the one which is not tee'd but is on the end of the pipe, will stop cloning when tee errors out. Does not matter which system is in which order, once tee errors out one the ones inside the tee statement will continue cloning. How do I fix this?
_________________
Ever picture systemd as what runs "The Borg"?
Back to top
View user's profile Send private message
The_Great_Sephiroth
Veteran
Veteran


Joined: 03 Oct 2014
Posts: 1602
Location: Fayetteville, NC, USA

PostPosted: Mon Mar 11, 2019 4:18 pm    Post subject: Reply with quote

OK, my desktop guy had burned the latest System Rescue CD. When I found out we went to the last version of the 5 series and now the command works like a charm, Is there a bug in the newer versions of tee? Odd that it fails in System Rescue CD 6 but not in System Rescue CD 5. Either way it is cloning so my big issue has been resolved, but this does prevent me from upgrading to newer live distros. System Rescue CD is Gentoo-based, if my memory serves me.
_________________
Ever picture systemd as what runs "The Borg"?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54099
Location: 56N 3W

PostPosted: Mon Mar 11, 2019 4:24 pm    Post subject: Reply with quote

The_Great_Sephiroth,

System Rescue CD is Arch based from version 6. Older ones are indeed Gentoo based.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Mon Mar 11, 2019 5:09 pm    Post subject: Reply with quote

We even have a thread about the move to Arch Linux: https://forums.gentoo.org/viewtopic-t-1092708-highlight-.html
Back to top
View user's profile Send private message
toralf
Developer
Developer


Joined: 01 Feb 2004
Posts: 3921
Location: Hamburg

PostPosted: Mon Mar 11, 2019 8:18 pm    Post subject: Reply with quote

Hhm, under Gentoo indeed just the last file is filled:
Code:
$ ls | tee >a >b >c >d | wc
      0       0       0

$ ls -l ?
-rw-r--r-- 1 tinderbox tinderbox  0 Mar 11 21:17 a
-rw-r--r-- 1 tinderbox tinderbox  0 Mar 11 21:17 b
-rw-r--r-- 1 tinderbox tinderbox  0 Mar 11 21:17 c
-rw-r--r-- 1 tinderbox tinderbox 34 Mar 11 21:17 d
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Mon Mar 11, 2019 8:31 pm    Post subject: Reply with quote

toralf: you're right that
Code:
$ ls | tee >a >b >c >d

won't work. But try
Code:
$ ls | tee a >(cat > b) >(cat > c) >(cat > d) >e

You will get the result below:
Code:
$ ls -l ?

-rw-r----- 1 tinderbox tinderbox  857 11. Mär 21:27 a
-rw-r----- 1 tinderbox tinderbox  857 11. Mär 21:27 b
-rw-r----- 1 tinderbox tinderbox  857 11. Mär 21:27 c
-rw-r----- 1 tinderbox tinderbox  857 11. Mär 21:27 d
-rw-r----- 1 tinderbox tinderbox  857 11. Mär 21:27 e

It's a combination of tee's ability to write to multiple files and the shell's feature 'process substitution'.


Last edited by mike155 on Mon Mar 11, 2019 8:51 pm; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54099
Location: 56N 3W

PostPosted: Mon Mar 11, 2019 8:50 pm    Post subject: Reply with quote

The_Great_Sephiroth,

When you get that working, you will find that its a lot faster with a bigger block size. Try bs=1M
Most drives these days have a 4k block size too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
toralf
Developer
Developer


Joined: 01 Feb 2004
Posts: 3921
Location: Hamburg

PostPosted: Mon Mar 11, 2019 8:53 pm    Post subject: Reply with quote

mike155 wrote:
It's a combination of tee's ability to write to multiple files and the shell's feature 'process substitution'.
Thx!
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 6051
Location: Removed by Neddy

PostPosted: Mon Mar 11, 2019 9:15 pm    Post subject: Reply with quote

NeddySeagoon wrote:
The_Great_Sephiroth,

System Rescue CD is Arch based from version 6. Older ones are indeed Gentoo based.
but why would tee fail, thats part of coreutils. Surely Arch couldn't have screwed that up...

just means I need to pull my finger out
_________________
Quote:
Removed by Chiitoo
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Mon Mar 11, 2019 9:22 pm    Post subject: Reply with quote

Naib wrote:
but why would tee fail

I guess it's not tee, but the shell, which fails. 'info tee' shows the details: process substitution (the ‘>(command)’ syntax), works with ‘zsh’, ‘bash’, and ‘ksh’, but not with ‘/bin/sh’.
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 6051
Location: Removed by Neddy

PostPosted: Mon Mar 11, 2019 9:43 pm    Post subject: Reply with quote

how did they screw that up? this is an embarrassment
_________________
Quote:
Removed by Chiitoo
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Mon Mar 11, 2019 10:19 pm    Post subject: Reply with quote

Hmm. I just started SystemRescueCD 6.0.2 in a virtual machine. The defaults shell is bash and the statement below works as expected:
Code:
ls | tee a >(cat > b) >(cat > c) >(cat > d) >e

It must be something else...
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Mon Mar 11, 2019 10:56 pm    Post subject: Reply with quote

I tried to reproduce that error. Since I don't have 30 remote servers, I used the statements below to copy a tar archive from a client machine to my server:
Code:
cd /
tar cf - /usr/share | tee >(ssh user@10.0.0.1 "dd of=/d/scratch/a") >(ssh user@10.0.0.1 "dd of=/d/scratch/b") | ssh user@10.0.0.1 "dd of=/d/scratch/c"

If I run that statement on my Gentoo Linux client, I get the error message:
Code:
tee: 'standard output': Resource temporarily unavailable

A 'ls' on the destination directory shows:
Code:
$ ls -la a b c
-rw-r--r-- 1 user user 1327042560 11. Mär 23:41 a
-rw-r--r-- 1 user user 1327042560 11. Mär 23:41 b
-rw-r--r-- 1 user user    2107392 11. Mär 23:40 c

It's probably better to run:
Code:
tar cf - /usr/share | tee >(ssh user@10.0.0.1 "dd of=/d/scratch/a") >(ssh user@10.0.0.1 "dd of=/d/scratch/b") >(ssh user@10.0.0.1 "dd of=/d/scratch/c") >/dev/null

If I do it that way, I don't get any errors an a 'ls' on the destination directory shows:
Code:
$ ls -la a b c
-rw-r--r-- 1 user user 1327042560 11. Mär 23:49 a
-rw-r--r-- 1 user user 1327042560 11. Mär 23:49 b
-rw-r--r-- 1 user user 1327042560 11. Mär 23:49 c

Conclusion: the error is not related to SystemRescueCD. It occurs on Gentoo Linux machines, too.


Last edited by mike155 on Mon Mar 11, 2019 11:09 pm; edited 2 times in total
Back to top
View user's profile Send private message
The_Great_Sephiroth
Veteran
Veteran


Joined: 03 Oct 2014
Posts: 1602
Location: Fayetteville, NC, USA

PostPosted: Mon Mar 11, 2019 10:57 pm    Post subject: Reply with quote

Crap, does that move mean systemd? If so, I just stopped using them. Also, these were 500GB disks with 512b sectors, and I prefer to copy a sector at a time. I even saw a new 1TB disk with 512b sectors. Strange, I know.
_________________
Ever picture systemd as what runs "The Borg"?
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Mon Mar 11, 2019 11:31 pm    Post subject: Reply with quote

I tried to find out what happened and ran the statement below:
Code:
cd /
tar cf - /usr/share | strace -f /tmp/log tee >(ssh user@10.0.0.1 "dd of=/d/scratch/a") >(ssh user@10.0.0.1 "dd of=/d/scratch/b") | ssh user@10.0.0.1 "dd of=/d/scratch/c"

The strace log shows what happened:
Code:
read(0, "...", 8192) = 8192
write(1, "...", 8192) = 8192
write(3, "...", 8192) = 8192
write(4, "...", 8192) = 8192
read(0, "...", 8192) = 8192
write(1, "...", 8192) = 8192
write(3, "...", 8192) = 8192
write(4, "...", 8192) = 8192
read(0, "...", 8192) = 8192
write(1, "...", 8192) = -1 EAGAIN (Resource temporarily unavailable)
write(2, "tee: ", 5)                    = 5
write(2, "'standard output'", 17)       = 17
openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, ": Resource temporarily unavailab"..., 34) = 34
write(2, "\n", 1)                       = 1
write(3, "...", 8192) = 8192
write(4, "...", 8192) = 8192
read(0, "...", 8192) = 8192
write(3, "...", 8192) = 8192
write(4, "...", 8192) = 8192
read(0, "...", 8192) = 8192
write(3, "...", 8192) = 8192
write(4, "...", 8192) = 8192

tee happily reads 8kB blocks from STDIN and writes them to STDOUT (which is a pipe to the last ssh command) and to its two output files (which are asynchronously running ssh processes).

After a while, the write to STDOUT fails with EAGAIN - which means: pipe is full. And that's strange. I wouldn't expect EAGAIN in a normal pipe. I looked at the source code of tee. There's no handler for EAGAIN.

The culprit seems to be SSH. According to this thread: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209509, SSH sets STDIN to non-blocking mode, which would explain why tee receives EAGAIN if the pipe is full.


Last edited by mike155 on Tue Mar 12, 2019 12:14 am; edited 1 time in total
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Mon Mar 11, 2019 11:57 pm    Post subject: Reply with quote

@The_Great_Sephiroth: the problem is not related to SystemRescueCD. I can't tell you why the command you posted worked in the past - maybe it was sheer luck?

But I think you can safely use the statement below. Just send the output of the pipe to /dev/null:
Code:
dd if=/dev/sda bs=1M | tee >(ssh root@192.168.2.29 "dd of=/dev/sda bs=1M") >(ssh root@192.168.2.30 "dd of=/dev/sda bs=1M") >(ssh root@192.168.2.31 "dd of=/dev/sda bs=1M") >/dev/null
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Tue Mar 12, 2019 12:25 am    Post subject: Reply with quote

The_Great_Sephiroth wrote:
Crap, does that move mean systemd? If so, I just stopped using them. Also, these were 500GB disks with 512b sectors, and I prefer to copy a sector at a time. I even saw a new 1TB disk with 512b sectors. Strange, I know.

Not that strange, there's no need for advanced format disks until you get to 2TB (2³¹ × 512). Even so, it's more efficient to use a larger bs= value like 4K (syscalls still add up, and you're multiplying that by the overhead of three ssh connections...)
Back to top
View user's profile Send private message
The_Great_Sephiroth
Veteran
Veteran


Joined: 03 Oct 2014
Posts: 1602
Location: Fayetteville, NC, USA

PostPosted: Thu Apr 11, 2019 6:48 pm    Post subject: Reply with quote

I meant to reply here and forgot to do so. Using any of the 4 and 5 series System Rescue discs my command works fine. Starting on 6, it fails. That is all I need to know for now. I did try piping to /dev/null, but that too failed on 6.

The reason I used 512b blocks is because the HDDs have 512b physical sectors. Older machines. I do one sector at a time so, should it fail, I know which sector failed and I can manually test it and even resum starting on that sector or a few before it, thus saving time.
_________________
Ever picture systemd as what runs "The Borg"?
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Thu Apr 11, 2019 9:18 pm    Post subject: Reply with quote

This is certainly an interesting problem.
I just posted this because I want to know how this ends up...
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Thu Apr 11, 2019 10:23 pm    Post subject: Reply with quote

The_Great_Sephiroth:

I tested the statement below under SystemRescueCD 6 and it works.
Code:
dd if=/dev/sda | tee >(ssh root@192.168.2.29 "dd of=/dev/sda") >(ssh root@192.168.2.30 "dd of=/dev/sda") >/dev/null

I also explained in my last post why it works and why your original statement doesn't work: OpenSSH sets STDIN to non-blocking mode and therefore we cannot pipe the output of tee to ssh.

Do I understand correctly that this statement doesn't work for you under SystemRescueCD 6?
Back to top
View user's profile Send private message
Akkara
Bodhisattva
Bodhisattva


Joined: 28 Mar 2006
Posts: 6702
Location: &akkara

PostPosted: Fri Apr 12, 2019 8:07 am    Post subject: Reply with quote

This thread reminds me of some tee troubles I had in the past.

I believe the problem comes from when the pipe(s) aren't being read in a more or less synchronous manner by all the processes. In my case I was trying to transcode audio into multiple formats in parallel with only a single disk-read/decode feeding the chain. And I ran into incomplete encoding and various errors.

Unfortunately this was a long time ago and I don't recall clearly what the solution was. But it involved reading up on the "-p" option to tee (use info tee which has better documentation), and, if I recall, creative use of dd, probably with the conv=fullblock option in there somewhere.

Sorry this is all rather vague. But I hope it might give some ideas on how to proceed.
_________________
Many think that Dilbert is a comic. Unfortunately it is a documentary.
Back to top
View user's profile Send private message
The_Great_Sephiroth
Veteran
Veteran


Joined: 03 Oct 2014
Posts: 1602
Location: Fayetteville, NC, USA

PostPosted: Wed Apr 24, 2019 7:21 pm    Post subject: Reply with quote

Thank you for the info. I am on my Android tablet now, but will check it out later today.

Also, is that a gnome I see? You know, I'm Tauren...
_________________
Ever picture systemd as what runs "The Borg"?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum