Timeouts and stream buffering with GNU coreutils
29 Jul 2021 14:26:32 FFR, softwareUse timeout(1) to terminate application that runs for too long (and to sleep
for a fraction of a second), and stdbuf(1) to disable buffering.
Until now I’ve compiled a minimal tool (named usleep) when I’ve needed a script
to sleep for a fraction of a second. About a month ago I learned about
timeout.
I was working on a script that ran a commmand, and terminate it if it was
running for too long. I was doing this by forking the process, capturing its
pid, sleeping for a little bit, and checking whether the process was still
alive. The approach was fragile at best. After a bit of investigation I
discovered timeout. Not only did it solve my issue out of the box, but it
also made another hack obsolete.
% timeout 0.2 curl -s https://example.com >/dev/null || echo fail
% timeout .1 sleep 1
I often develop and test stream processors by generating output on command line. Sometime I want to time the output generation, and sometimes I need time resolution better than whole seconds. Until a month ago I used a trivial C application. Now that’s no longer needed.
Today I also discovered stdbuf. stdbuf can set stdin/stdout/stderr buffer
size to zero. This can also be super useful.
Today I was running deleting a large set of
BorgBackup archives, and wanted an estimate of
how long it would take. I knew how many lines of output Borg would produce, and
how many of those would be about actual work. I could use pv(1) to calculate
the estimate. There was a problem, however. The informative part of Borg output
was 1/5th of the entire output, and this made the ETA widely inaccurate. I
could use sed to consume the irrelevant bits, but now the output would be
buffered and pv would not receive realtime data. stdbuf to the rescue!
# without magic
% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1; done) | pv -l -s12
1
2
3
4
1
2
3.00 0:00:01 [5.97 /s] [==================> ] 50% ETA 0:00:01
4
5.00 0:00:02 [1.99 /s] [========================> ] 66% ETA 0:00:01
6
70.0 0:00:03 [1.99 /s] [==============================> ] 83% ETA 0:00:00
8
12.0 0:00:04 [2.00 /s] [====================================>] 100% ETA 0:00:00
# skip the head
% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1; done) | sed 1,4d \
| pv -l -s8
0.00 0:00:04 [0.00 /s] [> ] 0% ETA 0:00:00
zsh: exit 124 ( seq 4; for a in $(seq 8); do; echo $a; timeout .5 sleep 1; done; ) |
zsh: terminated sed 1,4d |
zsh: exit 32 pv -l -s8
# skip the head, don't fail
% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1 || true; done) | \
sed 1,4d | pv -l -s8
1.00 0:00:04 [0.00 /s] [> ] 0% ETA 0:00:00
2
3
4
5
6
7
8
8.00 0:00:04 [1.99 /s] [====================================>] 100%
# skip the beginning, don't fail, don't buffer
% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1 || true; done) \
| stdbuf -o0 sed 1,4d | pv -l -s8
1
2
3.00 0:00:01 [1.99 /s] [========> ] 25% ETA 0:00:03
4
5.00 0:00:02 [1.99 /s] [==================> ] 50% ETA 0:00:02
6
7.00 0:00:03 [1.99 /s] [===========================> ] 75% ETA 0:00:01
8
8.00 0:00:04 [1.99 /s] [====================================>] 100%
Hooray. \o/