wnd's weblog

Timeouts and stream buffering with GNU coreutils

29 Jul 2021 14:26:32 FFR, software

Use timeout(1) to terminate application that runs for too long (and to sleep for a fraction of a second), and stdbuf(1) to disable buffering.

Until now I’ve compiled a minimal tool (named usleep) when I’ve needed a script to sleep for a fraction of a second. About a month ago I learned about timeout.

I was working on a script that ran a commmand, and terminate it if it was running for too long. I was doing this by forking the process, capturing its pid, sleeping for a little bit, and checking whether the process was still alive. The approach was fragile at best. After a bit of investigation I discovered timeout. Not only did it solve my issue out of the box, but it also made another hack obsolete.

% timeout 0.2 curl -s https://example.com >/dev/null || echo fail
% timeout .1 sleep 1

I often develop and test stream processors by generating output on command line. Sometime I want to time the output generation, and sometimes I need time resolution better than whole seconds. Until a month ago I used a trivial C application. Now that’s no longer needed.

Today I also discovered stdbuf. stdbuf can set stdin/stdout/stderr buffer size to zero. This can also be super useful.

Today I was running deleting a large set of BorgBackup archives, and wanted an estimate of how long it would take. I knew how many lines of output Borg would produce, and how many of those would be about actual work. I could use pv(1) to calculate the estimate. There was a problem, however. The informative part of Borg output was 1/5th of the entire output, and this made the ETA widely inaccurate. I could use sed to consume the irrelevant bits, but now the output would be buffered and pv would not receive realtime data. stdbuf to the rescue!

# without magic

% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1; done) | pv -l -s12
1
2
3
4
1
2
3.00  0:00:01 [5.97 /s] [==================>                   ] 50% ETA 0:00:01
4
5.00  0:00:02 [1.99 /s] [========================>             ] 66% ETA 0:00:01
6
70.0  0:00:03 [1.99 /s] [==============================>       ] 83% ETA 0:00:00
8
12.0  0:00:04 [2.00 /s] [====================================>] 100% ETA 0:00:00



# skip the head

% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1; done) | sed 1,4d \
	| pv -l -s8
0.00  0:00:04 [0.00 /s] [>                                     ]  0% ETA 0:00:00
zsh: exit 124    ( seq 4; for a in $(seq 8); do; echo $a; timeout .5 sleep 1; done; ) | 
zsh: terminated  sed 1,4d |
zsh: exit 32     pv -l -s8



# skip the head, don't fail

% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1 || true; done) | \
	sed 1,4d | pv -l -s8
1.00  0:00:04 [0.00 /s] [>                                     ]  0% ETA 0:00:00
2
3
4
5
6
7
8
8.00  0:00:04 [1.99 /s] [====================================>] 100%



# skip the beginning, don't fail, don't buffer

% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1 || true; done) \
	| stdbuf -o0 sed 1,4d | pv -l -s8
1
2
3.00  0:00:01 [1.99 /s] [========>                             ] 25% ETA 0:00:03
4
5.00  0:00:02 [1.99 /s] [==================>                   ] 50% ETA 0:00:02
6
7.00  0:00:03 [1.99 /s] [===========================>          ] 75% ETA 0:00:01
8
8.00  0:00:04 [1.99 /s] [====================================>] 100%

Hooray. \o/