Random thoughts on software, technology, and stuff

Timeouts and stream buffering with GNU coreutils

2021-07-29T14:26:32+03:00

Use timeout(1) to terminate application that runs for too long (and to sleep for a fraction of a second), and stdbuf(1) to disable buffering.

Until now I’ve compiled a minimal tool (named usleep) when I’ve needed a script to sleep for a fraction of a second. About a month ago I learned about timeout.

I was working on a script that ran a commmand, and terminate it if it was running for too long. I was doing this by forking the process, capturing its pid, sleeping for a little bit, and checking whether the process was still alive. The approach was fragile at best. After a bit of investigation I discovered timeout. Not only did it solve my issue out of the box, but it also made another hack obsolete.

% timeout 0.2 curl -s https://example.com >/dev/null || echo fail
% timeout .1 sleep 1

I often develop and test stream processors by generating output on command line. Sometime I want to time the output generation, and sometimes I need time resolution better than whole seconds. Until a month ago I used a trivial C application. Now that’s no longer needed.

Today I also discovered stdbuf. stdbuf can set stdin/stdout/stderr buffer size to zero. This can also be super useful.

Today I was running deleting a large set of BorgBackup archives, and wanted an estimate of how long it would take. I knew how many lines of output Borg would produce, and how many of those would be about actual work. I could use pv(1) to calculate the estimate. There was a problem, however. The informative part of Borg output was 1/5th of the entire output, and this made the ETA widely inaccurate. I could use sed to consume the irrelevant bits, but now the output would be buffered and pv would not receive realtime data. stdbuf to the rescue!

# without magic

% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1; done) | pv -l -s12
1
2
3
4
1
2
3.00  0:00:01 [5.97 /s] [==================>                   ] 50% ETA 0:00:01
4
5.00  0:00:02 [1.99 /s] [========================>             ] 66% ETA 0:00:01
6
70.0  0:00:03 [1.99 /s] [==============================>       ] 83% ETA 0:00:00
8
12.0  0:00:04 [2.00 /s] [====================================>] 100% ETA 0:00:00



# skip the head

% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1; done) | sed 1,4d \
	| pv -l -s8
0.00  0:00:04 [0.00 /s] [>                                     ]  0% ETA 0:00:00
zsh: exit 124    ( seq 4; for a in $(seq 8); do; echo $a; timeout .5 sleep 1; done; ) | 
zsh: terminated  sed 1,4d |
zsh: exit 32     pv -l -s8



# skip the head, don't fail

% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1 || true; done) | \
	sed 1,4d | pv -l -s8
1.00  0:00:04 [0.00 /s] [>                                     ]  0% ETA 0:00:00
2
3
4
5
6
7
8
8.00  0:00:04 [1.99 /s] [====================================>] 100%



# skip the beginning, don't fail, don't buffer

% (seq 4; for a in $(seq 8); do echo $a; timeout .5 sleep 1 || true; done) \
	| stdbuf -o0 sed 1,4d | pv -l -s8
1
2
3.00  0:00:01 [1.99 /s] [========>                             ] 25% ETA 0:00:03
4
5.00  0:00:02 [1.99 /s] [==================>                   ] 50% ETA 0:00:02
6
7.00  0:00:03 [1.99 /s] [===========================>          ] 75% ETA 0:00:01
8
8.00  0:00:04 [1.99 /s] [====================================>] 100%

Hooray. \o/

Photos from Marburg/Frankfurt, Germany, December 2012

2020-05-17T15:38:36+03:00

My first trip to anoter country with snow at location, more than 100 km from Finland. Free accommodation was available in Marburg, so that’s where I’d be going. No particular purpose, but local activities may have included a bit of geocaching.

See the photos.

Photos from Norway, August 2012

2020-04-26T17:17:53+03:00

A seven day roadtrip to Norway. 3500 kilometres by an estate car with three passengers (including driver), a tent, sleeping bags, and great scenery. Admittedly much time was spent driving in Finland but it’s the thought that counts.

See the photos.

SSH connection multiplexing

2020-04-07T00:22:25+03:00

OpenSSH has built-in connection multiplexing. I’ve been using SSH for two decades, and only now I realise SSH comes with connection multiplexing. See ssh_config option ControlMaster.

I use SSH daily. I create and use SSH tunnels daily. To name a few uses, I use SSH tunnels for database connections, HTTP connections, and for SSH connections. I used to connect to x11vnc over an SSH tunnel. I create tunnels that require two hops. However, never have I used SSH connection multiplexing. Until tonight.

Host foo
        Hostname        example.com
        ControlPath     /var/run/user/1000/%r@%h:%p
        ControlMaster   auto
        ControlPersist  10m

This simple config will /automagically/ start a master process on first connection to foo, and share that for future connections. Master is killed 10 minutes after the last client disconnect.

I am sure there are benefits in having dedicated process for each connection. I am sure there are drawbacks in connection multiplexing. For now, I’m just happy to learn something new.

Photos from Paris, France, March 2011

2020-01-14T23:13:11+02:00

Random pictures from seemingly random locations.

See the photos.

TCP port scanner

2020-01-13T20:37:32+02:00

portscan is super simple TCP port scanner for a particular purpose. It exists because I was too lazy to read nmap manual about timing options and I wanted to scan a host for open ports. Also, it was a perfect opportunity to have another learning experience with Golang. This tool will trigger each and every IDS along the way. You should use nmap instead.

Should you want to proceed against all the odds, you can find the sources at GitLab).

Photos from Tallinn, Estonia, August 2010

2020-01-13T20:22:26+02:00

Small set of photos from Tallinn, Estonia, taken back in August 2010.

See the photos.

xmlstream

2019-12-27T02:20:18+02:00

xmlstream is a Golang package for streaming XML output. It was created to replace raw text output of gpx-to-kml. Conveniently both also served as a little exercise at Golang. Little documentation exists as of yet.

You can find the sources at GitLab.

Website overhaul

2019-12-26T02:27:11+02:00

wnd.katei.fi website is now generated with Jekyll with photos handled mostly by Sigal. Jekyll is used unmodified, but with two custom plugins. I had to modify Sigal slightly to make passing its output to Jekyll more bearable. Both use custom themes. Finally, there’s a Makefile to orchestrate builds, link checks, and pushing changes live. All this means site layout has changed dramatically. While I personally strongly dislike browsing Teh Web on mobile, I’ve spent quite a bit of effort to improve readability on such devices. It’s not perfect still.

As I decommissioned my personal server, I had to relocate my website(s) as well. At the time I wanted to go with minimal effort, and not worry about my dynamically generated weblog.

On 23 September 2019, with the warm summer days only a distant memory, I started working on a replacement. I turned to Jekyll to build the bulk of it, and Sigal to build the photo galleries. All this meant I’d spend a bit of time to revamp the entire website.

Neither tool comes without limitations. Jekyll loves file suffixes. One clearly can’t do without. This issue could be mitigated with HTTP server configuration, but I don’t know if it’s worth the trouble. I also lost autogenerated post category and date collections. Eventually, I resorted to manually maintained page for the former, and decided to leave the latter for another time - likely to be forgotten.

As for Sigal, it is clearly designed to generate a finished product and not something to be passed for another processor, like Jekyll. Also, it seems only I care about providing image dimensions for <img> tag as no ready-made theme for Sigal generate these. Similarly it seems that JavaScript is a requirement these days. Eventually I managed to hack a functional theme myself. As for lost functionality, for now, I’ll go without photo tags and originals.

Then there are the plugins for Jekyll - not to mention changes to Sigal. With some motivation I’ll get to clean up and release these later. The entire site still works (perfectly) without JavaScript, but there are some placed where JavaScript is used when available. This mostly affects mobile.

In addition to Jekyll and Sigal, mobile web also turned to be a major issue. It wasn’t that my layout was using fixed sizes, but rather the fact that mobiles would render paragraph text to different sizes depending on context. I also took the opportunity to give a shot at some new(er) technologies and techniques. These mostly affect my photos, and not all shown or accessable by conventional means. :-)

I didn’t anticipate the entire process to take three months of calendar time. It did. Obviously most of it was procrastination. Then there was my reluctance to release anything with know issues. There are some rough edges; just try switching between portrait and landscape. Old links should mostly work, but not all. I may end up changing some links later. File timestamps are wrong.

Still, with all the know issues, I’ve finally decided to release. This is one step on my journey to accept compromise when I absolutely don’t have to. Maybe this lowers the bar for tying the loose ends at another occasion. Maybe that means more actual content. Maybe.

HTML and mobile browsers

2019-10-24T23:06:54+03:00

I don’t understand HTML rendering anymore. I understand h1, h2, p structure. However it seems that on mobile not all equal text is created equal.

While I’ve been trying to rebuild my website will slightly more modern tools, I’ve come to realise browsers today have a lot more say in how content is rendered. Not only do developers have much more control over specifics of how stuff is rendered, but now browsers have their tentacles in this mess too.

Admittedly in general mobile (or “responsive”) browsers are doing a fine job resizing text elements to better fit on a small screen. They cleverly resize elements to reduce the amount of otherwise empty wasted space. If you look at my work on desktop I find it agreeable. If I look at the same page on mobile using its desktop mode, there’s tons of empty space. In mobile mode, most of the waste is gone and result looks good.

As good as the logic is, sometimes its behaviour seems rather arbitrary to me. https://drafts.csswg.org/css-size-adjust/ explains the culprit nicely. That is, if you’d catch the fine details in it. I don’t. Let me demonstrate my pain.

Plain HTML page, no CSS. Header, optinally a text paragraph, lesser header, a div with text (optionally) and some links. How large the glyphs are rendered in the divs depend on the content. I can’t quite figure out the logic. Go figure.

See how the page with text and the other without text renders on your device.

Below are captures from developer mode of Chromium 76.0.3809.100 in mobile mode. The third image is a capture from desktop view for comparison.