The hidden gems of moreutils
It seems no matter how long I work with the command line, every once in a while
I find handy utilities I've never encountered before. Most people have heard
about the bird coreutils, that's where utilities such as
echo, cat, and others come from. But did you know about moreutils?
ts
Say I want to trace every program invocation (~exec()) on the system. I can do
that with execsnoop from the BPF Compiler Collection.
# execsnoop
ls 1586 1401 0 /usr/bin/ls --color=auto -F
git 1587 1401 0 /usr/bin/git rev-parse ...
Apparently every time I touch a terminal, it runs Git to determine whether I'm
within a Git repository just so that it can show me the current branch in the
prompt. Surely that doesn't slow things down. What it doesn't show me is the
time of the invocation though. I can easily add it with ts.
# execsnoop | ts
Dec 19 12:39:25 ls 1791 1401 0 /usr/bin/ls --color=auto -F
Dec 19 12:39:25 git 1792 1401 0 /usr/bin/git rev-parse ...
I now realize that execsnoop does have the -T flag which adds invocation
times so there's no need for ts in this case but I've already written these
examples so yeah. ts can also convert time stamps into relative times. This is
especially handy when looking through log files.
$ journalctl --since yesterday --priority emerg..warning
Dec 18 19:30:47 vm sudo[5068]: user: 3 incorrect password attempts
$ journalctl --since yesterday --priority emerg..warning | ts -r
18h7m ago vm sudo[5068]: user: 3 incorrect password attempts
sponge
Have you ever wanted to modify a file, save the result into the same file, and got quickly disappointed with the result?
$ echo 1\n3\n2 > file.txt
$ sort file.txt > file.txt
$ wc -l file.txt
0 file.txt # empty
Remember, it is the shell which is responsible for redirecting the output, not
the individual commands. In this case, when the shell sees > file.txt, it
opens file.txt for writing (or creates it if necessary). Crucially, it also
opens the file with the O_TRUNC flag which instantly truncates (empties) the
file. When sort later opens the file to do the actual work, it finds the file
empty and exits.
The most common workaround is to first redirect the output to a temporary file
and then move it back to the original name. And that's exactly what sponge
does behind the scenes for you.
echo 1\n3\n2 > file.txt
sort file.txt | sponge file.txt
Interestingly, some of the commands from coreutils have a "-o" flag. I could
have just written sort -o file.txt file.txt. Oh well.
vidir
This is probably what I use most often. Running vidir opens your $EDITOR or
$VISUAL with files / directories of the specified directory (or the current
directory) and allows you to edit them. If you change the name, it will rename
it. If you delete a file row, the file gets deleted. To delete an entire
directory with everything in it, delete its row and all sub entries.
vidir *.pdf
fd -t f | vidir -
What if you've made so many great changes that you want to just quit the editor
without applying any of them? You just need to quit the editor with exit code
1. In Vim / Helix you do that with :cq. This by the way works in pretty
much all cases where a command invokes your editor (e.g. when writing a commit
message).
If your distribution doesn't provide moreutils, there's also qmv from
renameutils.
vipe
This is a bit dirty but can be useful sometimes. Imagine you want to process a bunch of files but cannot get the damned file name regex right and so you have a few extra files in the output. You could just write the output to a file, edit it, and continue from there but there a more fragile option - edit the output right between the pipes!
fd | vipe | ...
In this case, the output from fd will be buffered by vipe and passed to your
configured text editor. You can make any changes you want and once you quit, the
data will be passed down the pipe if any. The downside is that if you want to
run the command again, you will need to also edit output again. It's probably a
better idea to just write it into a file. At least in this scenario.
pee
I'm going to mention this not because I have an actual use case for this but
just because it took me a second to realize what it actually does. pee takes
its stdin and passes it to all commands given as arguments. It then gathers
their output and sends that as its own output. It runs the commands using
popen, so it's actually passing them to /bin/sh -c (which on most systems is
a symlink to Bash).
$ echo "Alice" | pee "xargs echo Hello" "xargs echo Hi"
Hello Alice
Hi Alice
I think those are all the commands I regularly use / I've used. There are more
available, such as chronic (print output only if the command failed) or ifne
(run command only if there's non empty stdin) but I haven't needed them yet.
Besides more commands in moreutils, there are even more of these command sets such as evenmoreutils or num-utils.