Thursday, September 29, 2016

Be more intelligent with your `sleep`--or-- how to overengineer your scripts

The universal solution to waiting for something to be ready in shell scripting is the `sleep` command.

Here, we're waiting for a dir to be created (say from a `yum install httpd` going on in another terminal), so we can `ls` the contents, perhaps as part of a script that configures httpd.

sleep 6
ls /etc/init.d/httpd

But is there a better way? What if the directory exists almost immediately? You've wasted nearly 6 seconds unnecessarily, which if you do this a lot in your script, adds a bunch of time.

We need a way to continuously query if the resource exists. And it needs to be with a command that sets an error exit code if the resource is not found. Let's use `stat` to do this.

Leveraging the fact that `stat` will set a non-zero exit code on failure

while true; do
  stat /etc/init.d/httpd
  if [ $? -eq 0 ]; then # check if the stat was successful
    break
  fi
done
ls /etc/init.d/httpd

'true' is a command, whose output is nothing and whose exit status is 0. In terms of performance we'd probably like not to call a binary, but `true` is a shell built-in. The no-op operator `:` accomplishes the same thing, ex: `while :; do`.

We can simplify the if statement by using the && operator which executes the following command break, if stat exits without error (sets a 0 status)

while true; do
  stat /etc/init.d/httpd && break
done
ls /etc/init.d/httpd

Instead of stat, we can use the `test` command (aliased as `[`). Here we check for a file using -f.

while true; do
  if [ -f /etc/init.d/httpd ]; then break; fi
done
ls /etc/init.d/httpd

However, in these examples, if the file never exists, the loop will never exit.

So, instead we can define a timeout and a shorter `sleep` interval, and a counter (i) to track the iterations:

i=0
while [ "$i" -lt 6 ]; do
  if [ -f /etc/init.d/httpd ]; then break; fi
  sleep 1
  (( i++ )) # built-in arithmetic
done
ls /etc/init.d/httpd

Quote i for protection. Use the break keyword to escape the while loop.

Alternatively, use Bash's built-in arithmetic:
i=0
while (( i < 6 )); do
  if [ -f /etc/init.d/httpd ]; then break; fi
  sleep 1
  (( i++ ))
done
ls /etc/init.d/httpd

This is a good start. Calling test (even as a builtin) in an infinite loop is also wasteful. If you are in bash, you can use the [[ keyword, which has the added benefit of protecting you against unquoted variables in a comparison.

i=0
while (( i < 6 )); do
  if [[ -f /etc/init.d/httpd ]]; then break; fi

  sleep 1
  (( i++ ))
done
ls /etc/init.d/httpd

There is a bug here. ls will run no matter whether the file was found or not.

So now we exit the loop, but how do we notify the caller that it failed? The `break` statement does not return a non-zero. As far as the shell is concerned, the loop completed. We can use a RETVAL variable that we set explicitly to 0 when it succeeds, and 143 to mean "file not found".

i=0
while (( i < 6 )); do
  if [[ -f "$path" ]];then
      RETVAL=0
  else
      RETVAL=143
  fi
    
  [[ "$RETVAL" -eq 0 ]] && break

  sleep 1
  (( i++ ))
done
[[ "$RETVAL" -eq 0 ]] && ls /etc/init.d/httpd || echo "ERR: file never created"

I quote variables in the [[ ]] here even though it's not actually required.

Which works fine, but we can simplify by simply unsetting RETVAL if there's a success. The test is expressed with [[ -z $xxx ]]. Also it is best practice to send error messages to STDERR (file descriptor --or fd-- #2), using >&2.

i=0
while (( i < 6 )); do
  if [[ -f "$path" ]];then
    unset RETVAL
  else
    RETVAL=143
  fi
    
  [[ -z "$RETVAL" ]] && break

  sleep 1
  (( i++ ))
done
[[ -z "$RETVAL" ]] && ls /etc/init.d/httpd || echo "ERR: file never created" >&2

Now, let's abstract the variables, and set an exit status.

i=0; path="/etc/init.d/httpd"; timeout=6
while (( i < $timeout )); do
  if [[ -f "$path" ]]; then
    unset RETVAL
  else
    RETVAL=143
  fi
    
  [[ -z "$RETVAL" ]] && break

  sleep 1
  (( i++ ))
done
[[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created" >&2; exit $RETVAL )

Notice that the `exit` is called from a subshell, as if it is called in the current context, it will exit your interactive shell which is annoying and undesirable. There is no way to use return in this context since this is not yet in a function.

Testing with a bogus file.

$ path=/etc/bogusfile
$ while (( i < $timeout )); do
if [[ -f "$path" ]]; then
>   unset RETVAL
> else
>   RETVAL=143
> fi
> [[ -z "$RETVAL" ]] && break
> sleep 1
> (( i++ ))
> done

$ [[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created"; exit $RETVAL)
ERR: file never created
$ echo $?
143

Now put it in a reusable function!

_testforfile () {
  local i=0
  local timeout="$1"
  local path="$2"

  while (( i < $timeout )); do
    if [[ -f "$path" ]];then
      unset RETVAL
    else
      local RETVAL=143
    fi

    [[ -z "$RETVAL" ]] && break

    sleep 1
    (( i++ ))
  done
  [[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created" >&2; return "$RETVAL" )
}

Note the use of the `local` keyword so we don't have our custom variables pollute the invoking environment, and the change of exit to return. This function accepts two parameters as input. Call it like so:

_testforfile 6 /etc/init.d/httpd

Now, we can enhance this by making sure that at least the first argument is numeric, and even set that to a default value of 5 if it was not provided.

_testforfile () {
  local i=0
  local timeout="${2:-5}"
  local path="$1"

  local re='^[0-9]+$'
  if ! [[ $timeout =~ $re ]]; then
    echo "ERR: Timeout was not a number" >&2
    return 1
  fi

  while (( i < $timeout )); do
    if [[ -f "$path" ]];then
      unset RETVAL
    else
      local RETVAL=143
    fi
    
    [[ -z "$RETVAL" ]] && break
    
    sleep 1
    (( i++ ))
  done
  [[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created" >&2; return "$RETVAL" )
}

re is a regular expression used in conjunction with the =~ operator.

In action:
$ _testforfile 47d /etc/hosts
ERR: Timeout was not a number

And maybe some tests in a future post.

Thursday, September 22, 2016

Kill an X application when you have multiple X servers running

Mistakenly started a screensaver on my Chromoting session. Needless to say, a real pain when you have to unlock your local workstation AND the remote end after a minute of inactivity.

Of course, now you have duplicate screensavers running, and it's not apparent in the process listing which one belongs to which X server. I certainly don't want to kill the real screensaver on :0, leaving my local console unlocked!

$ ps ax|grep screensave
  6166 ?        S     16:30 xautolock -time 1 -locker xscreensaver-command -lock -detectsleep -corners -+00 -cornerdelay 1
 17200 ?        S      9:25 xscreensaver -nosplash
 25747 ?        S      7:13 xautolock -time 1 -locker xscreensaver-command -lock -detectsleep -corners -+00 -cornerdelay 1
 56143 pts/17   SN+    0:00 grep --color=auto screensave
140723 ?        S      0:07 xscreensaver -nosplash

Seeing as the DISPLAY var is a part of the environment where the X application was invoked, it should be in:

/proc/<PID>/environ

cat /proc/<PID>/environ gives a whole block of unbroken text (unsure why this is not broken), but if you look closely you'll find the DISPLAY variable in there.

The awk command was borrowed from elsewhere on the Internet. It basically says, break things apart using '=' as a field separator (FS) and \0 as a return separator (RS). I tried investigating this further, seeing as '\0' is the null return used in some GNU utils including xargs, went down the following path:

This seemed like the most straightforward way to break it out:
/bin/echo -e $(cat /proc/137572/environ)

However the output was still not broken into lines. Oh well.

Note the use of /bin/echo as echo is usually a shell built-in.

Anyway, let's use awk like I mentioned before, by substituting the PIDs I found above.


PID=17200
$ awk 'BEGIN{FS="="; RS="\0"}  $1=="DISPLAY" {print $2; exit}' /proc/$PID/environ
:0

Well that's not it.

PID=140723
$ awk 'BEGIN{FS="="; RS="\0"}  $1=="DISPLAY" {print $2; exit}' /proc/$PID/environ
:20

Bingo.

Tie it together loosely:
for PID in $(pgrep xscreensa); do echo -n $PID; awk 'BEGIN{FS="="; RS="\0"}  $1=="DISPLAY" {print $2; exit}' /proc/$PID/environ;done
6166:0
25747:20

Now I can quickly tell which processe below to which DISPLAY.


Saturday, September 3, 2016

Simple field searching with MongoDB

Tons of articles detailing full-text search. But that's not what we want.

For us UNIX folks, we just want to "grep" a field:

db.collection.find( {path: /serverpush.html/ } )

The slashes are essentially wildcards.

9menu vs ratmenu vs dmenu

Today I played around with some of the X11 popup menu options available out there, after I had started using i3 as my window manager. i3's simplicity is great, but it also left a bit of a hole in my toolset, as I often used a system-wide context menu, or root menu, to get access to shortcuts, such as.
  1. go into dynamic submenu that contains a list of recent clipboard contents to get them back in the copy-paste buffer
  2. navigate files and folders quickly through submenus
  3. use xfreerdp to connect to the windows machine hostname that I currently have highlighted in another document
  4. call a special paste function that types my clipboard contents (in some remote sessions such as Java-based ones, copy-paste does not work)
  5. take the current X selection and remove http:// and https:// from it (due to Chrome undermining copy and inserting the protocol into copied hostnames

Summary

ratmenu

  • has a "prev" function to recall the previous menu in a hierarchy
  • will not take STDIN or process a script to generate menus
  • no mouse support
  • no hinting that it's a popup (though you can tell your tiling WM to pop it out)
  • easy config

9menu

  • mouse support
  • no text search to select items
  • easy config
  • will take STDIN?
  • window hints for a popover

dmenu

  • full text search to select items
  • no back option, but you can simply call the previous dmenu
  • no mouse support
  • scripts can be used to generate menus
  • a bit tricky to get it to recognize your bashrc

Ideally, there would be a standalone port of the Openbox root menu. However, the code looks very intertwined with Openbox itself. In addition, I find the XML of the Openbox root menu rc.xml to be pretty messy, and would probably re-implement it with a json or plaintext structure.

So, without being able to port the Openbox root menu, the best option for me is 9menu. I've successfully written dynamic menus for it, though nothing too complicated

Errors running iodined server

iodine does not need an actual IPv6 interface, but will fail if AF_INET6 is not enabled, so you should change your kernel params as so:

# sysctl -w net.ipv6.conf.all.disable_ipv6=0

It was likely net.ipv6.conf.all.disable_ipv6=1.

I encountered this problem on Google Compute Engine, which surprisingly does not enable IPv6 by default.